
Reputation
Badges 1
149 × Eureka!So, to summarize:
PipelineController works with default image, but it incurs overhead 4-5 min It doesn't work with any other image
I can add issue on Github
pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code
AgitatedDove14 is it expected behavior?
Sorry for the delay
Not reproduced, but caught another error when running pipeline_from_tasks.py
` Traceback (most recent call last):
File "pipeline_from_tasks.py", line 31, in <module>
pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact')
File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step
base_task_project, base_task_name))
ValueError: Could not find ...
Refactoring is to account for the new project names. And also to resolve the project name depending on the version of a client
AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step
to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?
Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS
tab?
(this is an answer to the previous message)
if fails during add_step
stage for the very first step, because task_overrides
contains invalid keys
The pipeline is initialized like thispipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone
` step_base_task = Task.get_task(project_name=cfg[name]['base_project'],
task_name=...
SuccessfulKoala55 sorry, that was a bug on my side. It was just referring to another class named Model
I just happened to spawn multiple OutputModels
within a single script which is being run in a single task. That is, I see dozens of models in Models
tab in web UI. What I want is to delete most of them (along with the files in S3), preserving the spawning task
this is the same thing as in the previous thread. I suggest that we move there
this is how I implemented it by myself. Looks like clearml functionality is quite opinionated and requires some tweaks every time I try to replace my own stuff with it
This way I would want to keep track of 3 OutputModel
s and call update_weights
3 times every update - and probably do 2 redundant uploadings
I initialize tasks not as functions, but as scripts from different repositories, with different images
AgitatedDove14
`
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.
- Make sure you pushed the requested commit:
(repository='git@...', branch='main', commit_id='...', tag='', docker_cmd='registry.gitlab.com/...:...', en...
for https cloning, deploy token is needed
I still haven't figured out how to make files downloaded this way visible for future get_local_copy
calls though
in order to work with ssh cloning, one has to manually install openssh-client to the docker image, looks like that
of course, I use custom images all the time, the question was how to do it for a pipeline 😆 setting private attributes directly doesn't look as good practice
In short, what helped isgitlab+deploy-token
in gitlab url
Searching by model ID is good idea, but how do I fetch it from the code? In principle, InputModels are rarely defined automatically, so I could look up for the ID manually...
where is it in the docs?
OK, I managed to launch the example and it works
@<1523701435869433856:profile|SmugDolphin23> could you please review it further? Is it acceptable to be merged?
@<1523701435869433856:profile|SmugDolphin23> maybe I could make a pull request ? Is there any community guideline how to make pull requests to ClearML?
If I keep track of 3 OutputModels
simultaneously, the weights would need to shift between them every epoch (like, updated weights for top-1, then top-1 becomes top-2, top-2 becomes top-3 etc)