So, to summarize:
PipelineController works with default image, but it incurs overhead 4-5 min It doesn't work with any other image
I can add issue on Github

pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code

AgitatedDove14 is it expected behavior?

Sorry for the delay

Not reproduced, but caught another error when running pipeline_from_tasks.py
` Traceback (most recent call last):
File "pipeline_from_tasks.py", line 31, in <module>
pipe.add_step(name='stage_data', base_task_project='examples', base_task_name='pipeline step 1 dataset artifact')
File "/home/kirillfish/.local/lib/python3.6/site-packages/clearml/automation/controller.py", line 276, in add_step
base_task_project, base_task_name))
ValueError: Could not find ...

Refactoring is to account for the new project names. And also to resolve the project name depending on the version of a client

AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?

Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS tab?

AgitatedDove14 Yes this would be great

(this is an answer to the previous message)

if fails during add_step stage for the very first step, because task_overrides contains invalid keys

The pipeline is initialized like this
pipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone

` step_base_task = Task.get_task(project_name=cfg[name]['base_project'],
task_name=...

SuccessfulKoala55 sorry, that was a bug on my side. It was just referring to another class named Model

I just happened to spawn multiple OutputModels within a single script which is being run in a single task. That is, I see dozens of models in Models tab in web UI. What I want is to delete most of them (along with the files in S3), preserving the spawning task

this is the same thing as in the previous thread. I suggest that we move there

this is how I implemented it by myself. Looks like clearml functionality is quite opinionated and requires some tweaks every time I try to replace my own stuff with it

This way I would want to keep track of 3 OutputModel s and call update_weights 3 times every update - and probably do 2 redundant uploadings

I initialize tasks not as functions, but as scripts from different repositories, with different images

AgitatedDove14
`
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.

1. Make sure you pushed the requested commit:
   (repository='git@...', branch='main', commit_id='...', tag='', docker_cmd='registry.gitlab.com/...:...', en...

for https cloning, deploy token is needed

I still haven't figured out how to make files downloaded this way visible for future get_local_copy calls though

in order to work with ssh cloning, one has to manually install openssh-client to the docker image, looks like that

When I run locally w/o clearml, it doesn't freeze

of course, I use custom images all the time, the question was how to do it for a pipeline 😆 setting private attributes directly doesn't look as good practice

In short, what helped is
gitlab+deploy-tokenin gitlab url

Searching by model ID is good idea, but how do I fetch it from the code? In principle, InputModels are rarely defined automatically, so I could look up for the ID manually...

where is it in the docs?

OK, I managed to launch the example and it works

@<1523701435869433856:profile|SmugDolphin23> could you please review it further? Is it acceptable to be merged?

@<1523701435869433856:profile|SmugDolphin23> maybe I could make a pull request ? Is there any community guideline how to make pull requests to ClearML?

If I keep track of 3 OutputModels simultaneously, the weights would need to shift between them every epoch (like, updated weights for top-1, then top-1 becomes top-2, top-2 becomes top-3 etc)