The pipeline is initialized like this
pipe = PipelineController(project=cfg['pipe']['project_name'], name='pipeline-{}'.format(name_postfix), version='1.0.0', add_pipeline_tags=True) pipe.set_default_execution_queue('my-queue')
Then for each step I have a base task which I want to clone

` step_base_task = Task.get_task(project_name=cfg[name]['base_project'],
task_name=...

pipeline launches on the server anyway (appears on the web UI)

There is literally Models tab in each project

What exactly we need to copy? I believe we have already copied everything, but it keeps throwing "Fetch experiment failed" error

For experiments with no artifacts, everything seems to work properly

When I launch tasks with a pipeline, they keep complaining about missing pip packages. I run it inside a docker container, and I'm sure these packages are present inside it (when I launch the container locally, run python3 and import them, it works like charm). Any ideas how to fix this?

It worked, thank you!

so yeah, in short, target_project cannot do that

Thank you

AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?

Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS tab?

You can try to spin the "services" queue without docker support, if there is no need for containers it will accelerate the process.

With pipe.start(queue='services') , it still tries to run some docker for some reason
1633799714110 kirillfish-ROG-Strix-G512LW-G512LW info ClearML Task: created new task id=a4b0fbc6a1454947a06be4e48eda6740 ClearML results page: `
1633799714974 kirillfish-ROG-Strix-G512LW-G512LW info ClearML new version available: upgrade to v1.1.2 is recommended!

...

I found out this happens with any other image except the default one, regardless of whether I set it with pipe._task.set_base_docker

The image is not needed to run the pipeline logic, I do it just to reduce overhead. Otherwise it would take too long to just build the default image on every launch

of course, I use custom images all the time, the question was how to do it for a pipeline 😆 setting private attributes directly doesn't look as good practice

StorageManager class seems to lack methods for removal as well

I launch everything in docker mode, and since it builds an image on every run, it builds default nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 image, which incurs heavy overhead. What if I want to give it my custom lightweight image instead? The same way I do for all individual tasks

but the question was about the pipeline controller, not individual tasks.

PipelineContoller._task.set_base_docker ?? :good-thinking:

After I set base docker for pipeline controller task, I cannot clone the repo...

cloning base tasks and modyfing their parameters

AgitatedDove14
`
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.

1. Make sure you pushed the requested commit:
   (repository='git@...', branch='main', commit_id='...', tag='', docker_cmd='registry.gitlab.com/...:...', en...

When I run locally w/o clearml, it doesn't freeze

So, to summarize:
PipelineController works with default image, but it incurs overhead 4-5 min It doesn't work with any other image
I can add issue on Github

never tried to start it locally in clearml

CostlyOstrich36 thank you for the answer! Maybe I just can delete old models along with corresponding tasks, seems to be easier

RotundHedgehog76 We were using clearml-server on kubernetes cluster, so I just reached out to our devops to change nginx settings and re-deploy it

CostlyOstrich36 idk, I need to share it to see

how do I share it?

@<1523701070390366208:profile|CostlyOstrich36> Yes, I'm self deployed, and the company I want to share it with is also self deployed

CostlyOstrich36 so it's the same problem as https://clearml.slack.com/archives/CTK20V944/p1636373198353700?thread_ts=1635950908.285900&cid=CTK20V944

any idea what is missing?

Maybe displaying 9 or 10 by default would be enough + clearly visible and thick scrollbar to the right