https://clearml.slack.com/archives/C01GBQJL5KL/p1633868735006500

never tried to start it locally in clearml

When I run locally w/o clearml, it doesn't freeze

doesn't target_project force the same project on all pipeline steps?

No, I just want to register a new model in the storage. I need to create a separate task for this right?

ok, got it, thanks

Before the code I shared, there were some lines like this

step_base_task = Task.get_task(project_name=cfg[name]['base_project'],
                               task_name=cfg[name]['base_name'])
export_data = step_base_task.export_task()

... modify export_data in-place ...

task = Task.import_task(export_data)
pipe.add_step(base_task_id=task.id, clone_base_task=False, ...)

@<1523701205467926528:profile|AgitatedDove14> yeah, I'll try calling task.reset() before add_step
No, IMO it's better to leave task_overrides arguments with "." - the same structure as in the dictionary we get from export_data - this is more intuitive

so I had a base_task for every step. Then I wanted to modify a shitton of things, and I found no better way than to export, modify, and import to another task. Then this second task serves as step without cloning (that was my intention)

maybe being able to change 100% of things with task_overrides would be the most convenient way

We digressed a bit from the original thread topic though 😆 About clone_base_task=False .

I ended up using task_overrides for every change, and this way I only need 2 tasks (a base task and a step task, thus I use clone_base_task=True and it works as expected - yay!)

So, the problem I described in the beginning can be reproduced only this way:

• to have a base task
• export_data - modify - import_data - have a second task
• pass the second task to add_step with `cl...

In principle, I can modify almost everything with task_overrides , omitting export part, and it's fine. But seems that by exporting I can change more things, for example project_name

I think it's still an issue, not critical though, because we have another way to do it and it works

@<1523701205467926528:profile|AgitatedDove14> this last advice works, thank you!

but maybe add a lot more examples of using it to documentation

SparklingElephant70 in WebUI Execution/SCRIPT PATH

Also from python client

No, when I run the pipeline from the console on my local machine, it for some reason launches on clearml-services hostname (despite of the fact I specified the queue with the desired agent with pipe.set_default_execution_queue in my code)

Yes, look like it fails on 2 different containers at least

What is the solution then? What exactly has helped?

` 1633204284443 clearml-services INFO Executing: ['docker', 'run', '-t', '-l', 'clearml-worker-id=clearml-services:service:58186f9e975f484683a364cf9ce69583', '-l', 'clearml-parent-worker-id=clearml-services', '-e', 'NVIDIA_VISIBLE_DEVICES=none', '-e', 'CLEARML_WORKER_ID=clearml-services:service:58186f9e975f484683a364cf9ce69583', '-e', 'CLEARML_DOCKER_IMAGE=', '-v', '/tmp/.clearml_agent.pgsygoh2.cfg:/root/clearml.conf', '-v', '/root/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/root/.cl...

on the machine I build images? Docker version 20.10.8, build 3967b7d

Oh, I need to ask the guy who deployed it

Yes, I'll try it straightaway

I regularly run into the same problem when I launch pipelines locally (for remote execution)

However, when I clone the pipeline from web UI and launch it once again, it works. Is there a way to bypass this?

Solved. The problem was a trailing space before the image name in the Image section in web UI. I think you should probably strip the string before proceeding to environment building step, to avoid this annoying stuff to happen. Of course, users could check twice before launching, but this thing will come up every once in a while regardless

It doesn't install anything with pip during launch, I'm assuming it should take everything from the container itself (otherwise there would be a huge overhead). It simply fails trying to import things in the script

File "preprocess.py", line 4, in <module> from easydict import EasyDict as edict ModuleNotFoundError: No module named 'easydict'

When I launch tasks with a pipeline, they keep complaining about missing pip packages. I run it inside a docker container, and I'm sure these packages are present inside it (when I launch the container locally, run python3 and import them, it works like charm). Any ideas how to fix this?

Yeah, classic. What helped me is this
https://clearml.slack.com/archives/CTK20V944/p1636452811364800?thread_ts=1635950908.285900&cid=CTK20V944