Yeah, but do I need to call add_files first?

I found out this happens with any other image except the default one, regardless of whether I set it with pipe._task.set_base_docker

The image is not needed to run the pipeline logic, I do it just to reduce overhead. Otherwise it would take too long to just build the default image on every launch

look like I created a new task for every epoch ...

No, I just want to register a new model in the storage. I need to create a separate task for this right?

so I had a base_task for every step. Then I wanted to modify a shitton of things, and I found no better way than to export, modify, and import to another task. Then this second task serves as step without cloning (that was my intention)

Before the code I shared, there were some lines like this

step_base_task = Task.get_task(project_name=cfg[name]['base_project'],
                               task_name=cfg[name]['base_name'])
export_data = step_base_task.export_task()

... modify export_data in-place ...

task = Task.import_task(export_data)
pipe.add_step(base_task_id=task.id, clone_base_task=False, ...)

but maybe add a lot more examples of using it to documentation

of course, I use custom images all the time, the question was how to do it for a pipeline 😆 setting private attributes directly doesn't look as good practice

doesn't target_project force the same project on all pipeline steps?

The lower task is created during import_task , the upper one - during actual execution of the step, several minutes after

what's the difference between set_default_upload_uri and set_upload_destination ?

base task is in draft status, so when I call import_data it imports draft status as well, am I right?

(this is an answer to the previous message)

task = Task.import_task(export_data)
pipe.add_step(
    name=name,
    base_task_id=task.id,
    parents=parents,
    task_overrides={'script.branch': 'main', 'script.version_num': '', },
    execution_queue=pipe_cfg['step_queue'],
    cache_executed_step=True,
    clone_base_task=False
)

SuccessfulKoala55 It gives the correct name, that is running_best_pth !

@<1523701070390366208:profile|CostlyOstrich36> on a remote agent, yes, running the task from the interface

ok, got it, thanks

So now it works smoothly

Is there a way to simplify it with ClearML, not make it more complicated?

It doesn't install anything with pip during launch, I'm assuming it should take everything from the container itself (otherwise there would be a huge overhead). It simply fails trying to import things in the script

File "preprocess.py", line 4, in <module> from easydict import EasyDict as edict ModuleNotFoundError: No module named 'easydict'

AgitatedDove14 Yes, this is exactly what I was looking for and was running into 413 Request Entity Too Large error during add_files . This is what helped to solve this:
https://github.com/allegroai/clearml/issues/257#issuecomment-736020929

CostlyOstrich36 so it's the same problem as https://clearml.slack.com/archives/CTK20V944/p1636373198353700?thread_ts=1635950908.285900&cid=CTK20V944

for https cloning, deploy token is needed

specifying target storage for OutputModel from the config is bad idea: it should be project-specific, not agent-specific

AgitatedDove14
`
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.

1. Make sure you pushed the requested commit:
   (repository='git@...', branch='main', commit_id='...', tag='', docker_cmd='registry.gitlab.com/...:...', en...

You should add the wrong key to task_overrides , not the wrong parameter to add_step

Thank you

i need it so much man

As for now it seems to query only by project name and model name.