any idea what is missing?

@<1523701205467926528:profile|AgitatedDove14> yeah, I'll try calling task.reset() before add_step
No, IMO it's better to leave task_overrides arguments with "." - the same structure as in the dictionary we get from export_data - this is more intuitive

SparklingElephant70 then use task_overrides argument, like this
task_overrides={'script.branch': 'main', 'script.version_num': '', 'script.diff': '', 'project': Task.get_project_id(project_name=cfg[name]['base_project']), 'name': 'task-name', 'container.image': 'registry.gitlab.com/image:tag'}there must be some schema to change script name as well

ok, thanks, I'll give it a try

there seems to be no way to change default_output_uri from the code.

Dataset.create calls Task.create which in turn accepts add_task_init_call flag. Task.init accepts output_uri , but we cannot add arguments with add_task_init_call , so we cannot change output_uri from Dataset.create , right?

I initialize tasks not as functions, but as scripts from different repositories, with different images

the pipeline

maybe being able to change 100% of things with task_overrides would be the most convenient way

AgitatedDove14 Yes, this is exactly what I was looking for and was running into 413 Request Entity Too Large error during add_files . This is what helped to solve this:
https://github.com/allegroai/clearml/issues/257#issuecomment-736020929

CostlyOstrich36 idk, I need to share it to see

how do I share it?

Or maybe there is a log with something more informative which I could check up

cloning base tasks and modyfing their parameters

CostlyOstrich36 nice! thank you

OK, I managed to launch the example and it works

So, to summarize:
PipelineController works with default image, but it incurs overhead 4-5 min It doesn't work with any other image
I can add issue on Github

try add_step(..., task_overrides={'project_name': 'my-awesome-project', ...})

@<1523701070390366208:profile|CostlyOstrich36> On the screenshot, the upper task has the lower task as parent

of course, I use custom images all the time, the question was how to do it for a pipeline 😆 setting private attributes directly doesn't look as good practice

pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code

You should add the wrong key to task_overrides , not the wrong parameter to add_step

AgitatedDove14 yeah, that makes sense, thank you. That means I need to pass a single zip file to path argument in add_files , right?

The files themselves are not on S3 yet, they are stored locally. That's what I want: register a new dataset and upload the data itself to S3

Changing sdk.development.default_output_uri in clearml.conf seems to be bad idea, because different datasets will likely have different addresses on S3

i need it so much man

so that the way of doing it would be like this:
all_models = Model.query_models(projeect_name=..., task_name=..., tags=['running-best-checkpoint']) all_models = sorted(all_models, key=lambda x: extract_epoch(x)) for model in all_models[:-num_to_preserve]: Model.remove(model, delete_weights_file=True)

where is it in the docs?

I found exactly that, question is solved

It worked, thank you!

so yeah, in short, target_project cannot do that

AgitatedDove14 any ideas? 😀

AgitatedDove14 Yes this would be great