but for datasets there seems to be a way to do it from the code

specifying target storage for OutputModel from the config is bad idea: it should be project-specific, not agent-specific

what's the difference between set_default_upload_uri and set_upload_destination ?

ok, got it, thanks

The lower task is created during import_task , the upper one - during actual execution of the step, several minutes after

pipeline controller itself is stuck at running mode forever all step tasks are created but never enqueued
I can share some code

CostlyOstrich36 thank you for the answer! Maybe I just can delete old models along with corresponding tasks, seems to be easier

yes

yeah, during a run

Is there a way to simplify it with ClearML, not make it more complicated?

creates all the step tasks in draft mode and then stucks

I still haven't figured out how to make files downloaded this way visible for future get_local_copy calls though

I suspect this has something to do with task.output_model

AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?

Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS tab?

I see the task on the web UI, but get Fetch experiment failed when I click on it, as I described. It even fetches the correct ID by it's name. I'm almost sure it will be present in mongodb

But I still cannot launch my own pipelines

You are right, I had [None] as parents in one of the tasks. Now this error is gone

pipeline launches on the server anyway (appears on the web UI)

hm, not quite clear how it is implemented. For example, this is how I do it now (explicitly)

@<1523701435869433856:profile|SmugDolphin23> could you please give me a link to it? I can't find it on github... Here I see only one comment
None

AgitatedDove14 any ideas? 😀

@<1523701435869433856:profile|SmugDolphin23> could you please review it further? Is it acceptable to be merged?

sorry, no GH issue, just a link to this thread (I saw other contributors did this and got their PR merged, hehe)

I don't think so. it is solved by installing openssh-client to the docker image or by adding deploy token to the cloning url in web ui

In short, what helped is
gitlab+deploy-tokenin gitlab url

where is it in the docs?

in order to work with ssh cloning, one has to manually install openssh-client to the docker image, looks like that

any idea what is missing?

is there a some sort of OutputModel.remove method? Docs say there isn't