Suppose I have the following scenario (real-world project, real ML pipeline scenario)

• I have separate projects for different steps (ETL, train, test, tensorrt conversion...). Every step has it's own git repository, docker image, branch etc
• For quite a long time all the steps were not functioning as parts of an automated pipeline. For example, collaborative experimentation (training and validation steps). We were just focusing on reproducibility/versioning etc
• After some time, we decided...

base task is in draft status, so when I call import_data it imports draft status as well, am I right?

@<1523701070390366208:profile|CostlyOstrich36> On the screenshot, the upper task has the lower task as parent

https://clearml.slack.com/archives/C01GBQJL5KL/p1633868735006500

No, I just want to register a new model in the storage. I need to create a separate task for this right?

e.g. if I want to store only top-3 running best checkpoints

For datasets it's easily done with a dedicated project, a separate task per dataset, and Artifacts tab within it

@<1523701435869433856:profile|SmugDolphin23> could you please give me a link to it? I can't find it on github... Here I see only one comment
None

add_files . There is no upload call, because add_files uploads files by itself, if I got it correctly

CostlyOstrich36 thank you for the quick answer! I tried it but there is still 413 Request Entity Too Large error, as if it still uses a default fileserver

SuccessfulKoala55 Turns out we have copied elasticsearch database as well. Also it seems that the error is thrown only for experiments with artifacts

@<1523701435869433856:profile|SmugDolphin23> maybe I could make a pull request ? Is there any community guideline how to make pull requests to ClearML?

I don't think so. it is solved by installing openssh-client to the docker image or by adding deploy token to the cloning url in web ui

CostlyOstrich36 so it's the same problem as https://clearml.slack.com/archives/CTK20V944/p1636373198353700?thread_ts=1635950908.285900&cid=CTK20V944

AgitatedDove14 is it expected behavior?

ok, thanks

` clearml_name = os.path.basename(save_path)

output_model_best = OutputModel(
task=task,
name=clearml_name,
tags=['running-best'])

output_model_best.update_weights(
save_path,
upload_uri=params.clearml_aws_checkpoints,
target_filename=clearml_name
) `

Refactoring is to account for the new project names. And also to resolve the project name depending on the version of a client

PipelineContoller._task.set_base_docker ?? :good-thinking:

For experiments with no artifacts, everything seems to work properly

this is so cursed, it's 10:30 pm

this is the same thing as in the previous thread. I suggest that we move there

for https cloning, deploy token is needed

sorry, no GH issue, just a link to this thread (I saw other contributors did this and got their PR merged, hehe)

SuccessfulKoala55 sorry, that was a bug on my side. It was just referring to another class named Model

There is literally Models tab in each project

never tried to start it locally in clearml

doesn't target_project force the same project on all pipeline steps?

@<1523701205467926528:profile|AgitatedDove14> yeah, I'll try calling task.reset() before add_step
No, IMO it's better to leave task_overrides arguments with "." - the same structure as in the dictionary we get from export_data - this is more intuitive