clearml package version: '1.8.3'

Sure, AgitatedDove14 !

I will get to it next week. Thank you for the answer!

AgitatedDove14
Hello, Martin. Any news about this issue?

We really want to use ClearML for datasets that are hundreds GB worth of data.

Are you saying the ClearML is not able to do that?

y. In the second case you run a script that creates a task with

Task.create()

which creates a draft task with execution parameters, output uri etc (Nothing in configuration I assume? Please check

In the second case I only call Task.create , that specifies docker, repository, commit, script path and so on (but doesn't specify output_uri or tags . They should be set in train.py , when the Task.Init is called).

Afterwards on the remote ma...

CostlyOstrich36 It seems to be a critical bug.

Do you happen to know a support channel, that can help with that?

I guess I can simplify it a little.
Basically there are two scenarios:

on local machine
task = Task.Init(output_uri="...", tags=["tag"])Result: everything works. Remote uri is used, tags are set

on local machine
task = clearml.Task.create(...) clearml.Task.enqueue(task, queue_name=queue)> on remote clearml agent, the same code is called
task = Task.Init(output_uri="...", tags=["tag"])Result: init params are ignored, remote uril is not set, tags are empty

Does it mak...

UPD: it doesn't solve anything 😞
This approach just creates a separate task corresponding to enque_task.py script. But the task that is being run on clearml agent still ignored outpur_uri 😞

I figured the problem.

Reason:
If you create a clearml task and put it into queue, all further Task.init call arguments from clearml worker will be ignored.

Solution:
enque_task.py
task = clearml.Task.create(...) task.init(remote_uri=..., tags=...) clearml.Task.enqueue(task, queue_name=queue)
train.py
task = Task.Init(<whatever, all this args will be ignored>)

@<1523701087100473344:profile|SuccessfulKoala55>

So basically my problem was that I couldn't specify ouput_uri with Task.creaate .

I ended up with a solution to just use CLI version of clearml-task that allows for specifying output_uri (but not tags, though).

Thank you! I had doubts about it.

Oh. I just tested it in the new version, it really works now. Thank you very much!

Sorry for reporting a bug without testing on the newest version

I think the more general solution would be creating integration with Zapier.

This way user would be able to customize his own triggers for events like "on experiment status changed", "on tag added" and so on

The first automation I would write for myself would probably be "on tag added" -> "if tag in Jira ids" -> "reference the experiment URL in Jira issue"

Error:
---------------------------------------------------------------------------
NotFound                                  Traceback (most recent call last)
Cell In[1], line 3
      1 import clearml
      2 ds = clearml.Dataset.get("1a607bbeb31b4e2c8033112950827d8e")
----> 3 out_ds = clearml.Dataset.create(
      4     dataset_project="test",
      5     dataset_name=f"test",
      6     output_uri="

",
      7 )

File ~/miniconda3/envs/vfm38/lib/python3.8/site-packages/cl...

Looks like opening a dataset that is stored on gcp bucket and then trying to create another dataset on the same bucket creates issues.

Maybe there is a hotfix to reset clearml storage manager cache in the middle of the script?