Hey @<1523701205467926528:profile|AgitatedDove14> ,
sorry, I am quite new to slack... forgot to submit my changes of the answer...

When you are saying parallel what do you mean? from multiple machines ?

yes, or (because I deployed clearml using helm in kubernetes) from the same machine, but multiple pods (tasks).

Once a dataset was finalized the only way to add files is to add another version that inherits from the previous one (i.e. the finalized version becomes the parent of the new version)
If you are worried about multiple versions, just like in git you have squeeze

okay, great. thank you so much!

The correct way would be to pas the Dataset ID, then other task would simple get it with Dataset.get
No need to worry about re-download, everything is automatically cached.

Sounds good, thanks for clarification.

Hi @<1661542579272945664:profile|SaltySpider22> I'm not sure I understand the answer to my parallel quesion

@<1523701205467926528:profile|AgitatedDove14>

When you are saying parallel what do you mean? from multiple machines ?

yes, or (because I deployed clearml using helm in kubernetes) from the same machine, but multiple pods (tasks).

Oh now I see, long story short, no 😞 the correct way of doing that is every node/pod creates it's own dataset,
then when you are done, you create a new version with the X datasets that you created as parents, the newly created version is just "meta" it basically tells the system how to combine the previously generated datasets (i.e. no data is actually re-uploaded).
Version tree should look something like

 [x]
  |
+-+--+---+
|    |   |
[a] [b] [c]

to question 1:
passing Dataset artifacts between tasks seems to be not possible, getting the following error message:

TypeError: cannot pickle '_thread.lock' object. 

So i guess its not possible to upload files from different tasks in parallel to the dataset, before finalizing it.

Hi @<1661542579272945664:profile|SaltySpider22>

question 1: are parallel writes to a dataset with the same version possible?

When you are saying parallel what do you mean? from multiple machines ?

Whats the recommended way to append the dataset in a future version?

Once a dataset was finalized the only way to add files is to add another version that inherits from the previous one (i.e. the finalized version becomes the parent of the new version)
If you are worried about multiple versions, just like in git you have squeeze 🙂

passing Dataset artifacts between tasks seems to be not possible,

The correct way would be to pas the Dataset ID, then other task would simple get it with Dataset.get
No need to worry about re-download, everything is automatically cached.
Make sense ?