AgitatedDove14 from what I gather there is a lightly documented concept of “multi_instance_support” https://github.com/allegroai/clearml/blob/90854fa4a516fcb38ea0a5ec23894c5a3b6bbc4f/clearml/automation/controller.py#L3296 .
Do you think it can work?
Interesting question, maybe something like:
r = 
for i in things_to_do:
create some stuff to do:
results = 
for step in range(10):
r = process_sub_list(list(range(step*10, (step+1)*10)))
push into one list with all result, this will actually wait for them to be completed
merged = 
for r in results:
from what I gather there is a lightly documented concept
Yes ... 😞 the reason for it is that actually one could do:
if name == 'main':
pipeline(2) `Basically rerunning the pipeline 3 times
This support was added as some users found a use case for it, but I think this would be a rare one
AgitatedDove14 it’s pretty much similar to your proposal but with pipelines instead of tasks, right?