Unanswered
Hi, I Have A Question About The Pipeline, Especially About The Parallelism Part.
We Are Considering Implementing A Use Case And Are Interested In Knowing Whether It Can Be Efficiently Managed Using Clearml Pipeline.
Our Use Case Involves A Dataset That
Hi Jason, yes this can be done. Your pipeline code will look like this:
Execution of preprocessing task
for i in range(125):
Execution of data splitting and inference task(s); each of the 125 tasks have the same base task name but different names, e.g. name = "inference_task" + str(i)
<end loop>
ids = ["${inference_task_" + str(i) + ".id}" for i in range(125)]
Execution of aggregation task with the ids passed in as some part of parameter_override e.g. "General/inference_ids": '[' + ','.join(ids) + ']', as a string that can be processed in the task script itself.
Let me know if you have any further questions; thanks!
142 Views
0
Answers
one year ago
one year ago