Unanswered
I Am Struggling A Bit To Understand The Use Case Of A Pipeline:
Let Say You Have Step1 -> Step2 -> Step3
What Is The Point To Use Pipeline Feature Versus Having A Single Task That Do Those Steps One After Another ???
Caching can be a reason. Say you do some heavy data loading / processing in step 1. Now you're developing step 2.
It'd be nice not to have to re-run Step 1 every time you want to test a change to step 2.
You could find a way to simply write your output of step1 to disk and do everything in one step, or you could let ClearML handle that caching for you--with the added benefit that others collaborating remotely can also use the outputs of steps you've cached with ClearML
163 Views
0
Answers
one year ago
one year ago