There are some questions in this channel already regarding pipeline V2. Is there any tutorial or changelog or examples I can refer to?
AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?
Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS tab?
MelancholyElk85 I'll talk to the guy in charge and try! You can find main points here 🙂 https://docs.google.com/document/d/1QlPiDO2EzDq_HRvuVhYwPh5F_IsjgmmRTlPNdHik10k/edit
SubstantialElk6 Yes! We should have it up, together with the previous one uploaded soon!
It's decorators. It's functions as steps, It's the ability to add metric tracking for pipelines. It's a better way to move artifacts between steps (way less boilerplate code). It's pipeline instance versioning (for the upcoming UI changes), it's better step parallelizations (If steps are not dependent on each other, they will automatically be parallelized) and I've probably missed some features....
Very nice for small pipelines (where every step could be put into a function in a single repository)
Ah, ok, pipeline v2 is just about decorators? This example has already been there for a while
I could insert some updated info to my conference talk if you share the recording by tomorrow morning 😄
MelancholyElk85 , yes nesting pipelines is possible. as for flatting it afterwords, maybe AgitatedDove14 knows? I'm pretty sure it can't be done though
Hi MelancholyElk85 , documentation is a soft spot for us, we're trying to be better but we aren't always 🙂 https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py this is an example, if you check the pipeline folder you'll see others (like pipeline_from_function.py