There are some questions in this channel already regarding pipeline V2. Is there any tutorial or changelog or examples I can refer to?
It's decorators. It's functions as steps, It's the ability to add metric tracking for pipelines. It's a better way to move artifacts between steps (way less boilerplate code). It's pipeline instance versioning (for the upcoming UI changes), it's better step parallelizations (If steps are not dependent on each other, they will automatically be parallelized) and I've probably missed some features....
MelancholyElk85 , yes nesting pipelines is possible. as for flatting it afterwords, maybe AgitatedDove14 knows? I'm pretty sure it can't be done though
SubstantialElk6 Yes! We should have it up, together with the previous one uploaded soon!
Hi MelancholyElk85 , documentation is a soft spot for us, we're trying to be better but we aren't always 🙂 https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py this is an example, if you check the pipeline folder you'll see others (like pipeline_from_function.py
AnxiousSeal95 We can make a nested pipeline, right? Like if the top pipeline calls add_step to create steps from tasks, and then we decompose any single step further and create a sub-pipeline from decorators there. We should be able to do that, because PipelineController is itself a task, right?
Also, is there a way to unfold such nested pipeline into a flat pipeline? So that only a single pipeline task is created, and it draws a single detailed DAG in PLOTS tab?
I could insert some updated info to my conference talk if you share the recording by tomorrow morning 😄
MelancholyElk85 I'll talk to the guy in charge and try! You can find main points here 🙂 https://docs.google.com/document/d/1QlPiDO2EzDq_HRvuVhYwPh5F_IsjgmmRTlPNdHik10k/edit
Very nice for small pipelines (where every step could be put into a function in a single repository)
Ah, ok, pipeline v2 is just about decorators? This example has already been there for a while