Hi @<1523701070390366208:profile|CostlyOstrich36> , I'm using pipeline from tasks, am I able to do the same as pipeline from decorator ?
I do this a lot. pipeline params spawn K number of nodes, that collect just like you drew. No decorator being used here, just referencing tasks by id or name/project. I do not use continue on fail at all.
I do this with functions that have the contract ( f(pipe: PipelineController, **kwargs) -> PipelineController
) and a for-loop.
just be aware DAG creation slows down pretty quickly after a dozen or so such loops.
All the images below were made with the same pipeline (just evolved some names over time) + different parameter options. Including making the "collect results" step optional.
I suggest you see this example - None
And see how you can implement an if statement into this sample to basically create 'branches' in the pipeline 🙂
Hi @<1546303293918023680:profile|MiniatureRobin9> , if you use pipelines from decorator your can certainly set if statements to decide where/how to go
Okay, I found it : I put an if statement with the parameter and continue_on_fail=True in the steps, and it worked. But I had to specify the continue_on_fail in add_step to make it work.