@<1559711623147425792:profile|PlainPelican41> , I think you should switch to using pipelines from decorators for easier management of your tasks/code. Creating pipelines from pipelines according to your scenario (difficulty managing code) would only lead to more difficulty tracking the code executions - How would you show pipelines of pipelines results? How many depths would be enabled? How should data aggregation work? This introduces many problems without any benefit.
I think restructuring the way you manage your pipelines might be the way 🙂
I'm not sure pipeline of pipelines would work out of the box without some hacks. What is your use case though, why do you need a pipeline of pipelines? In the end it would abstract into a single large pipeline anyways
@<1559711623147425792:profile|PlainPelican41> what is the exact use-case? Why isn't a single pipeline enough? Logically a pipeline of pipelines is equivalent to a single pipeline...
I'm thinking maybe you can do it by chaining tasks somehow, but I think that is not the correct way (Never tried it myself, as I said, in the end it all abstracts to a single pipeline).
Maybe check out pipelines with functions or even better decorators, this might be an easier solution for you and you can create very complex pipelines with it. Think about using loops and if statements to create steps and progress your pipeline
Hi @<1559711623147425792:profile|PlainPelican41> , what exactly are you trying to do and what is your use case?
earlier you've said I can managed to do so with some hacks. do u have an example or can u explain what I need to do 🙂 ?
maybe some hacks will be ok for now too
currently yes.
but as I said I want to create a pipeline of pipelines
exactly because of the reason I described above:
None
I'm using the PipelineController to ctreate a pipeline.
now I want to add 2 other pipelines to this main-pipeline using add_step (so instead of task-id I'm giving it a pipeline-id.
but it doesn't work and always throws error.
If I'm doing the same thing with task ids all works.. so I guess there is another correct way to do a pipeline of pipelines
let's say I have pipeline_1 which has 10 tasks in it.
and I also have pipeline_2 with has 10 tasks in it.
I want a new_pipeline that will run pipeline_1 and after that will run pipeline_2.
If I cannot do pipeline of pipelines, it means that I need to create a pipeline containing 20 tasks using add_step !
and In the future if I would like to add pipeline_3 to all of this, I'll need again to use add_step to all the 20 task + the ones of pipeline_3.
I makes all the process very uncomfortable as we scales to larger automations and tasks. So I need a way to make it simple and more readable using a pipeline of pipelines
