Hi,
I am using TaskScheduler to start a pipeline once per hour, that sequentially executes tasks.
My current solution is that I have three execution queues, with three different worker processes.
Process 1: Executes the TaskScheduler,
Process 2: Executes the Pipeline,
Process 3: Executes the individual tasks of the pipeline (e.g. data ingestion, preparation, training).

Can I reduce the amount of queues/workers needed? Right now the TaskScheduler blocks one worker entirely and it won’t start a second TaskScheduler or the Pipeline-Exection.

Thank you!
Best,
Valentin