SubstantialElk6 ,

We were trying with 'from task' at the moment. But the question apply to all methods.

You can specify this using add_function_step(..., execution_queue="<QUEUE>")

Make certain tasks in the pipeline run in the same container session, instead of spawning new container sessions? (To improve efficiency)

I'm not sure this is possible currently. This could a be nice feature request. Maybe open a github request?

What type of pipeline steps are you running? From task, decorator or function?

We were trying with 'from task' at the moment. But the question apply to all methods.

If they're all running on the same container why not make them the same task and do things in parallel?

The tasks were created by different teams and their tasks content is rather independent and modular. Usage of them is usually optional. For example, task1 performs 'image whitening', task2 performs 'image resize'.

Hi SubstantialElk6 ,

Define prior to running the pipeline, which tasks to be running on which remote queue using which images?

What type of pipeline steps are you running? From task, decorator or function?

Make certain tasks in the pipeline run in the same container session, instead of spawning new container sessions? (To improve efficiency)

If they're all running on the same container why not make them the same task and do things in parallel?