Hi,
I'm using PipelineController to launch remote pipelines from a local orchestration script. For each input file, I create a pipeline like this sequentially:
for file in files:
pipeline = PipelineController(...)
pipeline.add_step(...)
pipeline.start(queue="default")
However, once .start() is called, ClearML prints:
ClearML Terminating local execution process - continuing execution remotely
...and the local Python process exits immediately.
I tried isolating each pipeline launch in a separate subprocess (via subprocess.run() from a Poetry-managed script), but the subprocess still gets terminated as soon as pipeline.start() is invoked.
PipelineController ) from a local script without ClearML killing the process?execute_remotely() so that I can launch many in sequence or in parallel?Hi @<1861218295315697664:profile|FlutteringLobster45> ! You should use PipelineController.create and PipelineController.enqueue in this case, to create a pipeline without executing it at all remotely and enqueue it for remote execution
Thank you Eugen, I was trying to use both methods but it resulted in an error earlier - so my fallback was a bash script.
I will try again and post my error logs if I'm unsuccessful there.
