Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'm using `PipelineController` to launch remote pipelines from a local orchestration script. For each input file, I create a pipeline like this sequentially: ```for file in files: pipeline = PipelineController(...) pipeline.add_step(...) pip

Hi,

I'm using PipelineController to launch remote pipelines from a local orchestration script. For each input file, I create a pipeline like this sequentially:

for file in files:
   pipeline = PipelineController(...)
   pipeline.add_step(...)
   pipeline.start(queue="default")

However, once .start() is called, ClearML prints:

ClearML Terminating local execution process - continuing execution remotely

...and the local Python process exits immediately.

I tried isolating each pipeline launch in a separate subprocess (via subprocess.run() from a Poetry-managed script), but the subprocess still gets terminated as soon as pipeline.start() is invoked.

  • How can I launch multiple pipelines (via PipelineController ) from a local script without ClearML killing the process?
  • Is there a supported way to enqueue remote pipelines without using execute_remotely() so that I can launch many in sequence or in parallel?
    Thanks in advance! 🙌
  
  
Posted one month ago
Votes Newest

Answers 2


Thank you Eugen, I was trying to use both methods but it resulted in an error earlier - so my fallback was a bash script.
I will try again and post my error logs if I'm unsuccessful there.

  
  
Posted 28 days ago

Hi @<1861218295315697664:profile|FlutteringLobster45> ! You should use PipelineController.create and PipelineController.enqueue in this case, to create a pipeline without executing it at all remotely and enqueue it for remote execution

  
  
Posted 28 days ago