Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone. I Have A Strange Problem With Pipelines: When I Start A Pipeline To Run In A Queue The Pipeline Is Stuck At The First Pipeline Component. That Is, The Pipeline Itself Is Started And Run (By The Worker Who Observes The Queue) Until The Point W

Hi everyone.
I have a strange problem with pipelines: When I start a pipeline to run in a queue the pipeline is stuck at the first pipeline component. That is, the pipeline itself is started and run (by the worker who observes the queue) until the point when the first component is called. I can see the component being scheduled in the queue as the next experiment to run, but it is never enqueued. I get no error, the pipeline is just stuck. This happens with multiple pipelines I tested, including clearml example pipelines. All pipelines run normally with .run_locally(). The agent runs in docker mode. Standard experiments run like charm with my setup. Any ideas about the cause of this or how I should inspect this further?

  
  
Posted 5 months ago
Votes Newest

Answers 3


Hi @<1627478122452488192:profile|AdorableDeer85> might you have been able to resolve this? I also gave the same challenge with remote execution

  
  
Posted 3 months ago

Hi @<1627478122452488192:profile|AdorableDeer85> , can you provide a code snippet that reproduces this?

What happens if you remove the run_locally() ?
And just making sure - the first pipeline step isn't even pushed into a queue? It remains in 'draft' mode?

  
  
Posted 5 months ago

Hi @<1523701070390366208:profile|CostlyOstrich36> , sorry for my late reply.

My description was not clear enough, what I mean is:

  1. I run a pipeline with run_locally() . This can be any pipeline e.g. the example pipelines like this one None . The pipeline completes.
  2. Then I open the successfully run pipeline in the UI and rerun it in a queue. The queue is observed by agents running in docker mode.
  3. The overall pipeline is "taken out" of the queue and shown as running on the worker. BUT then the pipeline execution does not proceed. I see the first pipeline component (that is the first single step) being enqueued as the next experiment to come but it is never dequeued and never started.
  
  
Posted 5 months ago
321 Views
3 Answers
5 months ago
3 months ago
Tags
Similar posts