Hi Everyone. I Have A Strange Problem With Pipelines: When I Start A Pipeline To Run In A Queue The Pipeline Is Stuck At The First Pipeline Component. That Is, The Pipeline Itself Is Started And Run (By The Worker Who Observes The Queue) Until The Point W

Answered

Hi everyone.
I have a strange problem with pipelines: When I start a pipeline to run in a queue the pipeline is stuck at the first pipeline component. That is, the pipeline itself is started and run (by the worker who observes the queue) until the point when the first component is called. I can see the component being scheduled in the queue as the next experiment to run, but it is never enqueued. I get no error, the pipeline is just stuck. This happens with multiple pipelines I tested, including clearml example pipelines. All pipelines run normally with .run_locally(). The agent runs in docker mode. Standard experiments run like charm with my setup. Any ideas about the cause of this or how I should inspect this further?

  				
Posted 
	12 months ago

					More
				  		
  Report
		
					AdorableDeer85
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1627478122452488192:profile|AdorableDeer85> might you have been able to resolve this? I also gave the same challenge with remote execution

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					BeefyOwl35
				
					0
					 × 1

Hi @<1523701070390366208:profile|CostlyOstrich36> , sorry for my late reply.

My description was not clear enough, what I mean is:

I run a pipeline with run_locally() . This can be any pipeline e.g. the example pipelines like this one None . The pipeline completes.
Then I open the successfully run pipeline in the UI and rerun it in a queue. The queue is observed by agents running in docker mode.
The overall pipeline is "taken out" of the queue and shown as running on the worker. BUT then the pipeline execution does not proceed. I see the first pipeline component (that is the first single step) being enqueued as the next experiment to come but it is never dequeued and never started.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					AdorableDeer85
				
					0
					 × 1

Hi @<1627478122452488192:profile|AdorableDeer85> , can you provide a code snippet that reproduces this?

What happens if you remove the run_locally() ?
And just making sure - the first pipeline step isn't even pushed into a queue? It remains in 'draft' mode?

  				
Posted 
	12 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

778 Views

3 Answers

12 months ago

10 months ago