Hi All, I'M Trying To Create Pipeline With Decorator. When I Run The Code To Push The Pipeline To Clearml Server, It Seems The Pipeline Is Pending In "Services" Queue. But My Worker Is Listening To "Default" Queue, So The Pipeline Is Not Executed. I Read

Answered

Hi all,
I'm trying to create pipeline with decorator. When I run the code to push the pipeline to clearml server, it seems the pipeline is pending in "services" queue. But my worker is listening to "default" queue, so the pipeline is not executed. I read in the doc https://clear.ml/docs/latest/docs/pipelines/pipelines_sdk_function_decorators and there are 2 arguments:
default_queue - The default https://clear.ml/docs/latest/docs/fundamentals/agents_and_queues#what-is-a-queue in which to enqueue all pipeline steps
pipeline_execution_queue - The queue in which to enqueue the pipeline controller task
so these 2 queues supposed to be different, right? How should I execute my pipeline? Should I set the pipeline_execution_queue to "default" or should I let my worker to listen on "services" queue?
Thanks

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyWoodpecker71
				
					0
					 × 1

Votes Newest

Answers 2

thank CostlyOstrich36 so should I point the clearml-agent to listen to "services" queue instead of "default" queue? Would it be able to process the pipeline steps?
If I understand correctly, the agent will spin additional pod to run the steps in the pipeline, right?

Also just another question: How do I set the git credential for the agent? In the values file https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml-agent/values.yaml
look like I can set git user and pass, but can I set ssh key instead?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyWoodpecker71
				
					0
					 × 1

Hi PunyWoodpecker71 ,

It's best to run the pipeline controller in the services queue because the assumption is that the controller doesn't require much compute power as opposed to steps that can be resource exhausting (depends on pipeline of course)

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

1K Views

2 Answers

2 years ago

one year ago