ClearML FAQ | [Auto Scaler / Api Client Does Not See Tasks In Queue]

Answered

[Auto Scaler / Api Client Does Not See Tasks In Queue]

[Auto scaler / API client does not see tasks in queue]
We had used the AWS auto scaler (based on the aws_autoscaler.py script in the repo) and it worked great.
Now we have implemented a GCP auto scaler script after we have switched clouds (happy to share that btw, it is very lightweight.)
However, the clearml.automation.auto_scaler.AutoScaler object does not seem to see the tasks in the specified queue - any idea why that is?

Specifically, I’ve tested the code:

autoscaler.api_client.queues.get_by_id("QUEUE ID").entries # Returns []
autoscaler.api_client.queues.get_all() # Successfully returns all the queues, including the "a100-auto-spot" one

I had queued 2 experiments into that queue and when I started an agent by hand using clearml-agent daemon --queue a100-auto-spot --gpus all --detached and it starts working on the tasks right away (still loving the Agent functionality btw 😉 )

Any help is appreciated!

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1523701087100473344:profile|SuccessfulKoala55> , sorry there was a mistake on my end - clearml.conf pointed to the wrong URL 🙈

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Hi @<1523702496097210368:profile|ScantChimpanzee51> , this is very strange 😕 ... can you confirm the queue ID you're sending is identical to the one returned by get_all?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

897 Views

2 Answers

one year ago

one year ago

Tags

Similar posts

I Am Using Version 1.16.3, And There Is A New Phenomenon That When I Run A Controller With Several Tasks Or Even Just One When The Task Ends, It Is Once Again Assigned To The Queue It Was In And Is Executed Repeatedly This Phenomenon Was Not In Previous V

Hi Everyone, I Am New To Clearml And Am Trying To Use Clearml Agent With A Docker Image For Run A Pipeline Remotely. My Agent Is Running And It Is Able To Pick Up The Tasks From The Queue. It Starts Running The Task Inside Docker. However, It Exits The T

Hey All, In Amazon Aws, Using The Auto Scaler, I Want To Mount My Efs On All Ec2 Workers Who Get Spawned By The Auto Scaler. Assuming I Edited My Security Group Rules, And Updated My Iam Role So It Could Access The Efs, How Do I Mount The Efs To The Machi

Hello. In Which Yaml Files For Kubernetes And Where Should I Specify The Aws File_Server And Credentials For Clearml-Agent So That When Running Tasks Remotely In The Queue, The Artifacts Are Saved In The S3 Bucket?

Hey Everyone, I'M New To Clearml Pipelines And My Tasks Are Always In Pending. 1) I Have Created Clearml Agent And It Is Connected To The Main Clearml Server. 2) Agent Configured To Use Docker With Mounted Volumes, Does Listen To The "Services" Queue And

Another Question About The Aws Auto_Scaler, I Managed To Get The Scaler Service Running, It Listens To A Queue Called "Aws" (Yes, Very Imaginative), When I Queue A Simple Task I See That It'S Being Handled But The Task Is Stuck As Pending. When Checking T

Hey There! I Kindly Ask For Your Swarm Knowledge On Clearml Pipelines. I'M Trying To Setup A Simple Pipeline With A Controller Running On The Service-Queue And Three Tasks, That Are Added With

Hi All, What Is The Best Way To Monitor Failer Clearml Agent That Kill All Tasks In Queue?

Is It Normal For The Agent To Just Give 0 Output When Run In Docker? At Most It Gives Me "Using Environment Access Key...." But It'S Not Pulling Tasks From A Queue And Zero Other Output Also Is There A Docker Version Requirement?

[Distributed Training] Hi, I Have A Clearml Setup With K8Sglue That Spins Up Pods Of 4 Gpus When Picking Tasks Off The Clearml Queue. We Would Now Want To Proceed With Multi-Node Training, And Some Of The Examples We Are Trying Are Here.