Hi All, I Have An Ongoing Issue With Queues.

Answered

Hi all, I have an ongoing issue with queues.

I send a task to my_queue which does not have any listening agents. This is done using task = Task.init(...) and task.execute_remotely("my_queue") .
When I look in Queues ( None ) I see my task and I see that the task is in Pending state in projects.
After a few minutes (it varies from a minute to 6 minutes), the task disappears from the queue. There are no listening agents so there is no reason for it to be gone. Furthermore, when I look at the Task, it is still in a Pending state. I have confirmed this using the python SDK.
This has led me to conclude that there is a bug in the backend for the queues.
This is a major issue for us because we use the AWS Autoscaler for basically all our compute and when I send a job to the queue, the Autoscaler creates a new instance but by the time that instance is ready, the Task is no longer in the queue (but is still in Pending!) so it doesn't get picked up.

Has anyone else experienced this behaviour?

Cheers,
James

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					AmusedCat74
				
					0
					 × 1

Votes Newest

Answers

Hi,

I've managed to fix it.

Basically, I had a tracker running on our queues to ensure that none of them were lagging. This was using get_next_task from APIClient().queues .

If you call get_next_task it removes the task from the queue but does not put it into another state. I think because typically get_next_task is immediately followed by something to make the task run in the daemon or delete it.

Hence you end up in this weird state were the task thinks its queued because nothing modified the status but the queue doesn't think it has a task. A bit of an edge case but potentially worth looking at.

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					AmusedCat74
				
					0
					 × 1

Write your answer

497 Views

1 Answer

9 months ago

Hi, Does Anyone Faced This Problem? I'M Trying To Run A Task From A Queue. The Same Task Is Running Locally Just Fine. Moreover, I Have An Experience Running Large Scale Of Experiments From Clearml'S Queues, This Problem Is New. After Setting The Env, I G

Another Question About The Aws Auto_Scaler, I Managed To Get The Scaler Service Running, It Listens To A Queue Called "Aws" (Yes, Very Imaginative), When I Queue A Simple Task I See That It'S Being Handled But The Task Is Stuck As Pending. When Checking T

My Gcp Auto Scalar Spins Up The Box, But The Task Does Not Get Executed, The Status Remains Pending, I Did Use The Same Schedular Long Ago And It Was Working Fine, Is It Possible That Somehow Worker Or The Ques Or May Be The Schedular Got Messed For Some

How Will I Add The Agent, So That It Listens To The Particular Que..My Task Goes To Pending And Seems Like No Worker Is Running.

My Task Is In Pending State, After I Enqueued The Task Its In Pending State, Need Help!!

Hi We Just Got The Aws Autoscaler To Create A New Instance When You Enqueue A Task To The Relevant Queue. However, For Some Reason The Task Itself Is Never Run, It Stays In The Pending State. When Looking At The Worker Details, It Says "No Queues Curren

Hi I Have A Problem (Presumably) With K8Sglue-Agents Working Roughly 50% Of The Time. Basically What Happens Is When I Launch A Pipeline From Command Line, It Ends Up In "Pending" State And Dissapears From All Of The Queues. Initially It First Appears On

Hi I Upgraded Some Of My K8S Glue To Use The Latest Clearml-Agent 1.1.0 And Receive This Error When The Agent Pulls A Task. Pulling Task Xxxbbbxxxbb Launching On Kubernetes Cluster Pushing Task Xxxbbbxxxbb Into Temporary Pending Queue Kubernetes Scheduli

Hi, We Noted That Using K8S Glue, There Are Some Situations Where The Task Cannot Be Registered As Error And Will Be Stuck At Pending. An Example Of One Situation Is When The Task Is Pulling A Docker Image That Doesn'T Exist. Is There A Way To Catch Such