Hello Everyone, We’Re Encountering A Persistent Issue With Our Autoscaler Setup And Could Really Use Some Help. Despite Having The Autoscaler Running And The Queue (Default_Cpu) Properly Populated (87 Jobs Pending), The Tasks Are Never Picked Up And Exe

Unanswered

Hi,
It looks like the same issue is happening. It seems to be caused by the recent update of the clearml-agent package to version 2.0.0 .
When I start the queue locally, the agent appears in the list but doesn't pick up any tasks. On the agent side, I get the following error:

FATAL ERROR:
Traceback (most recent call last):
File "***.venv/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 2128, in daemon
self.run_tasks_loop(
File "***.venv/lib/python3.12/site-packages/clearml_agent/commands/worker.py", line 1464, in run_tasks_loop
self.monitor.setup_daemon_cluster_report(worker_id=self.worker_id, max_workers=1)
File "***.venv/lib/python3.12/site-packages/clearml_agent/helper/resource_monitor.py", line 283, in setup_daemon_cluster_report
self._cluster_report.pending = True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'pending'

  				
Posted 
	one month ago

					More
				  		
  Report
		
					IdealCamel90
				
					0
					 × 1

41 Views

0 Answers

one month ago