Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

Answered

Hey everyone! I've been using ClearML for few years now and love it. Recently started getting this error when I try to enqueue a task:

Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
2025-07-11 08:58:33,914 - clearml.log - WARNING - failed logging task to backend (5 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)
Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
2025-07-11 08:59:33,945 - clearml.log - WARNING - failed logging task to backend (1 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)

This works whether I try to enqueue the task from python via Task.enqueue(cloned_task.id, queue_name='gcp-l4') or whether I do it manually in ClearML server frontend (right click a 'Draft' task and click 'Enqueue')

I also get this error when I try to create the template task:

2025-07-11 10:35:21,353 - clearml.log - WARNING - failed logging task to backend (2 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)
ClearML Terminating local execution process - continuing execution remotely

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

Votes Newest

Answers 8

Can you add a full log from startup of both Elastic and apiserver containers?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

@<1855782498290634752:profile|AppetizingFly3>

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

@<1856144902816010240:profile|SuccessfulCow78> can you please help provide

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

Hi @<1523705004920147968:profile|CloudySwallow27> , what are you seeing in the apiserver container?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

It seems to an a memory issue w/ the VM that hosts clearml filling up. I am trying to delete some experiments but now i get:

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

I get that error whether I select "Remove all related artifacts and debug samples from ClearML file server" or not

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					SuccessfulCow78
				
					0

Hi @<1523701070390366208:profile|CostlyOstrich36> .

At first I was getting alot of errors there showing it couldnt connect to clearml-elastic (urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='elasticsearch', port='9200'): Read timed out. (read timeout=60)) so i reset both apiserver adn clearml-elastic.
After the reset, when I tried deleting in the frontend, I got more the informative error on the frontend: General data error (TransportError(503, 'search_phase_execution_exception', '[clearml][172.20.0.2:9300][indices:data/read/search[phase/query]]')) . In the docker logs on apiserver, I am getting: :

[2025-07-14 17:51:07,439] [9] [WARNING] [elasticsearch] POST

 [status:503 request:0.009s]
[2025-07-14 17:51:07,449] [9] [WARNING] [elasticsearch] POST

 [status:503 request:0.010s]
[2025-07-14 17:51:07,512] [9] [WARNING] [elasticsearch] POST

 [status:503 request:0.012s]

Not sure what to do from here though...

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CloudySwallow27
				
					0
					 × 1

Write your answer

532 Views

8 Answers

2 months ago