Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Everyone! I'Ve Been Using Clearml For Few Years Now And Love It. Recently Started Getting This Error When I Try To Enqueue A Task:

Hey everyone! I've been using ClearML for few years now and love it. Recently started getting this error when I try to enqueue a task:

Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
2025-07-11 08:58:33,914 - clearml.log - WARNING - failed logging task to backend (5 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)
Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
Action failed <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>
2025-07-11 08:59:33,945 - clearml.log - WARNING - failed logging task to backend (1 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)

This works whether I try to enqueue the task from python via Task.enqueue(cloned_task.id, queue_name='gcp-l4') or whether I do it manually in ClearML server frontend (right click a 'Draft' task and click 'Enqueue')

I also get this error when I try to create the template task:

2025-07-11 10:35:21,353 - clearml.log - WARNING - failed logging task to backend (2 lines, <404/0: events.add_batch (<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
)>)
ClearML Terminating local execution process - continuing execution remotely
  
  
Posted one month ago
Votes Newest

Answers 8


Hi @<1523705004920147968:profile|CloudySwallow27> , what are you seeing in the apiserver container?

  
  
Posted one month ago

ok

  
  
Posted one month ago

Can you add a full log from startup of both Elastic and apiserver containers?

  
  
Posted one month ago

Hi @<1523701070390366208:profile|CostlyOstrich36> .

  • At first I was getting alot of errors there showing it couldnt connect to clearml-elastic (urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='elasticsearch', port='9200'): Read timed out. (read timeout=60)) so i reset both apiserver adn clearml-elastic.
  • After the reset, when I tried deleting in the frontend, I got more the informative error on the frontend: General data error (TransportError(503, 'search_phase_execution_exception', '[clearml][172.20.0.2:9300][indices:data/read/search[phase/query]]')) . In the docker logs on apiserver, I am getting: :
[2025-07-14 17:51:07,439] [9] [WARNING] [elasticsearch] POST 
 [status:503 request:0.009s]
[2025-07-14 17:51:07,449] [9] [WARNING] [elasticsearch] POST 
 [status:503 request:0.010s]
[2025-07-14 17:51:07,512] [9] [WARNING] [elasticsearch] POST 
 [status:503 request:0.012s]

Not sure what to do from here though...

  
  
Posted one month ago

It seems to an a memory issue w/ the VM that hosts clearml filling up. I am trying to delete some experiments but now i get:
image

  
  
Posted one month ago

I get that error whether I select "Remove all related artifacts and debug samples from ClearML file server" or not

  
  
Posted one month ago

@<1856144902816010240:profile|SuccessfulCow78> can you please help provide

  
  
Posted one month ago

@<1855782498290634752:profile|AppetizingFly3>

  
  
Posted one month ago