Hi @<1523705004920147968:profile|CloudySwallow27> , what are you seeing in the apiserver container?
Can you add a full log from startup of both Elastic and apiserver containers?
Hi @<1523701070390366208:profile|CostlyOstrich36> .
- At first I was getting alot of errors there showing it couldnt connect to clearml-elastic (
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='elasticsearch', port='9200'): Read timed out. (read timeout=60)
) so i reset both apiserver adn clearml-elastic.
- After the reset, when I tried deleting in the frontend, I got more the informative error on the frontend:
General data error (TransportError(503, 'search_phase_execution_exception', '[clearml][172.20.0.2:9300][indices:data/read/search[phase/query]]'))
. In the docker logs on apiserver, I am getting: :
[2025-07-14 17:51:07,439] [9] [WARNING] [elasticsearch] POST
[status:503 request:0.009s]
[2025-07-14 17:51:07,449] [9] [WARNING] [elasticsearch] POST
[status:503 request:0.010s]
[2025-07-14 17:51:07,512] [9] [WARNING] [elasticsearch] POST
[status:503 request:0.012s]
Not sure what to do from here though...
It seems to an a memory issue w/ the VM that hosts clearml filling up. I am trying to delete some experiments but now i get:
I get that error whether I select "Remove all related artifacts and debug samples from ClearML file server" or not
@<1856144902816010240:profile|SuccessfulCow78> can you please help provide
@<1855782498290634752:profile|AppetizingFly3>