Hi devs :slightly_smiling_face: I am still having an issue that has been there for a long time: When doing a large batch of deletes (e.g. 100s of experiments with millions of steps each), the clearml-server API will become unresponsive for minutes to hour

Answered

Hi devs 🙂
I am still having an issue that has been there for a long time: When doing a large batch of deletes (e.g. 100s of experiments with millions of steps each), the clearml-server API will become unresponsive for minutes to hours.
This is something that I have experienced with every version so far. Do you have any idea, why it happens (maybe something like index rebuilding?) and how to mitigate it?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1523701868901961728:profile|ReassuredTiger98> , this simply might be an issue of too few handler processes un the ClearML apiserver (unless the pressure is on the databases). You can easily change that (the default is 8 ) using the CLEARML_GUNICORN_WORKERS environment variable passed to the apiserver service)

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Hi @<1523701087100473344:profile|SuccessfulKoala55> I tested it and this seems not to be the issue. When looking at my server I can see that ElasticSearch utilizes a single core with 100%. This is the only observation that I made. May it be that elasticsearch is the issue?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Hi Jake, thank you very much for the suggestion. I will try that!

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ReassuredTiger98
				
					0
					 × 1

Write your answer

485 Views

3 Answers

4 months ago