Hi, We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17 Suddenly All Experiments Were Removed. After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead

Unanswered

Thanks CostlyOstrich36
Actually I was able to find IP of the machine where API call was triggered in web logs and found the user who run the delete action.
User tried to remove only archived experiments in his project( tried several times and got some errors ) and that is what we see in API call - somehow Clearml removed all server experiments 🤔
Any idea why this might happened if user only run "delete archived experiments of his project" in WEB UI ?
xx.xxx.xxx.xx - - [21/Nov/2022:17:39:14 +0000] "POST /api/v2.17/tasks.delete_many HTTP/1.1" 499 0 " " "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" "-"

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					LackadaisicalHedgehong78
				
					0
					 × 1

173 Views

0 Answers

2 years ago

one year ago