Unanswered
Hi,
We Have Clearml Version Webapp: 1.3.0-165 • Server: 1.3.0-165 • Api: 2.17
Suddenly All Experiments Were Removed.
After Checking Logs We See That Before 6 Days Elasticsearch Container Was Terminated Due To Out Of Memory And New One Was Created Instead
We tried with copying on test machine and it worked(delete and then restore tasks in DB - it appears again in UI).
When did same on prod - nothing happened.
Also we see all data in the fileserver is not removed and mongo shows 17000 tasks so it looks like tasks were removed from UI but still appear in Mongo DB and locally at file system.
CostlyOstrich36
` > db.task.count()
17262
db.task__trash.count()
383
db.task__trash__trash.count()
15
db.task__trash.aggregate([ {$merge: "task"}])
db.task__trash__trash.count()
15
db.task__trash.count()
385
db.task.count()
17647 `
164 Views
0
Answers
2 years ago
one year ago