Hi @<1523701601770934272:profile|GiganticMole91> , As long as experiments are deleted then their associated scalars are deleted as well.
I'd check the ES container for logs. Additionally, you can always beef up the machine with more RAM to give elastic more to work with.
Hi @<1523701070390366208:profile|CostlyOstrich36>
Is 87G a lot for an index? Enough that you would consider adding more RAM?
And also, how can I check that we are not storing scalars for deleted tasks? ClearML used to write a lot of errors in the cleanup script, although that seems to have been fixed in recent updates
Can confirm that for me usually increasing RAM solves the problem. ES is sometimes very aggressive.
Any tips on how to check if we are storing data on deleted tasks? Maybe @<1722061389024989184:profile|ResponsiveKoala38> knows? Is there a field on each scalar that I can cross check with ClearML?
Hi @<1523701601770934272:profile|GiganticMole91> , each scalar document in ES has a "task" field that is a task ID. The below query will show you the first 10 documents for the task ID:
curl -XGET "localhost:9200/<the scalar index name>/_search?q=task:<task ID>&pretty"
@<1722061389024989184:profile|ResponsiveKoala38> cool, thanks! I guess it will then be straightforward to script then.
What is your gut feeling regarding the size of the index? Is 87G a lot for an elastisearch index?
@<1523701601770934272:profile|GiganticMole91> Thats rookie numbers. We are at 228 GB for elastic now
What you want is to have a service script that cleans up archived tasks, here is what we used: None
@<1590514584836378624:profile|AmiableSeaturtle81> that’s the service we are using :-)
How much RAM have you assigned to your elastic service?
7 out of 30 GB is currently used and is quite stable
Which version of the server are you running?
WebApp: 1.16.0-494 • Server: 1.16.0-494 • API: 2.30
But be careful, upgrading is extremely dangerous
Yes, I tried updating recently, it costed me a full days work of rolling back versions until I found something that worked 😅
@<1590514584836378624:profile|AmiableSeaturtle81> this was last time i tried: https://clearml.slack.com/archives/CTK20V944/p1725534932820309