@<1523701601770934272:profile|GiganticMole91> Thats rookie numbers. We are at 228 GB for elastic now
@<1722061389024989184:profile|ResponsiveKoala38> cool, thanks! I guess it will then be straightforward to script then.
What is your gut feeling regarding the size of the index? Is 87G a lot for an elastisearch index?
Hi @<1523701601770934272:profile|GiganticMole91> , each scalar document in ES has a "task" field that is a task ID. The below query will show you the first 10 documents for the task ID:
curl -XGET "localhost:9200/<the scalar index name>/_search?q=task:<task ID>&pretty"
Hi @<1523701601770934272:profile|GiganticMole91> , As long as experiments are deleted then their associated scalars are deleted as well.
I'd check the ES container for logs. Additionally, you can always beef up the machine with more RAM to give elastic more to work with.
Any tips on how to check if we are storing data on deleted tasks? Maybe @<1722061389024989184:profile|ResponsiveKoala38> knows? Is there a field on each scalar that I can cross check with ClearML?
WebApp: 1.16.0-494 • Server: 1.16.0-494 • API: 2.30
But be careful, upgrading is extremely dangerous
Hi @<1523701070390366208:profile|CostlyOstrich36>
Is 87G a lot for an index? Enough that you would consider adding more RAM?
And also, how can I check that we are not storing scalars for deleted tasks? ClearML used to write a lot of errors in the cleanup script, although that seems to have been fixed in recent updates
Can confirm that for me usually increasing RAM solves the problem. ES is sometimes very aggressive.
@<1590514584836378624:profile|AmiableSeaturtle81> that’s the service we are using :-)
How much RAM have you assigned to your elastic service?
7 out of 30 GB is currently used and is quite stable
Yes, I tried updating recently, it costed me a full days work of rolling back versions until I found something that worked 😅
Which version of the server are you running?
@<1590514584836378624:profile|AmiableSeaturtle81> this was last time i tried: https://clearml.slack.com/archives/CTK20V944/p1725534932820309
What you want is to have a service script that cleans up archived tasks, here is what we used: None