looking into ES index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b

docs.count docs.deleted store.size pri.store.size
2118131043     29352476    265.1gb        265.1gb

sounds we're hitting some ES limitation?

Yeah, indeed that it. ClearML should still be able to read the data from it, but it can't add new data. In general I would advise to maintain your instance, delete old stuff periodically and make sure you don't get to this state 🙂

unfortunately I couldn't fix this
the ES state in hectic, can't delete anything
clearml is still live, read-only mode, all existing indices are readable
new jobs can't write to this clearml server

@<1523701842515595264:profile|PleasantOwl46> were you able to fix this?
@<1523701087100473344:profile|SuccessfulKoala55> what might be a fix for this?