Yeah, indeed that it. ClearML should still be able to read the data from it, but it can't add new data. In general I would advise to maintain your instance, delete old stuff periodically and make sure you don't get to this state 🙂
looking into ES index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b
docs.count docs.deleted store.size pri.store.size
2118131043 29352476 265.1gb 265.1gb
sounds we're hitting some ES limitation?
unfortunately I couldn't fix this
the ES state in hectic, can't delete anything
clearml is still live, read-only mode, all existing indices are readable
new jobs can't write to this clearml server
@<1523701842515595264:profile|PleasantOwl46> were you able to fix this?
@<1523701087100473344:profile|SuccessfulKoala55> what might be a fix for this?