So I'd allocate more storage for the elasticsearch PV, and try to restart the service
You can probably see the error in the ES logs
PleasantGiraffe85 this error originates form the ES service
So, a new question 🙂
I get a message:clearml.log - WARNING - failed logging task to backend (2 lines, <500/4: events.add_batch/v1.0 (Critical server error! server reports low or insufficient disk space. please resolve immediately by allocating additional disk space or freeing up storage space. (metrics, logs and all indexed data is in read-only mode!): reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>)
Obviously the volume isn't enough - but which one? I have 6 volumes connected to the ClearML:
agent-services api-server fileserver mongodb redis elasticsearchI tried resizing all of them... but the warning persist 😕 is there anywhere I can see a proper log or so?
So I'd allocate more storage for the elasticsearch PV, and try to restart the service
You can probably see the error in the ES logs
PleasantGiraffe85 this error originates form the ES service