Hi, I tried to run an experiment and got this error.
clearml.log - WARNING - failed logging task to backend (1 lines, <500/4: events.add_batch/v1.0 (Critical server error! server reports low or insufficient disk space. please resolve immediately by allocating additional disk space or freeing up storage space. (metrics, logs and all indexed data is in read-only mode!): reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];)>)
Well, the error is pretty clear:Critical server error! server reports low or insufficient disk space. please resolve immediately by allocating additional disk space or freeing up storage space. (metrics, logs and all indexed data is in read-only mode!
This is something that needs to be addressed on the machine level. Deleting experiments tries modifying metrics/logs etc. but these are read-only at this stage...
