back up and running again, thanks for your help

It looks like not all the containers are up... Try sudo docker ps and see if the apiserver container is restarting...

not entirely sure on this as we used the custom AMI solution, is there any documentation on it?

Morning, we got to 100% used which is what triggered this investigation. When we initially looked at overlay2 it was using 8GB, so now seems to be acceptable.

so am I right in thinking it's just the mount points that are missing?based on the output of df above

it looks like clearml-apiserver and clearml-fileserver are continually restarting

I believe you can set it on a 'per container' way as well.

thanks Stef, with max-size do you set it for every running service separately, or can you set it once?

yeah, that's usually the case when you get an empty dashboard

Check sudo docker logs <container-name>

no, they are still rebooting. i've looked in /opt/clearml/logs/apiserver.log no errors

also, is there a list anywhere with the mount points that are needed?

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such