incidentally we turn off the server every evening as it's not used overnight, we've not faced issues with it starting up in the morning or noticed any data loss

container_name:
  logging:
    options:
      max-size: 10m

hhrrmm.. in the initial problem, you mentioned that the /var/lib/docker/overlay2 was growing large in size.. but.. 4GB seems "fine" for docker images.. I wonder .. does your nvme0n1p1 ever report like 85% or 90% used or do you think that the 4GB is a lot ? when you restart the server, does the % used noticeably drop ? that would suggest tmp files inside the docker image itself which.. is possible with docker (weird but, possible)

it looks like clearml-apiserver and clearml-fileserver are continually restarting

🤔 i'll add the logging max_size now and monitor over the next week

thanks @<1523715084633772032:profile|AlertBlackbird30> this is really informative. Nothing seems to be particularly out of the ordinary though

3.7G	/var/lib/
3.7G	/var/lib/docker
3.0G	/var/lib/docker/overlay2

followed by a whole load of files that are a few hundred KBs in size, nothing huge though

I added this to each of the containers

    logging:
      options:
	    max-file: 5
        max-size: 10m

think I found the issue, a typo in apiserver.conf

After making the change yesterday to the docker-compose file, the server is completely unusable - this is all I see for the /dashboard screen

Howdy and Morning @<1687643893996195840:profile|RoundCat60> .. docker when using overlay2 doesn't have it's mount points show up in a 'df' btw, they will only appear in a 'df -a', mostly because since they are simply 'overlays', they don't (technically) consume any space (I mean, the files are still in the /var/lib but not for the space counting practices used by df)

this is why I was suggesting a find, maybe with a 'du' .. actually.. let me try that here.. 2s

It looks like not all the containers are up... Try sudo docker ps and see if the apiserver container is restarting...

back up and running again, thanks for your help

Not necessarily, is there any data in those directories?