yep, in most of them:

/opt/clearml/config
apiserver.conf
clearml.conf

/opt/clearml/data/elastic_7
/nodes

/opt/clearml/data/fileserver
<empty>

/opt/clearml/data/mongo/configdb
<empty>

/opt/clearml/data/mongo/db
collection/index files, /diagnostic.data, /journal etc

/opt/clearml/data/redis 
dump.rdb

/opt/clearml/logs
apiserver.log.x, filserver.log (0 bytes)

so am I right in thinking it's just the mount points that are missing?based on the output of df above

not yet, going to try and fix it today.

if I do a df I see this, which is concerning:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  928K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/nvme0n1p1   20G  7.9G   13G  40% /
tmpfs           790M     0  790M   0% /run/user/1000

so it looks like the mount points are not created. When do these get created? I thought using an AMI these would have already been setup?

container_name:
  logging:
    options:
      max-size: 10m

so yes indeedly ..

sudo find /var/lib/ -type d -exec du -s -x -h {} \; | grep G | more

seems to give saner results.. of course, in your case, you may also want to grep M for megabyte

also, is there a list anywhere with the mount points that are needed?

I believe you can set it on a 'per container' way as well.

think I found the issue, a typo in apiserver.conf

I think that if these directories are not mounted, you should first of all take care not to shut down the server. You'll probably want to exec /bin/bash into the mongo and elastic containers, and copy their data outside to the host storage

incidentally we turn off the server every evening as it's not used overnight, we've not faced issues with it starting up in the morning or noticed any data loss

Not necessarily, is there any data in those directories?

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such

Basically whatever was under the old /opt/trains/ folder is required, you can see the list here: None

@<1687643893996195840:profile|RoundCat60> you set it once, inside the docker-compose itself.. it will affect all docker containers but, to be honest, docker tends to log everything

no, they are still rebooting. i've looked in /opt/clearml/logs/apiserver.log no errors

In the publicly available AMI these are created. However, if you used a previously released Trains AMI and upgraded to ClearML, part of the upgrade process was to create those directories (required by the new docker-compose.yml ), as explained here: None

we turn off the server every evening...

In that case the issue is definitely not related to the mount points

btw - if you remove the docker-compose changes, do the containers start normally?

yeah, that's usually the case when you get an empty dashboard

Try looking at their logs

... from the AMI creation script:

# prepare directories to store data
sudo mkdir -p /opt/clearml/data/elastic_7
sudo mkdir -p /opt/clearml/data/redis
sudo mkdir -p /opt/clearml/data/mongo/db
sudo mkdir -p /opt/clearml/data/mongo/configdb
sudo mkdir -p /opt/clearml/logs
sudo mkdir -p /opt/clearml/config
sudo mkdir -p /opt/clearml/data/fileserver
sudo chown -R 1000:1000 /opt/clearml/data/elastic_7

So it seems the AMI is using the correct directories... Do you have these?

it looks like clearml-apiserver and clearml-fileserver are continually restarting

🤔 i'll add the logging max_size now and monitor over the next week

you will probably want to find the culprit, so a find should work wonders. I probably suspect elasticsearch first. It tends to go nuts 😕

Check sudo docker logs <container-name>

After making the change yesterday to the docker-compose file, the server is completely unusable - this is all I see for the /dashboard screen

Oh, that's strange. I'll run one of those soon to see if there's anything wrong with them

Good point @<1523715084633772032:profile|AlertBlackbird30> 👍

strange, I used one of the publicly available AMIs for ClearML (we did not upgrade from the Trains AMI as started fresh)

Hi @<1687643893996195840:profile|RoundCat60> ,
We've actually never had to address this issue. Can you find out what exactly is growing in size? I'd like to make sure this is not due to the containers storing data internally (causing docker to store more and more snapshots) - this is an unhealthy situation that might also indicate that volumes are not mounted correctly (i.e. data that should be stored externally is actually stored internally)