Thanks AppetizingMouse58 . I managed to fix it by removing docker completely and reinstalling it.

Hi WittyOwl57 , there is a chance that the reason is in this setting: Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log ...
First it say says about invalid log option that may require further investigation. Second the file that it tries to write to is logs/gc.log and it is not under the $clearml folder where you give the write permissions to the ES user. I would try cancelling JVM logging at all or specifying the full path to the file so that it would be under the folder that has 1000:1000 owner. Please see more here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#gc-logging

I didn't add that to the script since the effect is persistent (i.e. it only needs to be done once, right?) In any case, I checked that multiple times and it was as expected.

WittyOwl57 what about? vm.max_map_count echo "vm.max_map_count=262144" > /tmp/99-clearml.conf
sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
sudo sysctl -w vm.max_map_count=262144
sudo service docker restart `https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac (5)

Hi SuccessfulKoala55 ,yes, I am running as sudo. It's not my first time setting this up either. Just that on this vm (which is a particularly large instance) I get this issue which has me stumped.

WittyOwl57 maybe you have a filesystem permissions issue? Are you running this as root (or using sudo)?
Error opening log file 'logs/gc.log': Permission denied

WittyOwl57 not much info here but since it seems to fail on the Java options, I suspect AgitatedDove14 is correct and the most likely cause is not enough RAM?

WittyOwl57 could it be the EC2 instance is too small (i.e. not enough storage / memory) ?

I actually put all the commands in a script. The failure mode is exactly the same. I have no idea what to do next.
` #!/bin/bash

clearml_root=$1

if [[ $# -gt 0 ]]; then
echo Using "$1" as root
else
echo No root argument was provided, using /datadrive1
clearml_root=/datadrive1
fi

clearml="$clearml_root/clearml"

rm -R "$clearml"

mkdir -p "$clearml"/data/elastic_7
mkdir -p "$clearml"/data/mongo_4/db
mkdir -p "$clearml"/data/mongo_4/configdb
mkdir -p "$clearml"/data/redis
mkdir -p "$clearml"/logs
mkdir -p "$clearml"/config
mkdir -p "$clearml"/data/fileserver

dc_yaml="$clearml"/docker-compose.yaml

curl -o "$dc_yaml"
sed -i 's#/opt/clearml#'"$clearml"'#g' "$dc_yaml"

chown -R 1000:1000 "$clearml"

docker-compose -f "$dc_yaml" down
docker-compose -f "$dc_yaml" up -d `

Hi AgitatedDove14 , I deleted everything in /opt/clearml as per the docs. Should I delete anything else?

Hi WittyOwl57
Are you starting a new server from scratch or is it running on previously stored data?