See the watermark settings under elasticsearcg

The Elastic defaults are to go into this mode when less than 10 or 5% of the disk space is available. I assume logs? Perhaps we should change the instructions to use a large EBS disk size or change the default EBS size in the Trains AMI

Where is the docker-compose file? It's not at /opt (again, I didn't place it anywhere, I'm just using the ami)

The Trains Server Elastic instance has gone into read-only mode due to insufficient disk space

See https://allegroai-trains.slack.com/archives/CTK20V944/p1604982548295300

Change to lower than what you have http://free.in your disk 🙂

I've updated the default docker-compose to use 2gb in these watermarks

this is the df -h output

And the docker-compose you currently use there?

you want to see its contents?

worked

cluster.routing.allocation.disk.watermark.low:

Perhaps it's configured with absolute numbers... like 10GB free space?

SuccessfulKoala55

How did it come to this? I didn't configure anything, I'm using the trains AMI, with the suggested instance type

Check the free disk space

I only have like 40 tasks including the example ones

Try ~

this?

Or should I change all three of them?

SuccessfulKoala55 here it is

I assume it's in the home dir

All three 🙂