See https://allegroai-trains.slack.com/archives/CTK20V944/p1604982548295300

Change to lower than what you have http://free.in your disk 🙂

SuccessfulKoala55 here it is

cluster.routing.allocation.disk.watermark.low:

The Trains Server Elastic instance has gone into read-only mode due to insufficient disk space

See the watermark settings under elasticsearcg

And the docker-compose you currently use there?

All three 🙂

this is the df -h output

worked

I assume it's in the home dir

How did it come to this? I didn't configure anything, I'm using the trains AMI, with the suggested instance type

this?

Perhaps it's configured with absolute numbers... like 10GB free space?

Check the free disk space

SuccessfulKoala55

Where is the docker-compose file? It's not at /opt (again, I didn't place it anywhere, I'm just using the ami)

you want to see its contents?

The Elastic defaults are to go into this mode when less than 10 or 5% of the disk space is available. I assume logs? Perhaps we should change the instructions to use a large EBS disk size or change the default EBS size in the Trains AMI

I've updated the default docker-compose to use 2gb in these watermarks

Try ~

I only have like 40 tasks including the example ones

Or should I change all three of them?