Unanswered
Hi All,
I’M Running Experiments Using Clearml.
The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:
From what I understand, by default the ES has a low disk waterkmark set at 95% of the disk capacity. Once reached the shard is transitioned to a read only mode. Since you have a large disk of 1.8Tb the remaining 85Gb is below the 5%.
Basically you need to set the following env vars in elasticsearch service in the docker compose:- cluster.routing.allocation.disk.watermark.low=10gb - cluster.routing.allocation.disk.watermark.high=10gb - cluster.routing.allocation.disk.watermark.flood_stage=10gb
148 Views
0
Answers
2 years ago
one year ago