Yeah, the agent and trains itself are configured to keep retrying for a fairly long time, so a server downtime (as long as it's not down for hours) will not affect your experiment.
Thanks, SuccessfulKoala55 So if that's the case what will happened if I will stop server (docker-compose stop) while agent is running an experiment? I tested it, and it seems that the experiment is continue running without any issues.
Having said that, we use the publicly-available dockers for elastic, redis and mongodb, and you can absilutely read their documentation and see the available backup procedures for each one. For example, for mongodb and elastic:
https://docs.mongodb.com/manual/core/backups/
https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
Hi AstonishingSeaturtle47 ,
As specified in section #6 in the Trains-Server launching instructions ( https://github.com/allegroai/trains-server/blob/master/docs/install_linux_mac.md#launching-the-trains-server-docker-in-linux-or-macos ), the data folder contains data for elastic, redis, mongodb and the fileserver. Of those, the fileserver data is totally safe to copy while the system it up. As for elastic, redis and mongodb, copying the files while the system is up is not considered safe - you can try it, but I wouldn't recommend it (any activity while you do it might cause you to end up with corrupted data).