So the solution would be to custom develop a backup routine that is fetching the data increments from Elastic Search, MongoDB & Redis directly?
Yes. However I don't think you need to backup Redis as it holds only data related to the currently running agents.
Hi ObedientTurkey46 see below 🙂
Hi, I have two instances of clearml-server that I would like to merge into one. Is there any way to do that without loosing tracked experiments, datasets or artefacts?
This is not trivial, you would have to merge the databases somehow.
Additionally, do you if it is possible to backup clearml-server
without
shutting it down? If shutdown is mandatory, I have to force the Datascientists to stop trainings for the time of backup, which would not be great.
I think that's the only way to do it with the OS unless you procedurally backup the databases.
Elastic Search, MongoDB & Redis. They are all inside containers on the machine running the server
Thank you so much for your reply! Regarding merging - we have decided to overwrite one server by the other, so this issue is solved.
What Database is clearml using? How to access them?