Sure 🙂 - Elastic takes a long time to reindex 😄
You mean the migration from Trains Server 0.15 to 0.16?
how long? 😅
I am now stuck inCopying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b
for more then 40 min 😥
Do you have a lots of data on the server?
Hi AppetizingMouse58 , I had around 200GB when I started the migration now I have 169GB/
And yes, It looks it is growing was 9.4GB and now 9.5G
The script is running for more then 45 min, does it regular?
it took around 50 min but it now finish the backup..Thanks
I think you're good - ES takes a long time reindexing very large amounts of data, especially if you machine is not a monster 🙂
Hi SuccessfulKoala55 Thanks for the replay..
So for now, if I like to upgrade to the latest trains-server but on another machine and keep all the data.
what is the best practice?
Thanks again 🙂
SuccessfulKoala55 it still stuck on the same line .. does it should be like this?
To move your data from one machine to another, see "Backing up and restoring data and configuration" in https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/
Hey SuccessfulKoala55 again, how long does the migration script should take?
Hi CooperativeFox72 ,
The ClearML Server was not yet released - it should be out within several days. Once it's out, the upgrade instructions will include a section addressing your exact situation.
In short, yes, the same process would apply, with a tiny additional step 🙂
It is stuck, or do you keep seeing output from the script?
yes it looks like this.. I just wanted to understand if it is should be so slow.. or I did something wrong
it stuck on Creating the backup file: /opt/trains/data/elastic_5.tar.gz
Hi CooperativeFox72 , how much free space do you have on your disk now? If you run du on your /opt/trains/data/elastic_7 folder in let's say 5 mins intervals do you see the folder size is growing?
Well, 30GB might take a long time... AppetizingMouse58 do you have an idea?