I think you're good - ES takes a long time reindexing very large amounts of data, especially if you machine is not a monster 🙂
yes it looks like this.. I just wanted to understand if it is should be so slow.. or I did something wrong
Hi AppetizingMouse58 , I had around 200GB when I started the migration now I have 169GB/
And yes, It looks it is growing was 9.4GB and now 9.5G
Hi CooperativeFox72 , how much free space do you have on your disk now? If you run du on your /opt/trains/data/elastic_7 folder in let's say 5 mins intervals do you see the folder size is growing?
Well, 30GB might take a long time... AppetizingMouse58 do you have an idea?
SuccessfulKoala55 it still stuck on the same line .. does it should be like this?
how long? 😅
I am now stuck inCopying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b
for more then 40 min 😥
Sure 🙂 - Elastic takes a long time to reindex 😄
it took around 50 min but it now finish the backup..Thanks
it stuck on Creating the backup file: /opt/trains/data/elastic_5.tar.gz
It is stuck, or do you keep seeing output from the script?
Do you have a lots of data on the server?
The script is running for more then 45 min, does it regular?
You mean the migration from Trains Server 0.15 to 0.16?
Hey SuccessfulKoala55 again, how long does the migration script should take?
To move your data from one machine to another, see "Backing up and restoring data and configuration" in https://allegro.ai/docs/deploying_trains/trains_server_linux_mac/
Hi SuccessfulKoala55 Thanks for the replay..
So for now, if I like to upgrade to the latest trains-server but on another machine and keep all the data.
what is the best practice?
Thanks again 🙂
Hi CooperativeFox72 ,
The ClearML Server was not yet released - it should be out within several days. Once it's out, the upgrade instructions will include a section addressing your exact situation.
In short, yes, the same process would apply, with a tiny additional step 🙂