Thanks for the update. What can be seen from the log is that for some reason after copying of couple of indices Elasticsearch 7 becomes unavailable. I think we can find the reasons in the Elasticsearch 7 logs. I can send you the instructions on how to proceed (it requires a minimal change to the upgrade script so that the upgrade containers are not removed after the script run and inspection of ES7 logs)
Hi H4dr1en, there is a chance that the problem is that in parallel reindexing of data. You can try to replace parallel=max(docker_resources.cpus // 2, 1)
at line 190 with
parallel=1
I think you will need to remove the /opt/trains/data/elastic_7 folder before script restart
I should also rename /opt/trains/data/elastic_migrated_2020-08-11_15-27-05 folder to /opt/trains/data/elastic before running the migration tool right?
We just uploaded the new update script into
https://github.com/allegroai/trains-server/releases/download/0.16.0/trains-server-0.16.0-migration.zip
It has several improvements and there is a chance that it will overcome the issue that you are facing. Also, please check that you have enough disk space for copying of ES data.
There should be a log file in the directory where you run the script. It contains more info. Can you please send me the log?
Thanks! Unfortunately still not working, here is the log file:
Great:) The let's try to get the logs. Maybe we can get without changing the upgrade script. Please run 'sudo docker ps -a' if you see the exited container with name 'elastic-upgrade-7' then please save its logs to the file with the below command and send the file to me:
docker logs <container_id_for_elastic-upgrade-7> >& elastic_logs.txt
AppetizingMouse58 After some thoughts, we decided to install from scratch 0.16, with no data migration, because we believe this was an edge case not worth spending efforts on. Thank you very much for your help there, very appreciated. You guys rock! 🙂
JitteryCoyote63 thanks for the kind words 🙂 . If you can spare the time, I'd appreciate the ES logs - for the sake of other community members, I really like to make sure the migration process is as resilient and smooth as possible 🙏
as for disk space: I have 21Gb available (8Gb used), /opt/trains/data folder is about 600Mo
Enjoy the new version:) Would still be interesting to see what caused ES7 to stop responding.