IdealPanda97 It seems that expired ES5 license is the reason for both the upgrade failing and for inability to run the Trains v0.15. The license is free but the ways to renew it are different between ES5 and ES6/7. For the ES5 the procedure is more complicated and described in the medium article that I sent earlier. In the attached thread another user has applied it and it solved the issue. The article describe 2 possible solutions: turning of the xpack when running Elastic and retrieving the new free license and installing it with curl. The second way is better since it will be good for both running the old docker and for the upgrade script. You have to do it only once for your existing ES5.
AppetizingMouse58 I updated my lincense, but it doesnt work for me. And I have not --extra_source-env argument in elastic_upgrade.py
AppetizingMouse58 I tried it, but now I have new errors with shards (
AppetizingMouse58 When I tried to go to http://localhost:9200/_xpack/license/start_basic , I received this err msg: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_xpack","index_uuid":"_na_","index":"_xpack"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_xpack","index_uuid":"_na_","index":"_xpack"},"status":404}
AppetizingMouse58 thanks a lot! After I deleted the indices, I was able to migrate the data and update the trains version to 0.16.0
Oh, I see:( it turned out that --extra-source-env option was not officially released yet. But the script that supports it can be downloaded from here: https://github.com/allegroai/trains-server/files/5080286/upgrade.zip
Thanks! In this log it mentions that the source elastic 5 has failed during the reindex process. Can you also share the logs from the 'elastic-upgrade' service?
IdealPanda97 What can be seen now is that some of the indices (at least queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08) are in the corrupted state. This can be the result of abnormal termination of ES or some other situation. The queue metrics index is not particularly important but there maybe other indices that are also corrupted. To map the cluster and indices state you can issue the following commands (with the running ES5 docker container). Look for the "red" statuses in the output:
curl " http://localhost:9200/_cluster/health?pretty "
curl " http://localhost:9200/_cluster/health?level=indices&pretty "
In this case I have err: {"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}(
Another option that should work for the upgrade script is to pass an environment variable that disable the xpack (the feature that requires licensing) for the ES5 docker container. It can done as following:
python elastic_upgrade.py --extra-source-env xpack.security.enabled=false
Yes, the command would be like this: curl -XDELETE " http://localhost:9200/queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08 "
If you decide to delete the "red" indices then you can proceed with the command above issuing it for each problematic index. The queue metrics index is not very important but the second one "events-logs" contains all the log messages produced by your tasks in August. You will still have debug images and scalar metrics reported by these tasks but the log messages themselves will be lost. If they are important then I would consider trying to restore from the backup first and checking if you can get to the cluster with no "red" indices.
Sorry, I did not write it properly. You need to run the following curl command from the command line:
curl -XPOST ' http://localhost:9200/_xpack/license/start_basic '
Just a moment, it seems that this api is supported only on ES6 or newer. From the other discussion in this channel: for ES5 you have to download the basic license and install it as described in this article: https://medium.com/@ospaarmann/tidbits-solving-the-elasticsearch-x-pack-license-issue-in-docker-d15bb22d82fd
For running of the old version of Trains the same setting can be added to elasticsearch environment section in the docker compose
Hi IdealPanda97 , can you share the logs for the 'elastic-upgrade-7' docker container? According to the upgrade log there was some problem with Elasticsearch during indices copy.
In the attached file my new log from elastic-upgrade, and some info about the elastic cluster.
Strange:( What version of ElasticSearch do you currently use (before the upgrade)? Can you please share your docker compose file?
AppetizingMouse58 In general, I first had the trains version 0.15.1, it worked. I decided to upgrade to version 0.16.0, made a data backup, but did not make the data migration, and I got the following error (the screenshot). Then I tried to make a migration, but then there was a problem, I already wrote about it. After that, I tried to roll back by downloading http://docker-compose.ml from git (with tag 0.15.1) and making docker-compose-f /opt/trainz/docker-compose.yml pull , and also restore the data folder. But now this version doesn't work for me either.
Here is the thread with solving the same issue: https://allegroai-trains.slack.com/archives/CTK20V944/p1596724607016500
There is a "License expired" message for the Elasticsearch 5. Try running the following command when your old trains docker is up:
http://localhost:9200/_xpack/license/start_basic
If it returns an OK result then rerun the upgrade process again.
AppetizingMouse58 yes, I see two indices with red status. And now should I delete them? If yes, command for that like this:curl -XDELETE localhost:9200/queue_metrics/d1bd92a3b039400cbafc60a7a5b1e52b_2020-08?