Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! I Have Some Problems With Data Migration Process. My Error Log In The Attached Files.

Hi! I have some problems with data migration process. My error log in the attached files.

  
  
Posted 3 years ago
Votes Newest

Answers 27


Oh, I see:( it turned out that --extra-source-env option was not officially released yet. But the script that supports it can be downloaded from here: https://github.com/allegroai/trains-server/files/5080286/upgrade.zip

  
  
Posted 3 years ago

AppetizingMouse58 In general, I first had the trains version 0.15.1, it worked. I decided to upgrade to version 0.16.0, made a data backup, but did not make the data migration, and I got the following error (the screenshot). Then I tried to make a migration, but then there was a problem, I already wrote about it. After that, I tried to roll back by downloading http://docker-compose.ml from git (with tag 0.15.1) and making docker-compose-f /opt/trainz/docker-compose.yml pull , and also restore the data folder. But now this version doesn't work for me either.

  
  
Posted 3 years ago

Here is the thread with solving the same issue: https://allegroai-trains.slack.com/archives/CTK20V944/p1596724607016500

  
  
Posted 3 years ago

Hi IdealPanda97 , can you share the logs for the 'elastic-upgrade-7' docker container? According to the upgrade log there was some problem with Elasticsearch during indices copy.

  
  
Posted 3 years ago

AppetizingMouse58

  
  
Posted 3 years ago

Amount of memory

  
  
Posted 3 years ago

There is a "License expired" message for the Elasticsearch 5. Try running the following command when your old trains docker is up:
http://localhost:9200/_xpack/license/start_basic

  
  
Posted 3 years ago

AppetizingMouse58 When I tried to go to http://localhost:9200/_xpack/license/start_basic , I received this err msg: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_xpack","index_uuid":"_na_","index":"_xpack"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_xpack","index_uuid":"_na_","index":"_xpack"},"status":404}

  
  
Posted 3 years ago

IdealPanda97 What can be seen now is that some of the indices (at least queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08) are in the corrupted state. This can be the result of abnormal termination of ES or some other situation. The queue metrics index is not particularly important but there maybe other indices that are also corrupted. To map the cluster and indices state you can issue the following commands (with the running ES5 docker container). Look for the "red" statuses in the output:
curl " http://localhost:9200/_cluster/health?pretty "
curl " http://localhost:9200/_cluster/health?level=indices&pretty "

  
  
Posted 3 years ago

AppetizingMouse58 thanks a lot! After I deleted the indices, I was able to migrate the data and update the trains version to 0.16.0

  
  
Posted 3 years ago

AppetizingMouse58 yes, sure

  
  
Posted 3 years ago

AppetizingMouse58 yes, I see two indices with red status. And now should I delete them? If yes, command for that like this:
curl -XDELETE localhost:9200/queue_metrics/d1bd92a3b039400cbafc60a7a5b1e52b_2020-08?

  
  
Posted 3 years ago

Enjoy the new version!

  
  
Posted 3 years ago

Yes, the command would be like this: curl -XDELETE " http://localhost:9200/queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08 "
If you decide to delete the "red" indices then you can proceed with the command above issuing it for each problematic index. The queue metrics index is not very important but the second one "events-logs" contains all the log messages produced by your tasks in August. You will still have debug images and scalar metrics reported by these tasks but the log messages themselves will be lost. If they are important then I would consider trying to restore from the backup first and checking if you can get to the cluster with no "red" indices.

  
  
Posted 3 years ago

Thanks! In this log it mentions that the source elastic 5 has failed during the reindex process. Can you also share the logs from the 'elastic-upgrade' service?

  
  
Posted 3 years ago

If it returns an OK result then rerun the upgrade process again.

  
  
Posted 3 years ago

Just a moment, it seems that this api is supported only on ES6 or newer. From the other discussion in this channel: for ES5 you have to download the basic license and install it as described in this article: https://medium.com/@ospaarmann/tidbits-solving-the-elasticsearch-x-pack-license-issue-in-docker-d15bb22d82fd

  
  
Posted 3 years ago

Sorry, I did not write it properly. You need to run the following curl command from the command line:
curl -XPOST ' http://localhost:9200/_xpack/license/start_basic '

  
  
Posted 3 years ago

In this case I have err: {"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}(

  
  
Posted 3 years ago

Strange:( What version of ElasticSearch do you currently use (before the upgrade)? Can you please share your docker compose file?

  
  
Posted 3 years ago

IdealPanda97 It seems that expired ES5 license is the reason for both the upgrade failing and for inability to run the Trains v0.15. The license is free but the ways to renew it are different between ES5 and ES6/7. For the ES5 the procedure is more complicated and described in the medium article that I sent earlier. In the attached thread another user has applied it and it solved the issue. The article describe 2 possible solutions: turning of the xpack when running Elastic and retrieving the new free license and installing it with curl. The second way is better since it will be good for both running the old docker and for the upgrade script. You have to do it only once for your existing ES5.

  
  
Posted 3 years ago

Another option that should work for the upgrade script is to pass an environment variable that disable the xpack (the feature that requires licensing) for the ES5 docker container. It can done as following:
python elastic_upgrade.py --extra-source-env xpack.security.enabled=false

  
  
Posted 3 years ago

For running of the old version of Trains the same setting can be added to elasticsearch environment section in the docker compose

  
  
Posted 3 years ago

AppetizingMouse58 I updated my lincense, but it doesnt work for me. And I have not --extra_source-env argument in elastic_upgrade.py

  
  
Posted 3 years ago

In the attached file my new log from elastic-upgrade, and some info about the elastic cluster.

  
  
Posted 3 years ago

AppetizingMouse58 I tried it, but now I have new errors with shards (

  
  
Posted 3 years ago
542 Views
27 Answers
3 years ago
one year ago
Tags