Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Got Some Errors While Running Migration Script From Es5 To Es7:

Got some errors while running migration script from ES5 to ES7:
2020-08-11 15:21:50,130 Running on: Linux 2020-08-11 15:21:50,227 Docker allocated memory: 16GB 2020-08-11 15:21:50,227 Docker cores: 4 2020-08-11 15:21:50,227 Creating the backup file: /opt/trains/data/elastic_5.tar.gz 2020-08-11 15:23:29,435 Creating the target directory: /opt/trains/data/elastic_7 2020-08-11 15:23:29,435 Stopping containers. 2020-08-11 15:23:29,950 Starting the upgrade containers. 0ecfc0823f71151dc7168cc35f0ce06f670fc9ccca9d8f98f43401cd83cddf67 Unable to find image 'docker.elastic.co/elasticsearch/elasticsearch:7.6.2' locally 7.6.2: Pulling from elasticsearch/elasticsearch c808caf183b6: Pull complete d6caf8e15a64: Pull complete b0ba5f324e82: Pull complete d7e8c1e99b9a: Pull complete 85c4d6c81438: Pull complete 3119218fac98: Pull complete 914accf214bb: Pull complete Digest: sha256:59342c577e2b7082b819654d119f42514ddf47f0699c8b54dc1f0150250ce7aa Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.6.2 99b6200e5459f601680bc14726fd7a3eedd7f2252fa803b81218217efc4c7446 2020-08-11 15:24:32,369 Waiting for the upgrade containers to start (30sec). 2020-08-11 15:25:02,924 Copying indices from v5 to v7 2020-08-11 15:25:02,933 6 indices will be copied. 2020-08-11 15:25:03,637 Running with a parallel factor of 2 2020-08-11 15:25:03,638 Copying index events-log-d1bd92a3b039400cbafc60a7a5b1e52b 2020-08-11 15:25:03,642 Copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07 2020-08-11 15:25:05,213 Error when copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07: 503 Server Error: Service Unavailable for url: 2020-08-11 15:25:05,213 queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07 could not be copied. 2020-08-11 15:25:05,213 Copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07 2020-08-11 15:25:06,257 Error when copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07: 503 Server Error: Service Unavailable for url: 2020-08-11 15:25:06,257 worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07 could not be copied. 2020-08-11 15:25:06,257 Copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 2020-08-11 15:25:14,089 events-log-d1bd92a3b039400cbafc60a7a5b1e52b copied successfully. 2020-08-11 15:25:14,089 Copying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b 2020-08-11 15:25:15,023 Error when copying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b: 503 Server Error: Service Unavailable for url: 2020-08-11 15:25:15,024 events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b could not be copied. 2020-08-11 15:25:15,024 Copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 2020-08-11 15:26:06,156 queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 copied successfully. 2020-08-11 15:26:32,593 worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 copied successfully. 2020-08-11 15:26:32,594 Finished copying indices. 2020-08-11 15:26:47,606 Stopping the upgrade containers. 2020-08-11 15:26:47,684 Stopping: elastic-upgrade 2020-08-11 15:27:04,725 Stopping: elastic-upgrade-7 2020-08-11 15:27:05,825 Renaming the source directory /opt/trains/data/elastic to /opt/trains/data/elastic_migrated_2020-08-11_15-27-05. 2020-08-11 15:27:05,826 Upgrade completed.

  
  
Posted 3 years ago
Votes Newest

Answers 16


still same errors 😕

  
  
Posted 3 years ago

We just uploaded the new update script into
https://github.com/allegroai/trains-server/releases/download/0.16.0/trains-server-0.16.0-migration.zip
It has several improvements and there is a chance that it will overcome the issue that you are facing. Also, please check that you have enough disk space for copying of ES data.

  
  
Posted 3 years ago

AppetizingMouse58 After some thoughts, we decided to install from scratch 0.16, with no data migration, because we believe this was an edge case not worth spending efforts on. Thank you very much for your help there, very appreciated. You guys rock! 🙂

  
  
Posted 3 years ago

Yes, correct.

  
  
Posted 3 years ago

JitteryCoyote63 thanks for the kind words 🙂 . If you can spare the time, I'd appreciate the ES logs - for the sake of other community members, I really like to make sure the migration process is as resilient and smooth as possible 🙏

  
  
Posted 3 years ago

Enjoy the new version:) Would still be interesting to see what caused ES7 to stop responding.

  
  
Posted 3 years ago

sure, will be happy to debug that 🙂

  
  
Posted 3 years ago

Thanks! Unfortunately still not working, here is the log file:

  
  
Posted 3 years ago

here it is

  
  
Posted 3 years ago

Taking a look 🙂

  
  
Posted 3 years ago

as for disk space: I have 21Gb available (8Gb used), /opt/trains/data folder is about 600Mo

  
  
Posted 3 years ago

Thanks for the update. What can be seen from the log is that for some reason after copying of couple of indices Elasticsearch 7 becomes unavailable. I think we can find the reasons in the Elasticsearch 7 logs. I can send you the instructions on how to proceed (it requires a minimal change to the upgrade script so that the upgrade containers are not removed after the script run and inspection of ES7 logs)

  
  
Posted 3 years ago

I should also rename /opt/trains/data/elastic_migrated_2020-08-11_15-27-05 folder to /opt/trains/data/elastic before running the migration tool right?

  
  
Posted 3 years ago

There should be a log file in the directory where you run the script. It contains more info. Can you please send me the log?

  
  
Posted 3 years ago

Hi H4dr1en, there is a chance that the problem is that in parallel reindexing of data. You can try to replace parallel=max(docker_resources.cpus // 2, 1)
at line 190 with
parallel=1
I think you will need to remove the /opt/trains/data/elastic_7 folder before script restart

  
  
Posted 3 years ago

Great:) The let's try to get the logs. Maybe we can get without changing the upgrade script. Please run 'sudo docker ps -a' if you see the exited container with name 'elastic-upgrade-7' then please save its logs to the file with the below command and send the file to me:
docker logs <container_id_for_elastic-upgrade-7> >& elastic_logs.txt

  
  
Posted 3 years ago
594 Views
16 Answers
3 years ago
one year ago
Tags