Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have A Problem After Updating Clearml-Server To The Most Recent Version. Elasticsearch Has Been Updated From

Hi, I have a problem after updating clearml-server to the most recent version. Elasticsearch has been updated from elasticsearch:7.6.2 to elasticsearch:7.16.2 when I start the clearml-server, I see only part of the experiments which are pretty old (a few months old) and don't see the newest ones. I tried reverting to the old version, but getting some errors related to index format in ElasticSearch:
clearml-elastic | Likely root cause: org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_hw"))): 10 (needs to be between 7 and 9)I'm interested in how to fix this problem. Thank you!

  
  
Posted 2 years ago
Votes Newest

Answers 26


At some point we switched from Mongo DB v3.6 to v4.4. Upgrading from old versions require a migration of mongo data. Did you run the upgrade script as described below? Were there any errors?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/

  
  
Posted 2 years ago

Hmm, I'm not sure which version the server was prior to upgrade. Old version had http://docker.elastic.co/elasticsearch/elasticsearch:7.6.2 and mongo:3.6.5

  
  
Posted 2 years ago

Hmm, I might have messed something up. It seems that the backup has been done after the update, and not before. If I don't want to mess something up, what are my alternative options?

  
  
Posted 2 years ago

Backup what you currently have

  
  
Posted 2 years ago

Okay, I think everything is OK now!
Thank you very much for your help. I'll make sure that backup is performed on regular basis.

  
  
Posted 2 years ago

This explains the issue I think. The recovery path would be as follows:
Put down the running containers Restore both mongo and elastic data from the backup Run the old version docker containers and make sure that all the data is there Put down the containers Run the upgrade script Start the new version

  
  
Posted 2 years ago

I didn't see that, I didn't run it

  
  
Posted 2 years ago

For example doing a fresh installation of the previous version and using backup data so you have a fresh state. Then try to upgrade again.

  
  
Posted 2 years ago

I do have backup of data. What's the exact process of rolling back?

  
  
Posted 2 years ago

I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)

  
  
Posted 2 years ago

AppetizingMouse58 , might have some input here 🙂

  
  
Posted 2 years ago

Is there a way to receive updates when new version of ClearML is out?

  
  
Posted 2 years ago

Okay, backup in progress

  
  
Posted 2 years ago

Is there a way to reindex data in Elastic?

  
  
Posted 2 years ago

We all do eventually 😛

  
  
Posted 2 years ago

What's weird to me is that only part of the data has been migrated, and the other part is not available after update

  
  
Posted 2 years ago

About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?

I don't see these tasks at all

  
  
Posted 2 years ago

Thank you!

  
  
Posted 2 years ago

Hi SoggyBeetle95 , from what version of clearml did you upgrade? About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?

  
  
Posted 2 years ago

And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂

  
  
Posted 2 years ago

Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?

  
  
Posted 2 years ago

SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available

  
  
Posted 2 years ago

Should I try to do it now?

  
  
Posted 2 years ago

How do I remove the current version?

  
  
Posted 2 years ago

Backup is done

  
  
Posted 2 years ago

Had to learn that the hard way, obviously 😄

  
  
Posted 2 years ago
1K Views
26 Answers
2 years ago
one year ago
Tags