Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Have A Problem After Updating Clearml-Server To The Most Recent Version. Elasticsearch Has Been Updated From

Hi, I have a problem after updating clearml-server to the most recent version. Elasticsearch has been updated from elasticsearch:7.6.2 to elasticsearch:7.16.2 when I start the clearml-server, I see only part of the experiments which are pretty old (a few months old) and don't see the newest ones. I tried reverting to the old version, but getting some errors related to index format in ElasticSearch:
clearml-elastic | Likely root cause: org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_hw"))): 10 (needs to be between 7 and 9)I'm interested in how to fix this problem. Thank you!

  
  
Posted one year ago
Votes Newest

Answers 26


Backup is done

  
  
Posted one year ago

Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?

  
  
Posted one year ago

Hmm, I'm not sure which version the server was prior to upgrade. Old version had http://docker.elastic.co/elasticsearch/elasticsearch:7.6.2 and mongo:3.6.5

  
  
Posted one year ago

I do have backup of data. What's the exact process of rolling back?

  
  
Posted one year ago

I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)

  
  
Posted one year ago

Hmm, I might have messed something up. It seems that the backup has been done after the update, and not before. If I don't want to mess something up, what are my alternative options?

  
  
Posted one year ago

Is there a way to receive updates when new version of ClearML is out?

  
  
Posted one year ago

Should I try to do it now?

  
  
Posted one year ago

Had to learn that the hard way, obviously 😄

  
  
Posted one year ago

Backup what you currently have

  
  
Posted one year ago

Okay, backup in progress

  
  
Posted one year ago

AppetizingMouse58 , might have some input here 🙂

  
  
Posted one year ago

Hi SoggyBeetle95 , from what version of clearml did you upgrade? About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?

  
  
Posted one year ago

About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?

I don't see these tasks at all

  
  
Posted one year ago

I didn't see that, I didn't run it

  
  
Posted one year ago

This explains the issue I think. The recovery path would be as follows:
Put down the running containers Restore both mongo and elastic data from the backup Run the old version docker containers and make sure that all the data is there Put down the containers Run the upgrade script Start the new version

  
  
Posted one year ago

Thank you!

  
  
Posted one year ago

Okay, I think everything is OK now!
Thank you very much for your help. I'll make sure that backup is performed on regular basis.

  
  
Posted one year ago

Is there a way to reindex data in Elastic?

  
  
Posted one year ago

At some point we switched from Mongo DB v3.6 to v4.4. Upgrading from old versions require a migration of mongo data. Did you run the upgrade script as described below? Were there any errors?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/

  
  
Posted one year ago

How do I remove the current version?

  
  
Posted one year ago

For example doing a fresh installation of the previous version and using backup data so you have a fresh state. Then try to upgrade again.

  
  
Posted one year ago

We all do eventually 😛

  
  
Posted one year ago

SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available

  
  
Posted one year ago

What's weird to me is that only part of the data has been migrated, and the other part is not available after update

  
  
Posted one year ago

And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂

  
  
Posted one year ago
595 Views
26 Answers
one year ago
one year ago
Tags