Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Would Like To Switch From The Elastic-Search Service In The Docker-Compose Of The Clearml-Server To An Externally Managed, Scalable Elastic-Search Cluster. I Have Two Questions:

Hi, I would like to switch from the elastic-search service in the docker-compose of the clearml-server to an externally managed, scalable elastic-search cluster. I have two questions:
How should I change the docker-compose.yaml to point to the ES cluster? How can I migrate the data from the 1-node ES cluster to the ES cluster?

  
  
Posted 3 years ago
Votes Newest

Answers 22


Hi JitteryCoyote63 ,
In the docker-compose file, you have an environment setting for the apiserver service host and port ( CLEARML_ELASTIC_SERVICE_HOST and CLEARML_ELASTIC_SERVICE_PORT ) - changing those will allow you to point the server to another ES service

  
  
Posted 3 years ago

Migration of the data is basicallt reindexing from a different server, I guess you can read about it here: https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html

  
  
Posted 3 years ago

Thanks! I would like to use this opportunity to split the indices into multiple shards, as explained here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html#indices-split-index

  
  
Posted 3 years ago

I am not sure I can do both operations at the same time (migration + splitting), do you think it’s better to do splitting first or migration first?

  
  
Posted 3 years ago

It can be done ay the same time, it all depends on the mapping template you set ion the new cluster

  
  
Posted 3 years ago

ha nice, where can I find the mapping template of the original clearml so that I can copy and adapt?

  
  
Posted 3 years ago

There's an ES REST Api command to get the current template

  
  
Posted 3 years ago

ha nice thanks 🙏

  
  
Posted 3 years ago

SuccessfulKoala55 I was able to recreate the indices in the new ES cluster. I specified number_of_shards: 4 for the events-log-d1bd92a3b039400cbafc60a7a5b1e52b index. I then copied the documents from the old ES using the _reindex API. The index is 7.5Gb on one shard.
Now I see that this index on the new ES cluster is ~19.4Gb 🤔 The index is divided into the 4 shards, but each shard is between 4.7Gb and 5Gb!
I was expecting to have the same index size as in the previous env (7.5Gb) splitted across the 4 shards ( 7.5 / 4 = ~1,8). What am I missing?

  
  
Posted 3 years ago

The number of documents in the old and the new env are the same though 🤔 I really don’t understand where this extra space used comes from

  
  
Posted 3 years ago

PS: in the new env, I’v set num_replicas: 0, so I’m only talking about primary shards…

  
  
Posted 3 years ago

Well, probably an optimization side-effect

  
  
Posted 3 years ago

You mean it will resolve by itself in the following days or should I do something? Or there is nothing to do and it will stay this way?

  
  
Posted 3 years ago

This https://discuss.elastic.co/t/index-size-explodes-after-split/150692 seems to say for the _split API such situation happens and solves itself after a couple fo days, maybe the same case for me?

  
  
Posted 3 years ago

SuccessfulKoala55

In the docker-compose file, you have an environment setting for the apiserver service host and port (CLEARML_ELASTIC_SERVICE_HOST and CLEARML_ELASTIC_SERVICE_PORT) - changing those will allow you to point the server to another ES service

The ES cluster is running in another machine, how can I set its IP in CLEARML_ELASTIC_SERVICE_HOST ? I would need to add host to the networks of the apiserver service somehow? How can I do that?

  
  
Posted 3 years ago

If this host is accessible from the machine running ClearML server (for example http://some-host-name ), you can just use the hostname

  
  
Posted 3 years ago

It's an external server, so no need to add anything to the docker-compose network

  
  
Posted 3 years ago

The host is accessible, I can ping it and even run curl " http://internal-aws-host-name:9200/_cat/shards " and get results from the local machine

  
  
Posted 3 years ago

the api-server shows when starting:
clearml-apiserver | [2021-07-13 11:09:34,552] [9] [INFO] [clearml.es_factory] Using override elastic host clearml-apiserver | [2021-07-13 11:09:34,552] [9] [INFO] [clearml.es_factory] Using override elastic port 9200 ... clearml-apiserver | [2021-07-13 11:09:38,407] [9] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 1 of 4. Waiting for 30sec clearml-apiserver | [2021-07-13 11:10:08,414] [9] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 2 of 4. Waiting for 30sec clearml-apiserver | [2021-07-13 11:10:38,443] [9] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 3 of 4. Waiting for 30sec clearml-apiserver | [2021-07-13 11:11:08,468] [9] [ERROR] [clearml.app_sequence] Error connecting to Elasticsearch: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4866d5f6a0>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4866d5f6a0>: Failed to establish a new connection: [Errno -2] Name or service not known) clearml-apiserver | Traceback (most recent call last): clearml-apiserver | File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main clearml-apiserver | "__main__", mod_spec) clearml-apiserver | File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code clearml-apiserver | exec(code, run_globals) clearml-apiserver | File "/opt/clearml/apiserver/server.py", line 10, in <module> clearml-apiserver | AppSequence(app).start(request_handlers=RequestHandlers()) clearml-apiserver | File "/opt/clearml/apiserver/server_init/app_sequence.py", line 40, in start clearml-apiserver | self._init_dbs() clearml-apiserver | File "/opt/clearml/apiserver/server_init/app_sequence.py", line 90, in _init_dbs clearml-apiserver | and get_last_server_version() < Version("0.16.0") clearml-apiserver | TypeError: '<' not supported between instances of 'Version' and 'Version'

  
  
Posted 3 years ago

ha wait, I removed the http:// in the host and it worked 🎉

  
  
Posted 3 years ago

🙂

  
  
Posted 3 years ago

http vs https is controlled by the secure flag, I think

  
  
Posted 3 years ago