I want to use clearml, the old one's mongo is 3.65...

And i try to upgrade elastic by your script, but it cannot success, the log shows :

SuccessfulKoala55 I get all the trains server experiments record in new clearml server. Maybe it's due to vm.max_map_count or access of /opt/clearml, or the failed ES upgrade...

Also, from the log, I see your ES version is 5.6:

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]

and I'm not sure why - in v0.16, the ES version is already 7.6, which makes me think your original version was v0.15, also in the new docker-compose the version is 7.16, so maybe you are not using the updated docker-compose file?

It confused me too, but ES dir in my 0.16 version is /opt/trains/data/elastic_7

But when i click view_all, i can see all the experiments

Cam you perhaps send the docker-compose in the current server? (已編輯)

Sorry I don't undestand what' your meaning...

I know why, because i used the old apiserver.conf, i set fixed_users. After log in http://xxx:8080/login , it's ok now

But without navigation bar, it's quiet inconvenient 😢 ...Do you known why? SuccessfulKoala55

Do you mean i send the old one to you?

Hi SuccessfulKoala55 I find why the ES version is not upgraded to 7.16, because the script provided here is 7.6 https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_es7_migration

here is my old docker-compose

SuccessfulKoala55 Ok, thanks, i will have a try.

My docker-compose is lateset, download by this:
curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml

I tried as below, it can works fine:
copy(scp) /opt/trains to target machine upgrade mongo by this: https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/ mv /opt/trains /opt/clearml Increase vm.max_map_count ;Grant access to the Dockers; get lastest docker-compose.yml and pull/upBut the web ui dashboard is still weird, anyway it can work now

HI: AgitatedDove14
2. I mean if my server break down, and i start a new server in another machine, can i migrate my backup experiments to the new server?
3. Not only change the info in web ui, can i connect to the old experiment , and report a new graph to that?

AgitatedDove14 OK, i see, thanks so much!

Ok , i will start a new experiment to see if the error will be still there? Sorry i dont really get how to show the trains-apiserver log

AppetizingMouse58 After i modify the max_buckets to 200000000000, I cannot see any information in web ui. Here is the log:

The log is produced after i go to the web ui and reproduce the error of "Failed to get Plot Charts"

AppetizingMouse58 Ok, this is the full log, here seems a error:

Ok, i have reverte the change ,here is two log:

May be its due to my curl?

AppetizingMouse58 here:

AppetizingMouse58

I have remake curl, this commad "sudo curl -L " https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compos" will work

https://stackoverflow.com/questions/28287261/connection-timeout-with-elasticsearch
How to set the timeout?

That's ok, i have just start the server. Could you tell me how to delete this index?

This is the second command

AppetizingMouse58

AppetizingMouse58 Great, Thanks so much! You have done a great work.
Another question, how to configure elasticsearch to run as a cluster with 2 or more nodes on the same or different machine 😅