And i try to upgrade elastic by your script, but it cannot success, the log shows :
Hi SubstantialBaldeagle49 , in order to upgrade to v1.2, you will need to run a migration, see https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/
Hi SuccessfulKoala55 I have run this in step 2
Correct me if I'm wrong, but you wrote:
- update /opt/trains/mongo to /opt/trains/mongo_4 in target machine
which does not mention running the upgrade script?
Also, from the log, I see your ES version is 5.6:at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]
and I'm not sure why - in v0.16, the ES version is already 7.6, which makes me think your original version was v0.15, also in the new docker-compose the version is 7.16, so maybe you are not using the updated docker-compose file?
If you only want to move the server to another machine and not upgrade the server version, you should just copy the folder and restart the server there, but change nothing
Sorry , my poor english, it means upgrade by script.
Good news is it's fine now, I try to upgrade ES (although it fails), and i try to go through the necessary steps in this: https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac
SuccessfulKoala55 I get all the trains server experiments record in new clearml server. Maybe it's due to vm.max_map_count or access of /opt/clearml, or the failed ES upgrade...
Also, from the log, I see your ES version is 5.6:
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]
and I'm not sure why - in v0.16, the ES version is already 7.6, which makes me think your original version was v0.15, also in the new docker-compose the version is 7.16, so maybe you are not using the updated docker-compose file?
It confused me too, but ES dir in my 0.16 version is /opt/trains/data/elastic_7
Hi SuccessfulKoala55 I find why the ES version is not upgraded to 7.16, because the script provided here is 7.6 https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_es7_migration
This is the migration step. once done, the latest docker compose should use the 7.16 image
7.6 and 7.16 do not require data migration, 7.16 can simply be started after the migration to 7.6 was done
OK i will try again, this time i won't upgrade ES, But do increase vm.max_map_count and open access of /opt/clearm. Why i do again is i found the web ui is fine in deployed machine, but it's weird in my local machine:
This only means the UI is loading but can't access the server
But when i click view_all, i can see all the experiments
Where's "view all" in that screenshot?
Cam you perhaps send the docker-compose in the current server?
Cam you perhaps send the docker-compose in the current server? (已編輯)
Sorry I don't undestand what' your meaning...
My docker-compose is lateset, download by this:curl
https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
I mean the old one, in the non-migrated server
I want to use clearml, the old one's mongo is 3.65...
I know, I just want to understand the old one's status
Do you mean i send the old one to you?
so the process to move to a new machine and upgrade should be:
Shut down the server Copy the data folder to a new machine Run the migration script from <=v1.1 to v1.2 - the script should automatically create the new mongo_4
folder Get the new docker-compose Start the new server
I tried as below, it can works fine:
copy(scp) /opt/trains to target machine upgrade mongo by this: https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/ mv /opt/trains /opt/clearml Increase vm.max_map_count
;Grant access to the Dockers; get lastest docker-compose.yml and pull/upBut the web ui dashboard is still weird, anyway it can work now
But without navigation bar, it's quiet inconvenient 😢 ...Do you known why? SuccessfulKoala55