Reputation
Badges 1
37 × Eureka!I tried as below, it can works fine:
copy(scp) /opt/trains to target machine upgrade mongo by this: https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/ mv /opt/trains /opt/clearml Increase vm.max_map_count
;Grant access to the Dockers; get lastest docker-compose.yml and pull/upBut the web ui dashboard is still weird, anyway it can work now
That's ok, i have just start the server. Could you tell me how to delete this index?
Also, from the log, I see your ES version is 5.6:
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]
and I'm not sure why - in v0.16, the ES version is already 7.6, which makes me think your original version was v0.15, also in the new docker-compose the version is 7.16, so maybe you are not using the updated docker-compose file?
It confused me too, but ES dir in my 0.16 version is /opt/trains/data/elastic_7
But without navigation bar, it's quiet inconvenient 😢 ...Do you known why? SuccessfulKoala55
SuccessfulKoala55 I get all the trains server experiments record in new clearml server. Maybe it's due to vm.max_map_count or access of /opt/clearml, or the failed ES upgrade...
Hi SuccessfulKoala55 I have run this in step 2
This is the second command
SuccessfulKoala55 Thanks, If restarting server wont stop running experiments, then what i say is not necessary !
AppetizingMouse58 Ok, this is the full log, here seems a error:
And actually the problem here is round doesn't work before tolist
SuccessfulKoala55 It seems ok
I know why, because i used the old apiserver.conf, i set fixed_users. After log in http://xxx:8080/login , it's ok now
Ok , i will start a new experiment to see if the error will be still there? Sorry i dont really get how to show the trains-apiserver log
AppetizingMouse58 Great, Thanks so much! You have done a great work.
Another question, how to configure elasticsearch to run as a cluster with 2 or more nodes on the same or different machine 😅
Can i backup my experiments?
AppetizingMouse58 Ok , i see, thanks!
SuccessfulKoala55 Thanks for your reply, How can i fix this?
SuccessfulKoala55
Do you mean even if the json is so large, if i use Logger.report_scatter2d()
, it wont cause TransportError(429, 'circuit_breaking_exception
?
May be its due to my curl?
AppetizingMouse58 Thanks so much!, Could u tell why does this happen? If it happen next time , is there any other solution?
Hi GrumpyPenguin23 ,
Thanks your reply.
In 2, dose that mean i should develop another tool to parse the json standalone? 3 is a good solutionAnd i think what i actually need is the solution to integrate application based server,such as plotly dash or bokeh server in web ui?
GrumpyPenguin23
It seems the solution 3 is the fastest way, and i can reuse my code easily. It works now~ That's quite interesting, but the learning curve seems a bit steep for me.Do you have any material or suggestion for learning that?
I found plotly dash cannot be exported to html file, so it may cannot be used here
GrumpyPenguin23 Ok, i will open a request now!