SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available
Is there a way to receive updates when new version of ClearML is out?
Okay, I think everything is OK now!
Thank you very much for your help. I'll make sure that backup is performed on regular basis.
This explains the issue I think. The recovery path would be as follows:
Put down the running containers Restore both mongo and elastic data from the backup Run the old version docker containers and make sure that all the data is there Put down the containers Run the upgrade script Start the new version
At some point we switched from Mongo DB v3.6 to v4.4. Upgrading from old versions require a migration of mongo data. Did you run the upgrade script as described below? Were there any errors?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/
About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
I don't see these tasks at all
Hmm, I'm not sure which version the server was prior to upgrade. Old version had http://docker.elastic.co/elasticsearch/elasticsearch:7.6.2 and mongo:3.6.5
Hi SoggyBeetle95 , from what version of clearml did you upgrade? About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
AppetizingMouse58 , might have some input here 🙂
Is there a way to reindex data in Elastic?
Had to learn that the hard way, obviously 😄
And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂
What's weird to me is that only part of the data has been migrated, and the other part is not available after update
Hmm, I might have messed something up. It seems that the backup has been done after the update, and not before. If I don't want to mess something up, what are my alternative options?
I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)
For example doing a fresh installation of the previous version and using backup data so you have a fresh state. Then try to upgrade again.
I do have backup of data. What's the exact process of rolling back?
Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?