I do have backup of data. What's the exact process of rolling back?
At some point we switched from Mongo DB v3.6 to v4.4. Upgrading from old versions require a migration of mongo data. Did you run the upgrade script as described below? Were there any errors?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/
I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)
Hi SoggyBeetle95 , from what version of clearml did you upgrade? About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
What's weird to me is that only part of the data has been migrated, and the other part is not available after update
Had to learn that the hard way, obviously 😄
Hmm, I might have messed something up. It seems that the backup has been done after the update, and not before. If I don't want to mess something up, what are my alternative options?
AppetizingMouse58 , might have some input here 🙂
This explains the issue I think. The recovery path would be as follows:
Put down the running containers Restore both mongo and elastic data from the backup Run the old version docker containers and make sure that all the data is there Put down the containers Run the upgrade script Start the new version
Hmm, I'm not sure which version the server was prior to upgrade. Old version had http://docker.elastic.co/elasticsearch/elasticsearch:7.6.2 and mongo:3.6.5
Okay, I think everything is OK now!
Thank you very much for your help. I'll make sure that backup is performed on regular basis.
For example doing a fresh installation of the previous version and using backup data so you have a fresh state. Then try to upgrade again.
Is there a way to reindex data in Elastic?
And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂
SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available
Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?
Is there a way to receive updates when new version of ClearML is out?
About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
I don't see these tasks at all