About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
I don't see these tasks at all
I do have backup of data. What's the exact process of rolling back?
Is there a way to receive updates when new version of ClearML is out?
I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)
And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂
At some point we switched from Mongo DB v3.6 to v4.4. Upgrading from old versions require a migration of mongo data. Did you run the upgrade script as described below? Were there any errors?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/
This explains the issue I think. The recovery path would be as follows:
Put down the running containers Restore both mongo and elastic data from the backup Run the old version docker containers and make sure that all the data is there Put down the containers Run the upgrade script Start the new version
What's weird to me is that only part of the data has been migrated, and the other part is not available after update
Hi SoggyBeetle95 , from what version of clearml did you upgrade? About the tasks that disappeared: you do not see these tasks at all or you see these tasks with no results?
Hmm, I'm not sure which version the server was prior to upgrade. Old version had http://docker.elastic.co/elasticsearch/elasticsearch:7.6.2 and mongo:3.6.5
Okay, I think everything is OK now!
Thank you very much for your help. I'll make sure that backup is performed on regular basis.
Had to learn that the hard way, obviously 😄
AppetizingMouse58 , might have some input here 🙂
Is there a way to reindex data in Elastic?
For example doing a fresh installation of the previous version and using backup data so you have a fresh state. Then try to upgrade again.
Hmm, I might have messed something up. It seems that the backup has been done after the update, and not before. If I don't want to mess something up, what are my alternative options?
SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available
Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?