Hi JitteryCoyote63 , this can be done using an ES update_by_query
command. To so that, you'll have to send the command (using curl
for example) to the ES service (externally, if the 9200 port is open for external access), or by executing bash inside the ES container using
sudo docker exec -it clearml-elasticsearch /bin/bash `
JitteryCoyote63 my previous comment was actually related to the events (debug images etc.) logged by your experiment
Hi SuccessfulKoala55 , will I be able to update all references to the old s3 bucket using this command?
For models, you can connect to the MongoDB service (same command, but using the clearml-mongo
container), and issue a manual mongo command
Are you interested in updating both model URLs and debug Image URLs?
But please do back up your data before trying it 🙂
Yes, I would like to update all references to the old bucket unfortunately… I think I’ll simply delete the old s3 bucket, wait or his name to be available again and recreate it where on the other aws account and move the data there. This way I don’t have to mess with clearml data - I am afraid to do something wrong and loose data
Well, in case you do want to try it out:
For models, assuming your original URLs start with s3://<old-bucket-name>/
, you'll need a JS script, something like:db.model.find({uri:{$regex:/^s3/}}).forEach(function(e,i) { e.uri = e.uri.replace("s3://<old-bucket-name>/","s3://<new-bucket-name>/"); db.model.save(e); });
And than:
` sudo docker exec -it clearml-mongo /bin/bash
paste script into this file and exit nano
nano script.js
this should run the script against the backend db
mongo backend script.js `
Please take care : since I'm not in front of computer right now, I can't test it, so please back up your data before trying it , and let me know if you encounter any errors...
If you want the appropriate instructions for debug images, just let me know 🙂
Thanks a lot for the solution SuccessfulKoala55 ! I’ll try that if the solution “delete old bucket, wait for its name to be available, recreate it with the other aws account, transfer the data back” fails