As I’ve used it, rsync is akin to scp, but provides more control.
There are two things I want to accomplish with this:
- I want a reliable location to store my tasks without needing to worry about running out of space.
- I want to be able to access these stored tasks in the clearml ui when they are in this storage
While rsync would let me backup the file server contents, like cp’ing them into a rclone of onedrive. I don’t think it would let me view tasks that I have put on onedrive.
Thank you!
I wasn’t getting my hopes up on storing tasks elsewhere :)
For the models & artifacts is there a parameter to change the default directory of saving and loading to something else?
Hi @<1538330703932952576:profile|ThickSeaurchin47> , if you have rsync, can't you just rsync the fileserver data folder to onedrive?
This answered my question, I ended up setting up minio running off of a external hard drive which backups to onedrive.
Though I am still having some problems getting the client to connect to minio (see recent thread)
Sorry for the delay on responding to this, I had some family craziness.
Oh, I see. Well tasks are stored in the mongodb and elasticsearch databases, so unless you can find a way to make them use remote storage on onedrive (I don't think that's possible), it won't be possible.
As for artifacts and models, if you want that the same way the SDK and WebApp do it, this requires adding support for that in the WebApp code and in the SDK code (as these remote object storage functions are client-side, and not provided by the clearml server)
@<1538330703932952576:profile|ThickSeaurchin47> sorry, I missed this somehow 😞 - do you mean in the SDK? The SDK can choose to store models and artifacts on any object storage (not in the clearml server). The server itself only has the fileserver storage, but it is not up to the server which storage will be used - it's an SDK choice.
For clarification, I want to store artifacts and models (maybe tasks too) like how the webui lets you specify a external storage s3 bucket when you make a folder, but with onedrive.
I.e in such a way I can still see them on the ui, but don’t necessarily have them stored on the server
If a external storage is not specified it will use internal correct?
So if I give up on having specific folder paths for each project within onedrive I could just treat it as a alternative path for internal artifact storage.