Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hello, I Have A Self-Hosted Clearml Instance. I Want To Save All Recorded Artifacts & Models On Onedrive Instead Of S3/Gstore/Local. I Have A Rclone Mount Of Onedrive On My Clearml Server, And I’M Running The Linux Docker Container For Clearml. I’Ve Loo

Hello, I have a self-hosted clearml instance. I want to save all recorded artifacts & models on onedrive instead of s3/gstore/local.

I have a rclone mount of onedrive on my clearml server, and I’m running the Linux docker container for clearml. I’ve looked at path substitution as described here https://clear.ml/docs/latest/docs/integrations/storage but it seems to be regarding cached models for the client.

How should I go about saving models and artifacts recorded from tasks on onedrive?

Posted one year ago
Votes Newest

Answers 10

As I’ve used it, rsync is akin to scp, but provides more control.

There are two things I want to accomplish with this:

  • I want a reliable location to store my tasks without needing to worry about running out of space.
  • I want to be able to access these stored tasks in the clearml ui when they are in this storage
    While rsync would let me backup the file server contents, like cp’ing them into a rclone of onedrive. I don’t think it would let me view tasks that I have put on onedrive.
Posted one year ago

Oh, I see. Well tasks are stored in the mongodb and elasticsearch databases, so unless you can find a way to make them use remote storage on onedrive (I don't think that's possible), it won't be possible.
As for artifacts and models, if you want that the same way the SDK and WebApp do it, this requires adding support for that in the WebApp code and in the SDK code (as these remote object storage functions are client-side, and not provided by the clearml server)

Posted one year ago

For clarification, I want to store artifacts and models (maybe tasks too) like how the webui lets you specify a external storage s3 bucket when you make a folder, but with onedrive.

I.e in such a way I can still see them on the ui, but don’t necessarily have them stored on the server

Posted one year ago

If a external storage is not specified it will use internal correct?

So if I give up on having specific folder paths for each project within onedrive I could just treat it as a alternative path for internal artifact storage.

Posted one year ago

@<1538330703932952576:profile|ThickSeaurchin47> sorry, I missed this somehow 😞 - do you mean in the SDK? The SDK can choose to store models and artifacts on any object storage (not in the clearml server). The server itself only has the fileserver storage, but it is not up to the server which storage will be used - it's an SDK choice.

Posted one year ago

Sorry for the delay on responding to this, I had some family craziness.

Posted one year ago

Thank you

Posted one year ago

This answered my question, I ended up setting up minio running off of a external hard drive which backups to onedrive.

Though I am still having some problems getting the client to connect to minio (see recent thread)

Posted one year ago

Thank you!

I wasn’t getting my hopes up on storing tasks elsewhere :)

For the models & artifacts is there a parameter to change the default directory of saving and loading to something else?

Posted one year ago

Hi @<1538330703932952576:profile|ThickSeaurchin47> , if you have rsync, can't you just rsync the fileserver data folder to onedrive?

Posted one year ago
10 Answers
one year ago
one year ago