Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Hi Everyone, I Have Been Using Clearml Self Hosted For Experiment Visualisation - Logging Loss And Uploading Debug Samples Etc. Our Clearml Server Is Running On A Aws Instance, With The Fileserver And Databases (Mongo, Redis Etc) On That Instance. We Run

Hi Everyone,

I have been using ClearML self hosted for experiment visualisation - logging loss and uploading debug samples etc.
Our ClearML server is running on a AWS instance, with the fileserver and databases (mongo, redis etc) on that instance.
We run a lot of experiments and log a lot of data, and I would like to move the fileserver to S3, and potentially move the databases off the instance onto something that scales with the size of the data so I don't need to monitor any disk usage.
Has anyone got any tips or guides on how to set this up?

Thanks in advance :)

Posted one year ago
Votes Newest

Answers 2

Hi TimelyCrab1 , directing all your outputs to s3 is actually pretty easy. You simply need to configure api.files_server: <S3_BUCKET/SOME_DIR> in clearml.conf of all machines working on it.

Migrating existing data is more difficult since everywhere in the system everything is saved as links. I guess you could change the links in mongodb but I would advise against it.

Posted one year ago

TimelyCrab1 there were several discussions here and in the GitHub issues on how to migrate fileserver data (databases should be just fine if you just move or backup/restore the data)

Posted one year ago