Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey All. Thinking About Organisation Of Artefacts And How We Store Them In S3. We Have Been Using The

Hey all. Thinking about organisation of artefacts and how we store them in S3. We have been using the StorageManager to upload files to target S3 destinations. The issue with this approach is that you need to export the file to disk prior to the upload. Is there another way of exporting artefacts to target destinations which can do this without the prior export to disk?

Thinking to have a structure like:
raw data: <bucket>/raw-data/<partner>/<date>/
training/transformed data: <bucket>/training-data/<partner>/<date>/
models : <bucket>/models/<partner>/<date>/
data transformers: <bucket>/transformers/<partner>/<date>/
training-runs: <bucket>/training-runs/<partner>/<date>/

Another thing is that we have metadata exported for each artefact e.g. for raw data, you have:
partner name min timestamps in data max timestamp in data s3 location dataset ID task ID created by etc
The plan is to load the metadata exports into a separate s3 path to be loaded into the data warehouse so that we can query for analytics & deployment purposes. Any suggestions/insights on this?

  
  
Posted 2 years ago
Votes Newest

Answers 6


I'm specifically interested in the model-first queries you would like to do (as experiment-first queries are fully featured, we want to understand whats the best way to take that into models)

  
  
Posted 2 years ago

Thanks Ariel, will give it a watch now

  
  
Posted 2 years ago

Thanks GrumpyPenguin23 , will have a look shortly 🙂

  
  
Posted 2 years ago

Hi TenseOstrich47 sorry for the long wait, here is a video + code of how to put any sort of metadata inside your clearml model artifact 😉 We will also be improving this, so if you have feature requests we would love to hear about them
https://www.youtube.com/watch?v=WIZ88SmT58M&list=PLMdIlCuMqSTkXBApOMqg2S5IeVfnkq2Mj&index=12

  
  
Posted 2 years ago

Ideally, I want to avoid re-inventing the wheel so if this functionality already exists with some examples then it would be great if someone could point me to it

  
  
Posted 2 years ago

TenseOstrich47 as i might have stated earlier, I'm doing a low-key build of something like this. Thanks to your question I know what to focus on when showcasing 😇 ♥

Here's what's already been done https://youtu.be/xliX3IhNdmw

  
  
Posted 2 years ago