Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I'M New To Clearml, And Trying To Understand The Set_Metadata Method Of The Dataset Class. The Documentation Is Almost Non-Existent Does Anyone Have An Example Script? How Would I Add Data Using E.G. Pandas. I Assume One Column Would Relate To The

Hi All,
I'm new to ClearML, and trying to understand the set_metadata method of the Dataset class. The documentation is almost non-existent
Does anyone have an example script? How would I add data using e.g. pandas. I assume one column would relate to the filename, and the other columns containing the metadata. But, how would I reference the filename? Any help much appreciated. Thanks

  
  
Posted one year ago
Votes Newest

Answers 7


The metadata would relate to the entire dataset.

For your use case I think what's relevant is HyperDatasets

None

  
  
Posted one year ago

Thanks John. But does the metadata relate to the entire dataset or individual elements in the dataset?
For example, lets say I have a dataset of images, and I would like to attach metadata to each image - e.g. a "type" field which could have values 1,2,3,4,5...
How would the dataframe be constructed? I assume one column would contain an ID identifying the image, and the other column would be "type". If this is the case, what would the ID be?

  
  
Posted one year ago

Hi @<1574931891478335488:profile|DizzyButterfly4> , I think if you have a pandas object pd then the usage would be something like ds.set_metadata(metadata=pd, metadata_name="my pandas object")

I think you would be referencing the entire thing using the metadata_name parameter

  
  
Posted one year ago

@<1574931891478335488:profile|DizzyButterfly4> , what do you feel was lacking from the documentation? A usage example?

  
  
Posted one year ago

OK so if I understand correctly, you can only add metadata to items in the enterprise version.
Just to confirm, the screen shot in the dataops pages here refers to the enterprise version?
None

  
  
Posted one year ago

John, regarding documentation - what is mainly missing is how he dataframe / numpy / dict is structures. Yes, an example would be helpful

  
  
Posted one year ago

It looks like there is a mistake on the website. I'll let the ones in charge to fix it ASAP, thanks for the heads up!

  
  
Posted one year ago