Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Struggling For Following Points. 1. Trying To Update Model Metadata Through

Hi,
I am struggling for following points.

  1. Trying to update model metadata through APIClient .
    For which i am using POST /models.add_or_update_metadata
    As per the documentation to update the metadata there is a parameter metadata : < models.metadata_item > array.

I tried creating metadata object by doing import client.models.metadata_item .
But facing error: AttributeError: 'Models' object has no attribute 'metadata_item'

  1. Is there any way to add scalars (metric) value to MODELS through APIClient ?

  2. Tried setting the dataset compression by passing "ZIP_DEFLATED","ZIP_BZIP2" & "ZIP_LZMA" as string to upload .
    But facing error NotImplementedError: That compression method is not supported .

dataset.upload(show_progress=True,chunk_size=10,compression="ZIP_DEFLATED")

  1. Chunk Size dataset.upload(chunk_size):
    I am trying to understand how the Chunk size parameter from upload works.
    Following are the iteration i performed to upload two .npz files through add_files & upload

file1.npz -> 1.63 MB
file2.npz -> 315.01 MB

Iteraion 1 : dataset.upload(show_progress=True)
No chunk_size set.
In logs :
File compression and upload completed: total size 316.65 MiB, 1 chunk(s) stored (average size 316.65 MiB)

Iteraion 2 : dataset.upload(show_progress=True,chunk_size=10)
chunk_size set to 10 MB
In logs :
File compression and upload completed: total size 316.64 MiB, 2 chunk(s) stored (average size 158.32 MiB)

  
  
Posted one month ago
Votes Newest

Answers 7


There are only 2 chunks because we don't split large files into multiple chunks

  
  
Posted one month ago

Hi @<1654294820488744960:profile|DrabAlligator92> ,

Regarding the model metadata, you don't need to actually construct the types, just use a list of dictionaries, and they will be casted automatically, for example:

client.models.add_or_update_metadata(model="<model-id>", metadata=[{"key": "foo", "type": "int", "value": "1"}])
  
  
Posted one month ago

Regarding the compression, you need to use a zipfile constant, not a string, import it using from zipfile import ZIP_DEFLATED

  
  
Posted one month ago

Hi @<1654294820488744960:profile|DrabAlligator92> ! The way chunk size works is:
the upload will try to obtain zips that are smaller than the chunk size. So it will continuously add files to the same zip until the chunk size is exceeded. If the chunk size is exceeded, a new chunk (zip) is created. The initial file in this chunk is the file that caused the previous size to be exceeded (regardless of the fact that the file itself might exceed the size).
So in your case: am empty chunk is created. file1.npz is added to the chunk. The size is not exceeded. Then file2.npz would be added to the same zip, but the file is too large, so the chunk size would be exceeded. A new chunk is then created, containing file2.npz

  
  
Posted one month ago

@<1523701087100473344:profile|SuccessfulKoala55> Thanks for your response.
Currently we are targeting the APICLIENT for creating a model and adding scalar value.

from clearml.backend_api.session.client import APIClient
client = APIClient()
model_instance = client.models.create(name="sample_model",project="model_id",uri="sample_url")

The above code will create and return the model id.
It would be helpful if we can get a sample documentation or sample code snippet how we can use None
to add scalar value for above created model.

  
  
Posted one month ago

Hi @<1523701435869433856:profile|SmugDolphin23> Thanks for response.
As shared above i am initializing the chunk_size = 10 (i.e. 10MB)

dataset.upload(show_progress=True,chunk_size=10)

Uploading 2 files :
file1.npz -> 1.63 MB
file2.npz -> 315.01 MB

In logs i get below.
File compression and upload completed: total size 316.64 MiB, 2 chunk(s) stored (average size 158.32 MiB)

Now as per the explanation the chunk size is set to 10MB
First chunk will fill upto 10MB (file1.npz is 1.63 MB + Include file2.npz).
Second chunk will fill upto 10MB.
and so on.....
So here there should be more than 2 chunks right ?

  
  
Posted one month ago

Regarding reporting scalars, you can, but it won't be as easy as using the SDK calls - basically you can send a scalar event referencing the model (i.e. using client.events.add() )

  
  
Posted one month ago
137 Views
7 Answers
one month ago
one month ago
Tags