Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! For

Hi! For clearml.Dataset we can provide target storage (for example, in S3) with output_url parameter in the upload method.

Is there any similar way to provide target storage for OutputModel and InputModel ?

  
  
Posted 2 years ago
Votes Newest

Answers 23


MelancholyElk85 , you can specify the uri for that in your ~/clearml.conf file under sdk.development.default_output_uri

Please note that you don't provide target storage for InputModel since it's an input, and can be used only as an existing object in the system ๐Ÿ™‚

  
  
Posted 2 years ago

but for datasets there seems to be a way to do it from the code

  
  
Posted 2 years ago

specifying target storage for OutputModel from the config is bad idea: it should be project-specific, not agent-specific

  
  
Posted 2 years ago

Hmmmm, fair point.

  
  
Posted 2 years ago

what's the difference between set_default_upload_uri and set_upload_destination ?

  
  
Posted 2 years ago

I think the set_default_upload_uri is for all output models, while set_upload_destination is for a specific model/file

  
  
Posted 2 years ago

All output models in the script, that is ๐Ÿ™‚

  
  
Posted 2 years ago

ok, got it, thanks

  
  
Posted 2 years ago

CostlyOstrich36 and at what point the model weights will be actually uploaded?

  
  
Posted 2 years ago

MelancholyElk85 if you are manually adding models OutputModel, then when you call update_weights(...) upload will start in the background (if the process ends it will wait until the upload is competed). You can also specify auto_delete_file which will delete the local copy once the upload completes

  
  
Posted 2 years ago

And can I store models with no attachment to tasks? For example, original pretrained checkpoints

  
  
Posted 2 years ago

And can I store models with no attachment to tasks?

Assuming you have the Model ID :
model = InputModel(model_id='aabbcc') local_file_or_folder = model.get_weights()Is this what you are looking for?

  
  
Posted 2 years ago

AgitatedDove14 not exactly:
input: just a checkpoint file
output: a clearml model entity + stored weights on S3

  
  
Posted 2 years ago

Ohh, like a query based only of the stored url ?
Do you also has the creating Task ?

  
  
Posted 2 years ago

No, I just want to register a new model in the storage. I need to create a separate task for this right?

  
  
Posted 2 years ago

No, I just want to register a new model in the storage.

Is the model file is already uploaded, you can register it without a Task:
InputModel.import_model(...)https://github.com/allegroai/clearml/blob/b3a2b3425c5098ebfc0598c9dfb3e670d4a87706/clearml/model.py#L521

I need to create a separate task for this right?

If you want the model to be uploaded, then yes you have to create a Task.

  
  
Posted 2 years ago

Thanks. What is the difference between import_model and load_model ?

Why can I only call import_model with weights_url and not model name or ID? This means I need to call query_models first, if I got it right

  
  
Posted 2 years ago

load_model will get a link to a previously registered URL (i.e. it search a model pointing to the specific URL, if it finds it, it will get you the Model object)

  
  
Posted 2 years ago

Why can I only callย 

import_model

Actually creates a new Model object in the system
InputModel(id) will "load" a model based on the model id
Make sense ?

  
  
Posted 2 years ago

ok, got it, thanks

  
  
Posted 2 years ago
972 Views
23 Answers
2 years ago
one year ago
Tags