what's the difference between set_default_upload_uri
and set_upload_destination
?
How about this by the way?
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#outputmodelset_default_upload_uri
Thanks. What is the difference between import_model
and load_model
?
Why can I only call import_model
with weights_url
and not model name or ID? This means I need to call query_models
first, if I got it right
Why can I only callย
import_model
Actually creates a new Model object in the system
InputModel(id) will "load" a model based on the model id
Make sense ?
but for datasets there seems to be a way to do it from the code
No, I just want to register a new model in the storage. I need to create a separate task for this right?
MelancholyElk85 , you can specify the uri for that in your ~/clearml.conf
file under sdk.development.default_output_uri
Please note that you don't provide target storage for InputModel
since it's an input, and can be used only as an existing object in the system ๐
MelancholyElk85 if you are manually adding models OutputModel, then when you call update_weights(...)
upload will start in the background (if the process ends it will wait until the upload is competed). You can also specify auto_delete_file
which will delete the local copy once the upload completes
CostlyOstrich36 and at what point the model weights will be actually uploaded?
CostlyOstrich36 so it's the same problem as https://clearml.slack.com/archives/CTK20V944/p1636373198353700?thread_ts=1635950908.285900&cid=CTK20V944
And can I store models with no attachment to tasks? For example, original pretrained checkpoints
No, I just want to register a new model in the storage.
Is the model file is already uploaded, you can register it without a Task:InputModel.import_model(...)
https://github.com/allegroai/clearml/blob/b3a2b3425c5098ebfc0598c9dfb3e670d4a87706/clearml/model.py#L521
I need to create a separate task for this right?
If you want the model to be uploaded, then yes you have to create a Task.
load_model
will get a link to a previously registered URL (i.e. it search a model pointing to the specific URL, if it finds it, it will get you the Model object)
Ohh, like a query based only of the stored url ?
Do you also has the creating Task ?
All output models in the script, that is ๐
I think the set_default_upload_uri
is for all output models, while set_upload_destination
is for a specific model/file
AgitatedDove14 not exactly:
input: just a checkpoint file
output: a clearml model entity + stored weights on S3
specifying target storage for OutputModel
from the config is bad idea: it should be project-specific, not agent-specific
And can I store models with no attachment to tasks?
Assuming you have the Model ID :model = InputModel(model_id='aabbcc') local_file_or_folder = model.get_weights()
Is this what you are looking for?