MelancholyElk85 , you can specify the uri for that in your ~/clearml.conf
file under sdk.development.default_output_uri
Please note that you don't provide target storage for InputModel
since it's an input, and can be used only as an existing object in the system ๐
CostlyOstrich36 so it's the same problem as https://clearml.slack.com/archives/CTK20V944/p1636373198353700?thread_ts=1635950908.285900&cid=CTK20V944
but for datasets there seems to be a way to do it from the code
specifying target storage for OutputModel
from the config is bad idea: it should be project-specific, not agent-specific
How about this by the way?
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#outputmodelset_default_upload_uri
what's the difference between set_default_upload_uri
and set_upload_destination
?
I think the set_default_upload_uri
is for all output models, while set_upload_destination
is for a specific model/file
All output models in the script, that is ๐
CostlyOstrich36 and at what point the model weights will be actually uploaded?
MelancholyElk85 if you are manually adding models OutputModel, then when you call update_weights(...)
upload will start in the background (if the process ends it will wait until the upload is competed). You can also specify auto_delete_file
which will delete the local copy once the upload completes
And can I store models with no attachment to tasks? For example, original pretrained checkpoints
And can I store models with no attachment to tasks?
Assuming you have the Model ID :model = InputModel(model_id='aabbcc') local_file_or_folder = model.get_weights()
Is this what you are looking for?
AgitatedDove14 not exactly:
input: just a checkpoint file
output: a clearml model entity + stored weights on S3
Ohh, like a query based only of the stored url ?
Do you also has the creating Task ?
No, I just want to register a new model in the storage. I need to create a separate task for this right?
No, I just want to register a new model in the storage.
Is the model file is already uploaded, you can register it without a Task:InputModel.import_model(...)
https://github.com/allegroai/clearml/blob/b3a2b3425c5098ebfc0598c9dfb3e670d4a87706/clearml/model.py#L521
I need to create a separate task for this right?
If you want the model to be uploaded, then yes you have to create a Task.
Thanks. What is the difference between import_model
and load_model
?
Why can I only call import_model
with weights_url
and not model name or ID? This means I need to call query_models
first, if I got it right
load_model
will get a link to a previously registered URL (i.e. it search a model pointing to the specific URL, if it finds it, it will get you the Model object)
Why can I only callย
import_model
Actually creates a new Model object in the system
InputModel(id) will "load" a model based on the model id
Make sense ?