Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, So I'M Trying To Upload An Artefact To Clearml’S Fileserver(I Have A Self Hosted Clearml Server Running), I'Ve Uploaded The File Using Storagemanager.Upload_File(Path, Url) And Giving The Url As “

Hey,
So I'm trying to upload an artefact to clearml’s fileserver(I have a self hosted clearml server running),
I've uploaded the file using StorageManager.upload_file(path, url)
And giving the url as “ http://localhost:8081/test/model.pth
How do I access an uploaded file? (in this case it's a model)

  
  
Posted one year ago
Votes Newest

Answers 25


Hi WickedElephant66

So I'm trying to upload an artefact to clearml’s fileserver(I have a self hosted clearml server running),

Are you trying to upload an artifact? If so I would do:
task.upload_artifact('local file', artifact_object="/path/to/file")Or is it about Model files?
You can alst check how to upload artifacts / models here:
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts_retrieval.py
https://github.com/allegroai/clearml/blob/master/examples/reporting/model_reporting.py

  
  
Posted one year ago

But it seems to upload the model on noticing torch.save irrespective

  
  
Posted one year ago

you can specify the destination of the uploading like that :
when you initiate a task, you can set the parameter output_uri. If you set it to True, then the model will be uploaded to the uri specified in your conf file. Youcan also directly specify an url or you can use OutputModel.set_default_upload_uri or set_upload_destination ( https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#outputmodelset_default_upload_uri or https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel#set_upload_destination )

  
  
Posted one year ago

Also, does PipelineDecorator.upload_model store anything on the fileserver ? I can't seem to understand the use of PipelineDecorator.upload_model() apart from making a model appear on the pipeline task

  
  
Posted one year ago

So I am able to access it via sending requests to the clearml fileserver but, any way to access it from the dashboard(the main app)?

  
  
Posted one year ago

Do you mean from within a pipeline ? Do you manually report the model ? It might point to a local file, especially if it has been auto logged. That is what happens when you are saving your model (thus to the local file system) from your script.

  
  
Posted one year ago

Yep, that's exactly what's happening.
However here's what I want to do:
upload model to clearml’s fileserver and get the model url in the details for easy download

  
  
Posted one year ago

How do I provide an output storage destination for that stage of the pipeline?

  
  
Posted one year ago

yes but it is supposed to be logged in the task corresponding to the step the model is being saved from. monitor_model makes the logging to the main pipeline task.

  
  
Posted one year ago

David,
I haven't provided a monitor_model parameter

  
  
Posted one year ago

Oh

  
  
Posted one year ago

Hi WickedElephant66
When you are in the Projects section of the WebApp (second icon on the left), enter either "All Experiments" or any project you want to access to. Up on the center is the Models section. You csn find the url the model can be downloaded from, in the details, section

  
  
Posted one year ago

So the issue is that the model url points to the file location on my machine,
Is there a way for me to pass the model url something else?

  
  
Posted one year ago

Can PipelineDecorator.upload_model be used to store models on the clearml fileserver?

  
  
Posted one year ago

It depend if you have added the parameter monitor_models=['mymodel'] in the PipelineDecorator.component. If not, the model will not be logged. If you have, then torch.save will save your model to a local path, that will be logged into the main pipeline task

  
  
Posted one year ago

So I did exactly that, and the name and path of the model on the local repo is noted;
However, I want to upload it to the fileserver

  
  
Posted one year ago

Also, does clearml by default upload models if we save them using torch.save?

  
  
Posted one year ago

Are Kwargs supported in functions decorated as a pipeline component?

They are, but I think the main issue is the casting, without prior knowledge, everything will be a tring

  
  
Posted one year ago

Hey, thanks for the reply
I have another question ;
Are Kwargs supported in functions decorated as a pipeline component?

  
  
Posted one year ago

Basically I have a script that generates a pipeline report in pdf format, was wondering if that can be logged

  
  
Posted one year ago

Hey, is it possible for me to upload a pdf as an artefact?

Sure, just point to the file and it will upload it for you 🙂

  
  
Posted one year ago

I'm asking this because my kwargs is observed as an empty dict if printed

  
  
Posted one year ago

Hey, is it possible for me to upload a pdf as an artefact?

  
  
Posted one year ago

From a pipeline, you can use PipelineController.upload_model( name , path ) and specify in path the path you used to save your model from your script.

  
  
Posted one year ago

Hey David , I was able to get things uploaded to the fileserver by a change in the conf

  
  
Posted one year ago
596 Views
25 Answers
one year ago
one year ago
Tags