Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Getting A Bunch Of This Kind Of Log Messages "Clearml.Storage - Info - Starting Upload: /Tmp/.Clearml.Upload_Model_6Ou50Pb1.Tmp =>" I Am Pretty Sure They Happen As A Part Of The Model Initialization About 10 Of Those, My Guess Is That Every T

Hi all, I am getting a bunch of this kind of log messages "clearml.storage - INFO - Starting upload: /tmp/.clearml.upload_model_6ou50pb1.tmp =>" I am pretty sure they happen as a part of the model initialization about 10 of those, my guess is that every time a torch buffer is registered it uploads a copy (or something of this sort..) this is an unwanted behavior in my case.. how can I disable it?

  
  
Posted 3 years ago
Votes Newest

Answers 13


I added it to my code before Task.init or after both didn't seem to change anything

  
  
Posted 3 years ago

Hmm so there is a way to add callbacks (somewhat cumbersome, and we would live feedback) so you can filter them out.
What do you think, would that work?

  
  
Posted 3 years ago

Hi AgitatedDove14 , I am not uploading anything explicitly, and when I look at the UI Models tab I can only see the regular "{Project Name} - epoch={#}" and in addition "{Project Name} - {project_id}" so I am not sure what is really uploaded.. from the name of it it sounds like model weights and buffers (non-trainable)

  
  
Posted 3 years ago

am I supposed to change the WeightsFileHandler inplace?

  
  
Posted 3 years ago

well when returning None it works as expected, no model uploads

  
  
Posted 3 years ago

or should it be assigned somewhere

  
  
Posted 3 years ago

You can see the class here:
https://github.com/allegroai/clearml/blob/9b962bae4b1ccc448e1807e1688fe193454c1da1/clearml/binding/frameworks/init.py#L52

Basically you do:
` def my_callback(load_or_save, model):
# type: (str, WeightsFileHandler.ModelInfo) -> WeightsFileHandler.ModelInfo
assert load_or_save not in ('load', 'save')
# do something
if skip:
return None
return model

WeightsFileHandler.add_pre_callback(my_callback) `

  
  
Posted 3 years ago

RipeGoose2 you can put ut before/after the Task.init, the idea is for you to set it before any of the real training starts.
As for not effecting anything,
Try to add the callback and just have it returning None (which means skip over the model log process) let me know if this one works

  
  
Posted 3 years ago

Hi RipeGoose2
What exactly is being uploaded ? Are those the actual model weights or intermediate files ?

  
  
Posted 3 years ago

AgitatedDove14 Thanks, I am aware of the auto_connect_frameworks and it makes sense for torch.save (and probably other stuff) however if it is called about 10 times during model initialization it seem excessive and I would be happy to avoid this behavior during model initialization if possible. do you know what is actually triggering the uploads?

  
  
Posted 3 years ago

Yey 🙂 !
So now you can add some logic based on the model object passed as the second argument (see WeightsFileHandler.ModelInfo)
The easiest is based on the model name see model.local_model_path

  
  
Posted 3 years ago

RipeGoose2 models are automatically registered
i.e. added to the models artifactory, but it only points to where the files are stored
Only if you are passing the output_uri argument to the Task.init, they will be actually uploaded.
If you want to disable this behavior you can pass
Task.init(..., auto_connect_frameworks={'pytorch': False})

  
  
Posted 3 years ago

sounds like an ok option, any recommended reference for how to start?

  
  
Posted 3 years ago
526 Views
13 Answers
3 years ago
one year ago
Tags