Hi RipeGoose2
What exactly is being uploaded ? Are those the actual model weights or intermediate files ?
Hi AgitatedDove14 , I am not uploading anything explicitly, and when I look at the UI Models tab I can only see the regular "{Project Name} - epoch={#}" and in addition "{Project Name} - {project_id}" so I am not sure what is really uploaded.. from the name of it it sounds like model weights and buffers (non-trainable)
RipeGoose2 models are automatically registered
i.e. added to the models artifactory, but it only points to where the files are stored
Only if you are passing the output_uri argument to the Task.init, they will be actually uploaded.
If you want to disable this behavior you can passTask.init(..., auto_connect_frameworks={'pytorch': False})
AgitatedDove14 Thanks, I am aware of the auto_connect_frameworks and it makes sense for torch.save (and probably other stuff) however if it is called about 10 times during model initialization it seem excessive and I would be happy to avoid this behavior during model initialization if possible. do you know what is actually triggering the uploads?
Hmm so there is a way to add callbacks (somewhat cumbersome, and we would live feedback) so you can filter them out.
What do you think, would that work?
sounds like an ok option, any recommended reference for how to start?
You can see the class here:
https://github.com/allegroai/clearml/blob/9b962bae4b1ccc448e1807e1688fe193454c1da1/clearml/binding/frameworks/init.py#L52
Basically you do:
` def my_callback(load_or_save, model):
# type: (str, WeightsFileHandler.ModelInfo) -> WeightsFileHandler.ModelInfo
assert load_or_save not in ('load', 'save')
# do something
if skip:
return None
return model
WeightsFileHandler.add_pre_callback(my_callback) `
I added it to my code before Task.init or after both didn't seem to change anything
am I supposed to change the WeightsFileHandler inplace?
or should it be assigned somewhere
RipeGoose2 you can put ut before/after the Task.init, the idea is for you to set it before any of the real training starts.
As for not effecting anything,
Try to add the callback and just have it returning None (which means skip over the model log process) let me know if this one works
well when returning None it works as expected, no model uploads
Yey 🙂 !
So now you can add some logic based on the model object passed as the second argument (see WeightsFileHandler.ModelInfo)
The easiest is based on the model name see model.local_model_path