Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! Which Method Is Used To Delete Outputmodel From The Storage?

Hi! Which method is used to delete OutputModel from the storage?
https://clear.ml/docs/latest/docs/references/sdk/model_outputmodel/

  
  
Posted one year ago
Votes Newest

Answers 19


yeah, during a run

  
  
Posted one year ago

If I keep track of 3 OutputModels simultaneously, the weights would need to shift between them every epoch (like, updated weights for top-1, then top-1 becomes top-2, top-2 becomes top-3 etc)

  
  
Posted one year ago

if I just use plain boto3 to sync weights to/from S3, I just check how many files are stored in the location, and clear up the old ones

  
  
Posted one year ago

You mean you would like to delete an output model of a task if other models in the task surpass it?

  
  
Posted one year ago

During a run, correct?

  
  
Posted one year ago

How are you saving your models? torch.save ("<MODEL_NAME>") ?

  
  
Posted one year ago

If I'm not mistaken, models reflect the file names. So if you recycle the file names you recycle the models. So if you save torch.save(" http://top1.pt ") then later torch.save(" http://top2.pt ") and even later do torch.save(" http://top1.pt ") again, you will only have 2 OutputModels, not three. This way you can keep recycling the best models 🙂

  
  
Posted one year ago

this is how I implemented it by myself. Looks like clearml functionality is quite opinionated and requires some tweaks every time I try to replace my own stuff with it

  
  
Posted one year ago

` clearml_name = os.path.basename(save_path)

output_model_best = OutputModel(
task=task,
name=clearml_name,
tags=['running-best'])

output_model_best.update_weights(
save_path,
upload_uri=params.clearml_aws_checkpoints,
target_filename=clearml_name
) `

  
  
Posted one year ago

This way I would want to keep track of 3 OutputModel s and call update_weights 3 times every update - and probably do 2 redundant uploadings

  
  
Posted one year ago

Well, you can simply do the following:
Start with top 3 models named top1, top2, top3 Keep all 3 in disk cache during run Build logic to rate new model during run depending on it's standing compared to top 3 Decide on new standing of top 3 Perform update_weights_package on the relevant "new" top 3 models once per modelThis is only from the top of my head. I'm sure you could create something better without even the need to cache 3 models during the run

  
  
Posted one year ago

yes

  
  
Posted one year ago

is there a some sort of OutputModel.remove method? Docs say there isn't

  
  
Posted one year ago

CostlyOstrich36 thank you for the answer! Maybe I just can delete old models along with corresponding tasks, seems to be easier

  
  
Posted one year ago

Strictly speaking, there is only one training task, but I want to keep top-3 best checkpoints for it all the time

  
  
Posted one year ago

hm, not quite clear how it is implemented. For example, this is how I do it now (explicitly)

  
  
Posted one year ago

e.g. if I want to store only top-3 running best checkpoints

  
  
Posted one year ago

if the loss is lower than the best stored loss so far, add the new checkpoint and remove the top-4th

  
  
Posted one year ago

Is there a way to simplify it with ClearML, not make it more complicated?

  
  
Posted one year ago
369 Views
19 Answers
one year ago
10 months ago
Tags