Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone! I'M Using S3 For My Model Saving. During Hyperparameter Optimization My New Tasks Get Very Long Names Due To Override Parameters And Uploading Path Becomes Something Like This "/Traffic Lights Classification/

Hello everyone! I'm using S3 for my model saving. During hyperparameter optimization my new tasks get very long names due to override parameters and uploading path becomes something like this "/Traffic lights classification/ Train%3A General%2Fgeneral%2Fbatch_size_train=192 General%2Fgeneral%2Fmodel_name=resnet18 General%2Fimage_processing%2Faugmentation_version=2 General%2Ftrain%2Flearning_rate=0.00951 General%2Ftrain%2Floss_function=HuberLoss General%2Ftrain%2Foptimizer=yogi.1b46427f3a79406dbab354a71b410a88 /models/epoch=037-val_epoch_score=0.926.ckpt". S3 can't create a folder with such a long name in its bucket. How to fix such task naming?

  
  
Posted 2 years ago
Votes Newest

Answers 10


MinuteGiraffe30 , Hi ! 🙂

What if you try to manually create such a folder?

  
  
Posted 2 years ago

AgitatedDove14  
Stackoverflow says about the size of the file name, but here the problem is with the size of the folder name. Not sure if the file and folder limits are the same. The error that occurs says that An error occurred (XMinioInvalidObjectName) when calling the CompleteMultipartUpload operation: Object name contains unsupported characters. . At first I thought I was using the wrong characters in the task title, such as backslash, colon or space. But I managed to upload models with the task name including all these symbols when I used fewer override parameters in my hypeparameters optimization task. So I assumed that the problem is in the length of the folder. Here is an example.

  
  
Posted 2 years ago

Maybe you should make 

naming_function

 as public variable in 

SearchStrategy

 class or allow changing it in 

HyperParameterOptimizer

 class?

I like this idea, let's do that
Just making sure, you hit the 1024 character limit on S3 path?
If this is the case we should also fix the "artifact naming" to take that into account (it already does and has a limit, see here:
https://github.com/allegroai/clearml/blob/24464b7c1019f7a7b3149ecb80a379c5f82337a0/clearml/backend_interface/metrics/events.py#L351
Should we also limit the entire full path?

  
  
Posted 2 years ago

I run model training for weeks, where I train hundreds of models with different hyperparameters. Therefore, I need automatic naming of newly created tasks. Maybe there are some examples of how to set custom task naming scheme?

  
  
Posted 2 years ago

Maybe you should make naming_function as public variable in SearchStrategy class or allow changing it in HyperParameterOptimizer class?

  
  
Posted 2 years ago

Thanks MinuteGiraffe30 , fix will be pushed later today

  
  
Posted 2 years ago

Maybe it makes sense to limit or change number of parameters which participate in name generation or allow users to use their own naming scheme. https://github.com/allegroai/clearml/blob/24464b7c1019f7a7b3149ecb80a379c5f82337a0/clearml/automation/optimization.py#L720

  
  
Posted 2 years ago

I came up with a workaround to solve this problem, but since I run tasks remotely, where not my modified clearml code is used, this problem will remain until you fix it. Should I duplicate this issue in the github?

  
  
Posted 2 years ago
550 Views
10 Answers
2 years ago
one year ago
Tags