Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
What Is The Right Way To Increase Number Of Retries When Using

what is the right way to increase number of retries when using StorageManager.get_local_copy?

  
  
Posted 2 years ago
Votes Newest

Answers 10


I'm not sure since names of these parameters do not match with boto3 names, and num_download_attempt is passed https://github.com/allegroai/clearml/blob/3d3a835435cc2f01ff19fe0a58a8d7db10fd2de2/clearml/storage/helper.py#L1439 as container.config.retries

  
  
Posted 2 years ago

isn't this parameter related to communication with ClearML Server? I'm trying to make sure that checkpoint will be downloaded from AWS S3 even if there are temporary connection problems

there's https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig parameter in boto3, but I'm not sure if there's an easy way to pass this parameter to StorageManager

  
  
Posted 2 years ago

Oh you're right. My bad

  
  
Posted 2 years ago

DilapidatedDucks58 , I think this is what you want. You just configure it and don't need to changing anything in code.
https://github.com/allegroai/clearml/blob/92210e2c827ff20e06700807d25783691629808a/docs/clearml.conf#L106
You can pass any boto3 parameter here!

  
  
Posted 2 years ago

DilapidatedDucks58 long story short:
if you do:
` from clearml import StorageManager
from clearml.storage.helper import StorageHelper

StorageHelper.get(" ", retries=5) `It should make sure that all the other s3:// links of this bucket will use the same original configuration (i.e. retries)

If this workaround works let's make sure we add it into the conf file, wdyt ?

  
  
Posted 2 years ago

yeah, I think this is what I need!

  
  
Posted 2 years ago

let me know if it works 🙂

  
  
Posted 2 years ago

DilapidatedDucks58 , I think this is what you're looking for
https://github.com/allegroai/clearml/blob/master/docs/clearml.conf#L69

  
  
Posted 2 years ago

Looking for something else 🙂

  
  
Posted 2 years ago
1K Views
10 Answers
2 years ago
one year ago
Tags