Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, We'Re Facing An Error When Uploading Model Checkpoints To Clearml During Training (Using Clearml Version 1.9.0 And Pytorch Lightning 1.7.6), Anyone Knows How To Solve? Thanks! The Error: Clearml.Storage - Error - Failed Uploading: Httpsconnectionpool(

Hi, we're facing an error when uploading model checkpoints to clearml during training (using clearml version 1.9.0 and pytorch lightning 1.7.6), anyone knows how to solve? Thanks!
The error:
clearml.storage - ERROR - Failed uploading: HTTPSConnectionPool(host=' http://storage.googleapis.com ', port=443): Max retries exceeded with url: /upload/storage/v1/b/models/o?uploadType=resumable (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)')))

  
  
Posted one year ago
Votes Newest

Answers 4


Hi TightDog77 _

HTTPSConnectionPool(host='

', port=443): Max retries exceeded with url: /upload/storage/v1/b/models/o?uploadType=resumable (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)')))

This seems like a network error to GCP, (basically GCP python package thows it)
Are you always getting this error? is this something new ?

  
  
Posted one year ago

But this is clearml python package, it is not really related to the server. Could it be you also update the clearml package ?

  
  
Posted one year ago

This started happening once ClearML web got updated several times in the last 7 days.

  
  
Posted one year ago

After update the error seem to disappear , thanks !

  
  
Posted one year ago