ClearML FAQ | Hi, We'Re Facing An Error When Uploading Model Checkpoints To Clearml During Training (Using Clearml Version 1.9.0 And Pytorch Lightning 1.7.6), Anyone Knows How To Solve? Thanks! The Error: Clearml.Storage - Error

Answered

Hi, We'Re Facing An Error When Uploading Model Checkpoints To Clearml During Training (Using Clearml Version 1.9.0 And Pytorch Lightning 1.7.6), Anyone Knows How To Solve? Thanks! The Error: Clearml.Storage - Error - Failed Uploading: Httpsconnectionpool(

Hi, we're facing an error when uploading model checkpoints to clearml during training (using clearml version 1.9.0 and pytorch lightning 1.7.6), anyone knows how to solve? Thanks!
The error:
clearml.storage - ERROR - Failed uploading: HTTPSConnectionPool(host=' http://storage.googleapis.com ', port=443): Max retries exceeded with url: /upload/storage/v1/b/models/o?uploadType=resumable (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)')))

  				
Posted 
	one year ago

					More
				  		
  Report
		
					TightDog77
				
					0
					 × 1

Votes Newest

Answers 4

But this is clearml python package, it is not really related to the server. Could it be you also update the clearml package ?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Hi TightDog77 _

HTTPSConnectionPool(host='

', port=443): Max retries exceeded with url: /upload/storage/v1/b/models/o?uploadType=resumable (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)')))

This seems like a network error to GCP, (basically GCP python package thows it)
Are you always getting this error? is this something new ?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

After update the error seem to disappear , thanks !

  				
Posted 
	one year ago

					More
				  		
  Report
		
					TightDog77
				
					0
					 × 1

This started happening once ClearML web got updated several times in the last 7 days.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					TightDog77
				
					0
					 × 1

Write your answer

2K Views

4 Answers

one year ago