We Have An Nginx Container In Front Of Clearml, Functions Use Sort Of " Ssl Reverse Proxy" We Are Seeing This Errors With Jobs That Are Trying To Upload Some Artifacts To Fileserver

Answered

we have an nginx container in front of ClearML, functions use sort of " SSL reverse proxy"
we are seeing this errors with jobs that are trying to upload some artifacts to fileserver

2024-01-10 08:10:14,537 - clearml.metrics - WARNING - Failed uploading to

 (HTTPSConnectionPool(host='clearml-hrl-03.haifa.ibm.com', port=8081): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))))

clearml.conf contains api.verify_certificate : false
it happen only when a job tries to upload something to fileserver, saving metrics/scalrs/plots working without issues

anyone tackled this?

  				
Posted 
	one year ago

					More  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

Votes Newest

Answers 6

SuccessfulKoala55 seen this with 1.13.2
worth to try and upgrade to latest?

  				
Posted 
	one year ago

					More  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

clearml.conf contains

api.verify_certificate : false

PleasantOwl46 this is in the user's configuration file, right?

  				
Posted 
	one year ago

					More  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

What is the clearml package version used by the job?

  				
Posted 
	one year ago

					More  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

might be

  				
Posted 
	one year ago

					More  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

yep, again most jobs works .. the issue with when a job tries to upload artifacts to fileserver

  				
Posted 
	one year ago

					More  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

just confirming this with the user and will share it over here
I do recall in the past that latest version caused this, and downgrading to some prior version fixed the issue

let me get the info and will post back here
10x SuccessfulKoala55

  				
Posted 
	one year ago

					More  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

Write your answer

1K Views

6 Answers

one year ago