We Have An Nginx Container In Front Of Clearml, Functions Use Sort Of " Ssl Reverse Proxy" We Are Seeing This Errors With Jobs That Are Trying To Upload Some Artifacts To Fileserver

Answered

we have an nginx container in front of ClearML, functions use sort of " SSL reverse proxy"
we are seeing this errors with jobs that are trying to upload some artifacts to fileserver

2024-01-10 08:10:14,537 - clearml.metrics - WARNING - Failed uploading to

 (HTTPSConnectionPool(host='clearml-hrl-03.haifa.ibm.com', port=8081): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))))

clearml.conf contains api.verify_certificate : false
it happen only when a job tries to upload something to fileserver, saving metrics/scalrs/plots working without issues

anyone tackled this?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

Votes Newest

Answers 6

@<1523701087100473344:profile|SuccessfulKoala55> seen this with 1.13.2
worth to try and upgrade to latest?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

just confirming this with the user and will share it over here
I do recall in the past that latest version caused this, and downgrading to some prior version fixed the issue

let me get the info and will post back here
10x @<1523701087100473344:profile|SuccessfulKoala55>

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

might be

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

What is the clearml package version used by the job?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

clearml.conf contains

api.verify_certificate : false

@<1523701842515595264:profile|PleasantOwl46> this is in the user's configuration file, right?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

yep, again most jobs works .. the issue with when a job tries to upload artifacts to fileserver

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

Write your answer

719 Views

6 Answers

10 months ago