We Have An Nginx Container In Front Of Clearml, Functions Use Sort Of " Ssl Reverse Proxy" We Are Seeing This Errors With Jobs That Are Trying To Upload Some Artifacts To Fileserver

Answered

we have an nginx container in front of ClearML, functions use sort of " SSL reverse proxy"
we are seeing this errors with jobs that are trying to upload some artifacts to fileserver

2024-01-10 08:10:14,537 - clearml.metrics - WARNING - Failed uploading to

 (HTTPSConnectionPool(host='clearml-hrl-03.haifa.ibm.com', port=8081): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))))

clearml.conf contains api.verify_certificate : false
it happen only when a job tries to upload something to fileserver, saving metrics/scalrs/plots working without issues

anyone tackled this?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

Votes Newest

Answers 6

@<1523701087100473344:profile|SuccessfulKoala55> seen this with 1.13.2
worth to try and upgrade to latest?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

just confirming this with the user and will share it over here
I do recall in the past that latest version caused this, and downgrading to some prior version fixed the issue

let me get the info and will post back here
10x @<1523701087100473344:profile|SuccessfulKoala55>

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

yep, again most jobs works .. the issue with when a job tries to upload artifacts to fileserver

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					PleasantOwl46
				
					0
					 × 1

clearml.conf contains

api.verify_certificate : false

@<1523701842515595264:profile|PleasantOwl46> this is in the user's configuration file, right?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

What is the clearml package version used by the job?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

might be

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

219 Views

6 Answers

3 months ago