How To Use Zscaler (Or Custom Certificate) With Clearml ? I Installed The Zscaler Certificate Into The Os System.

Answered

How to use ZScaler (or custom certificate) with ClearML ?
I installed the ZScaler certificate into the OS system. curl is happy with it. boto3 too but then clearml is not :

WARNING:urllib3.connectionpool:Retrying (Retry(total=239, connect=240, read=240, redirect=240, status=240)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))': /auth.login
WARNING:urllib3.connectionpool:Retrying (Retry(total=238, connect=240, read=240, redirect=240, status=240)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))': /auth.login

And after Ctrl+C:

[...]
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/clearml/backend_api/utils.py", line 85, in send
    return super(SessionWithTimeout, self).send(request, **kwargs)
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  [Previous line repeated 5 more times]
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries.sleep()
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 432, in sleep
    self._sleep_backoff()
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 416, in _sleep_backoff
    time.sleep(backoff)
KeyboardInterrupt

Looks like connectionpool is not using the local system certificate ?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Votes Newest

Answers 11

Since the storage code does not use the same client/session for connecting to Azure, it will not use the certificate you provided in the env var, I think

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Is it because Azure is "whitelisted" in our network ? Thus need a different certificate ?? And how do I provide 2 differents certificate ? Is bundling them simple as a concat of 2 pem file ?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

See here: None

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Storage certificate are handled deperately

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

The question is why do you need a custom certificate for Azure at all?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> Actually it failed now: failed to talked to our storage in Azure:

ClearML Task: created new task id=c47dd71dea2f421db05647a21d78ed26



2024-01-25 21:45:23,926 - clearml.storage - ERROR - Failed uploading: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
2024-01-25 21:46:48,877 - clearml.storage - WARNING - Storage helper problem for .clearml.0149daec-7a03-4853-a0cd-a7e2b2958405.test: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)
Traceback (most recent call last):
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/clearml/storage/helper.py", line 2741, in check_write_permissions
    self.delete(path=dest_path)
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/clearml/storage/helper.py", line 2726, in delete
    return self._driver.delete_object(self.get_object(path))
  File "/mnt/c/data/suse-venv/lib64/python3.10/site-packages/clearml/storage/helper.py", line 1153, in delete_object
    container = object.container
AttributeError: 'NoneType' object has no attribute 'container'

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

The storage driver uses azure.storage.blob.BlobServiceClient - it's possible the package's implementation does not use the same env vars to obtain custom certificates and requires some other configuration

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> Thanks. Manage to get it working now with

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/zscaler.crt

(Ubuntu system)

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

@<1523701087100473344:profile|SuccessfulKoala55> I managed to make this working by:
concat the existing OS ca bundle and zscaler certificate. And set REQUESTS_CA_BUNDLE to that bundle file

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Hi @<1576381444509405184:profile|ManiacalLizard2> , you can set the REQUESTS_CA_BUNDLE env var to point to the certificate - this should do it

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

not sure ... providing Zscaler certificate seems to allow clearml to talk to our clearml server, hosted in azure, Task init worked. But then failed to connect to the storage account (Azure too) ...

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					ManiacalLizard2
				
					0
					 × 1

Write your answer

557 Views

11 Answers

9 months ago