Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Running Into Ssl Verification Issues With Trying To Upload Model Artifacts To Minio. We Are Running The Clearml Agent In A Container, Have Mounted A Ca Bundle To The Container And Referenced It On Env Vars So That Aws Cli/Boto And Requests Us

Hi all, I am running into SSL verification issues with trying to upload model artifacts to minio. We are running the clearml agent in a container, have mounted a CA bundle to the container and referenced it on env vars so that aws cli/boto and requests uses it. Here is a snippet of the docker-compose:

` version: "2.3"
services:

clearmlagent_0:
container_name: clearml-agent-0
image: <redacted>
restart: unless-stopped
entrypoint: ["clearml-agent", "daemon", "--queue", "default", "--cpu-only", "--foreground"]
volumes:
- /home/clearmlagent/clearml.conf:/home/clearmlagent/clearml.conf
- /home/clearmlagent/.ssh:/home/clearmlagent/.ssh
- /etc/pki/tls/certs/ca-bundle.crt:/etc/ssl/certs/ca-certificates.crt
environment:
CLEARML_WORKER_ID: "qa-clearml-1:0"
AWS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt `
The agent will successfully pick up and run a job but fails with SSL verification error when trying to upload the artifact to minio. I am able to successfully upload and get files via aws cli:

$ .local/bin/aws --endpoint-url https:/<redacted> s3 --profile minio cp test upload: ./test to
And also am able to run a simple python script using boto configured to use the minio endpoint.

So my question is, is the clearml-agent execution doing something to override these settings and how can I properly set the correct ca bundle for it to use?

  
  
Posted 2 years ago
Votes Newest

Answers 20


ok, ya something must override the standard configuration options

  
  
Posted 2 years ago

yes, I have secure: true, when set as false its a different error. and yes the agent is running in venv mode (as opposed to?)

  
  
Posted 2 years ago

Here is what the error is:
botocore.exceptions.SSLError: SSL validation failed for https://<redacted>/clearml/.clearml.test [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
And here is the applicable configuration in clearml.conf:

` sdk {
aws {
s3 {
key: "<redacted>"
secret: "<redacted>"
host: "<redacted>"

        credentials: [
            {
                bucket: "clearml"
                host: "<redacted>"
                key: "<redacted>"
                secret: "<redacted>"
                multipart: false
                secure: true
             }
        ]
    }
}
development {
    default_output_uri: "s3://<redacted url matching host above>/clearml"
}

} `
based on iterations before, I was getting issues reaching the minio server, I have rectified that and now its an ssl issue, but I have no idea what is misconfigured as awscli and boto3 both work fine when run within the same container to put/get objects from the same server and bucket

  
  
Posted 2 years ago

Sure thing :)
BTW could you maybe PR this argument (marked out) so that we know for next time?

Say here:
https://github.com/allegroai/clearml-agent/blob/5a6caf6399a0128ad81e8723d0a847e2ded5b75e/docs/clearml.conf#L287

  
  
Posted 2 years ago

Hi LooseClams37
From the docker compose, I see the agent is running in venv mode, is that correct?
Also notice that when configuring the minio credentials you can specify if this is an https connection (secure: true) which by default it is not.
See here: https://github.com/allegroai/clearml-agent/blob/5a6caf6399a0128ad81e8723d0a847e2ded5b75e/docs/clearml.conf#L287

  
  
Posted 2 years ago

Hmm can you try with additional configuration, next to "secure: true" in your clearml.conf, can you add "verify: false"

  
  
Posted 2 years ago

But first I want to make sure the verify argument is actually used, hence False

  
  
Posted 2 years ago

hey, that worked! what library is being used that reads that configuration?

  
  
Posted 2 years ago

happy to help document, can you provide where I should do that?

  
  
Posted 2 years ago

I can but that is not a configuration we would want to run with in production

  
  
Posted 2 years ago

can we also put the path to the CA?

  
  
Posted 2 years ago

k, re-running job now

  
  
Posted 2 years ago

going to try it again with a path to ca instead of false

  
  
Posted 2 years ago

and path works as well, thank you so much!

  
  
Posted 2 years ago

Thanks!
In the conf file, I guess this will be where ppl will look for it.

  
  
Posted 2 years ago

(Venv mode makes sense if running inside a container, if you need docker support you will need to mount the docker socket inside)
What is exactly the error you re getting from clearml? And what do you have in the configuration file?

  
  
Posted 2 years ago

can we also put the path to the CA?

Yes :)

  
  
Posted 2 years ago

I can but that is not a configuration we would want to run with in production
Agreed, I just want to isolate the issue. I think this is the bottom python interface missing some configuration or environment variables

  
  
Posted 2 years ago

hey, that worked! what library is being used that reads that configuration?

It's passed to boto3, but the pyhon interface and aws cli use different configuration, I guess, because otherwise it should have worked...

  
  
Posted 2 years ago
647 Views
20 Answers
2 years ago
one year ago
Tags