Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
We Have An Nginx Container In Front Of Clearml, Functions Use Sort Of " Ssl Reverse Proxy" We Are Seeing This Errors With Jobs That Are Trying To Upload Some Artifacts To Fileserver

we have an nginx container in front of ClearML, functions use sort of " SSL reverse proxy"
we are seeing this errors with jobs that are trying to upload some artifacts to fileserver

2024-01-10 08:10:14,537 - clearml.metrics - WARNING - Failed uploading to 
 (HTTPSConnectionPool(host='clearml-hrl-03.haifa.ibm.com', port=8081): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))))

clearml.conf contains api.verify_certificate : false
it happen only when a job tries to upload something to fileserver, saving metrics/scalrs/plots working without issues

anyone tackled this?

Posted 6 months ago
Votes Newest

Answers 6

clearml.conf contains

api.verify_certificate : false

@<1523701842515595264:profile|PleasantOwl46> this is in the user's configuration file, right?

Posted 6 months ago

@<1523701087100473344:profile|SuccessfulKoala55> seen this with 1.13.2
worth to try and upgrade to latest?

Posted 6 months ago

just confirming this with the user and will share it over here
I do recall in the past that latest version caused this, and downgrading to some prior version fixed the issue

let me get the info and will post back here
10x @<1523701087100473344:profile|SuccessfulKoala55>

Posted 6 months ago

What is the clearml package version used by the job?

Posted 6 months ago

yep, again most jobs works .. the issue with when a job tries to upload artifacts to fileserver

Posted 6 months ago

might be

Posted 6 months ago