Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone And Thanks Again For The Help, I Still Have No Success In Running Clearml Agent, It Just Gets Stuck Without Any Output, On Debug Mode For

Hi everyone and thanks again for the help, i still have no success in running clearml agent, it just gets stuck without any output, on debug mode for clearml-agent -d list i get the following output:
WARNING:urllib3.connectionpool:Retrying (Retry(total=238, connect=4, read=240, redirect=240, status=240)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /auth.login DEBUG:urllib3.connectionpool:Starting new HTTPS connection (3): <here is my internal clearml server>:8008 DEBUG:urllib3.util.retry:Incremented Retry for (url='/auth.login'): Retry(total=237, connect=4, read=240, redirect=240, status=240)any idea how to solve it?

  
  
Posted 2 years ago
Votes Newest

Answers 24


Hi ChubbyLouse32 , how did you configure the ClearML Server address?

  
  
Posted 2 years ago

ChubbyLouse32 could it be the configuration file is not passed to the agent machine itself ?
(were you able to run anything against this internal server? I mean to connect to it from code, clearml/cleamrl-agent) ?

  
  
Posted 2 years ago

yes i can communicate with the server, i managed to put tasks in the queue and retrieve them as well as running tasks with metrics reporting

  
  
Posted 2 years ago

both the SDK and the Agent use clearml.conf

  
  
Posted 2 years ago

ChubbyLouse32 does server URL configured in your clearml.conf file start with http:// or https:// ?

  
  
Posted 2 years ago

And how is your clearml.conf configured?

  
  
Posted 2 years ago

thanks

  
  
Posted 2 years ago

correct. just verified again now.

  
  
Posted 2 years ago

ChubbyLouse32 and this works when running python code and not when the agent is running ?
On the same machine ?

  
  
Posted 2 years ago

what do you mean?

  
  
Posted 2 years ago

where address is our server adderss starting with https://.. etc

  
  
Posted 2 years ago

Okay found the issue, to disable SSL verification global add the following env variable:
CLEARML_API_HOST_VERIFY_CERT=0(I will make sure we fix the actual issue with the config file)

  
  
Posted 2 years ago

Let me verify something in the code,

  
  
Posted 2 years ago

Im not sure what exactly your asking, someone else configured the server, im just using it

  
  
Posted 2 years ago

So what is the difference ? both running from the same machine ?

  
  
Posted 2 years ago

This makes no sense to me 😞
Both are reading the exact same file, and using the same session / flow ...
Maybe there is an error with the "verify_certificate" on the agent ?

  
  
Posted 2 years ago

` sdk.development.store_uncommitted_code_diff: false
api.verify_certificate : false
api {

web_server: <ADDRESS>:8080
api_server: <ADDRESS>:8008
files_server: <ADDRESS>:8081
credentials {
    "access_key" = "OMF..."
    "secret_key" = "oox..."
}

} `

  
  
Posted 2 years ago

yes i can communicate with the server, i managed to put tasks in the queue and retrieve them as well as running tasks with metrics reporting

Through the UI or python code ?

  
  
Posted 2 years ago

thanks it fixed it!

  
  
Posted 2 years ago

Yey!

  
  
Posted 2 years ago

python code

  
  
Posted 2 years ago

Or, what appears in <here is my internal clearml server> ? I don't need the exact IP, of course, but is it http ? https ?

  
  
Posted 2 years ago

sdk.development.store_uncommitted_code_diff: false api.verify_certificate : false api { web_server: https://<...>.com:8080 api_server: https://<...>.com:8008 files_server: https://<...>.com:8081 credentials { "access_key" = "OMF..." "secret_key" = "oox..." } }

  
  
Posted 2 years ago

i dont have agent configuration file if this might be the problem
all the machines share the same file system so i managed to do all the things i mentioned from different machines on the system

  
  
Posted 2 years ago