Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We’Re Randomly Getting The Following Message -

We’re randomly getting the following message -

2023-03-07 11:07:26,123 [WARNING] [urllib3.connectionpool]: Retrying (Retry(total=237, connect=237, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x14e80c430>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /v2.20/projects.get_all

We’re running ClearML on K8s. Not sure how to debug this. It’s not consistent unfortunately.
Any ideas?

  
  
Posted one year ago
Votes Newest

Answers 5


But this is not a matter of the server itself being configured or not (I'm sure it is), but something in the SDK changing the SDK's configuration regarding the server's address

  
  
Posted one year ago

Ah. Apparently getting a task ID while it’s running can cause this behaviour 🤔

  
  
Posted one year ago

That's what I thought @<1523701087100473344:profile|SuccessfulKoala55> , but the server URL is correct (and WebUI is functional and responsive).
In part of our code, we look for projects with a given name, and pull all tasks in that project. That's the crash point, and it seems to be related to having running tasks in that project.

  
  
Posted one year ago

But there's nothing of that sort happening. The process where it's failing is on getting tasks for a project.

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73> when do you mean exactly? This error indicates a mis-configured server URL, I can't imagine how this would be sporadic...

  
  
Posted one year ago