Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We’Re Randomly Getting The Following Message -

We’re randomly getting the following message -

2023-03-07 11:07:26,123 [WARNING] [urllib3.connectionpool]: Retrying (Retry(total=237, connect=237, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x14e80c430>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /v2.20/projects.get_all

We’re running ClearML on K8s. Not sure how to debug this. It’s not consistent unfortunately.
Any ideas?

  
  
Posted 11 months ago
Votes Newest

Answers 5


That's what I thought @<1523701087100473344:profile|SuccessfulKoala55> , but the server URL is correct (and WebUI is functional and responsive).
In part of our code, we look for projects with a given name, and pull all tasks in that project. That's the crash point, and it seems to be related to having running tasks in that project.

  
  
Posted 11 months ago

But this is not a matter of the server itself being configured or not (I'm sure it is), but something in the SDK changing the SDK's configuration regarding the server's address

  
  
Posted 11 months ago

@<1523701083040387072:profile|UnevenDolphin73> when do you mean exactly? This error indicates a mis-configured server URL, I can't imagine how this would be sporadic...

  
  
Posted 11 months ago

Ah. Apparently getting a task ID while it’s running can cause this behaviour 🤔

  
  
Posted 11 months ago

But there's nothing of that sort happening. The process where it's failing is on getting tasks for a project.

  
  
Posted 11 months ago
490 Views
5 Answers
11 months ago
11 months ago
Tags