Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I Got Several Error Messages Like This When Trying To Upload Data On The Clearml Which We Self-Hosted. Do You Have Any Ideas?

Hi everyone,
I got several error messages like this when trying to upload data on the ClearML which we self-hosted. Do you have any ideas?

WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f05b3d0a820>: Failed to establish a new connection: [Errno 111] Connection refused')': /
  
  
Posted one month ago
Votes Newest

Answers 7


@<1523701070390366208:profile|CostlyOstrich36> Thanks for quickly response.

Are you trying to upload data to the files server

Yes, I 'm trying to upload to the files server using

The code I'm using:

ds = Dataset.create(
            dataset_name="xxxx",
            dataset_project=f"A/B/C",
            dataset_tags=[task_tag],
        )
        ds.add_files(path=ng_path)
        ds.upload()
        ds.finalize(verbose=True)

The full log is the attachment.

  
  
Posted one month ago

Hi @<1638712150060961792:profile|SilkyCrocodile89> , it looks like a connectivity issue. Are you trying to upload data to the files server? Can you share the full log?

  
  
Posted one month ago

you mean too many connection to the file server then I could not connect more?

  
  
Posted one month ago

Not sure, I would ask the DevOps team to check

  
  
Posted one month ago

But you can see in the log that it manages to connect for a bit and then its interrupted

  
  
Posted one month ago

Looks like you're having some connectivity to the files server


2024-11-14 07:05:30,888 - clearml.storage - INFO - Uploading: 5.00MB / 12.82MB @ 35.88MBs from /tmp/state.vykhyxpt.json
2024-11-14 07:05:31,111 - clearml.storage - INFO - Uploading: 10.00MB / 12.82MB @ 22.36MBs from /tmp/state.vykhyxpt.json
1731567938707 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aaaf310>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aaaf5b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aa8aa60>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aa8aca0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567942714 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb5460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb5490>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb3310>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb35e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567943959 labserver info Uploading dataset changes (30545 files compressed to 13.07 MiB) to 

1731567944758 labserver info 2024-11-14 07:05:44,758 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 44.04MBs from /tmp/state.gkewkbrp.json
2024-11-14 07:05:44,805 - clearml.storage - INFO - Uploading: 5.00MB / 13.07MB @ 28.98MBs from /tmp/dataset.4e49382b63ef4e4aa5189a3e19dc4f03.z5prqb9n.zip
2024-11-14 07:05:45,218 - clearml.storage - INFO - Uploading: 10.00MB / 13.07MB @ 12.10MBs from /tmp/dataset.4e49382b63ef4e4aa5189a3e19dc4f03.z5prqb9n.zip
File compression and upload completed: total size 13.07 MiB, 1 chunk(s) stored (average size 13.07 MiB)
1731567945994 labserver info 2024-11-14 07:05:45,994 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 64.37MBs from /tmp/state.11sha1i0.json
1731567947091 labserver info 2024-11-14 07:05:47,090 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 65.74MBs from /tmp/state.fzbvb2c2.json
Updating statistics and genealogy
1731567950720 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77167c700>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77167c9d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77166a970>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77166a460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567953731 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77264a460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7070>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7520>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7fd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567957737 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7940>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7ee0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567965748 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8640>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8e50>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8550>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76b0dc3d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567968761 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff762e756d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7dffc8dc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7e6df0250>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7dfe99610>: Failed to establish a new connection: [Errno 111] Connection refused')': /

I'd suggest checking that

  
  
Posted one month ago

yes, that's exact what I'm concerned. Maybe it's the problem, because we're sharing the same CLearml for several projects and we log many things in the console, Clearml captures all.

  
  
Posted one month ago
132 Views
7 Answers
one month ago
one month ago
Tags