Unanswered
Hi Everyone,
I Got Several Error Messages Like This When Trying To Upload Data On The Clearml Which We Self-Hosted. Do You Have Any Ideas?
Looks like you're having some connectivity to the files server
2024-11-14 07:05:30,888 - clearml.storage - INFO - Uploading: 5.00MB / 12.82MB @ 35.88MBs from /tmp/state.vykhyxpt.json
2024-11-14 07:05:31,111 - clearml.storage - INFO - Uploading: 10.00MB / 12.82MB @ 22.36MBs from /tmp/state.vykhyxpt.json
1731567938707 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aaaf310>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aaaf5b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aa8aa60>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76aa8aca0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567942714 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb5460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb5490>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb3310>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff763eb35e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567943959 labserver info Uploading dataset changes (30545 files compressed to 13.07 MiB) to
1731567944758 labserver info 2024-11-14 07:05:44,758 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 44.04MBs from /tmp/state.gkewkbrp.json
2024-11-14 07:05:44,805 - clearml.storage - INFO - Uploading: 5.00MB / 13.07MB @ 28.98MBs from /tmp/dataset.4e49382b63ef4e4aa5189a3e19dc4f03.z5prqb9n.zip
2024-11-14 07:05:45,218 - clearml.storage - INFO - Uploading: 10.00MB / 13.07MB @ 12.10MBs from /tmp/dataset.4e49382b63ef4e4aa5189a3e19dc4f03.z5prqb9n.zip
File compression and upload completed: total size 13.07 MiB, 1 chunk(s) stored (average size 13.07 MiB)
1731567945994 labserver info 2024-11-14 07:05:45,994 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 64.37MBs from /tmp/state.11sha1i0.json
1731567947091 labserver info 2024-11-14 07:05:47,090 - clearml.storage - INFO - Uploading: 5.00MB / 8.33MB @ 65.74MBs from /tmp/state.fzbvb2c2.json
Updating statistics and genealogy
1731567950720 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77167c700>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77167c9d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77166a970>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77166a460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567953731 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff77264a460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7070>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7520>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7fd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567957737 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7940>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764fe7ee0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8460>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567965748 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8640>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8e50>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff764ff8550>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff76b0dc3d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
1731567968761 labserver error WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff762e756d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7dffc8dc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7e6df0250>: Failed to establish a new connection: [Errno 111] Connection refused')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7dfe99610>: Failed to establish a new connection: [Errno 111] Connection refused')': /
I'd suggest checking that
17 Views
0
Answers
one month ago
one month ago