2022-06-03 02:09:52 Jun 02 19:09:52 - Retrying (Retry(total=143, connect=143, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5cb1263940>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /v2.13/tasks.ping 2022-06-03 02:09:54 Jun 02 19:09:54 - Retrying (Retry(total=143, connect=143, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5ca95efac0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /v2.13/events.add_batch 2022-06-03 02:09:57 Jun 02 19:09:57 - Retrying (Retry(total=143, connect=143, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5ca95ef310>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /v2.13/events.add_batch 2022-06-03 02:09:58 2022-06-02 19:09:58,321 - clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###these were the latest ones.
yes it's manual one it was aborted by itself 1.1.2
ContemplativeGoat37
http://1.it seems the DNS resolving to the server fails? (Temporary failure in name resolution) Is this running on an agent, or manually ? "clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###" Is this you manually aborting the Task or is it aborting itslef due to the connectivity ?
4. what's the clearml/clearml-agent versions ?
ContemplativeGoat37 what are the exact errors you're getting?
ContemplativeGoat37 hi, any updates? I have a similar issue due executing clearml-data create command, also the status is stuck in “uploading”
And when I’m trying to add a file to dataset, this happens:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f20d7231430>: Failed to establish a new connection: [Errno 111] Connection refused')': / Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f20d7231670>: Failed to establish a new connection: [Errno 111] Connection refused')': /
ContemplativeGoat37 I think there was an issues just lije you described and it was solved in later versions, upgrade to the latest clearml package version, you should be fine 🙂
1.1.2. the new one I don’t remember, you can check the version that was in the last one in pypi 3 months ago