Hi @<1817731756720132096:profile|WickedWhale51> , are you using the Logger
module from the SDK?
Answered
Hey Team!
I Am Using
Hey team!
I am using clearml-serving
to serve a custom pipeline and I was testing the offline scenario when the internet goes off. Everything works well in terms of uploading the data once the internet is back, but I have an issue with logging. My logger gets quite spammed with the retries to reconnect - any ideas how I could suppress them on the ClearML side?
Logs:
Retrying (Retry(total=237, connect=237, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c6c748eafb0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /v2.23/events.add_batch
Retrying (Retry(total=236, connect=236, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c6d2c6c63b0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /v2.23/tasks.get_all
Retrying (Retry(total=236, connect=236, read=240, redirect=240, status=240)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c6c748d41f0>, 'Connection to XXX(host IP) timed out. (connect timeout=10.0)')': /v2.23/events.add_batch
Retrying (Retry(total=235, connect=235, read=240, redirect=240, status=240)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c6d2c6c7e80>, 'Connection to XXX(host IP) timed out. (connect timeout=10.0)')': /v2.23/tasks.get_all
I have tried the following code but still didn't help with these logs:
self._task = Task.init(project_name=project_name,
task_name=task_name,
task_type=task_type,
auto_connect_streams=False)
Thanks for the help! 🙌
445 Views
2
Answers
4 months ago
4 months ago
Tags
Similar posts