Running clearml_example.py in None reproduces the issue
My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:
auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.
@<1719524641879363584:profile|ThankfulClams64> , can you provide a small code snippet that reproduces this behaviour? Can you also test with the latest version of clearml
?
@<1719524641879363584:profile|ThankfulClams64> , are logs showing up without issue on the 'problematic' machine?
It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?
We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4