Yes it shows on the UI and has the first epoch for some of the metrics but that's it. It has run like 50 epochs, it says it is still running but there are no updates to the scalars or debug samples

Can you share any of the logs?

We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4

I'll update my clearml version. Unfortunately I do not have a small code snippet and it is not always repeatable. Is there some additional logging that can be turned on?

Thank you @<1719524641879363584:profile|ThankfulClams64> for opening the GI, hopefully we will be able to reproduce it and fox ot quickly

Just to make sure, did the logging to the clearml server work previously and stoped working at some point?

Can you try with auto_connect_streams=True ? Also, what version of clearml sdk are you using?

Not sure if this is helpful but this is what I get when I cntrl-c out of the hung script

^C^CException ignored in atexit callback: <bound method Reporter._handle_program_exit of <clearml.backend_interface.metrics.reporter.Reporter object at 0x70fd8b7ff1c0>>
Event reporting sub-process lost, switching to thread based reporting
Traceback (most recent call last):
  File "/home/richard/.virtualenvs/temp_clearml/lib/python3.10/site-packages/clearml/backend_interface/metrics/reporter.py", line 317, in _handle_program_exit
    self.wait_for_events()
  File "/home/richard/.virtualenvs/temp_clearml/lib/python3.10/site-packages/clearml/backend_interface/metrics/reporter.py", line 337, in wait_for_events
    return report_service.wait_for_events(timeout=timeout)
  File "/home/richard/.virtualenvs/temp_clearml/lib/python3.10/site-packages/clearml/backend_interface/metrics/reporter.py", line 129, in wait_for_events
    if self._empty_state_event.wait(timeout=1.0):
  File "/home/richard/.virtualenvs/temp_clearml/lib/python3.10/site-packages/clearml/utilities/process/mp.py", line 445, in wait
    return self._event.wait(timeout=timeout)
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 349, in wait
    self._cond.wait(timeout)
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 261, in wait
    return self._wait_semaphore.acquire(True, timeout)
KeyboardInterrupt: 

My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:

auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.

Can someone help with this?

Is this just the console output while training?

@<1719524641879363584:profile|ThankfulClams64> you could try using the compare function in the UI to compare the experiments on the machine the scalars are not reported properly and the experiments on a machine that runs the experiments properly. I suggest then replicating the environment exactly on the problematic machine. None

Correct, so I get something like this

ClearML Task: created new task id=6ec57dcb007545aebc4ec51eb5b34c67
======> WARNING! Git diff too large to store (2536kb), skipping uncommitted changes <======
ClearML results page: 

but that is all

@<1719524641879363584:profile|ThankfulClams64> , if you set auto_connect_streams to false nothing will be reported from your frameworks. With what frameworks are you working, tensorboard?

@<1719524641879363584:profile|ThankfulClams64> , can you provide a small code snippet that reproduces this behaviour? Can you also test with the latest version of clearml ?

I found that setting store_uncommitted_code_diff: false instead of true seems to fix the issue

Console logs

Running clearml_example.py in None reproduces the issue

What happens if you're running the reporting example from the ClearML github repository?

I will try with clearml==1.16.3rc2 and see if it still has the issue

I created an issue: None

Yes I see it in the terminal on the machine

No it completes and exists the script

@<1719524641879363584:profile|ThankfulClams64> , are logs showing up without issue on the 'problematic' machine?

Hi we are currently having the issue. There is nothing in the console regarding ClearML besides

ClearML Task: created new task id=0174d5b9d7164f47bd10484fd268e3ff
======> WARNING! Git diff too large to store (3611kb), skipping uncommitted changes <======
ClearML results page: 

The console logs continue to come in put no scalers or debug images show up.

I do have uncommitted code changes. I can try to check at some point if it would not have the problem without them. It seems like it could be repeated just by making a git repo with that script and adding a very large file. If I can repeat it is it best to open an issue in GitHub?

Yea I am fine not having the console logging. My issues is the scalers and debug images occasionally don't record to ClearML

Hi @<1719524641879363584:profile|ThankfulClams64> , stopping all processes should do that, there is no programmatic way of doing that specifically. Did you try calling task.close() for all tasks you're using?

It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?

I'm not sure how to even troubleshoot this.