I created an issue: None

Just to make sure, did the logging to the clearml server work previously and stoped working at some point?

It was working for me. Anyway I modified the callback. Attached is the script that has the issue for me whenever I add random_image_logger to the callbacks It only logs some of the scalars for 1 epoch. It then is stuck and never recovers. When I remove random_image_logger the scalars are correctly logged. Again this only on 1 computer, other computers we have logging work perfectly fine

We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4

When the script is hung at the end the experiment says failed in ClearML

So even if you abort it on the start of the experiment it will keep running and reporting logs?

Can you share any of the logs?

@<1719524641879363584:profile|ThankfulClams64> , if you set auto_connect_streams to false nothing will be reported from your frameworks. With what frameworks are you working, tensorboard?

@<1719524641879363584:profile|ThankfulClams64> , can you provide a small code snippet that reproduces this behaviour? Can you also test with the latest version of clearml ?

Console logs

I am on 1.16.2

    task = Task.init(project_name=model_config['ClearML']['project_name'],
                     task_name=model_config['ClearML']['task_name'],
                     continue_last_task=False,
                     auto_connect_streams=True)

Hi @<1719524641879363584:profile|ThankfulClams64> ,the logging is done by a separate process, I'm pretty sure it's not terminating all of the sudden. Did you manage to get a full log of such an experiment to share?

If you remove any reference of ClearML from the code on that machine, does it still hang?

I found that setting store_uncommitted_code_diff: false instead of true seems to fix the issue

When I try to abort an experiment. I get this in the log

clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###

but it does not stop anything it just continues to run

Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.

So I am only seeing values for the first epoch. It seems like it does not track all of them so maybe something is happening when it tries to log scalars.
I have seen it only log iterations but setting task.set_initial_iteration(0) seemed to fix that so it now seems to be logging the correct epoch
Tensorboard is correct and works. I have never seen an issue in the tensorboard logs

Not sure why that is related to saving images

It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?

Correct, so I get something like this

ClearML Task: created new task id=6ec57dcb007545aebc4ec51eb5b34c67
======> WARNING! Git diff too large to store (2536kb), skipping uncommitted changes <======
ClearML results page: 

but that is all

Does any exit code appear? What is the status message and status reason in the 'INFO' section?

I'll update my clearml version. Unfortunately I do not have a small code snippet and it is not always repeatable. Is there some additional logging that can be turned on?

Console output and also what you get on the ClearML task page under the console section

Any chance you have some uncommited code changes that, when not included, this works fine?

That makes sense... If you turn auto_connect_streams to false this mean that auto reporting will be disabled as per the documentation.. If you turn it to True then logging should resume.

Yea I am fine not having the console logging. My issues is the scalers and debug images occasionally don't record to ClearML

Hi @<1719524641879363584:profile|ThankfulClams64>

I am using ClearML Pro and pretty regularly I will restart an experiment and nothing will get logged to ClearML.
I use ClearML with pytorch 1.7.1, pytorch-lightning 1.2.2 and Tensorboard auto
All ClearML has the latest stable updates. (clearml 1.7.4, clearml-agent 1.7.2)

Is this still happening with the latest clearml ( clearml==1.16.3rc2 ) ?
What is the TB version?
I remember a fix regrading lightining support
Also just making sure, are you using the default lightning TB logger ?
How are you initializing the Task.init (i.e. could you copy here the code?)

Is this just the console output while training?

Okay I will do another run to capture the console output. We currently set auto_connect_streams to False to reduce the number of API calls. So there isn't really anything in the ClearML task page console section

My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:

auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.