If you remove any reference of ClearML from the code on that machine, does it still hang?

Hi @<1719524641879363584:profile|ThankfulClams64> , does the experiment itself show on the ClearML UI?

Can you try with auto_connect_streams=True ? Also, what version of clearml sdk are you using?

Yea I am fine not having the console logging. My issues is the scalers and debug images occasionally don't record to ClearML

Can someone help with this?

@<1719524641879363584:profile|ThankfulClams64> , can you provide a small code snippet that reproduces this behaviour? Can you also test with the latest version of clearml ?

Hi we are currently having the issue. There is nothing in the console regarding ClearML besides

ClearML Task: created new task id=0174d5b9d7164f47bd10484fd268e3ff
======> WARNING! Git diff too large to store (3611kb), skipping uncommitted changes <======
ClearML results page: 

The console logs continue to come in put no scalers or debug images show up.

Okay I will do another run to capture the console output. We currently set auto_connect_streams to False to reduce the number of API calls. So there isn't really anything in the ClearML task page console section

It seems similar to this None is it possible saving too many model weights causes metric logging thread to die?

The console logging still works. Aborting the task was in the log but did not work and the process continued until I killed it.

Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.

Just to make sure, did the logging to the clearml server work previously and stoped working at some point?

So even if you abort it on the start of the experiment it will keep running and reporting logs?

What happens if you're running the reporting example from the ClearML github repository?

I just created a new virtual environment and the problem persists. There are only two dependencies clearml and tensorflow. @<1523701070390366208:profile|CostlyOstrich36> what logs are you referring to?

I created an issue: None

@<1719524641879363584:profile|ThankfulClams64> , if you set auto_connect_streams to false nothing will be reported from your frameworks. With what frameworks are you working, tensorboard?

Hi @<1719524641879363584:profile|ThankfulClams64> , stopping all processes should do that, there is no programmatic way of doing that specifically. Did you try calling task.close() for all tasks you're using?

Then we also connect two dictionaries for configs

    task.connect(model_config)
    task.connect(DataAugConfig)

When I try to abort an experiment. I get this in the log

clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ###

but it does not stop anything it just continues to run

This was on the same machine I am having issues with it logs scalars correctly using the example code, but when I add in that callback which just logs a random image to tensorboard I don't get any scalars logged

I'm not sure how to even troubleshoot this.

I'm not sure if it still reports logs. But it will continue running on the machine

Yes I see it in the terminal on the machine

Console logs

Thank you @<1719524641879363584:profile|ThankfulClams64> for opening the GI, hopefully we will be able to reproduce it and fox ot quickly

We are running the same code on multiple machines and it just randomly happens. Currently we are having the issue on 1 out of 4

Do you also see the same in the terminal itself on the machine?

My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:

auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.

Yes it shows on the UI and has the first epoch for some of the metrics but that's it. It has run like 50 epochs, it says it is still running but there are no updates to the scalars or debug samples