Thanks ThankfulClams64 having a code that can reproduce it is exactly what we need.
One thing I might have missed and is very important , what is your tensorboard package version?

Yes it is logging to the console. The script does hang whenever it completes all the epochs when it is having the issue.

Then we also connect two dictionaries for configs

    task.connect(model_config)
    task.connect(DataAugConfig)

Console logs

sometimes I get no scalars, but the console logging always seems to be working

I just created a new virtual environment and the problem persists. There are only two dependencies clearml and tensorflow. CostlyOstrich36 what logs are you referring to?

ThankfulClams64 , are logs showing up without issue on the 'problematic' machine?

I do have uncommitted code changes. I can try to check at some point if it would not have the problem without them. It seems like it could be repeated just by making a git repo with that script and adding a very large file. If I can repeat it is it best to open an issue in GitHub?

Is there someway to kill all connections of a machine to the ClearML server this does seem to be related to restarting a task / running a new task quickly after a task fails or is aborted