So even if you abort it on the start of the experiment it will keep running and reporting logs?

I'm not sure if it still reports logs. But it will continue running on the machine

I found that setting store_uncommitted_code_diff: false instead of true seems to fix the issue

Not sure why that is related to saving images

Any chance you have some uncommited code changes that, when not included, this works fine?

I do have uncommitted code changes. I can try to check at some point if it would not have the problem without them. It seems like it could be repeated just by making a git repo with that script and adding a very large file. If I can repeat it is it best to open an issue in GitHub?

Running clearml_example.py in None reproduces the issue

I created an issue: None

Thank you @<1719524641879363584:profile|ThankfulClams64> for opening the GI, hopefully we will be able to reproduce it and fox ot quickly