Hi TrickyRaccoon92
Yes please update me once you can, I would love to be able to reproduce the issue so we could fix for the next RC 🙂

Got it to work now. Suspect something to do with a hydra issue, as both clearML hosted nor self-hosted option presented the same issue. Will update as I get a chance.

I might add, this error only started to occur upon upgrading from trains to clearmL

AgitatedDove14 , after some more pronging, the error seem to stem from the clearML server. The upload from client side does not seem to occur, or the server is not registering the uploads.

I've attempted to restart the server and pull the latest image, same error.

t seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard

Hmm that could point to an issue uploading the last images (which are larger than regular scalars) could you try flushing and waiting ?
i.e.
task.flush() sleep(45)

It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard.

Scalars plot fine, images do not find their way to the dashboard

Still having the same issue I'm afraid

Hmm can you test with the latest RC?
pip install clearml==0.17.6rc1

I'm running in the command line with Hydra and TFv1

I am able to register that the image exists, however, the push towards the clearML server just does not happen

Yes, I am aware of this. None of the values are being reported, neither the scalar, images in debug-samples, nor in the plots tab.

The issue only arises upon sending Images. (Both numpy, mpl and PIL)

BTW: they should appear under debug-samples Tab in the results

I should mention this is run within a TF v1 session context

This should not be connected.

everything gets stored as intended (to clearML dashboard)

So in jupyter it works? But from command line it does not ? what's the difference ?

I'll see if I can get time to do a reproducible example.

The issue only arises upon sending Images. (Both numpy, mpl and PIL)

And another detail, upon running the same code in a notebook session, everything gets stored as intended (to clearML dashboard)

I see, no i do not get anything. I either get pop-ups when trying to capture the mpl figure, and otherwise nothing happens when using report_image .

I should mention this is run within a TF v1 session context

If this is the case, then we do not change the maptplotlib backend
Also

I've attempted converting the 

mpl

 image to 

PIL

 and use 

report_image

  to push the image, to no avail.

What are you getting? error / exception ?

I'm manually executing on a local machine

Hi TrickyRaccoon92

TKinter

 is suddenly used as backend, and instead of writes to the dashboard I get popups per figure.

Are you running with an agent of manually executing the code ?

Likely related to: https://github.com/allegroai/clearml/issues/321 - clearML waits indefinitely on logs from TF v1