Hmm can you test with the latest RC?pip install clearml==0.17.6rc1
I'm running in the command line with Hydra and TFv1
If this is the case, then we do not change the maptplotlib backend
Also
I've attempted converting the
mpl
image to
PIL
and use
report_image
to push the image, to no avail.
What are you getting? error / exception ?
AgitatedDove14 , after some more pronging, the error seem to stem from the clearML server. The upload from client side does not seem to occur, or the server is not registering the uploads.
I've attempted to restart the server and pull the latest image, same error.
Hi TrickyRaccoon92
Yes please update me once you can, I would love to be able to reproduce the issue so we could fix for the next RC 🙂
I should mention this is run within a TF v1 session context
This should not be connected.
everything gets stored as intended (to clearML dashboard)
So in jupyter it works? But from command line it does not ? what's the difference ?
I am able to register that the image exists, however, the push towards the clearML server just does not happen
I'll see if I can get time to do a reproducible example.
The issue only arises upon sending Images. (Both numpy, mpl and PIL)
I'm manually executing on a local machine
Got it to work now. Suspect something to do with a hydra issue, as both clearML hosted nor self-hosted option presented the same issue. Will update as I get a chance.
I might add, this error only started to occur upon upgrading from trains to clearmL
Scalars plot fine, images do not find their way to the dashboard
t seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:
2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads
And there's no sign of updates on the dashboard
Hmm that could point to an issue uploading the last images (which are larger than regular scalars) could you try flushing and waiting ?
i.e.task.flush() sleep(45)
And another detail, upon running the same code in a notebook session, everything gets stored as intended (to clearML dashboard)
The issue only arises upon sending Images. (Both numpy, mpl and PIL)
BTW: they should appear under debug-samples Tab in the results
Likely related to: https://github.com/allegroai/clearml/issues/321 - clearML waits indefinitely on logs from TF v1
Hi TrickyRaccoon92
TKinter
is suddenly used as backend, and instead of writes to the dashboard I get popups per figure.
Are you running with an agent of manually executing the code ?
Yes, I am aware of this. None of the values are being reported, neither the scalar, images in debug-samples, nor in the plots tab.
I see, no i do not get anything. I either get pop-ups when trying to capture the mpl figure, and otherwise nothing happens when using report_image .
I should mention this is run within a TF v1 session context
It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:
2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads
And there's no sign of updates on the dashboard.