Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Encountered An Odd Bug. Upon Attempting To Write Images To Clearml (3D Projected, Matplotlib),

Encountered an odd bug. Upon attempting to write images to clearML (3D projected, matplotlib), TKinter is suddenly used as backend, and instead of writes to the dashboard I get popups per figure. I've attempted converting the mpl image to PIL and use report_image to push the image, to no avail. Have anyone experienced anything like this before?

  
  
Posted 3 years ago
Votes Newest

Answers 21


I'll see if I can get time to do a reproducible example.

The issue only arises upon sending Images. (Both numpy, mpl and PIL)

  
  
Posted 3 years ago

Hmm can you test with the latest RC?
pip install clearml==0.17.6rc1

  
  
Posted 3 years ago

Still having the same issue I'm afraid

  
  
Posted 3 years ago

Likely related to: https://github.com/allegroai/clearml/issues/321 - clearML waits indefinitely on logs from TF v1

  
  
Posted 3 years ago

It seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard.

  
  
Posted 3 years ago

The issue only arises upon sending Images. (Both numpy, mpl and PIL)

BTW: they should appear under debug-samples Tab in the results

  
  
Posted 3 years ago

I'm manually executing on a local machine

  
  
Posted 3 years ago

Hi TrickyRaccoon92

TKinter

 is suddenly used as backend, and instead of writes to the dashboard I get popups per figure.

Are you running with an agent of manually executing the code ?

  
  
Posted 3 years ago

And another detail, upon running the same code in a notebook session, everything gets stored as intended (to clearML dashboard)

  
  
Posted 3 years ago

If this is the case, then we do not change the maptplotlib backend
Also

I've attempted converting the 

mpl

 image to 

PIL

 and use 

report_image

  to push the image, to no avail.

What are you getting? error / exception ?

  
  
Posted 3 years ago

I see, no i do not get anything. I either get pop-ups when trying to capture the mpl figure, and otherwise nothing happens when using report_image .

I should mention this is run within a TF v1 session context

  
  
Posted 3 years ago

Yes, I am aware of this. None of the values are being reported, neither the scalar, images in debug-samples, nor in the plots tab.

  
  
Posted 3 years ago

I am able to register that the image exists, however, the push towards the clearML server just does not happen

  
  
Posted 3 years ago

t seems there is some async behavior going on. After ending a run, this prompt just hangs for a long time:

2021-04-18 22:55:06,467 - clearml.Task - INFO - Waiting to finish uploads

And there's no sign of updates on the dashboard

Hmm that could point to an issue uploading the last images (which are larger than regular scalars) could you try flushing and waiting ?
i.e.
task.flush() sleep(45)

  
  
Posted 3 years ago

Scalars plot fine, images do not find their way to the dashboard

  
  
Posted 3 years ago

AgitatedDove14 , after some more pronging, the error seem to stem from the clearML server. The upload from client side does not seem to occur, or the server is not registering the uploads.

I've attempted to restart the server and pull the latest image, same error.

  
  
Posted 3 years ago

Got it to work now. Suspect something to do with a hydra issue, as both clearML hosted nor self-hosted option presented the same issue. Will update as I get a chance.

  
  
Posted 3 years ago

I might add, this error only started to occur upon upgrading from trains to clearmL

  
  
Posted 3 years ago

Hi TrickyRaccoon92
Yes please update me once you can, I would love to be able to reproduce the issue so we could fix for the next RC 🙂

  
  
Posted 3 years ago

I should mention this is run within a TF v1 session context

This should not be connected.

everything gets stored as intended (to clearML dashboard)

So in jupyter it works? But from command line it does not ? what's the difference ?

  
  
Posted 3 years ago

I'm running in the command line with Hydra and TFv1

  
  
Posted 3 years ago
868 Views
21 Answers
3 years ago
one year ago
Tags