Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Seeing This In My Logs:

Hi all, I am seeing this in my logs:

2023-02-08 15:17:25,538 - clearml - WARNING - Failed decoding debug image [553, 509, 3]
2023-02-08 15:17:25,539 - clearml - WARNING - Error: I/O operation on closed file.

I believe this relates to plots not being uploaded to the ClearML server. Has anyone got any experience fixing this? I am a ClearML Pro user.

  
  
Posted one year ago
Votes Newest

Answers 11


Has there been any update on this? Not sure to see something related in the latest release notes

  
  
Posted one year ago

Further to this, I have inspected further. This is working as expected for ClearML 1.8.3 but not for ClearML 1.9.0.

I looked at the commits and found that a change had been made to the _decode_image method:

None

This aligns with the error message I'm seeing:

2023-02-08 15:17:25,539 - clearml - WARNING - Error: I/O operation on closed file.

Can this be actioned for the next release please?
Or is it suitable for me to raise my own PR?

Cheers,
James

  
  
Posted one year ago

I am using ClearML version 1.9.1. In code, I am creating a plot using matplotlib. I am able to see this in Tensorboard but it is not available in ClearML Plots

  
  
Posted one year ago

@<1529271085315395584:profile|AmusedCat74> , what happens if you try to run it with clearml 1.8.0?

  
  
Posted one year ago

Yep, that seems to be working fine.

  
  
Posted one year ago

Can you share a snippet just to reproduce this?

  
  
Posted one year ago

Thank you 👍

  
  
Posted one year ago

Hi @<1529271085315395584:profile|AmusedCat74> , what are you trying to do in code? What version of clearml are you using?

  
  
Posted one year ago

Could you give me some insight into why this is please?

  
  
Posted one year ago

Might be a bug. Let me check with the guys

  
  
Posted one year ago

The code is quite nested by I've tried to extract out the important parts ( summmary_writer is a tensorboard logger).

self.figure, (ax1, ax2, axc) = plt.subplots(1, 3, figsize=(total_width, total_height), facecolor="white")

self.summary_writer = self.tb_logger.experiment

self.summary_writer.add_figure(Partition.TRAINING.value, train_plot.figure, global_step=self.current_epoch + 1) 

The train_plot.figure is a matplotlib figure created using seaborn.

Let me know if this isn't useful and you'd like me to create a proper example.

I am using

tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
  
  
Posted one year ago