Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! Trying To Run The Following Very Basic Code. The First Few Parts Works As They Should:

Hi!

Trying to run the following very basic code.
The first few parts works as they should:
connecting the configuraiton first reporting of scalars reporting the confusion matrix
However, after trying to report a matplotlib figure, the logger stops reporting at all including the :
matplotlib figure second scalars reporting second confusion matrix
If I remove the matplotlib reporting, then it is all running as should.

I run it on a GPU machine (g4dn.4xlarge) with:
Python 3.7.10 matplotlib == 3.4.2 clearml == 1.0.4
Would love to get some help here…

Thx!

` from clearml import Task
import numpy as np
import matplotlib.pyplot as plt

task = Task.init(project_name='my-project-01', task_name='my-task-01')
logger = task.get_logger()
args = task.connect({"a":1, "b":2}, name='input')

for i in range(5):
logger.report_scalar('aa', 'bb', i, i)

confusion = np.random.randint(10, size=(10, 10))
logger.report_matrix(
"example_confusion1",
"ignored1",
iteration=0,
matrix=confusion,
xaxis="title X",
yaxis="title Y",
)

THE FOLLOWING 3 LINES CAUSE THE PROBLEM

fig = plt.figure()
img = np.random.randint(0, 255, size=(100, 100, 3))
plt.imshow(img)

removing the last 3 lines will make the following lines to run and report

logger.report_matplotlib_figure('a', 'b', fig)

for i in range(5, 10):
logger.report_scalar('aa', 'bb', i, i)

confusion = np.random.randint(10, size=(10, 10))
logger.report_matrix(
"example_confusion2",
"ignored2",
iteration=0,
matrix=confusion,
xaxis="title X",
yaxis="title Y",
) `

  
  
Posted 2 years ago
Votes Newest

Answers 14


Hi FunnyTurkey96
Any chance you can try to run with the latest form GitHub (i just tested your code and it seemed to work on my machine).
pip install git+

  
  
Posted 2 years ago

I cannot reproduce, tested with the same matplotlib version and python against the community server

  
  
Posted 2 years ago

Well taking the file out of my git repo folder it seems to work perfectly. May it have something to do with the git integration and the uncommitted changes?

  
  
Posted 2 years ago

It seems to be stuck in the following step:
` $ python eraseme.py

ClearML Task: created new task id=fa3db29498f241a19de81b75f787887e
ClearML results page:
======> WARNING! Git diff to large to store (1327kb), skipping uncommitted changes <======
2021-07-11 19:17:31,351 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
2021-07-11 19:17:32,821 - clearml.Task - INFO - Finished repository detection and package analysis
2021-07-11 19:17:32,822 - clearml.Task - INFO - Waiting to finish uploads `

  
  
Posted 2 years ago

seems like. I’m working with git only locally, with remote machine where the code actually runs (you know this pycharm pro remote). So for git integration I had to reproduce the git on the remote machine. and since it’s not updated regularly, the git diffs are large…

Do you have a better solution for the git integration for the way I work?

  
  
Posted 2 years ago

Hi FunnyTurkey96
what's the clearml server you are using ?

  
  
Posted 2 years ago

BTW:

======> WARNING! Git diff to large to store (1327kb), skipping uncommitted changes <======

This means all your git changes are stored as an artifact, which is consistent with the "wait for upload" message.

  
  
Posted 2 years ago

with remote machine where the code actually runs (you know this pycharm pro remote).

Are you using the pycharm plugin ? (to sync the local git changes with clearml)
https://github.com/allegroai/clearml-pycharm-plugin

  
  
Posted 2 years ago

Thx! fixing the too-big-diff (due to a jupyter notebook) solved the issues.

  
  
Posted 2 years ago

nope. still same problem.

  
  
Posted 2 years ago

Hmm let check again something.

  
  
Posted 2 years ago

I will try recreating my env.

  
  
Posted 2 years ago

2021-07-11 19:17:32,822 - clearml.Task - INFO - Waiting to finish uploads

I'm assuming a very large uncommitted changes 🙂

  
  
Posted 2 years ago

And anyway, the git diffs are logged long before, and still the reporting doesn’t upload, and I don’t see why it should.

  
  
Posted 2 years ago
622 Views
14 Answers
2 years ago
one year ago
Tags
Similar posts