Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Folks, Good Morning

Hi folks, good morning 🙂
In our setup we have a set of queues that do not use any GPU resources. Yet, when I run an experiment in such queues, we see a Warning:
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
Is there a way to disable GPU monitoring for an agent?

  
  
Posted 2 years ago
Votes Newest

Answers 6


Thanks CostlyOstrich36 I was thinking more to a setting of the environment, for example the documentation mentions the "--cpu-only" flag (which I am not sure I can use as I am using the helm charts from AllegroAI, I don't think I can override the command), or to set the env var NVIDIA_VISIBLE_DEVICES to an empty string (which I did, but I can still see the message)

  
  
Posted 2 years ago

Hi SarcasticSquirrel56 ,

In Task.init() you have the parameter auto_resource_monitoring
https://clear.ml/docs/latest/docs/references/sdk/task#taskinit
You can specify there what to turn off

  
  
Posted 2 years ago

As much as possible, I'd like removing the burden off the shoulders of people writing their models

  
  
Posted 2 years ago

Hi folks, I think I found the issue, the documentation mention to set NVIDIA_VISIBLE_DEVICES to "", when in reality it should be "none" according to the code:

if Session.get_nvidia_visible_env() == 'none': # NVIDIA_VISIBLE_DEVICES set to none, marks cpu_only flag # active_gpus == False means no GPU reporting self._active_gpus = False

  
  
Posted 2 years ago

that disabled gpu for me

  
  
Posted 2 years ago

Hi CostlyOstrich36 , the document mention ResourceMonitor class, but when I check the code at https://github.com/allegroai/clearml/blob/master/clearml/utilities/resource_monitor.py
I don't see gpu monitoring can be turned off from the class interface

  
  
Posted 2 years ago