Thanks CostlyOstrich36 I was thinking more to a setting of the environment, for example the documentation mentions the "--cpu-only" flag (which I am not sure I can use as I am using the helm charts from AllegroAI, I don't think I can override the command), or to set the env var NVIDIA_VISIBLE_DEVICES to an empty string (which I did, but I can still see the message)
Hi folks, I think I found the issue, the documentation mention to set NVIDIA_VISIBLE_DEVICES to "", when in reality it should be "none" according to the code:
if Session.get_nvidia_visible_env() == 'none': # NVIDIA_VISIBLE_DEVICES set to none, marks cpu_only flag # active_gpus == False means no GPU reporting self._active_gpus = False
Hi SarcasticSquirrel56 ,
In Task.init() you have the parameter auto_resource_monitoring
https://clear.ml/docs/latest/docs/references/sdk/task#taskinit
You can specify there what to turn off
As much as possible, I'd like removing the burden off the shoulders of people writing their models
Hi CostlyOstrich36 , the document mention ResourceMonitor class, but when I check the code at https://github.com/allegroai/clearml/blob/master/clearml/utilities/resource_monitor.py
I don't see gpu monitoring can be turned off from the class interface
