My goal is to detect events when task does not uses allocated resources (e.g. GPU) for some period of time.
I am still trying to understand clearml api response.

Do you have any clue how can I get it from client.tasks.get_all(status=["in_progress"]) ?
If task has GPU allocated but not using it, would it be in in_progress status also? I want to collect those task.

I see task runtime info. I guess it's current utilization not allocation but not sure.

"runtime": {
"progress": "0",
"platform": "linux",
"python_version": "3.8.0",
"python_exec": "/root/.clearml/venvs-builds/3.8/bin/python",
"OS": "Linux-5.15.0-1013-gcp-x86_64-with-glibc2.27",
"processor": "x86_64",
"cpu_cores": 256,
"memory_gb": 1007.7,
"hostname": "",
"gpu_count": 1,
"gpu_type": "NVIDIA xxx -40GB",