ItchyJellyfish73 Have you looked at the https://clear.ml/docs/latest/docs/clearml_agent#dynamic-gpu-allocation ?

Thanks, this helps.

ItchyJellyfish73
Unfortunately this needs backend support, and only available in the enterprise version, what is your use case for it? (It was designed to allow out of the box bare-metal multi gpu dynamic allocation, think DGX with 8 GPUs that instead of spinning down agents when you want to change the queue->num-gpu mapping you can do it on the fly)

Thanks!

Then the dynamic gpu allocation is exactly what you need, I suggest talking to the sales ppl, I'm sure they can help. https://clear.ml/contact-us/

Yeah this is my case. But we have multiple machines with different number of gpus (from 1 to 8 )