Hi all, is there a way to schedule the tasks from the queue onto the GPU instances based on factors such as GPU utilisation, number of CPU cores present, fre...
3 years ago
CharmingPuppy6
1
Question,
4
Answers
Active since 10 January 2023
Last activity
one year ago
Reputation
0
Badges 1
4 × Eureka!
0
Hi All, Is There A Way To Schedule The Tasks From The Queue Onto The Gpu Instances Based On Factors Such As Gpu Utilisation, Number Of Cpu Cores Present, Free Memory Or Custom Parameters Such As Priority Of The Task, Estimated Time Etc?
This will be quite easy to implement using the clearml k8s glue, just use user-properties and change the template based on it. I can point to where you need to modify the code
I’m pretty new to this. So, it’d be great if you can do that.
3 years ago
0
Hi All, Is There A Way To Schedule The Tasks From The Queue Onto The Gpu Instances Based On Factors Such As Gpu Utilisation, Number Of Cpu Cores Present, Free Memory Or Custom Parameters Such As Priority Of The Task, Estimated Time Etc?
Can I assume we are talking Kubernetes under the hood for the resource allocation?
yes
The granularity offered by K8s (and as you specified) is sometimes way too detailed for a user, for example I know I want 4 GPUs but 100GB disk-space, no idea, just give me 3 levels to choose from (if any, actually I would prefer a default that is large enough, since this is by definition for temp cache only), and the same argument for number of CPUs..
While i agree that over-detailing makes u...
3 years ago
0
Hi All, Is There A Way To Schedule The Tasks From The Queue Onto The Gpu Instances Based On Factors Such As Gpu Utilisation, Number Of Cpu Cores Present, Free Memory Or Custom Parameters Such As Priority Of The Task, Estimated Time Etc?
Hey AgitatedDove14 ,
The way
clearml
is designed, is to have queues abstract different types pf resources.
Configuring multiple queues and multiple agents based on the resources can be a solution for many use-cases . But when the instances are non-homogeneous, there can be too many combinations of resources based no.of GPUs, no. of cores, disk space etc. that work for various workloads. I’m thinking that creating as many agents and queues can get messy for managing a...
3 years ago
0
Hi All, Is There A Way To Schedule The Tasks From The Queue Onto The Gpu Instances Based On Factors Such As Gpu Utilisation, Number Of Cpu Cores Present, Free Memory Or Custom Parameters Such As Priority Of The Task, Estimated Time Etc?
ReassuredTiger98 thanks for sharing those threads. I found them very insightful.
3 years ago