When using the ClearML agent, is it possible to let the agent decide if it should schedule a pod on a CPU or GPU? Using the k8s glue it would have to have:
tolerations: - key: nvidia.com/gpu value: present effect: NoSchedule
Otherwise k8s won't be able to auto schedule a gpu node while adding this by default would most likely schedule a GPU even if it's not needed
That's a cool idea. Then you pass the tolerations definition through a different pod template?
I'm using this feature, in this case i would create 2 agents, one with cpu only queue and the other with gpu queue. And then at the code level decide with queue to send to.
Then you pass the tolerations definition through a different pod template?
Yup.
