About nodeSelector you are right, one is for the agent pod while the other is used to spawn task pods
this will make autoscaler life easier knowiing exactly how much resources you need
on OSS it’s usually the only way to as many agent deployments for any queue you define
Perfect, thanks for the answers Valeriano. These small stuff are missing from the documentation, but I now feel much more confident in setting this up.
about clearml-agent, just set resources in basepodtemplate (cpu gpu ram) so you will have a specific definition
(and any queue has it’s only basepodtemplate)
Maybe @<1523701827080556544:profile|JuicyFox94> can answer some questions then…
For example, what’s the difference between agentk8sglue.nodeSelector and agentk8sglue.basePodTemplate.nodeSelector ?
Am I correct in understanding that the former decides the node type that runs the “scaler” (listening to the given agentk8sglue.queue ), and the latter for any new booted instance/pod, that will actually run the agent and the task?
Read: The former can be kept lightweight, as it does no heavy computations, the latter should have bigger resources?
Hi @<1523701083040387072:profile|UnevenDolphin73> , this is the K8s integration. You can find more here:
None
about autoscaling it’s a complex topic regarding platform management in this case. ClearML glue simply spawn pods with resources defined in template.
Yes, I’ve found that too (as mentioned, I’m familiar with the repository). My issue is still that there is documentation as to what this actually offers.
Is this simply a helm chart to run an agent on a single pod? Does it scale in any way? Basically - is it a simple agent (similiar to on-premise agents, running in the background, but here on K8s), or is it a more advanced one that offers scaling features? What is it intended for, and how does it work?
The official documentation are very sparse about all of this, and only offers the variables that one can tweak, rather than an explanation about it actually offers.
@<1523701083040387072:profile|UnevenDolphin73> , basically, it scales to as many pods as you like. Very similar to the autoscaler but on top of K8s
We’re using karpenter (more magic keywords for me), so my understanding is that that will manage the scaling part.
Anything else you’d recommend paying attention to when setting the clearml-agent helm chart?
Does it make sense to you to run several such glue instances, to manage multiple resource requirements?
I think that's what's there. In the Scale & Enterprise version ClearML usually works together with customers to provide a glue layer for K8s or even SLURM
Right, so where can one find documentation about it?
The repo just has the variables with not much explanations.
But... Which queue does it listen to, and which type of instances will it use etc
it’s usually needed for autoscaler to decide when and how to scale up and down
in Enterprise we support multiqueueing but it’s a different story
It's all configured by the helm chart, it is the glue layer between K8s & ClearML
Hey @<1523701070390366208:profile|CostlyOstrich36> , thanks for the reply!
I’m familiar with the above repo, we have the ClearML Server and such deployed on K8s.
What’s lacking is documentation regarding the clearml-agent helm chart. What exactly does it offer, etc.
We’re interested in e.g. using karpenter to scale our deployments per demand, effectively replacing the AWS autoscaler.
I think this is what you're looking for - the agent integration
None
you will probably need a metrics-server on your k8s
how you cluster reacts is about scaling infra as much as needed (karpenter or any other cloud autoscaler should work)