Hi @<1523701083040387072:profile|UnevenDolphin73> , this is the K8s integration. You can find more here:
None
Hey @<1523701070390366208:profile|CostlyOstrich36> , thanks for the reply!
I’m familiar with the above repo, we have the ClearML Server and such deployed on K8s.
What’s lacking is documentation regarding the clearml-agent helm chart. What exactly does it offer, etc.
We’re interested in e.g. using karpenter to scale our deployments per demand, effectively replacing the AWS autoscaler.
I think this is what you're looking for - the agent integration
None
Yes, I’ve found that too (as mentioned, I’m familiar with the repository). My issue is still that there is documentation as to what this actually offers.
Is this simply a helm chart to run an agent on a single pod? Does it scale in any way? Basically - is it a simple agent (similiar to on-premise agents, running in the background, but here on K8s), or is it a more advanced one that offers scaling features? What is it intended for, and how does it work?
The official documentation are very sparse about all of this, and only offers the variables that one can tweak, rather than an explanation about it actually offers.
@<1523701083040387072:profile|UnevenDolphin73> , basically, it scales to as many pods as you like. Very similar to the autoscaler but on top of K8s
But... Which queue does it listen to, and which type of instances will it use etc
It's all configured by the helm chart, it is the glue layer between K8s & ClearML
Right, so where can one find documentation about it?
The repo just has the variables with not much explanations.
I think that's what's there. In the Scale & Enterprise version ClearML usually works together with customers to provide a glue layer for K8s or even SLURM
Maybe @<1523701827080556544:profile|JuicyFox94> can answer some questions then…
For example, what’s the difference between agentk8sglue.nodeSelector and agentk8sglue.basePodTemplate.nodeSelector ?
Am I correct in understanding that the former decides the node type that runs the “scaler” (listening to the given agentk8sglue.queue ), and the latter for any new booted instance/pod, that will actually run the agent and the task?
Read: The former can be kept lightweight, as it does no heavy computations, the latter should have bigger resources?
About nodeSelector you are right, one is for the agent pod while the other is used to spawn task pods
about autoscaling it’s a complex topic regarding platform management in this case. ClearML glue simply spawn pods with resources defined in template.
how you cluster reacts is about scaling infra as much as needed (karpenter or any other cloud autoscaler should work)
We’re using karpenter (more magic keywords for me), so my understanding is that that will manage the scaling part.
Anything else you’d recommend paying attention to when setting the clearml-agent helm chart?
you will probably need a metrics-server on your k8s
it’s usually needed for autoscaler to decide when and how to scale up and down
about clearml-agent, just set resources in basepodtemplate (cpu gpu ram) so you will have a specific definition
this will make autoscaler life easier knowiing exactly how much resources you need
Does it make sense to you to run several such glue instances, to manage multiple resource requirements?
on OSS it’s usually the only way to as many agent deployments for any queue you define
(and any queue has it’s only basepodtemplate)
in Enterprise we support multiqueueing but it’s a different story
Perfect, thanks for the answers Valeriano. These small stuff are missing from the documentation, but I now feel much more confident in setting this up.