PR incoming

ReassuredArcticwolf33 pls try this one and let me know if this is working for you

I didn’t get the point, if you need to change it, you can just override image name and tag by Helm

that sc is not set as default

annotations:
http://nginx.ingress.kubernetes.io/proxy-body-size: "0"
http://nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
http://nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

I don’t think it’s possible to setup queues in advance with any ClearML chart env var but I’m not 100% sure. SuccessfulKoala55 can you pls clarify this?

woah.. superfast 😄

s o kubectl -n clearml get ingress

an implementation of this kind is interesting for you or do you suggest to fork? I mean, I don't want to impact your time reviewing

You can’t write on readonly replica is about MongoDB. I guess you are using a multiple replica setup. In this case the mongodb dependency chart have a lot of parameters to tweak the system and maybe also an arbiter is good for you. But this is a huge topic regarding mongodb specific k8s setups.

I suggest to try exec into agent pod and try to do some kubectl command like a simple kubectl get pod

Basically you can install lates clearml chart

I need to investigate, ScrawnyLion96 can you pls open an issue on https://github.com/allegroai/clearml-helm-charts ?

Ok that’s weird

can you also show output of kubectl get po of the namespace where you installaed clearml?

checking more

Interesting use case, maybe we can create multiple k8s agents for different queues

I’m going to investigate (and fix it if possible) in some day

Hi ApprehensiveSeahorse83 , today we released clearml-agent chart that just installs glue agent. My suggestion is to disable k8s glue and any other agent from the clearml chart and install more than one clearml-agent chart in different namespaces. In this way you will be able to have k8s glue for every queue (cpu and gpu).

Hi everyone, I just fixed releases so new charts containing this fix are published. ty!

ok let me check this one

on OSS it’s usually the only way to as many agent deployments for any queue you define

still need time because I have two very busy days

Hey ApprehensiveSeahorse83 , I didn’t forget about you, it’s just a busy time for me; will answer during the day after a couple of more tests on my testing env.

regardless of this I probably need to add some more detailed explanations on credentials configs

Did you added volumeMounts to right section? it should be under basePodTemplate in override file.

clearml and agent are in the same namespace?

Hi, in k8s autoscaling must be managed by cloud pro user autoscaler. When the clearml-agent bound to related queue will spawn a new task pod with configured resources, k8s will adapt. On AWS you can start here https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html

In this case I suggest to give a try to k8s-glue that is there by default in latest chart version

Ty, I have other stuff that I'd like to send but it's better to get these eventually merged first so I can proceed to shiny news PR in the near future 😄