and I will give you feedback here

I don’t think it’s related how agent talk with apiserver or fileserver. It’s more related the fact agent pod internal kubectl cannot contact kubernetes apiserver

I need to evaluate a better way to handle image versioning for the future, btw latest version should be fixed. Apologies folks!

SuccessfulKoala55 after looking at the issue I’m a bit confused 😄 ; as far as I can see there is no way to pass any parameter to clearml-agent in daemon mode to push log to stdout. Can you confirm it? (If yes I need to find some workaround)

Will try to reproduce in next couple of hours, will give you feedback here asap

BoredBluewhale23 I can reproduce the issue, working on it

I’m going to investigate (and fix it if possible) in some day

(internally it will generate a file called services.conf in /opt/clearml/config folder in apiserver pod with content added) /cc SuccessfulKoala55

It's in values.yaml but yes, I need to improve this part, I agree

ok the issue must be there, After first creation nothing is there

(just to understand where are the ingress rules)

kubectl get ingress -A

apiserver: additionalConfigs: services.conf: |should be
apiserver: additionalConfigs: apiserver.conf: |in this way he pod will mount a file called apiserver.conf instead of services.conf that is not the right filename for auth.

it will be easier for me to reproduce

Just a quick suggestion since I have some more insight on the situation. Maybe you can look at Velero, it should be able to migrate data. If not you can simply create a new fresh install, scale everything to zero, then create some debug pod mounting old and new pvc and copy data between the two. More complex to say it than do it.

these are steps for major upgrade to latest chart version

If you have ALB you will just need to add some annotations on ingress rules depending on your setup. btw for now , since you already have everything in place, I suggest to just add values to /etc/hosts and see if it works

I suggest to try exec into agent pod and try to do some kubectl command like a simple kubectl get pod

Will cook something asap

can you post output of
kubectl get po -A -n clearmlpls?

any comment on the issue on GH is more than welcome

so you installed chart version 3.10.1

can you change the path in ALB healthcheck pls?

I need to investigate, ScrawnyLion96 can you pls open an issue on https://github.com/allegroai/clearml-helm-charts ?

with that said, the problem here is ambassador svc I think, still trying some trick

# Point to the internal API server hostname APISERVER= `

Path to ServiceAccount token

SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

Read this Pod's namespace

NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

Read the ServiceAccount bearer token

TOKEN=$(cat ${SERVICEACCOUNT}/token)

Reference the internal certificate authority (CA)

CACERT=${SERVICEACCOUNT}/ca.crt

Explore the API with TOKEN

curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${A...

usually you can see if you are getting timeouts or wrong http code

yep but this is not how it should work with inpod

it can help debugging

ok but describing the pod you should have, at least, the Ending cause