that's what I wanted to ask, while the proper networking is setup (I don't manage the cluster),
can I do tests using the .kube/config?

I guess yes but honestly I’m not sure you will get the right results

I wouldn’t say it’s related RBAC because the issue seems Networking realted so connection timed out

because kubectl inside pod uses inpod method

probably
is not accessible from your pod

try this into the pod

OK, I'll report that, and see if we can get it fixed

accessing apiserver from a pod doesn’t require kubeconfig

OK, thanks a lot, I'll try to get the networking thing sorted (and then I am sure I'll have lots more many doubts 😂 )

probably you will see it’s not capable of doing it and it should be related k8s config

# Point to the internal API server hostname APISERVER= `

Path to ServiceAccount token

SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

Read this Pod's namespace

NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

Read the ServiceAccount bearer token

TOKEN=$(cat ${SERVICEACCOUNT}/token)

Reference the internal certificate authority (CA)

CACERT=${SERVICEACCOUNT}/ca.crt

Explore the API with TOKEN

curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api `

Thanks for pitching in JuicyFox94 . For the connectivity, I used the "public" names for the various server
(e.g. we set clearml.internal.domain.name, clearml-apiserver.internal.domain.name and clearml-apiserver.internal.domain.name)

So in the agent values.yaml I set the parameters:
# -- Reference to Api server url apiServerUrlReference: " ` "

-- Reference to File server url

fileServerUrlReference: " "

-- Reference to Web server url

webServerUrlReference: " " `to the actual domain name.

Should I try using the internal kubernetes domain names instead, like
clearml-fileserver.clearml.svc.cluster.localinstead?

I don’t think it’s related how agent talk with apiserver or fileserver. It’s more related the fact agent pod internal kubectl cannot contact kubernetes apiserver

ah I see, I'll give it a try then

I suggest to try exec into agent pod and try to do some kubectl command like a simple kubectl get pod

because while I can run kubectl commands from within the agent pod, clearml doesn't seem to pick the right value:

2022-08-05 12:09:47 task 29f1645fbe1a4bb29898b1e71a8b1489 pulled from 51f5309bfb1940acb514d64931ffddb9 by worker k8s-agent-cpu 2022-08-05 12:12:59 Running kubectl encountered an error: Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2022-08-05 15:15:07 task 29f1645fbe1a4bb29898b1e71a8b1489 pulled from 51f5309bfb1940acb514d64931ffddb9 by worker k8s-agent-cpu

thanks for the help!

Effectively kubectl commands don't work from within the agent pod, I'll try to figure out why

you can workaround the issue mlunting the kubeconfig but I guess the issue is someway to be investigated

Thanks Valeriano, so copying the .kube/config file from a node from which I can run kubectl I could run kubectl commands correctly

yes, the curl returned a 503 error

JuicyFox94 apparently to make it work I'll have to add a "kubeconfig" file, but I can't see any obvious way to mount it in the agent pod, am I wrong?

btw it’s an infra issue

just my two cents

this means network issues at some level

exactly

yep but this is not how it should work with inpod

if it turns 503 it’s not network but something on top of it

from https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/