I used the values from the dashboard/configuration/api keys

vs using aws dns

ok yes, this is the problem

curl --insecure -sw %{http_code} -o /dev/null │ │ init-k8s-glue waiting for apiserver │ │ init-k8s-glue + [ 000 -ne 200 ]

I don't see any requests

This points to configuration, specifically maybe it is directed to a different server?!

the worker is now in the dashboard

I think this is VPN related now

Seems like it is routing fine

Yep I updated those as well

I think if I use the local service URL this problem is fixed

the API url works fine, returns 200

I don't see any requests

I just opened a shell with the api and tried to curl my files URL, and the curl just hangs. no response

that is the containerinit logs from k8glueagent

BoredHedgehog47 can you provide some logs, this is odd..

I can see this log message in the nginx controller
"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26

` * Serving Flask app 'fileserver' (lazy loading)

• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
  [2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
  WARNING: This is a development server. Do not use it in a production deployment. `

And these ?
https://github.com/allegroai/clearml-helm-charts/blob/19a6785a03b780c2d22da1e79bcd69ac9ffcd839/charts/clearml-agent/values.yaml#L48

perhaps the 405 is from nginx

These are the logs from the fileserver pod

thank you for the help!

I verified in the pod yaml it is set correctly

perhaps I need to use localhost

I think the issue is the pod to pod comms can't resolve my route53 dns records

Nice! 🙂

is everything on the same network?

so its not the files server, its every server

Then it tries to curl the files API and gets a 405

maybe a cors issue?

yep that fixed it using references like clearml-webserver.clearml.svc.cluster.local:80