And these ?
https://github.com/allegroai/clearml-helm-charts/blob/19a6785a03b780c2d22da1e79bcd69ac9ffcd839/charts/clearml-agent/values.yaml#L48

I used the values from the dashboard/configuration/api keys

Then it tries to curl the files API and gets a 405

I verified in the pod yaml it is set correctly

the API url works fine, returns 200

Yep I updated those as well

I can see this log message in the nginx controller
"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26

I don't see any requests

This points to configuration, specifically maybe it is directed to a different server?!

BoredHedgehog47 can you provide some logs, this is odd..

These are the logs from the fileserver pod

perhaps the 405 is from nginx

I don't see any requests

maybe a cors issue?

I just opened a shell with the api and tried to curl my files URL, and the curl just hangs. no response

I think the issue is the pod to pod comms can't resolve my route53 dns records

that is the containerinit logs from k8glueagent

so its not the files server, its every server

is everything on the same network?

Seems like it is routing fine

perhaps I need to use localhost

vs using aws dns

I think if I use the local service URL this problem is fixed

I think this is VPN related now

ok yes, this is the problem

yep that fixed it using references like clearml-webserver.clearml.svc.cluster.local:80

the worker is now in the dashboard

thank you for the help!

Nice! 🙂

` * Serving Flask app 'fileserver' (lazy loading)

• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
  [2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
  WARNING: This is a development server. Do not use it in a production deployment. `

curl --insecure -sw %{http_code} -o /dev/null │ │ init-k8s-glue waiting for apiserver │ │ init-k8s-glue + [ 000 -ne 200 ]