I can see this log message in the nginx controller"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26
I just opened a shell with the api and tried to curl my files URL, and the curl just hangs. no response
I think if I use the local service URL this problem is fixed
Then it tries to curl the files API and gets a 405
so its not the files server, its every server
that is the containerinit logs from k8glueagent
BoredHedgehog47 can you provide some logs, this is odd..
` * Serving Flask app 'fileserver' (lazy loading)
- Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead. - Debug mode: off
[2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
WARNING: This is a development server. Do not use it in a production deployment. `
I think the issue is the pod to pod comms can't resolve my route53 dns records
yep that fixed it using references like clearml-webserver.clearml.svc.cluster.local:80
These are the logs from the fileserver pod
I verified in the pod yaml it is set correctly
curl --insecure -sw %{http_code}
-o /dev/null │ │ init-k8s-glue waiting for apiserver │ │ init-k8s-glue + [ 000 -ne 200 ]
I don't see any requests
This points to configuration, specifically maybe it is directed to a different server?!
I used the values from the dashboard/configuration/api keys