Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Why Am I Getting A 403 From File Server When The K8 Glue Agent Is Initializing ?

Why am I getting a 403 from file server when the k8 glue agent is initializing ?

  
  
Posted one year ago
Votes Newest

Answers 30


I verified in the pod yaml it is set correctly

  
  
Posted one year ago

Then it tries to curl the files API and gets a 405

  
  
Posted one year ago

I don't see any requests

This points to configuration, specifically maybe it is directed to a different server?!

  
  
Posted one year ago

These are the logs from the fileserver pod

  
  
Posted one year ago

maybe a cors issue?

  
  
Posted one year ago

is everything on the same network?

  
  
Posted one year ago

curl --insecure -sw %{http_code} -o /dev/null │ │ init-k8s-glue waiting for apiserver │ │ init-k8s-glue + [ 000 -ne 200 ]

  
  
Posted one year ago

vs using aws dns

  
  
Posted one year ago

I don't see any requests

  
  
Posted one year ago

I think the issue is the pod to pod comms can't resolve my route53 dns records

  
  
Posted one year ago

I think this is VPN related now

  
  
Posted one year ago

I just opened a shell with the api and tried to curl my files URL, and the curl just hangs. no response

  
  
Posted one year ago

Yep I updated those as well

  
  
Posted one year ago

that is the containerinit logs from k8glueagent

  
  
Posted one year ago

` * Serving Flask app 'fileserver' (lazy loading)

  • Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
  • Debug mode: off
    [2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
    WARNING: This is a development server. Do not use it in a production deployment. `
  
  
Posted one year ago

the API url works fine, returns 200

  
  
Posted one year ago

BoredHedgehog47 can you provide some logs, this is odd..

  
  
Posted one year ago

I can see this log message in the nginx controller
"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26

  
  
Posted one year ago

perhaps I need to use localhost

  
  
Posted one year ago

perhaps the 405 is from nginx

  
  
Posted one year ago

I think if I use the local service URL this problem is fixed

  
  
Posted one year ago

so its not the files server, its every server

  
  
Posted one year ago

Seems like it is routing fine

  
  
Posted one year ago

yep that fixed it using references like clearml-webserver.clearml.svc.cluster.local:80

  
  
Posted one year ago

ok yes, this is the problem

  
  
Posted one year ago

the worker is now in the dashboard

  
  
Posted one year ago

thank you for the help!

  
  
Posted one year ago

Nice! 🙂

  
  
Posted one year ago

I used the values from the dashboard/configuration/api keys

  
  
Posted one year ago
773 Views
30 Answers
one year ago
one year ago
Tags