Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Why Am I Getting A 403 From File Server When The K8 Glue Agent Is Initializing ?

Why am I getting a 403 from file server when the k8 glue agent is initializing ?

  
  
Posted 2 years ago
Votes Newest

Answers 30


I can see this log message in the nginx controller
"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26

  
  
Posted 2 years ago

maybe a cors issue?

  
  
Posted 2 years ago

vs using aws dns

  
  
Posted 2 years ago

I just opened a shell with the api and tried to curl my files URL, and the curl just hangs. no response

  
  
Posted 2 years ago

I think if I use the local service URL this problem is fixed

  
  
Posted 2 years ago

Then it tries to curl the files API and gets a 405

  
  
Posted 2 years ago

so its not the files server, its every server

  
  
Posted 2 years ago

I think this is VPN related now

  
  
Posted 2 years ago

that is the containerinit logs from k8glueagent

  
  
Posted 2 years ago

perhaps the 405 is from nginx

  
  
Posted 2 years ago

Seems like it is routing fine

  
  
Posted 2 years ago

BoredHedgehog47 can you provide some logs, this is odd..

  
  
Posted 2 years ago

` * Serving Flask app 'fileserver' (lazy loading)

  • Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
  • Debug mode: off
    [2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
    WARNING: This is a development server. Do not use it in a production deployment. `
  
  
Posted 2 years ago

is everything on the same network?

  
  
Posted 2 years ago

the API url works fine, returns 200

  
  
Posted 2 years ago

I think the issue is the pod to pod comms can't resolve my route53 dns records

  
  
Posted 2 years ago

the worker is now in the dashboard

  
  
Posted 2 years ago

ok yes, this is the problem

  
  
Posted 2 years ago

yep that fixed it using references like clearml-webserver.clearml.svc.cluster.local:80

  
  
Posted 2 years ago

Nice! 🙂

  
  
Posted 2 years ago

These are the logs from the fileserver pod

  
  
Posted 2 years ago

perhaps I need to use localhost

  
  
Posted 2 years ago

I verified in the pod yaml it is set correctly

  
  
Posted 2 years ago

curl --insecure -sw %{http_code} -o /dev/null │ │ init-k8s-glue waiting for apiserver │ │ init-k8s-glue + [ 000 -ne 200 ]

  
  
Posted 2 years ago

Yep I updated those as well

  
  
Posted 2 years ago

I don't see any requests

This points to configuration, specifically maybe it is directed to a different server?!

  
  
Posted 2 years ago

I don't see any requests

  
  
Posted 2 years ago

thank you for the help!

  
  
Posted 2 years ago

I used the values from the dashboard/configuration/api keys

  
  
Posted 2 years ago
1K Views
30 Answers
2 years ago
one year ago
Tags