Hi @<1523701827080556544:profile|JuicyFox94> I figured out what the problem is! For some recent experimentation I set an acces_key and secret_key as environment variables in my os. When I deleted them everything worked fine so the environment variables overwrote the keys given by the clearml.conf. Is that the desired default behaviour?
And just one tip for everbody having similar problems: Switch to using the SDK instead of the CLI for better debugging. This helped me to find the cause of my issue. :)
ok so we should see the call in ingress controller logs, can you pls share them?
The clearml-data call results in these two lines in the ingress logs. Is that sufficient or would you like to have a larger section of the log?
2024/03/26 16:07:10 [warn] 2879#2879: *1151249 upstream sent duplicate header line: "server: clearml", previous value: "Server: Werkzeug/3.0.1 Python/3.9.18", ignored while reading response header from upstream, client: ***.***.***.22, server: api.clearml.****.com, request: "GET /auth.login HTTP/1.1", upstream: "
", host: "api.clearml.****.com"
77.23.243.22 - CN1YAZ1L7J4CY6GO8WQW [26/Mar/2024:16:07:10 +0000] "GET /auth.login HTTP/1.1" 401 347 "-" "python-requests/2.30.0" 449 0.005 [default-clearml-apiserver-8008] [] 10.213.150.109:8008 347 0.005 401 03f88a11d9264e7f078e8370a5e33dd2
url returning 401 is same as one exposed by ingress (if you are using them ofc)?
adding @<1523701087100473344:profile|SuccessfulKoala55> to the conversation because I’m not totally sure the problem relies on ingress, it looks to be a bad token but it shouldn’t since init was good