so it seems to be how istio is handling the routing. if we use the internal service address in the nginx configs it seems to go ok.
looks like he hardcoded the api fqdn in proxy_set_header Host instead of using $host
It looks somewhat familiar ... 😞
SuccessfulKoala55 any idea?
but there are a bunch of 405 errors in the log
the issue moving forward is if we restart the pod we will have to manually update that again. similar to what we were doing prior to the http 1.1 setting being put in place.
FloppyDeer99 what am I seeing in the screenshot ?
But why the url in es is different from it in web UI?
They are not really different, but sometimes the "url quote" is an issue (this is the process a browser will take a string url like a/b and convert it to a%2fb ),
I remember that there was an issue involving double quoting (this is when you have: a/b -> a%2fb -> a%252fb ), notice the last one replace "%" with "%25" as in your example...
Let me know if the latest clearml version solves it
Hi FloppyDeer99
Since this thread is a bit old, I might have missed something 🙂
Are we saying the links are not working in the UI ?
(notice the links themselves are generated by the clearml package, so if there was a bug, still not sure here, then old links will remain invalid until manually fixed) Can you verify that the latest clearml generates working links?
AgitatedDove14 , I will try it with latest version. But why the url in es is different from it in web UI? I have checked the codes that evnets.debug_images api which will return url to web UI will get the url from es.
code commit
Hi BurlySeagull48 AgitatedDove14 , does upgrading clearml package and images to 1.0.4 really work? https://clearml.slack.com/archives/CTK20V944/p1628112607184300?thread_ts=1628108536.180100&cid=CTK20V944
istio routes based off of hostname so the app hostname was being passed causing the loop
seems to only happen when going through the app
we got it running. waiting for my coworker to give me the summary of what he did... he said it was something in the nginx config though
Hi AgitatedDove14 ,I try it with Clearml 1.1.4, but it fails again.
looks like at the end of the day we removed proxy_set_header Host $host; and use the fqdn for the proxy_pass line
if i hit the api directly it returns fine
` {
"meta": {
"id": "aac901e3e58c4381852b0fe1d227c732",
"trx": "aac901e3e58c4381852b0fe1d227c732",
"endpoint": {
"name": "login.supported_modes",
"requested_version": "2.13",
"actual_version": "1.0"
},
"result_code": 200,
"result_subcode": 0,
"result_msg": "OK",
"error_stack": "",
"error_data": {
}
},
"data": {
"authenticated": false,
"basic": {
"enabled": false,
"guest": {
"enabled": false
}
},
"server_errors": {
"es_connection_error": false,
"missed_es_upgrade": false
},
"sso": {
},
"sso_providers": [
]
}
} `that's what the browser returns
I meet the same problem.
I curl elastic and get the following result(picture 1).
I check the response of debug images from web UI, and I get the picture2.
version
I deploy clearml in 1.1.1 with helm, and use 1.0.4 clearml package.
looks like at the end of the day we removed
proxy_set_header Host $host;
and use the fqdn for the proxy_pass line
And did that solve the issue?
Yes, the webserver doesn't know where the api server is, it will access /api and then the nginx running the webapp will do the routing (reverse proxy)
I think that for some reason it is failing to do that (actually similarly to the stackoverflow you linked)
he said it was something in the nginx config though
That makes sense 🙂
istio expects a certain host and whatever is getting passed by the pod isn't right.