the issue moving forward is if we restart the pod we will have to manually update that again. similar to what we were doing prior to the http 1.1 setting being put in place.
but there are a bunch of 405 errors in the log
Hi BurlySeagull48 AgitatedDove14 , does upgrading clearml package and images to 1.0.4 really work? https://clearml.slack.com/archives/CTK20V944/p1628112607184300?thread_ts=1628108536.180100&cid=CTK20V944
looks like at the end of the day we removed
proxy_set_header Host $host;
and use the fqdn for the proxy_pass line
And did that solve the issue?
looks like at the end of the day we removed proxy_set_header Host $host;
and use the fqdn for the proxy_pass line
Hi AgitatedDove14 ,I try it with Clearml 1.1.4, but it fails again.
looks like he hardcoded the api fqdn in proxy_set_header Host
instead of using $host
if i hit the api directly it returns fine
so it seems to be how istio is handling the routing. if we use the internal service address in the nginx configs it seems to go ok.
he said it was something in the nginx config though
That makes sense 🙂
It looks somewhat familiar ... 😞
SuccessfulKoala55 any idea?
we got it running. waiting for my coworker to give me the summary of what he did... he said it was something in the nginx config though
seems to only happen when going through the app
the issue moving forward is if we restart the pod we will have to manually update that again.
Can't you map the nginx configuration file ? (making the changes persistent across pods)
I meet the same problem.
I curl elastic and get the following result(picture 1).
I check the response of debug images from web UI, and I get the picture2.
version
I deploy clearml in 1.1.1 with helm, and use 1.0.4 clearml package.
` {
"meta": {
"id": "aac901e3e58c4381852b0fe1d227c732",
"trx": "aac901e3e58c4381852b0fe1d227c732",
"endpoint": {
"name": "login.supported_modes",
"requested_version": "2.13",
"actual_version": "1.0"
},
"result_code": 200,
"result_subcode": 0,
"result_msg": "OK",
"error_stack": "",
"error_data": {
}
},
"data": {
"authenticated": false,
"basic": {
"enabled": false,
"guest": {
"enabled": false
}
},
"server_errors": {
"es_connection_error": false,
"missed_es_upgrade": false
},
"sso": {
},
"sso_providers": [
]
}
} `that's what the browser returns
Yes, the webserver doesn't know where the api server is, it will access /api and then the nginx running the webapp will do the routing (reverse proxy)
I think that for some reason it is failing to do that (actually similarly to the stackoverflow you linked)
istio routes based off of hostname so the app hostname was being passed causing the loop
Hi FloppyDeer99
Since this thread is a bit old, I might have missed something 🙂
Are we saying the links are not working in the UI ?
(notice the links themselves are generated by the clearml package, so if there was a bug, still not sure here, then old links will remain invalid until manually fixed) Can you verify that the latest clearml generates working links?
AgitatedDove14 , I will try it with latest version. But why the url in es is different from it in web UI? I have checked the codes that evnets.debug_images
api which will return url to web UI will get the url from es.
code commit
FloppyDeer99 what am I seeing in the screenshot ?
But why the url in es is different from it in web UI?
They are not really different, but sometimes the "url quote" is an issue (this is the process a browser will take a string url like a/b
and convert it to a%2fb
),
I remember that there was an issue involving double quoting (this is when you have: a/b
-> a%2fb
-> a%252fb
), notice the last one replace "%" with "%25" as in your example...
Let me know if the latest clearml version solves it