the issue moving forward is if we restart the pod we will have to manually update that again. similar to what we were doing prior to the http 1.1 setting being put in place.

but there are a bunch of 405 errors in the log

he's still messing around though

Hi BurlySeagull48 AgitatedDove14 , does upgrading clearml package and images to 1.0.4 really work? https://clearml.slack.com/archives/CTK20V944/p1628112607184300?thread_ts=1628108536.180100&cid=CTK20V944

looks like at the end of the day we removed 

proxy_set_header Host $host;

  and use the fqdn for the proxy_pass line

And did that solve the issue?

looks like at the end of the day we removed proxy_set_header Host $host; and use the fqdn for the proxy_pass line

Hi  AgitatedDove14 ,I try it with Clearml 1.1.4, but it fails again.

looks like he hardcoded the api fqdn in proxy_set_header Host instead of using $host

if i hit the api directly it returns fine

so it seems to be how istio is handling the routing. if we use the internal service address in the nginx configs it seems to go ok.

he said it was something in the nginx config though

That makes sense 🙂

It looks somewhat familiar ... 😞
SuccessfulKoala55 any idea?

have you seen this happen before?

we got it running. waiting for my coworker to give me the summary of what he did... he said it was something in the nginx config though

seems to only happen when going through the app

Nice!

yeah i can and it comes back 200

Cab you do:
curl

the issue moving forward is if we restart the pod we will have to manually update that again.

Can't you map the nginx configuration file ? (making the changes persistent across pods)

webserver

I meet the same problem.
I curl elastic and get the following result(picture 1).
I check the response of debug images from web UI, and I get the picture2.

version

I deploy clearml in 1.1.1 with helm, and use 1.0.4 clearml package.

` {
"meta": {
"id": "aac901e3e58c4381852b0fe1d227c732",
"trx": "aac901e3e58c4381852b0fe1d227c732",
"endpoint": {
"name": "login.supported_modes",
"requested_version": "2.13",
"actual_version": "1.0"
},
"result_code": 200,
"result_subcode": 0,
"result_msg": "OK",
"error_stack": "",
"error_data": {

}

},
"data": {
"authenticated": false,
"basic": {
"enabled": false,
"guest": {
"enabled": false
}
},
"server_errors": {
"es_connection_error": false,
"missed_es_upgrade": false
},
"sso": {

},
"sso_providers": [
  
]

}
} `that's what the browser returns

Yes, the webserver doesn't know where the api server is, it will access /api and then the nginx running the webapp will do the routing (reverse proxy)
I think that for some reason it is failing to do that (actually similarly to the stackoverflow you linked)

istio routes based off of hostname so the app hostname was being passed causing the loop

we removed that and it was all good

Hi FloppyDeer99
Since this thread is a bit old, I might have missed something 🙂
Are we saying the links are not working in the UI ?
(notice the links themselves are generated by the clearml package, so if there was a bug, still not sure here, then old links will remain invalid until manually fixed) Can you verify that the latest clearml generates working links?

AgitatedDove14 , I will try it with latest version. But why the url in es is different from it in web UI? I have checked the codes that evnets.debug_images api which will return url to web UI will get the url from es.

code commit

FloppyDeer99 what am I seeing in the screenshot ?

But why the url in es is different from it in web UI?

They are not really different, but sometimes the "url quote" is an issue (this is the process a browser will take a string url like a/b and convert it to a%2fb ),
I remember that there was an issue involving double quoting (this is when you have: a/b -> a%2fb -> a%252fb ), notice the last one replace "%" with "%25" as in your example...
Let me know if the latest clearml version solves it

yes it did