Ok thanks for reply

hmm
that would be great.

I actually got the idea to disable istio for the container. I think that'll work considering api-server doesn't need to be accessed directly from outside.

I'll test that and then your approach. If nothing works then I'll post again 😄

Thanks for your help Jake

Hey Martin.
I’m deploying to k8s and want to know if PVC is necessary for Redis.

Yes.
ClearML has all these awesome dashboards. I was thinking maybe there will be some deploy to prod/staging/shadow button so you can do everything from there. (Like Gitlab CI/CD if you have seen that)
That would be ideal.

Hi AgitatedDove14

For example version control, A/B testing, shadow testing, rollback etc..
We can use conventional solutions that we already use. Like Helm and Istio.

But for example is there any tools that gives more insight into model performance in production? or gives more knobs and settings specific for AI.
using common microservice deployment frameworks will be limiting.

I found some solutions (KubeFlow, Seldon Core, http://run.ai , MLFlow, …) but haven’t got to look at them really.

wow man you're the best!
SuccessfulKoala55

Yes I see. But the current env doesn't reach there because it cannot be decoded by pyhocon.

of course because the tuple is valid in Python.
but there's no way I can supply a tuple in the HOCON format (of json for that matter(

no...

That wouldn't have crossed my mind especially when it's not in the docs.

there's no log of connecting to elasticsearch now 😕
does it mean it is okay?

the api server doesn't come up though. readiness probe fails.

no. tried many variations. I'm not sure if it's reading the variable or not. because wait times don't change either

nope no luck

Thanks!

I want to find a good deployment framework (if exists) and integrate it into our ClearML workflow).
like get latest release with prod tag from clearml api and apply that to k8s.
something along those lines

it is indeed parsing because invalid hocon leads to error. but nothing changes about retry times etc

oh great!
let me take a look at this. There’s a lot of custom tooling so integration is very important.

cc: FiercePenguin76 ^^^

I changed them and tried some combinations to no avail...
I assume it should change the retry pattern too. but it doesn't change.. I feel I'm missing something obvious here

That's right

yes. Tuple is not valid. like json. only scalar, string, dict or array

It’s probably unrelated 😬 I’ll keep you posted.

Ok I think I might have found the problem.

curl works with the same url and creds from the container...

The login prompt doesn’t appear (I tried removing cookies and all too)

webserver deployment:

https://github.com/allegroai/clearml-server/blob/babfcbb70701fc725448556d1e691cdfd1292949/apiserver/es_factory.py

https://allegro.ai/clearml/docs/docs/deploying_clearml/clearml_server_config.html

ah ok. So only solution is changing the hardcoded one…
I will be looking into docs to find the right env variable. If you know it I’d appreciate you telling me 😄 SuccessfulKoala55