Huh it works now…!

webserver deployment:

yep it was unrelated.. sorry
Thanks for your help

nope no luck

It's still not connecting XD
but parsing is successful 😄

hmm
that would be great.

The K8sIntegration can be made into a nice operator for example and deeply integrate into ClearML itself. :grinning_face_with_star_eyes:

How is it going to access to actual pod? Is it a headless service?

there's no log of connecting to elasticsearch now 😕
does it mean it is okay?

the api server doesn't come up though. readiness probe fails.

I think the whole project can be more cloud friendly. I spent a lot of time adapting it to our k8s environment. I am willing also to contribute. I think a roadmap should be created for more k8s integration and then we can start. 🙂

SuccessfulKoala55 I’m getting 405 on api calls with the configuration you proposed. (btw I think USER_KEY is right, with a single underscore)

Thanks!

curl works with the same url and creds from the container...

of course because the tuple is valid in Python.
but there's no way I can supply a tuple in the HOCON format (of json for that matter(

no...

│ urllib3.exceptions.LocationParseError: Failed to parse: ' https://elastic:xxxxxxxxxxxxxxxxx@clearml-elasticsearch-es-http ', label empty or too long │
` │ [2021-05-11 13:35:54,816] [8] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 1 of 4. Waiting for 30sec ...

I am already changing the chart, it would be no problem. But which env vars?

I actually got the idea to disable istio for the container. I think that'll work considering api-server doesn't need to be accessed directly from outside.

I'll test that and then your approach. If nothing works then I'll post again 😄

Thanks for your help Jake

I’m not using templates for k8s glue. I’m using the default operation mode which uses kubectl run. Should I use templates and specify a service in there to be able to connect to the pods?

- name: CLEARML__APISERVER__PRE_POPULATE__ENABLED value: "false" - name: CLEARML__APISERVER__PRE_POPULATE__ZIP_FILES value: /opt/clearml/db-pre-populate - name: CLEARML_SERVER_DEPLOYMENT_TYPE value: helm-cloud

The rest are clearly credentials…

no. apparently changing these variables causes something to fail.

Awesome info! Thanks!
What env variable should I use?
Also maybe I can block unauthenticated access to that endpoint too, I guess clearml-agent doesn’t use it!

I’m using cloud-ready helm chart (with some modifications)
So for api-server deployment:

we’re running it with the older helm chart if that matters. anyways I can’t see anything related to Gunicorn in chart or configs.

Hey Martin.
I’m deploying to k8s and want to know if PVC is necessary for Redis.

The dashboard doesn’t load

I'm using a simple pyhocon code to test it and it won't decode. So it won't reach the code to ovverride settings at all.

oh great!
let me take a look at this. There’s a lot of custom tooling so integration is very important.

cc: FiercePenguin76 ^^^

I changed them and tried some combinations to no avail...
I assume it should change the retry pattern too. but it doesn't change.. I feel I'm missing something obvious here

yes. Tuple is not valid. like json. only scalar, string, dict or array