![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/DisgustedDove53.png)
Reputation
Badges 1
63 × Eureka!yep it was unrelated.. sorry
Thanks for your help
It's still not connecting XD
but parsing is successful 😄
The K8sIntegration can be made into a nice operator for example and deeply integrate into ClearML itself. :grinning_face_with_star_eyes:
How is it going to access to actual pod? Is it a headless service?
there's no log of connecting to elasticsearch now 😕
does it mean it is okay?
the api server doesn't come up though. readiness probe fails.
I think the whole project can be more cloud friendly. I spent a lot of time adapting it to our k8s environment. I am willing also to contribute. I think a roadmap should be created for more k8s integration and then we can start. 🙂
SuccessfulKoala55 I’m getting 405 on api calls with the configuration you proposed. (btw I think USER_KEY is right, with a single underscore)
curl works with the same url and creds from the container...
of course because the tuple is valid in Python.
but there's no way I can supply a tuple in the HOCON format (of json for that matter(
│ urllib3.exceptions.LocationParseError: Failed to parse: '
https://elastic:xxxxxxxxxxxxxxxxx@clearml-elasticsearch-es-http ', label empty or too long │
` │ [2021-05-11 13:35:54,816] [8] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 1 of 4. Waiting for 30sec ...
I am already changing the chart, it would be no problem. But which env vars?
I actually got the idea to disable istio for the container. I think that'll work considering api-server doesn't need to be accessed directly from outside.
I'll test that and then your approach. If nothing works then I'll post again 😄
Thanks for your help Jake
I’m not using templates for k8s glue. I’m using the default operation mode which uses kubectl run. Should I use templates and specify a service in there to be able to connect to the pods?
- name: CLEARML__APISERVER__PRE_POPULATE__ENABLED value: "false" - name: CLEARML__APISERVER__PRE_POPULATE__ZIP_FILES value: /opt/clearml/db-pre-populate - name: CLEARML_SERVER_DEPLOYMENT_TYPE value: helm-cloud
The rest are clearly credentials…
no. apparently changing these variables causes something to fail.
Awesome info! Thanks!
What env variable should I use?
Also maybe I can block unauthenticated access to that endpoint too, I guess clearml-agent doesn’t use it!
I’m using cloud-ready helm chart (with some modifications)
So for api-server deployment:
we’re running it with the older helm chart if that matters. anyways I can’t see anything related to Gunicorn in chart or configs.
Hey Martin.
I’m deploying to k8s and want to know if PVC is necessary for Redis.
I'm using a simple pyhocon code to test it and it won't decode. So it won't reach the code to ovverride settings at all.
oh great!
let me take a look at this. There’s a lot of custom tooling so integration is very important.
cc: FiercePenguin76 ^^^
I changed them and tried some combinations to no avail...
I assume it should change the retry pattern too. but it doesn't change.. I feel I'm missing something obvious here
yes. Tuple is not valid. like json. only scalar, string, dict or array