In k8s there’s no services but just clearml-agent (k8sglue). you can set any definition you want for spawned pods in this section: https://github.com/allegroai/clearml-helm-charts/blob/503ab437adc5d4f9b7b1037e2af143d47da24048/charts/clearml-agent/values.yaml#L132
Thanks! To clarify, all the agent does is then spawn new nodes to cover the tasks?
yes, exactly, agent creates and manages task pod lifecycle
i.e. It does not process tasks on its own?
@<1523701827080556544:profile|JuicyFox94> we have it up and running, hurray 🙂
One thing I noticed in the k8s logs is frequent warnings about Python 3.6..? Is the helm chart built with that Python version?
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.utils import int_from_bytes
/usr/local/lib/python3.6/dist-packages/jwt/utils.py:7: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from
import EllipticCurve
I’ll also post this on the main channel -->
And actually it fails on quite many tasks for us with this Python 3.6.
I tried to set up a different image ( agent8sglue.defaultContainerImage: "ubuntu:20.04" ) but that did not change much.
I suspect the culprit is agentk8sglue.image , which is set to tag 1.24-21 of clearml-agent-k8s-base . That image is quite very old… Any updates on that? 🤔