I’ll also post this on the main channel -->
yes, exactly, agent creates and manages task pod lifecycle
@<1523701827080556544:profile|JuicyFox94> we have it up and running, hurray 🙂
One thing I noticed in the k8s logs is frequent warnings about Python 3.6..? Is the helm chart built with that Python version?
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.utils import int_from_bytes
/usr/local/lib/python3.6/dist-packages/jwt/utils.py:7: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from
import EllipticCurve
In k8s there’s no services but just clearml-agent (k8sglue). you can set any definition you want for spawned pods in this section: https://github.com/allegroai/clearml-helm-charts/blob/503ab437adc5d4f9b7b1037e2af143d47da24048/charts/clearml-agent/values.yaml#L132
And actually it fails on quite many tasks for us with this Python 3.6.
I tried to set up a different image ( agent8sglue.defaultContainerImage: "ubuntu:20.04" ) but that did not change much.
I suspect the culprit is agentk8sglue.image , which is set to tag 1.24-21 of clearml-agent-k8s-base . That image is quite very old… Any updates on that? 🤔
Thanks! To clarify, all the agent does is then spawn new nodes to cover the tasks?
i.e. It does not process tasks on its own?