Hi there,

Our team started using clearml a few months ago and we've recently deployed an AWS EKS k8s cluster with the hopes of deploying a clearml-agent. I've been able to install the agent on the cluster using:

helm install clearml-agent-gpu allegroai/clearml-agent \
    --set clearml.agentk8sglueKey=<removed> \
    --set clearml.agentk8sglueSecret=<removed> \
    --set agentk8sglue.defaultContainerImage="python:3.11-bullseye" \
    --set agentk8sglue.nodeSelector.nodegroup="clearml-agent" \
    --set agentk8sglue.queue="gpu-queue" \
    --set agentk8sglue.basePodTemplate.nodeSelector.nodegroup="clearml-gpu" \
    --set agentk8sglue.basePodTemplate.env[0].name=CLEARML_AGENT_GIT_USER \
    --set agentk8sglue.basePodTemplate.env[0].value=username \
    --set agentk8sglue.basePodTemplate.env[1].name=CLEARML_AGENT_GIT_PASS \
    --set agentk8sglue.basePodTemplate.env[1].valueFrom.secretKeyRef.name=git-password \
    --set agentk8sglue.basePodTemplate.env[1].valueFrom.secretKeyRef.key=git-password \
    --set clearml.clearmlConfig="
    agent {
      package_manager: {
        type: poetry;
        poetry_version: 1.8.2
      }
      force_git_ssh_protocol: false;
      disable_requirements_auto_install: true
    }"

I'm currently encountering two issues:

• --set agentk8sglue.defaultContainerImage="python:3.11-bullseye" does not seem to change the container that gets used. Here are the logs from the pod:

Executing task id [9296dbfe38384daf958911e9155a8bca]:
repository = git@gitlab.com:<company_git_repo>.git
branch = <branch_name>
version_num = f0052fa186cab812a4aa07c05e088d466eb41ff7
tag =
docker_cmd = ubuntu:18.04
entry_point = main.py
working_dir = scope-ml/scope_mllib/training

Python executable with version '3.11' requested by the Task, not found in path, using '/usr/bin/python3' (v3.6.9) instead

It still tries to use 'ubuntu:18.04', am I doing this correctly?
2. I've created a k8s secret with the gitlab personal access token, but it seems like it is still unable to git pull the repo that is needed. Here are the logs from the pod:

cloning: git@gitlab.com:<company_git_repo>.git
Using user/pass credentials - replacing ssh url 'git@gitlab.com:<company_git_repo>.git' with https url '

<company_git_repo>.git'

pulling git
Using SSH credentials - replacing https url '

<company_git_repo>.git' with ssh url '

<company_git_repo>.git'
fatal: could not read Username for '

': terminal prompts disabled
error: Could not fetch origin
git pull failed: Command '['git', 'fetch', '--all', '--tags', '--recurse-submodules']' returned non-zero exit status 1.
Repository cloning failed: Command '['git', 'fetch', '--all', '--tags', '--recurse-submodules']' returned non-zero exit status 1.
Task failed: stopping task (4) exception

Any assistance would be much appreciated thanks!!