Ok will try it

I will get back at you in 15mn if thats ok

for now:

• name: clearml-access-key
  value: CLEARML8AGENT9KEY1234567890ABCD
  - name: clearml-secret-key
  value: CLEARML-AGENT-SECRET-1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ123456
  - name: admin-password
  value: clearml123!

yes

Since with argo i can pass them as params

It's a bit hard for me to provide support here with the additional layer of Argo.
I assume the server is working fine and you can open the clearml UI and log in, right? If yes, would it be possible to extract the Agent part only, out of Argo, and proceed installing it through standard helm?

Hi @<1857232027015712768:profile|PompousCrow47> , are you using pods with a read-only-filesystem limitation?

Sorry we had a short delay on the deployment but

with these values:

clearml:
  agentk8sglueKey: "8888TMDLWYY7ZQJJ0I7R2X2RSP8XFT"
  agentk8sglueSecret: "oNODbBkDGhcDscTENQyr-GM0cE8IO7xmpaPdqyfsfaWearo1S8EQ8eBOxu-opW8dVUU"
  clearmlConfig: |-
    api {
        web_server: 

        api_server: 

        files_server: 

        credentials {
            "access_key" = "8888TMDLWYY7ZQJJ0I7R2X2RSP8XFT"
            "secret_key" = "oNODbBkDGhcDscTENQyr-GM0cE8IO7xmpaPdqyfsfaWearo1S8EQ8eBOxu-opW8dVUU"
        }
    }

agentk8sglue:
  # Try different image versions to avoid Python 3.6 regex issue
  image:
    repository: allegroai/clearml-agent-k8s-base
    tag: "latest"  # Use latest instead of specific version
    pullPolicy: Always

  # Essential server references
  apiServerUrlReference: "

"
  fileServerUrlReference: "

"
  webServerUrlReference: "

"

  # Disable certificate checking
  clearmlcheckCertificate: false

  # Queue configuration
  queue: default
  createQueueIfNotExists: true

  # Minimal resources
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi

sessions:
  svcType: NodePort
  externalIP: 192.168.70.211
  startingPort: 30100
  maxServices: 5
EOF

The following commands

helm repo add clearml 

helm repo update

helm install clearml-agent clearml/clearml-agent \
  --namespace clearml-prod \
  --values clearml-agent-values.yaml \
  --wait \
  --timeout 300s
"clearml" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "argo" chart repository
...Successfully got an update from the "clearml" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "nvidia" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: clearml-agent
LAST DEPLOYED: Mon Jul 21 15:11:38 2025
NAMESPACE: clearml-prod
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Glue Agent deployed.

then share what i found

I have separed the most crutial part. Its a container that runs the standard helm commands

example:
....
cat > /tmp/server-values.yaml <<EOF
global:
defaultStorageClass: $STORAGE_CLASS

        apiserver:

...
helm install clearml clearml/clearml
--namespace "$NS"
--values /tmp/server-values.yaml
--wait
--timeout "$TMO"

...

helm install clearml-agent clearml/clearml-agent
--namespace "$NS"
--values /tmp/simple-agent-values.yaml
--wait
--timeout 300s

these are the values :

clearml:
agentk8sglueKey: $ACCESS_KEY
agentk8sglueSecret: $SECRET_KEY
clearmlConfig: |-
api {
web_server: http://$NODE_IP:30080
api_server: http://$NODE_IP:30008
files_server: http://$NODE_IP:30081
credentials {
"access_key" = "$ACCESS_KEY"
"secret_key" = "$SECRET_KEY"
}
}

        agentk8sglue:
          # Try newer image version to fix Python 3.6 regex issue
          image:
            repository: allegroai/clearml-agent-k8s-base
            tag: "1.25-1"
            pullPolicy: Always

          apiServerUrlReference: "http://$NODE_IP:30008"
          fileServerUrlReference: "http://$NODE_IP:30081"
          webServerUrlReference: "http://$NODE_IP:30080"
          clearmlcheckCertificate: false
          queue: default
          createQueueIfNotExists: true

          # Keep resources minimal for testing
          resources:
            limits:
              cpu: 500m
              memory: 1Gi
            requests:
              cpu: 100m
              memory: 256Mi

        sessions:
          svcType: NodePort
          externalIP: $NODE_IP
          startingPort: 30100
          maxServices: 5

Yeah i know.. thats what i did for the github implementation, but for this i need them to be generated on the fly or via CLI that i can use argo to create if thats possible

I will try to create them on the UI and only run the Agent task on argo or so to see if it helps

It got an error then backed off

As far as i can test, the server is going ok, i had some isses with resources not loading but solved those. The bigger issue for now is agent and prob could propagate to the serving. Later on i plan on adding also gpu resouces to both so im not entirely sure on that part

clearml-apiserver-866ccf75f7-zr5wx 1/1 Running 0 37m
clearml-apiserver-asyncdelete-8dfb574b8-8gbcv 1/1 Running 0 37m
clearml-elastic-master-0 1/1 Running 0 37m
clearml-fileserver-86b8ddf6f6-4xnqd 1/1 Running 0 37m
clearml-mongodb-5f995fbb5-xmdpb 1/1 Running 0 37m
clearml-redis-master-0 1/1 Running 0 37m
clearml-webserver-c487cfcb-vv5z5 1/1 Running 0 37m

I assume the key and secret values here are redacted values and not the actual ones, right?

I will try :
1- update the agent with these values
2- run argo with those changes

@<1729671499981262848:profile|CooperativeKitten94> @<1857232027015712768:profile|PompousCrow47>

I figured it out for future reference this is a error regarding the Kubernetes Support on the agent : None

As for getting the credentials to lauch the agent the only way i can do it is via UI manually i could not get a way to get them via code

with the values on helm

 helm get values clearml-agent -n clearml-prod
USER-SUPPLIED VALUES:
agentk8sglue:
  apiServerUrlReference: 

  clearmlcheckCertificate: false
  createQueueIfNotExists: true
  fileServerUrlReference: 

  image:
    pullPolicy: Always
    repository: allegroai/clearml-agent-k8s-base
    tag: latest
  queue: default
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi
  webServerUrlReference: 

clearml:
  agentk8sglueKey: 8888TMDLWYY7ZQJJ0I7R2X2RSP8XFT
  agentk8sglueSecret: oNODbBkDGhcDscTENQyr-GM0cE8IO7xmpaPdqyfsfaWearo1S8EQ8eBOxu-opW8dVUU
  clearmlConfig: |-
    api {
        web_server: 

        api_server: 

        files_server: 

        credentials {
            "access_key" = "8888TMDLWYY7ZQJJ0I7R2X2RSP8XFT"
            "secret_key" = "oNODbBkDGhcDscTENQyr-GM0cE8IO7xmpaPdqyfsfaWearo1S8EQ8eBOxu-opW8dVUU"
        }
    }
sessions:
  externalIP: 192.168.70.211
  maxServices: 5
  startingPort: 30100
  svcType: NodePort
jcarvalho@kharrinhao:~$

Please replace those credentials on the Agent and try upgrading the helm release

Yes i am using those, they are hardcoded ones cause i will on a later stage generate them via a secure method

Hi! Im using just a plain Kubernetes cluster (kubeadm) running on Proxmox VM, and im using Argo to deploy the helm, in order to standarize it Let me know if you need any more details!

Hi @<1811208768843681792:profile|BraveGrasshopper38> , following up on your last message, are you running in an OpenShift k8s cluster?

The value field is a default argo falls back into if i dont provide any

I had no issues deploying via the Github but helm is quite more confusing

Also, in order to simplify the installation, can you use a simpler version of your values for now, something like this should work:

agentk8sglue:
  apiServerUrlReference: 

  clearmlcheckCertificate: false
  createQueueIfNotExists: true
  fileServerUrlReference: 

  queue: default
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi
  webServerUrlReference: 

clearml:
  agentk8sglueKey: <NEW_KEY>
  agentk8sglueSecret: <NEW_SECRET>
sessions:
  externalIP: 192.168.70.211
  maxServices: 5
  startingPort: 30100
  svcType: NodePort

Python regex error in k8s glue agent :

sre_constants.error: bad inline flags: cannot turn on global flag at position 92

• Issue is in clearml-agent k8s glue codebase (Python 3.6 compatibility)
• Not configuration-related - persists across different HOCON formats
• Affects image tags: 1.24-21 , 1.24-23 , latest

Cause when i check it references to 3y ago and i am following this: None

Oh, okay, not sure this will be the only issue but you'll need these credentials to be valid, since they are used by the ClearML Agent to connect to the ClearML Server 🙂
The easiest way to generate credentials is to open the ClearML UI in the browser, login with an Admin user, then navigate to the Settings located on the top right corner when clicking on the user icon. From there go to "Workspace" and click "Create new credentials" and use the value provided

Hey! @<1729671499981262848:profile|CooperativeKitten94> Is there any tips you can give me on this?

It seems like the most recent version supported for kubernetes is clearml-agent==1.9.2?

thanks again!

Oh no worries, I understand 😄
Sure, if you could share the whole values and configs you're using to run both the server and agent that would be useful.
Also what about other Pods from the ClearML server, are there any other crash or similar error referring to a read-only filesystem? Are the server and agent installed on the same K8s node?