Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys, I’M Trying To Deploy Clearml To K8S. When I Try To Execute A Pipeline (From The Examples) On A Remote Worker, I Get An

Hi guys, I’m trying to deploy ClearML to K8s.
When I try to execute a pipeline (from the examples) on a remote worker, I get an "aborted" run status and see the following logs in the agent:
ERROR: Could not push back task [87a84fa22a3d4e3fbc82db58ccd4a070] to k8s pending queue k8s_scheduler [9680e8b47e644286895ad8760937ee51], error: Validation error (Cannot skip setting execution queue for a task that is not enqueued or does not have execution queue)

any idea how to fix?

  
  
Posted 20 days ago
Votes Newest

Answers 3


found solution None

  
  
Posted 20 days ago

clearml:
  existingAgentk8sglueSecret: "clearml-agent-keys"
  existingClearmlConfigSecret: "clearml-agent-conf"

agentk8sglue:
  queue: "cpu"
  createQueueIfNotExists: true

  replicaCount: 1
  apiServerUrlReference: "
"
  fileServerUrlReference: "
"
  webServerUrlReference: "
"

  nodeSelector:
    custom.pool: clearml-pool
  tolerations:
    - key: "custom.pool"
      operator: "Equal"
      value: "clearml-pool"
      effect: "NoSchedule"
    - key: "custom.pool"
      operator: "Equal"
      value: "clearml-pool"
      effect: "NoExecute"

  extraEnvs:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef: { name: clearml-s3, key: AWS_ACCESS_KEY_ID }
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef: { name: clearml-s3, key: AWS_SECRET_ACCESS_KEY }

  basePodTemplate:
    resources:
      limits:
        cpu: "10"
        memory: "64Gi"
    nodeSelector:
      custom.pool: clearml-pool
    tolerations:
      - key: "custom.pool"
        operator: "Equal"
        value: "clearml-pool"
        effect: "NoSchedule"
      - key: "custom.pool"
        operator: "Equal"
        value: "clearml-pool"
        effect: "NoExecute"
  
  
Posted 20 days ago

queue cpu created in ui

  
  
Posted 20 days ago