Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I’M Trying To Run A Clearml Pipeline. I’M Using The "

Hi everyone,

I’m trying to run a ClearML pipeline. I’m using the " Pipeline from function" script and running it with the ClearML agent. However, the run fails even though I don’t get any errors in the terminal.

The following packages were successfully installed: Successfully installed Pillow-10.4.0 SQLAlchemy-2.0.34 annotated-types-0.7.0 boto3-1.35.16 botocore-1.35.19 clearml-1.16.4 exceptiongroup-1.2.2 greenlet-3.1.0 iniconfig-2.0.0 jmespath-1.0.1 joblib-1.4.2 pandas-2.2.2 pluggy-1.5.0 pydantic-2.7.4 pydantic-core-2.18.4 pydantic-settings-2.5.2 pytest-7.4.4 python-dotenv-1.0.1 pytz-2024.2 s3transfer-0.10.2 scikit-learn-1.5.1 scipy-1.13.1 structlog-24.4.0 tenacity-9.0.0 threadpoolctl-3.5.0 typing-extensions-4.12.2 tzdata-2024.1

Also, the virtual environment was added to the cache:
Adding venv into cache: /root/.clearml/venvs-builds/3.9

Despite this, the task is failing. Could someone help me with this?

Thanks!

  
  
Posted one month ago
Votes Newest

Answers 9


The clearML agent works within the Kubernetes cluster

  
  
Posted one month ago

Hi @<1669152726245707776:profile|ManiacalParrot65> , is this a specific task or the controller?

  
  
Posted one month ago

Its actually happens in both sometimes in the pipeline task and sometimes in pipeline controller

  
  
Posted one month ago

I just found out that ClearML Agent has a service mode. However, I'm currently using ClearML Agent with a Helm chart on Kubernetes (K8s). How can I start the agent in service mode in this setup?

  
  
Posted one month ago

Screenshot of the problem
image

  
  
Posted one month ago

@<1523701070390366208:profile|CostlyOstrich36> can you help me with that? I can provide you more information if you need 🙌

  
  
Posted one month ago

@<1669152726245707776:profile|ManiacalParrot65> could you please send your values file override for the Agent helm chart?

  
  
Posted one month ago

@<1729671499981262848:profile|CooperativeKitten94> here:

None

global:
imageRegistry: "docker-proxy.nexmart.com:5000"
clearml:
existingAgentk8sglueSecret: "clearml-agent-secret"
existingClearmlConfigSecret: "clearml-agent-secret"
agentk8sglue:
defaultContainerImage: "repo.nexmart.com:5000/nm-container-python:3.9"
apiServerUrlReference: " None "
fileServerUrlReference: " None "
webServerUrlReference: " None "

Use SA from default agent

serviceExistingAccountName: "test-clearml-clearml-agent-sa"
queue: "services"
initContainers:
resources:
requests:
memory: "50M"
cpu: "50m"
limits:
memory: "200M"
cpu: "1"
resources:
requests:
memory: "50M"
cpu: "10m"
limits:
memory: "500M"
cpu: "500m"
basePodTemplate:
env:
- name: "GL_PACA_PIPELINE_TRIGGER_TOKEN"
valueFrom:
secretKeyRef:
key: "gl_paca_trigger_token"
name: "clearml-agent-mlops-secret"
resources:
requests:
memory: "50M"
cpu: "10m"
limits:
memory: "750M"
cpu: "500m"

  
  
Posted one month ago

@<1729671499981262848:profile|CooperativeKitten94> I still have this problem even with other pipelines, can you please help me?

  
  
Posted one month ago
88 Views
9 Answers
one month ago
one month ago
Tags
Similar posts