Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I’M Trying To Run A Clearml Pipeline. I’M Using The "

Hi everyone,

I’m trying to run a ClearML pipeline. I’m using the " Pipeline from function" script and running it with the ClearML agent. However, the run fails even though I don’t get any errors in the terminal.

The following packages were successfully installed: Successfully installed Pillow-10.4.0 SQLAlchemy-2.0.34 annotated-types-0.7.0 boto3-1.35.16 botocore-1.35.19 clearml-1.16.4 exceptiongroup-1.2.2 greenlet-3.1.0 iniconfig-2.0.0 jmespath-1.0.1 joblib-1.4.2 pandas-2.2.2 pluggy-1.5.0 pydantic-2.7.4 pydantic-core-2.18.4 pydantic-settings-2.5.2 pytest-7.4.4 python-dotenv-1.0.1 pytz-2024.2 s3transfer-0.10.2 scikit-learn-1.5.1 scipy-1.13.1 structlog-24.4.0 tenacity-9.0.0 threadpoolctl-3.5.0 typing-extensions-4.12.2 tzdata-2024.1

Also, the virtual environment was added to the cache:
Adding venv into cache: /root/.clearml/venvs-builds/3.9

Despite this, the task is failing. Could someone help me with this?

Thanks!

  
  
Posted one year ago
Votes Newest

Answers 9


@<1729671499981262848:profile|CooperativeKitten94> here:

None

global:
imageRegistry: "docker-proxy.nexmart.com:5000"
clearml:
existingAgentk8sglueSecret: "clearml-agent-secret"
existingClearmlConfigSecret: "clearml-agent-secret"
agentk8sglue:
defaultContainerImage: "repo.nexmart.com:5000/nm-container-python:3.9"
apiServerUrlReference: " None "
fileServerUrlReference: " None "
webServerUrlReference: " None "

Use SA from default agent

serviceExistingAccountName: "test-clearml-clearml-agent-sa"
queue: "services"
initContainers:
resources:
requests:
memory: "50M"
cpu: "50m"
limits:
memory: "200M"
cpu: "1"
resources:
requests:
memory: "50M"
cpu: "10m"
limits:
memory: "500M"
cpu: "500m"
basePodTemplate:
env:
- name: "GL_PACA_PIPELINE_TRIGGER_TOKEN"
valueFrom:
secretKeyRef:
key: "gl_paca_trigger_token"
name: "clearml-agent-mlops-secret"
resources:
requests:
memory: "50M"
cpu: "10m"
limits:
memory: "750M"
cpu: "500m"

  
  
Posted one year ago

Screenshot of the problem
image

  
  
Posted one year ago

@<1729671499981262848:profile|CooperativeKitten94> I still have this problem even with other pipelines, can you please help me?

  
  
Posted one year ago

The clearML agent works within the Kubernetes cluster

  
  
Posted one year ago

@<1523701070390366208:profile|CostlyOstrich36> can you help me with that? I can provide you more information if you need 🙌

  
  
Posted one year ago

@<1669152726245707776:profile|ManiacalParrot65> could you please send your values file override for the Agent helm chart?

  
  
Posted one year ago

Its actually happens in both sometimes in the pipeline task and sometimes in pipeline controller

  
  
Posted one year ago

Hi @<1669152726245707776:profile|ManiacalParrot65> , is this a specific task or the controller?

  
  
Posted one year ago

I just found out that ClearML Agent has a service mode. However, I'm currently using ClearML Agent with a Helm chart on Kubernetes (K8s). How can I start the agent in service mode in this setup?

  
  
Posted one year ago
854 Views
9 Answers
one year ago
one year ago
Tags
Similar posts