Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, I Have A Question On How To Disable Running Pipelines In Hybrid Mode, So That Agent Only Considers The Code From Git And Not From Local Repo While Running A Task Remotely. I Am Trying To Run A Demo Pipeline Remotely Using Clearml-Agent. Some

Hi Everyone, I have a question on how to disable running pipelines in hybrid mode, so that agent only considers the code from git and not from local repo while running a task remotely.

I am trying to run a demo pipeline remotely using clearml-agent. Somehow agent is picking up uncommitted changes from the code which have not been pushed to git repository yet and neither are they part of docker image required to run the pipeline.

My agent setup as as follows:

The clearml.conf file for the agent

agent {
    docker_force_pull: true
    vcs_cache {
        enabled: false  # Disable VCS caching
    }
}

sdk: {
    development: {
        store_code_diff_from_remote: false  # Disable syncing code diffs from remote
        store_uncommitted_code_diff: false

    }
}

task {
    # Disable Git tracking and diff logging
    force_git: false
    vcs_diff: ""
}

And I'm starting the agent using

export CLEARML_CONFIG_FILE=clearml.conf
clearml-agent daemon --docker python:3.11.9-slim-bullseye --queue=default --foreground --cpu-only --git-user=*** --git-pass=***** 

On start, the agent has the setting, I have set up using above configs.

The pipeline repository has the following configurations
The clearml.conf file

api {
web_server:

api_server:

files_server:

}


sdk {
    apply_environment: true
    development {
        default_output_uri: ${CLEARML_DEFAULT_OUTPUT_URI}
        store_uncommitted_code_diff: false
        store_uncommitted_code_diff: false
        venv_update_enabled: true
        venv_update: false
    }

    environment {
        PYTHONPATH: ${PYTHONPATH}
        AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT}
        AZURE_STORAGE_KEY=${AZURE_STORAGE_KEY}
        CLEARML_API_ACCESS_KEY=${CLEARML_API_ACCESS_KEY}
        CLEARML_API_SECRET_KEY=${CLEARML_API_SECRET_KEY}
        CLEARML_DEFAULT_OUTPUT_URI=${CLEARML_DEFAULT_OUTPUT_URI}
        CLEARML_TASK_NO_REUSE=${CLEARML_TASK_NO_REUSE}
        CLEARML_CACHE_DIR=${CLEARML_CACHE_DIR}
        CLEARML_AGENT_SKIP_PIP_VENV_INSTALL: true
        CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL: true
    }
    azure.storage {
        containers: [
            {
                account_name: ${AZURE_STORAGE_ACCOUNT}
                account_key: ${AZURE_STORAGE_KEY}
                container_name: clearml
            }
        ]
    }
}

task {
    # Completely disable Git tracking (no repo info, no uncommitted changes)
    force_git: false
    # Additionally, disable diff logging just in case
    vcs_diff: ""
}

And my demo pipeline is as follows:

import os
from clearml import PipelineDecorator
from configs.clearml_pipeline_params import (
    clearml_component_params,
    clearml_pipeline_params,
    clearml_configs,
)


@PipelineDecorator.component(
    **clearml_component_params,  # Unpacking the common parameters
    return_values="component1_status",
)
def component1(x):
    print("test uncommitedd changes if picked")
    print(f'queue name ==>  {clearml_configs["CLEARML_PIPELINES_QUEUE"]}')
    print(f'dockerimage name ==>  {clearml_configs["CLEARML_DOCKER"]}')
    print(f'repo name ==>  {clearml_configs["CLEARML_REPO"]}')
    return f"{x} component 1 done"


@PipelineDecorator.component(
    **clearml_component_params,  # Unpacking the common parameters
    return_values="component2_status",
)
def component2(x):
    return f"{x} component 2 done"


@PipelineDecorator.component(
    **clearml_component_params,  # Unpacking the common parameters
    return_values="component3_status",
)
def component3(x):
    return f"{x} component 3 done"


@PipelineDecorator.pipeline(
    **clearml_pipeline_params,
    name="clearml-demo-pipleine-remote",
    project="clearml-demo-project-remote",
    return_value="demo status",
)
def run_test_demo(inp: str):
    print(f'queue name ==>  {clearml_configs["CLEARML_PIPELINES_QUEUE"]}')
    c1 = component1(inp)
    c2 = component2(c1)
    c3 = component3(c2)

    return "demo done"


if __name__ == "__main__":
    final_progress = run_test_demo("input")

where clearml pipeline configs are

clearml_configs = {# is a dictionary of values}

# Define reusable configurations as a dictionary
clearml_component_params = {
    "cache": False,
    "execution_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
    "docker": clearml_configs["CLEARML_DOCKER"],
    "repo": clearml_configs["CLEARML_REPO"],
    "repo_branch": clearml_configs["CLEARML_REPO_BRANCH"],
    "docker_args": clearml_configs["CLEARML_DOCKER_ARGS"],
}

clearml_pipeline_params = {
    "default_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
    "pipeline_execution_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
    "docker": clearml_configs["CLEARML_DOCKER"],
    "repo": clearml_configs["CLEARML_REPO"],
    "repo_branch": clearml_configs["CLEARML_REPO_BRANCH"],
    "docker_args": clearml_configs["CLEARML_DOCKER_ARGS"],
    "skip_global_imports": False,
}

my agent logs are

Executing task id [0d****]:
repository = https://****@dev.azure.com/****/****/_git/repo
branch = feature/clearml_integration
version_num = e55*****
tag =
docker_cmd = ****.azurecr.io/***:repo -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 -e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 -e
entry_point = src/pipelines/clearml_demo_pipeline_remote.py
working_dir = .

cloning: https://****@dev.azure.com/****/****/_git/repo

pulling git
pulling git completed
Note: switching to 'e55*****'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at e5** mounting volume explicitly in docker args for clearml
type: git
url: https://****@dev.azure.com/****/****/_git/repo
branch: HEAD
commit: e5******
root: /root/.clearml/venvs-builds/task_repository/repo
Applying uncommitted changes

I am not sure why the agent is picking up uncommitted changes.
Is there anyone who might be aware what I'm missing here? Greatly appreciate the help 🙏

  
  
Posted 2 months ago
Votes Newest

Answers 6


Hi @<1731483438642368512:profile|LoosePigeon2> , you need to set the following:

sdk: {
    development: {
        store_code_diff_from_remote: false
        store_uncommitted_code_diff: false

On the machine you're running your pipeline from

  
  
Posted 2 months ago

Hi, Thank you for the reply. However, I set these in my repo clearml.conf settings as suggested and it where I'm triggering that pipeline from, the same machine. it doesn't seem to work.

  
  
Posted 2 months ago

yes, it is there in my home folder from where the code is running

  
  
Posted 2 months ago

This will disable storing the uncommitted changes

  
  
Posted 2 months ago

What do you mean by repo cleaml.conf ?
Just to make sure, clearml.conf by default should sit in your home folder

  
  
Posted 2 months ago

You can also set just specific commit/branch in the component object - None

  
  
Posted 2 months ago