Hi Everyone, I have a question on how to disable running pipelines in hybrid mode, so that agent only considers the code from git and not from local repo while running a task remotely.
I am trying to run a demo pipeline remotely using clearml-agent. Somehow agent is picking up uncommitted changes from the code which have not been pushed to git repository yet and neither are they part of docker image required to run the pipeline.
My agent setup as as follows:
The clearml.conf file for the agent
agent {
docker_force_pull: true
vcs_cache {
enabled: false # Disable VCS caching
}
}
sdk: {
development: {
store_code_diff_from_remote: false # Disable syncing code diffs from remote
store_uncommitted_code_diff: false
}
}
task {
# Disable Git tracking and diff logging
force_git: false
vcs_diff: ""
}
And I'm starting the agent using
export CLEARML_CONFIG_FILE=clearml.conf
clearml-agent daemon --docker python:3.11.9-slim-bullseye --queue=default --foreground --cpu-only --git-user=*** --git-pass=*****
On start, the agent has the setting, I have set up using above configs.
The pipeline repository has the following configurations
The clearml.conf file
api {
web_server:
api_server:
files_server:
}
sdk {
apply_environment: true
development {
default_output_uri: ${CLEARML_DEFAULT_OUTPUT_URI}
store_uncommitted_code_diff: false
store_uncommitted_code_diff: false
venv_update_enabled: true
venv_update: false
}
environment {
PYTHONPATH: ${PYTHONPATH}
AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT}
AZURE_STORAGE_KEY=${AZURE_STORAGE_KEY}
CLEARML_API_ACCESS_KEY=${CLEARML_API_ACCESS_KEY}
CLEARML_API_SECRET_KEY=${CLEARML_API_SECRET_KEY}
CLEARML_DEFAULT_OUTPUT_URI=${CLEARML_DEFAULT_OUTPUT_URI}
CLEARML_TASK_NO_REUSE=${CLEARML_TASK_NO_REUSE}
CLEARML_CACHE_DIR=${CLEARML_CACHE_DIR}
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL: true
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL: true
}
azure.storage {
containers: [
{
account_name: ${AZURE_STORAGE_ACCOUNT}
account_key: ${AZURE_STORAGE_KEY}
container_name: clearml
}
]
}
}
task {
# Completely disable Git tracking (no repo info, no uncommitted changes)
force_git: false
# Additionally, disable diff logging just in case
vcs_diff: ""
}
And my demo pipeline is as follows:
import os
from clearml import PipelineDecorator
from configs.clearml_pipeline_params import (
clearml_component_params,
clearml_pipeline_params,
clearml_configs,
)
@PipelineDecorator.component(
**clearml_component_params, # Unpacking the common parameters
return_values="component1_status",
)
def component1(x):
print("test uncommitedd changes if picked")
print(f'queue name ==> {clearml_configs["CLEARML_PIPELINES_QUEUE"]}')
print(f'dockerimage name ==> {clearml_configs["CLEARML_DOCKER"]}')
print(f'repo name ==> {clearml_configs["CLEARML_REPO"]}')
return f"{x} component 1 done"
@PipelineDecorator.component(
**clearml_component_params, # Unpacking the common parameters
return_values="component2_status",
)
def component2(x):
return f"{x} component 2 done"
@PipelineDecorator.component(
**clearml_component_params, # Unpacking the common parameters
return_values="component3_status",
)
def component3(x):
return f"{x} component 3 done"
@PipelineDecorator.pipeline(
**clearml_pipeline_params,
name="clearml-demo-pipleine-remote",
project="clearml-demo-project-remote",
return_value="demo status",
)
def run_test_demo(inp: str):
print(f'queue name ==> {clearml_configs["CLEARML_PIPELINES_QUEUE"]}')
c1 = component1(inp)
c2 = component2(c1)
c3 = component3(c2)
return "demo done"
if __name__ == "__main__":
final_progress = run_test_demo("input")
where clearml pipeline configs are
clearml_configs = {# is a dictionary of values}
# Define reusable configurations as a dictionary
clearml_component_params = {
"cache": False,
"execution_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
"docker": clearml_configs["CLEARML_DOCKER"],
"repo": clearml_configs["CLEARML_REPO"],
"repo_branch": clearml_configs["CLEARML_REPO_BRANCH"],
"docker_args": clearml_configs["CLEARML_DOCKER_ARGS"],
}
clearml_pipeline_params = {
"default_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
"pipeline_execution_queue": clearml_configs["CLEARML_PIPELINES_QUEUE"],
"docker": clearml_configs["CLEARML_DOCKER"],
"repo": clearml_configs["CLEARML_REPO"],
"repo_branch": clearml_configs["CLEARML_REPO_BRANCH"],
"docker_args": clearml_configs["CLEARML_DOCKER_ARGS"],
"skip_global_imports": False,
}
my agent logs are
Executing task id [0d****]:
repository = https://****@dev.azure.com/****/****/_git/repo
branch = feature/clearml_integration
version_num = e55*****
tag =
docker_cmd = ****.azurecr.io/***:repo -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 -e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 -e
entry_point = src/pipelines/clearml_demo_pipeline_remote.py
working_dir = .
cloning: https://****@dev.azure.com/****/****/_git/repo
pulling git
pulling git completed
Note: switching to 'e55*****'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at e5** mounting volume explicitly in docker args for clearml
type: git
url: https://****@dev.azure.com/****/****/_git/repo
branch: HEAD
commit: e5******
root: /root/.clearml/venvs-builds/task_repository/repo
Applying uncommitted changes
I am not sure why the agent is picking up uncommitted changes.
Is there anyone who might be aware what I'm missing here? Greatly appreciate the help 🙏
