Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Trying To Use The Clearml-Agent In Docker Mode To Run An Experiment, But It Seems To Fail Passing The Clearml.Conf File To The Docker Container:

Hi, I am trying to use the clearml-agent in docker mode to run an experiment, but it seems to fail passing the clearml.conf file to the docker container:
` Executing: ['docker', 'run', '-t', '-l', 'clearml-worker-id=xyz', '-l', 'clearml-parent-worker-id=xyz', '-e', 'CLEARML_WORKER_ID=xyz', '-e', 'CLEARML_DOCKER_IMAGE=python:3.6', '-e', 'CLEARML_TASK_ID=xyz', '-v', '/tmp/.clearml_agent.qqa1_3og.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.z2mp4_g1:/.ssh', '-v', '/root/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/root/.clearml/pip-cache:/root/.cache/pip', '-v', '/root/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/root/.clearml/cache:/clearml_agent_cache', '-v', '/root/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', 'python:3.6', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; cp -Rf /.ssh -T ~/.ssh ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip==20.2.3" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=none $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id xyz']

[...]

Successfully installed PyYAML-6.0 attrs-22.2.0 certifi-2022.12.7 charset-normalizer-2.0.12 clearml-agent-1.5.1 distlib-0.3.6 filelock-3.4.1 furl-2.1.3 idna-3.4 importlib-metadata-4.8.3 importlib-resources-5.4.0 jsonschema-3.2.0 orderedmultidict-1.0.1 pathlib2-2.3.7.post1 platformdirs-2.4.0 psutil-5.9.4 pyjwt-2.4.0 pyparsing-3.0.9 pyrsistent-0.18.0 python-dateutil-2.8.2 requests-2.27.1 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.13 virtualenv-20.17.1 zipp-3.6.0
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python3.6 -m pip install --upgrade pip' command.
cp: -r not specified; omitting directory '/tmp/clearml.conf'
Using built-in ClearML default key/secret

clearml_agent: ERROR: Could not find host server definition (missing ~/clearml.conf or Environment CLEARML_API_HOST)
To get started with ClearML: setup your own clearml-server, or create a free account at and run clearml-agent init

Process failed, exit code 1 I am using both the clearml.conf file and some environment variables (such as CLEARML_API_ACCESS_KEY, CLEARML_WEB_HOST). I've set CLEARML_LOG_ENVIRONMENT='' ` . Am I doing something wrong? I am using clearml-agent version 1.4.1 (version 1.5.1 is installed in the docker container)

  
  
Posted one year ago
Votes Newest

Answers 13


I am using an old version of the aws autoscaler, so the instance has the following user data executed:
echo "{clearml_conf}" >>/root/clearml.conf ... python -m clearml_agent --config-file '/root/clearml.conf' daemon --detached --queue '{queue}' --docker --cpu-only

  
  
Posted one year ago

Ok so the problem was indeed the way docker was installed (with snap)

  
  
Posted one year ago

So it seems like it doesn't copy /root/clearml.conf and it doesn't pass the environment variables (CLEARML_API_HOST, CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY)

  
  
Posted one year ago

might be worth documenting 😄

  
  
Posted one year ago

This works well when I run the agent in virtualenv mode (remove --docker )

  
  
Posted one year ago

And you made sure to run clearml-agent init on the machine or to implement the clearml.conf manually?

  
  
Posted one year ago

How are you running the agent? Is it an ec2 machine or bare metal you have?

  
  
Posted one year ago

The rest of the configuration is set with env variables

  
  
Posted one year ago

in my clearml.conf, I only have:
sdk.aws.s3.region = eu-central-1 sdk.aws.s3.use_credentials_chain = true agent.package_manager.pip_version = "==20.2.3"

  
  
Posted one year ago

(docker was install with sudo snap install docker )

  
  
Posted one year ago

JitteryCoyote63 CostlyOstrich36 want to echo the statement that documentation should warn against the snap installation of docker.

I also want to highlight that the snap installation has major problems when it comes to volume mounting of priviliged paths etc. On my baremetal instance, even the docker-compose up started having problems which was resolved only when I switched to the docker documentation's installation.

  
  
Posted one year ago

When installed with http://get.docker.com , it works

  
  
Posted one year ago

in an ec2 machine

  
  
Posted one year ago
2K Views
13 Answers
one year ago
one year ago
Tags