Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey, Loving Clearml So Far. I Create An Agent With 1 Gpu And I Am Sending A Task To It. But It Says That It Couldn’T Create A Docker With Gpu Access. How Can I Fix That?

Hey, loving ClearML so far.

I create an agent with 1 gpu and I am sending a task to it. But it says that it couldn’t create a docker with gpu access.
How can I fix that?
` 1664547202698 adamastor:gpu1 INFO Executing: ['docker', 'run', '-t', '--gpus', '"device=1"', '-l', 'clearml-worker-id=adamastor:gpu1', '-l', 'clearml-parent-worker-id=adamastor:gpu1', '-e', 'CLEARML_WORKER_ID=adamastor:gpu1', '-e', 'CLEARML_DOCKER_IMAGE=nvcr.io/nvidia/pytorch:22.08-py3', '-e', 'CLEARML_TASK_ID=c7880c62222d4237ac8306046b266e05', '-v', '/tmp/.clearml_agent.rxaullv3.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/tmp/clearml_agent.ssh.xc1zvtgd:/root/.ssh', '-v', '/home/adamastor/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/adamastor/.clearml/pip-cache:/root/.cache/pip', '-v', '/home/adamastor/.clearml/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/home/adamastor/.clearml/cache:/clearml_agent_cache', '-v', '/home/adamastor/.clearml/vcs-cache:/root/.clearml/vcs-cache', '--rm', 'nvcr.io/nvidia/pytorch:22.08-py3', 'bash', '-c', 'echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id c7880c62222d4237ac8306046b266e05']

1664547207757 adamastor:gpu1 DEBUG docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]. `

  
  
Posted 2 years ago
Votes Newest

Answers 2


I am doing clearml-agent --docker … --foreground --gpus 1

  
  
Posted 2 years ago

The issue was that nvidia-docker2 was not installed on the machine where I was trying to run the agent. Following this guide fixed it:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

  
  
Posted 2 years ago
1K Views
2 Answers
2 years ago
one year ago
Tags