Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Correct Way To Configure Ssh Authentication For Git In Agent With Docker Mode

Correct way to configure ssh authentication for git in agent with docker mode
I am struggling with configuring ssh authentication in docker mode with agent.
My setup is that on machine with agent I have proper ssh config and in pip mode all works fine.
Here are my mounts for docker from debug log (plus envs related to ssh):
['docker', 'run', '-t', '--gpus', '"device=0,1"', '-v', '/tmp/ssh-XXXXXXxU2tbn/agent.963334:/tmp/ssh-XXXXXXxU2tbn/agent.963334', '-e', 'SSH_AUTH_SOCK=/tmp/ssh-XXXXXXxU2tbn/agent.963334', '-v', '/home/nino/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent.yz2vl5mi.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/home/nino/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/nino/.clearml/pip-cache:/home/testuser/.cache/pip', '-v', '/home/nino/.clearml/pip-download-cache:/home/testuser/.clearml/pip-download-cache', '-v', '/home/nino/.clearml/cache:/clearml_agent_cache', '-v', '/home/nino/.clearml/vcs-cache:/home/testuser/.clearml/vcs-cache', '-v', '/home/nino/.clearml/venvs-cache:/root/.clearml/venvs-cache',And config param from agent re ssh folder:
agent.docker_internal_mounts.ssh_folder = /home/testuser/.sshTo put it simple I want to mount host /home/nina/.ssh to /home/testuser/.ssh like it is done for pip-cache and so on but it does not do the stuff.

I have made it through explicit mount in extra_docker_arguments but I have a feeling that it was not designed for that purpose.

  
  
Posted 2 years ago
Votes Newest

Answers 7


GentleSwallow91 notice this part:
Hi Martin. Sorry - missed your reply.
Yeap I am aware that docker_internal_mounts is inside agent section.
'-v', '/tmp/ssh-XXXXXXnfYTo5/agent.8946:/tmp/ssh-XXXXXXnfYTo5/agent.8946', '-e', 'SSH_AUTH_SOCK=/tmp/ssh-XXXXXXnfYTo5/agent.8946',

It is creating a copy of the ssh folder and setting the SSH_AUTH_SOCK env to it. You can just map the entire ssh folder automatically by un-setting SSH_AUTH_SOCK before running the agent.
SSH_AUTH_SOCK= clearml-agent ...

  
  
Posted 2 years ago

Hi Martin. Sorry - missed your reply.
Yeap I am aware that docker_internal_mounts is inside agent section.
Here is the actual docker command from the log
INFO Executing: ['docker', 'run', '-t', '--gpus', '"device=0"', '-v', '/tmp/ssh-XXXXXXnfYTo5/agent.8946:/tmp/ssh-XXXXXXnfYTo5/agent.8946', '-e', 'SSH_AUTH_SOCK=/tmp/ssh-XXXXXXnfYTo5/agent.8946', '-l', 'clearml-worker-id=agent-gpu:gpu0', '-l', 'clearml-parent-worker-id=agent-gpu:gpu0', '-e', 'CLEARML_WORKER_ID=agent-gpu:gpu0', '-e', 'CLEARML_DOCKER_IMAGE=torch2022', '-e', 'CLEARML_TASK_ID=089d2a5d2b59443db53dfb5a884eb7f3', '-v', '/home/localuser/.gitconfig:/root/.gitconfig', '-v', '/tmp/.clearml_agent.twer6v9b.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/home/localuser/.clearml/apt-cache:/var/cache/apt/archives', '-v', '/home/localuser/.clearml/pip-cache:/home/testuser/.cache/pip', '-v', '/home/localuser/.clearml/pip-download-cache:/home/testuser/.clearml/pip-download-cache', '-v', '/home/localuser/.clearml/cache:/clearml_agent_cache', '-v', '/home/localuser/.clearml/vcs-cache:/home/testuser/.clearml/vcs-cache', '-v', '/home/localuser/.clearml/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'torch2022', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip==21.2.4" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=all $LOCAL_PYTHON -u -m clearml_agent execute --disable-monitoring --id 089d2a5d2b59443db53dfb5a884eb7f3']

  
  
Posted 2 years ago

and I run agent from local user and I would expect that settings to have effect -v /home/localuser/.ssh:/home/testuser/.ssh

It does not map it directly, it creates a temp copy in the host /tmp folder of the entire ".ssh" folder, than maps this folder inside the container:
https://github.com/allegroai/clearml-agent/blob/a5a797ec5e5e3e90b115213c0411a516cab60e83/clearml_agent/commands/worker.py#L3422
Notice that the "docker_internal_mounts" section is nested inside the "agent" section in the clearml.conf file.
Can you share the top Task log of a task your agent is running? (the first few lines contain the exact docker run command the agent is using)

  
  
Posted 2 years ago

Hi AgitatedDove14
Either I do smth wrong or it works only in theory 😃
Here are my mounts in agent clearml.conf
sdk_cache: "/clearml_agent_cache" ssh_folder: "/home/testuser/.ssh" pip_cache: "/home/testuser/.cache/pip" vcs_cache: "/home/testuser/.clearml/vcs-cache" venv_build: "/home/testuser/.clearml/venvs-builds" pip_download: "/home/testuser/.clearml/pip-download-cache"testuser is inside Docker container
and I run agent from local user and I would expect that settings to have effect -v /home/localuser/.ssh:/home/testuser/.ssh
but that does not happen...
Please advise

  
  
Posted 2 years ago

Here are my extra_docker_arguments that make the thing working:

GentleSwallow91 Nice!
BTW: in theory there should not need to be any need to add the specific: "-v","/home/nino/.ssh:/home/testuser/.ssh", the agent should do that automatically

  
  
Posted 2 years ago

Yeap. It is configured this way
force_git_ssh_protocol: trueBut I don't see the mount of .ssh
One thing though - my container is running on behalf of non-root user.
Here are my docker mounts: docker_internal_mounts { sdk_cache = /clearml_agent_cache # apt_cache = /var/cache/apt/archives ssh_folder = /home/testuser/.ssh pip_cache = /home/testuser/.cache/pip poetry_cache = /home/testuser/.cache/pypoetry vcs_cache = /home/testuser/.clearml/vcs-cache venv_build = /home/testuser/.clearml/venvs-builds pip_download = /home/testuser/.clearml/pip-download-cache }Host user has a different name from testuser
Here are my extra_docker_arguments that make the thing working:
extra_docker_arguments: ["-v","/home/nino/.ssh:/home/testuser/.ssh", "--privileged"]

  
  
Posted 2 years ago

I am struggling with configuring ssh authentication in docker mode

GentleSwallow91 Basically the agent will automatically mount the .ssh into the container , just make sure you set the following in the clearml.conf:
force_git_ssh_protocol: truehttps://github.com/allegroai/clearml-agent/blob/178af0dee84e22becb9eec8f81f343b9f2022630/docs/clearml.conf#L30

  
  
Posted 2 years ago