Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Cannot Get Clearml-Agent With Docker Containers To Work. Clearml Uses

I cannot get clearml-agent with docker containers to work. clearml uses docker run -v /home/.ssh:/root/.ssh to mount the ssh profile into the container. However, the files cannot be used in the container, since the owner of the files is the host user and not the docker root.

Bad owner or permissions on /root/.ssh/config fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Repository cloning failed: Command '['clone', 'git@github.com:mygroup/core.git', '/root/.clearml/vcs-cache/core.git.1c81c63da4814d427beb8f31bd60f95e/core.git', '--quiet', '--recursive']' returned non-zero exit status 128.

  
  
Posted 3 years ago
Votes Newest

Answers 30


-rw------- 1 1000 1000 0 Feb 28 23:41 config

  
  
Posted 3 years ago

However, I have not yet found a flexible solution other than ssh-agent forwarding.

And is it working?

  
  
Posted 2 years ago

I ran
docker run -it -v /home/hostuser/.ssh/:/root/.ssh ubuntu:18.04but cloning does not work and this is what ls -lah /root/.ssh gives inside the docker container:
-rw------- 1 1001 1001 1.5K Apr 8 12:28 authorized_keys -rw-rw-r-- 1 1001 1001 208 Apr 29 09:15 config -rw------- 1 1001 1001 432 Apr 8 12:53 id_ed25519 -rw-r--r-- 1 1001 1001 119 Apr 8 12:53 id_ed25519.pub -rw------- 1 1001 1001 432 Apr 29 09:16 id_gitlab -rw-r--r-- 1 1001 1001 119 Apr 29 09:25 id_gitlab.pub -rw-rw-r-- 1 1001 1001 3.1K Apr 29 09:33 known_hosts

  
  
Posted 2 years ago

(I think it is the empty config file)

  
  
Posted 3 years ago

Yea, but before in my original setup the config file was filled. I just added some lines to the config and now the error is back.

  
  
Posted 3 years ago

size 0

  
  
Posted 3 years ago

Yes

  
  
Posted 3 years ago

In the beginning my config file was not empty 😕

  
  
Posted 3 years ago

As in if it was not empty it would work?

  
  
Posted 3 years ago

could you remove it and test ?

  
  
Posted 3 years ago

👍

  
  
Posted 3 years ago

Now it works.

  
  
Posted 3 years ago

Maybe the problem is that I do not start my docker containers from the root user, so 1001 is a mapping inside the docker to my actual user. Could it be that on the host the owner if your .ssh files is called root ?

  
  
Posted 2 years ago

However, I have not yet found a flexible solution other than ssh-agent forwarding.

  
  
Posted 2 years ago

So it seems to be definitely a problem with docker and not with clearml. However, I do not get, why it works for you but on none of my machine (all Ubuntu 20.04 with docker 20.10)

  
  
Posted 2 years ago

that was an odd error

  
  
Posted 3 years ago

and config is empty ?

  
  
Posted 3 years ago

Yes (Mine isn't and it is working 🙂 )

  
  
Posted 3 years ago

Nothing changes, still bad owner or permissions.

  
  
Posted 3 years ago

hurray

  
  
Posted 3 years ago

For everyone who had the patience to read through everything, here is my solution to make clearml work with ssh-agent forwarding in the current version:
Start and ssh-agent Add ssh keys with ssh-add to agent echo $SSH_AUTH_SOCK and paste into clearml.conf as here: https://github.com/allegroai/clearml-agent/issues/45#issuecomment-779302144 (replace $SSH_AUTH_SOCKET with actually value) Move all the files except known_hosts out of ~/.ssh of the clearml-agent workstation. Start the clearml-agent.

  
  
Posted 2 years ago

btw: With the ssh agent forwarding I do not have any issues ( https://github.com/allegroai/clearml-agent/issues/45 )

  
  
Posted 3 years ago

Can you clone the git with the .ssh credentials on the host machine ?
If so, can you do the same manually inside a docker (i.e. spin a docker with mount -v /home/hostuser/.ssh:/root/.ssh) ?

  
  
Posted 2 years ago

Yea. Not using the config file does not seem like a good long-term solution for me. However, I still have no idea, why this error happens. But enough for today. Thank you a lot for your help!

  
  
Posted 3 years ago

Anyways, from my google search it seems that this is not something that is intuitive to fix.
Is there any progress on this: https://github.com/allegroai/clearml-agent/issues/45 ? This works on all my machines 🙂

  
  
Posted 2 years ago

The docker run command of the agent includes '-v', '/tmp/clearml_agent.ssh.8owl7uf2:/root/.ssh' and the file permissions are exactly the same.

  
  
Posted 2 years ago

AgitatedDove14 fyi I think this is the issue I have: https://stackoverflow.com/a/65526944/3038183

  
  
Posted 2 years ago

agent-forwarding is working just like your described here: https://github.com/allegroai/clearml-agent/issues/45 Looking forward to not having to use the absolute path in the future 🙂

  
  
Posted 2 years ago

So just tried again and still it does not work.
This is what is in .ssh on my clearml-agent
-rw------- 1 tim tim 1,5K Apr 8 14:28 authorized_keys -rw-rw-r-- 1 tim tim 208 Apr 29 11:15 config -rw------- 1 tim tim 432 Apr 8 14:53 id_ed25519 -rw-r--r-- 1 tim tim 119 Apr 8 14:53 id_ed25519.pub -rw------- 1 tim tim 432 Apr 29 11:16 id_gitlab -rw-r--r-- 1 tim tim 119 Apr 29 11:25 id_gitlab.pub -rw-rw-r-- 1 tim tim 3,1K Apr 29 11:33 known_hosts

  
  
Posted 2 years ago

When I change the owner and the group of the files to root it works.

  
  
Posted 2 years ago
590 Views
30 Answers
3 years ago
one year ago
Tags
Similar posts