So it seems to be definitely a problem with docker and not with clearml. However, I do not get, why it works for you but on none of my machine (all Ubuntu 20.04 with docker 20.10)

For everyone who had the patience to read through everything, here is my solution to make clearml work with ssh-agent forwarding in the current version:
Start and ssh-agent Add ssh keys with ssh-add to agent echo $SSH_AUTH_SOCK and paste into clearml.conf as here: https://github.com/allegroai/clearml-agent/issues/45#issuecomment-779302144 (replace $SSH_AUTH_SOCKET with actually value) Move all the files except known_hosts out of ~/.ssh of the clearml-agent workstation. Start the clearml-agent.

However, I have not yet found a flexible solution other than ssh-agent forwarding.

In the beginning my config file was not empty 😕

Now it works.

Yes (Mine isn't and it is working 🙂 )

Yea. Not using the config file does not seem like a good long-term solution for me. However, I still have no idea, why this error happens. But enough for today. Thank you a lot for your help!

👍

-rw------- 1 1000 1000 0 Feb 28 23:41 config

(I think it is the empty config file)

Yes

could you remove it and test ?

hurray

agent-forwarding is working just like your described here: https://github.com/allegroai/clearml-agent/issues/45 Looking forward to not having to use the absolute path in the future 🙂

However, I have not yet found a flexible solution other than ssh-agent forwarding.

And is it working?

The docker run command of the agent includes '-v', '/tmp/clearml_agent.ssh.8owl7uf2:/root/.ssh' and the file permissions are exactly the same.

When I change the owner and the group of the files to root it works.

Can you clone the git with the .ssh credentials on the host machine ?
If so, can you do the same manually inside a docker (i.e. spin a docker with mount -v /home/hostuser/.ssh:/root/.ssh) ?

As in if it was not empty it would work?

AgitatedDove14 fyi I think this is the issue I have: https://stackoverflow.com/a/65526944/3038183

that was an odd error

Yea, but before in my original setup the config file was filled. I just added some lines to the config and now the error is back.

and config is empty ?

I ran
docker run -it -v /home/hostuser/.ssh/:/root/.ssh ubuntu:18.04but cloning does not work and this is what ls -lah /root/.ssh gives inside the docker container:
-rw------- 1 1001 1001 1.5K Apr 8 12:28 authorized_keys -rw-rw-r-- 1 1001 1001 208 Apr 29 09:15 config -rw------- 1 1001 1001 432 Apr 8 12:53 id_ed25519 -rw-r--r-- 1 1001 1001 119 Apr 8 12:53 id_ed25519.pub -rw------- 1 1001 1001 432 Apr 29 09:16 id_gitlab -rw-r--r-- 1 1001 1001 119 Apr 29 09:25 id_gitlab.pub -rw-rw-r-- 1 1001 1001 3.1K Apr 29 09:33 known_hosts

btw: With the ssh agent forwarding I do not have any issues ( https://github.com/allegroai/clearml-agent/issues/45 )

So just tried again and still it does not work.
This is what is in .ssh on my clearml-agent
-rw------- 1 tim tim 1,5K Apr 8 14:28 authorized_keys -rw-rw-r-- 1 tim tim 208 Apr 29 11:15 config -rw------- 1 tim tim 432 Apr 8 14:53 id_ed25519 -rw-r--r-- 1 tim tim 119 Apr 8 14:53 id_ed25519.pub -rw------- 1 tim tim 432 Apr 29 11:16 id_gitlab -rw-r--r-- 1 tim tim 119 Apr 29 11:25 id_gitlab.pub -rw-rw-r-- 1 tim tim 3,1K Apr 29 11:33 known_hosts

Maybe the problem is that I do not start my docker containers from the root user, so 1001 is a mapping inside the docker to my actual user. Could it be that on the host the owner if your .ssh files is called root ?

Anyways, from my google search it seems that this is not something that is intuitive to fix.
Is there any progress on this: https://github.com/allegroai/clearml-agent/issues/45 ? This works on all my machines 🙂

Nothing changes, still bad owner or permissions.

size 0