Nothing changes, still bad owner or permissions.

For everyone who had the patience to read through everything, here is my solution to make clearml work with ssh-agent forwarding in the current version:
Start and ssh-agent Add ssh keys with ssh-add to agent echo $SSH_AUTH_SOCK and paste into clearml.conf as here: https://github.com/allegroai/clearml-agent/issues/45#issuecomment-779302144 (replace $SSH_AUTH_SOCKET with actually value) Move all the files except known_hosts out of ~/.ssh of the clearml-agent workstation. Start the clearml-agent.

So it seems to be definitely a problem with docker and not with clearml. However, I do not get, why it works for you but on none of my machine (all Ubuntu 20.04 with docker 20.10)

Now it works.

However, I have not yet found a flexible solution other than ssh-agent forwarding.

And is it working?

👍

Yea, but before in my original setup the config file was filled. I just added some lines to the config and now the error is back.

However, I have not yet found a flexible solution other than ssh-agent forwarding.

size 0

Maybe the problem is that I do not start my docker containers from the root user, so 1001 is a mapping inside the docker to my actual user. Could it be that on the host the owner if your .ssh files is called root ?

When I change the owner and the group of the files to root it works.

In the beginning my config file was not empty 😕

Anyways, from my google search it seems that this is not something that is intuitive to fix.
Is there any progress on this: https://github.com/allegroai/clearml-agent/issues/45 ? This works on all my machines 🙂

Yea. Not using the config file does not seem like a good long-term solution for me. However, I still have no idea, why this error happens. But enough for today. Thank you a lot for your help!

Can you clone the git with the .ssh credentials on the host machine ?
If so, can you do the same manually inside a docker (i.e. spin a docker with mount -v /home/hostuser/.ssh:/root/.ssh) ?

As in if it was not empty it would work?

-rw------- 1 1000 1000 0 Feb 28 23:41 config

(I think it is the empty config file)

AgitatedDove14 fyi I think this is the issue I have: https://stackoverflow.com/a/65526944/3038183

Yes (Mine isn't and it is working 🙂 )

and config is empty ?

that was an odd error

hurray

Yes

The docker run command of the agent includes '-v', '/tmp/clearml_agent.ssh.8owl7uf2:/root/.ssh' and the file permissions are exactly the same.

agent-forwarding is working just like your described here: https://github.com/allegroai/clearml-agent/issues/45 Looking forward to not having to use the absolute path in the future 🙂

could you remove it and test ?

So just tried again and still it does not work.
This is what is in .ssh on my clearml-agent
-rw------- 1 tim tim 1,5K Apr 8 14:28 authorized_keys -rw-rw-r-- 1 tim tim 208 Apr 29 11:15 config -rw------- 1 tim tim 432 Apr 8 14:53 id_ed25519 -rw-r--r-- 1 tim tim 119 Apr 8 14:53 id_ed25519.pub -rw------- 1 tim tim 432 Apr 29 11:16 id_gitlab -rw-r--r-- 1 tim tim 119 Apr 29 11:25 id_gitlab.pub -rw-rw-r-- 1 tim tim 3,1K Apr 29 11:33 known_hosts

I ran
docker run -it -v /home/hostuser/.ssh/:/root/.ssh ubuntu:18.04but cloning does not work and this is what ls -lah /root/.ssh gives inside the docker container:
-rw------- 1 1001 1001 1.5K Apr 8 12:28 authorized_keys -rw-rw-r-- 1 1001 1001 208 Apr 29 09:15 config -rw------- 1 1001 1001 432 Apr 8 12:53 id_ed25519 -rw-r--r-- 1 1001 1001 119 Apr 8 12:53 id_ed25519.pub -rw------- 1 1001 1001 432 Apr 29 09:16 id_gitlab -rw-r--r-- 1 1001 1001 119 Apr 29 09:25 id_gitlab.pub -rw-rw-r-- 1 1001 1001 3.1K Apr 29 09:33 known_hosts

btw: With the ssh agent forwarding I do not have any issues ( https://github.com/allegroai/clearml-agent/issues/45 )