-rw------- 1 1000 1000 0 Feb 28 23:41 config

Yes (Mine isn't and it is working 🙂 )

As in if it was not empty it would work?

AgitatedDove14 fyi I think this is the issue I have: https://stackoverflow.com/a/65526944/3038183

However, I have not yet found a flexible solution other than ssh-agent forwarding.

Now it works.

For everyone who had the patience to read through everything, here is my solution to make clearml work with ssh-agent forwarding in the current version:
Start and ssh-agent Add ssh keys with ssh-add to agent echo $SSH_AUTH_SOCK and paste into clearml.conf as here: https://github.com/allegroai/clearml-agent/issues/45#issuecomment-779302144 (replace $SSH_AUTH_SOCKET with actually value) Move all the files except known_hosts out of ~/.ssh of the clearml-agent workstation. Start the clearml-agent.

Yea. Not using the config file does not seem like a good long-term solution for me. However, I still have no idea, why this error happens. But enough for today. Thank you a lot for your help!

could you remove it and test ?

and config is empty ?

that was an odd error

btw: With the ssh agent forwarding I do not have any issues ( https://github.com/allegroai/clearml-agent/issues/45 )

So it seems to be definitely a problem with docker and not with clearml. However, I do not get, why it works for you but on none of my machine (all Ubuntu 20.04 with docker 20.10)

Yea, but before in my original setup the config file was filled. I just added some lines to the config and now the error is back.

(I think it is the empty config file)

hurray

However, I have not yet found a flexible solution other than ssh-agent forwarding.

And is it working?

I ran
docker run -it -v /home/hostuser/.ssh/:/root/.ssh ubuntu:18.04but cloning does not work and this is what ls -lah /root/.ssh gives inside the docker container:
-rw------- 1 1001 1001 1.5K Apr 8 12:28 authorized_keys -rw-rw-r-- 1 1001 1001 208 Apr 29 09:15 config -rw------- 1 1001 1001 432 Apr 8 12:53 id_ed25519 -rw-r--r-- 1 1001 1001 119 Apr 8 12:53 id_ed25519.pub -rw------- 1 1001 1001 432 Apr 29 09:16 id_gitlab -rw-r--r-- 1 1001 1001 119 Apr 29 09:25 id_gitlab.pub -rw-rw-r-- 1 1001 1001 3.1K Apr 29 09:33 known_hosts

Can you clone the git with the .ssh credentials on the host machine ?
If so, can you do the same manually inside a docker (i.e. spin a docker with mount -v /home/hostuser/.ssh:/root/.ssh) ?

agent-forwarding is working just like your described here: https://github.com/allegroai/clearml-agent/issues/45 Looking forward to not having to use the absolute path in the future 🙂

Maybe the problem is that I do not start my docker containers from the root user, so 1001 is a mapping inside the docker to my actual user. Could it be that on the host the owner if your .ssh files is called root ?

Yes

👍

The docker run command of the agent includes '-v', '/tmp/clearml_agent.ssh.8owl7uf2:/root/.ssh' and the file permissions are exactly the same.

When I change the owner and the group of the files to root it works.

In the beginning my config file was not empty 😕

size 0

Nothing changes, still bad owner or permissions.

Anyways, from my google search it seems that this is not something that is intuitive to fix.
Is there any progress on this: https://github.com/allegroai/clearml-agent/issues/45 ? This works on all my machines 🙂

So just tried again and still it does not work.
This is what is in .ssh on my clearml-agent
-rw------- 1 tim tim 1,5K Apr 8 14:28 authorized_keys -rw-rw-r-- 1 tim tim 208 Apr 29 11:15 config -rw------- 1 tim tim 432 Apr 8 14:53 id_ed25519 -rw-r--r-- 1 tim tim 119 Apr 8 14:53 id_ed25519.pub -rw------- 1 tim tim 432 Apr 29 11:16 id_gitlab -rw-r--r-- 1 tim tim 119 Apr 29 11:25 id_gitlab.pub -rw-rw-r-- 1 tim tim 3,1K Apr 29 11:33 known_hosts