Actually it is better to leave it as is, it will just automatically mount the .ssh folder into the container, i will make sure the docs point to this option first

It still didn't work. I created the ssh keys on the agent with ssh-keygen -C "<user>" , added the public key to bitbucket to allow it, and didn't specified the SSH_AUTH_SOCK now. Am I missing something? Do I need to setup an ssh-agent on host before spinning up the agent?

Did you you set 'force_git_ssh_protocol: true '?
https://github.com/allegroai/clearml-agent/blob/249b51a31bee97d63f41c6d5542e657962008b68/docs/clearml.conf#L39

yes

The error i'm getting is:
cloning: Using SSH credentials - replacing https url ' ' with ssh url ' ' 2022-12-16 14:33:39 fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.However I can successfuly clone the repo on the host vm, so the ssh key should be giving me the permisison.

Hmm can you run the agent in debug mode, and check the specific console log?
'''
clearml-agent --debug daemon --foreground ...

This morning I kiled the agent and spinned it up again with --foreground and it worked flawlesly. Then I tried without --foreground and it still worked. I didn't change anything else, weird. When I set up another worker in the future I will keep an eye on it