because when I was running both agents on my local machine everything was working perfectly fine

This is probably you (or someone) had set up ssh public key with your git repo sometime in the past