One thing though - I am running agent on behalf of a regular user.

Oh that might be credentials / docker service issue (i.e. the user might not have the ability to rn a docker with --gpus, but as you mentioned,, that seems like an arch thing 🙂 )