I guess that's a good point but really applicable if your training is CPU intensive. If your training is GPU intensive I guess most of the load goes on the GPU so running over VM (EC2 instances for example) shouldn't have much of a difference but this is worthy of testing.

I found this article talking about performance
https://blog.equinix.com/blog/2022/01/04/3-reasons-why-you-should-consider-running-containers-on-bare-metal/

But it doesn't really say what the difference in performance is.

Maybe SuccessfulKoala55 , JuicyFox94 or AgitatedDove14 might have some input on this interesting point