No problem!
Yes, I’m running manual mode and I only see one GPU tracked in the resource monitoring. I’m using train 0.16.4 .

Everything seems to work as it should, but if I run without trains, my process is only visible on the one GPU i made visible with CUDA_VISIBLE_DEVICES . If I run with Trains, it’s “registered” on all other devices as well if inspected with sudo fuser -v /dev/nvidia*