I have a set up an agent, on a GPU machine, and spun up the daemon in docker moder, and specifically specified a GPU that it will work with. The image is okay and I verified that by running docker run ... nvidia-smi and made sure it does have a valid output.
When I launch a task from an environment without GPU and use the task.execute_remotely to execute it on the agent, the agent pulls the task but the code does not run on the GPU (TensorFlow code). Could this be because the TF installed on the environment executing the line task.execute_remotely is the CPU version and therefor to replicate trains installs the same CPU variant instead of GPU? Or something else could be wrong here?
and the machine I have is 10.2.
I also tried nvidia/cuda:10.2-base-ubuntu18.04 which is the latest
https://hub.docker.com/layers/nvidia/cuda/10.1-cudnn7-runtime-ubuntu18.04/images/sha256-963696628c9a0d27e9e5c11c5a588698ea22eeaf138cc9bff5368c189ff79968?context=explore
the docker image is missing the cudnn which is a must for TF to work 🙂
