BTW:

Error response from daemon: cannot set both Count and DeviceIDs on device request.

Googling it points to a docker issue (which makes sense considering):
https://github.com/NVIDIA/nvidia-docker/issues/1026
What is the host OS?

Okay, I'll make sure we always qoute " , since it seems to work either way.
We will release an RC soon, with this fix.
Sounds good?

Also what is the docker vserion?

indeed, i managed to make a docker run command to work with the fix you mentioned ( docker run --gpus '"device=1,2"' nvidia/cuda:9.0-base nvidia-smi ) but trains-agent just appends to --gpus device= and there is no way to make the quoting like this

Docker version 19.03.7, build 7141c199a2 on Linux, btw

Are you using zsh by any chance?

yes

Hmm, let me check something

no, it's SUSE on a server, and bash

Are you working with venv or docker mode?

sorry, important info! Docker mode

Also notice that is you need all gpus you can pass 

--gpus all

yes, i know, but i need to use 2 out of 4 for a queue

Okay, checking...

Hi OutrageousGrasshopper93
Are you working with venv or docker mode?
Also notice that is you need all gpus you can pass --gpus all

Ubuntu? which version?

amazing and thanks! keep me posted

OutrageousGrasshopper93 is "--gpus all" working ?