general infrastructure question:
my company isn't using AWS for training, we have all our GPU's inhouse in our own servers, we have a problem where we want on one hand to give all the GPUs for the clearml-agent (ie' that they will be available for tasks) but on the other hand i want to give my developers the chance to develop on GPUS that arent being used.
the best case scenario of what i want is that the agent wouldn't give a GPU that has a task running on it, but i wasn't able to find if thats possible. my workaround that is that when my developers start their debugging process i will make a tool for them that restarts the daemon, but will change the --gpus argument to only include the gpu they arent working on (most of our servers have 2-8 gpus on them). in theory it should work, but i'm not sure what happens if lets say there is already a task running at GPU0 when a developer takes the daemon down. will the task keep running? is there a way to change which gpus are visible without taking down the daemon?
thanks!