Hi, I have a worker on a machine using gpus 0,1 and another worker on the same machine using gpus 0,1,2,3,4,5. A worker ran a task on gpus 0,1 but for some reason the second worker started additional task in queue on gpus 0,1,2,3,4,5, which caused both of the worker to fail. The first worker use a queue called 2_gpu and the second worker use a queue called 6_gpu. I was expecting the second worker to wait until the first finishes, given the GPUs are taken. How can I use trains-agent to overcome such cases? I think there are a few behaviors possible for this case (One behavior is waiting until all gpus are ready and then starting the second worker, meanwhile more 2 gpus only workers can run additional more tasks. Another behavior is waiting for the 2 gpu only task to finish while blocking all new 2 gpus tasks until the 6 gpu task finishes),
Could you maybe implement such automatic logics so we could choose from?
Anything would be better than the current state, haha.