HugePelican43 as AgitatedDove14 says, that's a slippery slope to out-of-memory land. If you have Nvidia A100 you can use multiple agents in MIG mode, sort of like containerized hardware if you never heard of it.
Other than that I do not recommend. Max out utilisation for each task instead.
Hi, I was just answering your previous question. can you explain a bit what you mean by "under utilized"? e.g. do you have 2 gpus and are using only one of them for a task?
or are maxing out resources but do not get to 100% utilization (which might be a data pipeline issue)
HugePelican43 sure you can, usually the limiting factor is memory, as it cannot be shared among processes, so if one allocated all memory the second process will crash with out of memory error
I have a single gpu to serve as an agent. Currently running a task on that gpu. Can I run another task on that same GPU?
No by definition the agent will only execute one Task at a time, you can spin a second agent on the same GPU :)
I was running a task, then I clone it. I queued the second one and it is pending till now.
does it mean I have low mem allocated for docker container?