No, the agent can be in any machine.
But the agent has to be running on the machine with gpu

Yes 🙂

VexedCat68 , you need to run the agent on a machine with gpu. The server doesn't need to have gpu 🙂

Regarding the queue:
Yes, if someone enqueues a task onto queue X, an agent listening to that queue will run the task.

Basically since I want to train AI Models right. I'm trying to set up the architecture where I can automate the process from data fetching to model training, and need GPU for training.

Thank you for the help guys.

Ok. I kind of have a confusion now. Suppose I have an agent listening to some Queue X. If someone else on some other machine enqueues their task on Queue X, will my agent run it?

Just to be absolutely clear.

Agent Listening on Machine A with GPU listening to Queue X.

Task enqueued onto queue X from Machine B with no GPU.

Task runs on Machine A and experiment gets published to server?

And multiple agents can listen to the same queue right?