. I was just wondering if instead of using local subprocesses, several agents could serve the same purpose (running several pipelines concurrently)
wouldn't --service-mode (read as multiple simultaneous Tasks on the same agent) solve the issue?
(BTW: if you set the pipeline component target queue to "services" , this is exactly what will happen)
GiganticTurtle0 Hi 🙂
You can use worker/agent interchangeably. When running an agent a new process is started. I think the only limitation on number of works running is the amount of the resources your system has to spare.
Regarding if it can be a thread I'll give you information soon.
Hi CostlyOstrich36 AgitatedDove14
Oh no, I am not trying to say that I am using each agent to run a single task. I have several agents listening to a number of queues so that they are busy most of the time. As we talked about, it is not possible to run multiple pipelines ( PipelineDecorator.pipeline ) simultaneously in a single process. That's why I had been testing locally launching pipelines in different subprocesses and this way I have managed to run several pipelines concurrently. I was just wondering if instead of using local subprocesses, several agents could serve the same purpose (running several pipelines concurrently)
So, for example, if I have a machine with 64 CPU cores, I will be able to run up to 64 agents (in case my system consists of only this machine), right?
GiganticTurtle0 notice that when you spin an agent with --services-mode, you basically let it run many Tasks at once (this is in contrast to the default behavior, when you have one Task per agent).
GiganticTurtle0 I'm not sure why a direct correlation between the number of agents and cores is an issue, but the issue is resources - each agent will run a task in a separate process (or docker container) - do you really want each one of your tasks to run on a single core shared with the agent process? Wouldn't you get better performance if each task has more cores to run with? What about GPUs?
Ok! I'll try to spin up an agent with the --service-mode command and I will give you feedback