I'm running the following agent:clearml-agent --config-file /clearml-cache/config/clearml-cpu.conf daemon --queue cpu default services --docker ubuntu:20.04 --cpu-only --services-mode 4 --detached
The goal is to have an agent that can run multiple cpu only tasks at the same time. I notices that when enqueueing multiple tasks, all except for one stay pending until the first one finished downloading all packages and started with code execution. And then task by task switch to "running" when the previous task is done with "execution preparation". Is there any way to not have this waiting time?
Well, when running in services mode, the agent starts running a task, and when the task actually starts running, goes to run the next one, so the behaviour your seeing is correct
Hi ClumsyElephant70 ,
Indeed there is - just pass a number immediately after the --services-mode command-line option (e.g. --services-mode 5 )
I see... Well, another way to greatly improve the execution time is to use smaller docker images with preinstalled requirements (mostly system stuff such as Python etc. and major python packages) - most of the time is spent on these installations
Ok, if I would like to have a different behaviour I would need one agent per task, right?
you can basically take one of the standard python+bullseye images, add some stuff to it, and use it as the default image
We run a lot of pipelines that are cpu only with some parallel steps. Its just about improving the execution time
there's also a tiny python+bullseye image, I think