I have a preprocessing task that is CPU parallelized, but I wanted to create a ClearML task for each instance since it reports some parameters that I want to later aggregate. Running this quickly requires 10-30 agents listening on the same queue

that's an interesting use case for services mode. I'll try it out. thanks!

HugeArcticwolf77 you can add --services-mode to the agent, and it will basically keep on spinning Tasks in parallel (unfortunately the open source version does not include a way to limit it to a maximum of concurrent Tasks)

Hi HugeArcticwolf77 , I don't think such a thing exists out of the box. What is your use case?