Hi FloppyDeer99 ,

It depends on you setup:
if you have on prem machines, you can start more than one clearml-agent on the machine with the resources and assign for example each gpu on the machine to a https://clear.ml/docs/latest/docs/clearml_agent#docker-mode . You can have the same for cloud machine, and if you are using the AWS you can run the https://clear.ml/docs/latest/docs/guides/services/aws_autoscaler/ as a service. K8S: there is a great example for k8s glue https://github.com/allegroai/clearml-agent/blob/master/examples/k8s_glue_example.py

I can solve it by implementing a custom scheduler which is used to watch the queue, pull the task and send it to a remote environment. This environment will prepare the prerequisites and run agent in execute mode.

Each of the above should do it for you 🙂