Hi,

I am trying to set up the GCP autoscaler. I created a GCP service account and granted it the following roles: Compute Admin, Service Account User, and Logs Writer. I then added its credentials under the GCP credentials section in ClearML.

VM instances are created successfully; however, they remain idle, do not pick up tasks from the default queue (which I specified in the configuration), and then shut down shortly afterward.

For the Docker base image, I am using gcr.io/deeplearning-platform-release/base-cu121:latest .
For the machine image, I am using: projects/ml-images/global/images/family/common-cu121-debian-11

The monitored queue is 'default', and I’ve set the Service Account Email to the correct email address of the service account with the mentioned roles. I have not defined an initialization script.

My additional ClearML configuration is: agent.extra_docker_arguments: ["--ipc=host", "--gpus", "all"]

Here is what happens when I launch teh app:
- The app instance is created.

- N number of instances are successfully created and launched on GCP.

- Several tasks are visible in the default queue.

However, after init, the VMs shut down almost right away and remain idle, with tasks still waiting in the queue. Log reports 0 active cloud instances.

Could you please advise on what might I be doing wrong? If you need any additional information about my configuration, I’d be happy to provide it.

Thanks in advance!