Hi TimelyPenguin76
Thanks for working on this. The clearml gcp autoscaler is a major feature for us to have. I can't really evaluate clearml without some means of instantiating multiple agents on GCP machines and I'd really prefer not to have to set up a k8 cluster with agents and manage scaling it myself.

I tried the settings above with two resources, one for default queue and one for the services queue (making sure I use that image you suggested above for both).
The autoscaler started up without instantiating VM nodes and the UI was updated showing zero VMs in use as there were no tasks pending.

I then enqued the pipeline from our conversation above (pipeline from decorators) by re-running it from the clearml pielines UI and the autoscaler crashed trying to spin up a node. Attached is the full log file.

looking at GCP, I see two VMs were started. Looking at the logs of one of the worker VMs, things don't seem to have gone right on that end either. I've attached the log.

This is the resource_configurations that the autoscaler task show I used:
[{"resource_name": "gcp_default", "machine_type": "n1-standard-1", "cpu_only": true, "gpu_type": null, "gpu_count": 1, "preemptible": false, "num_instances": 3, "queue_name": "default", "source_image": "projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20220131", "disk_size_gb": 100}, {"resource_name": "gcp_services", "machine_type": "n1-standard-1", "cpu_only": true, "gpu_type": null, "gpu_count": 1, "preemptible": false, "num_instances": 2, "queue_name": "services", "source_image": "projects/ubuntu-os-cloud/global/images/ubuntu-1804-bionic-v20220131", "disk_size_gb": 100}]the extra_vm_bash_script matches what you suggested I use:
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y curl -fsSL | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] lsb_release -cstest" sudo apt update sudo apt install docker-ce -y