I'M Currently Trying To Setup Gcp Autoscaler And Having Some Issues. I Have Setup The Permissions Of The Gcp Service Account Credentials File According To The Docs Here:

Answered

I'm currently trying to setup GCP autoscaler and having some issues. I have setup the permissions of the GCP service account credentials file according to the docs here: None

And I'm able to initially create the autoscaler and I can see the head node VM appear in GCP compute engine. But then it crashes not long after initializing. Attaching the corresponding console logs here.

The default source image seems a bit dated: "source_image":"projects/debian-cloud/global/images/debian-10-buster-v20210721" . Also, I'm wondering if it could be a permissions issue for the service account. Are Service Account User and Compute Admin still sufficient for the GCP autoscaler?

Let me know if I can provide any more details here to help troubleshoot. Appreciate any guidance. Thank you in advance for your support.

  				
Posted 
	one month ago

					More
				  		
  Report
		
					NonchalantCoral99
				
					0
					 × 1

Votes Newest

Answers 6

Well, this Machine Image ( projects/debian-cloud/global/images/debian-10-buster-v20210721 ) was the default value set by the WebUI (not something we specified). A ccording to the docs it should be an optional field? However, when I clear the field and try to re-launch I get this error:

googleapiclient.errors.HttpError: <HttpError 400 when requesting

 returned "Invalid value for field 'resource.disks[0].initializeParams.sourceImage': ''. The URL is malformed.". Details: "[{'message': "Invalid value for field 'resource.disks[0].initializeParams.sourceImage': ''. The URL is malformed.", 'domain': 'global', 'reason': 'invalid'}]">

  				
Posted 
	one month ago

					More
				  		
  Report
		
					NonchalantCoral99
				
					0
					 × 1

Can you try with a machine image with cuda (if you need) and docker pre-installed?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Thanks @<1523701070390366208:profile|CostlyOstrich36> , I was ultimately able to get it to run but had to build my own image, which included:

The right version of python
Docker
CUDA + NVIDIA drivers + kernel header for NVIDIA install
Virtual env
The right GPU machine types, region with availability

  				
Posted 
	one month ago

					More
				  		
  Report
		
					NonchalantCoral99
				
					0
					 × 1

Is it possible the image you used doesn't have docker? Did you find any errors in the log?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Here is the log from the GCP VM for extra context

  				
Posted 
	one month ago

					More
				  		
  Report
		
					NonchalantCoral99
				
					0
					 × 1

@<1523701070390366208:profile|CostlyOstrich36> , any thoughts on what might be happening with the autoscaler here?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					NonchalantCoral99
				
					0
					 × 1

Write your answer

281 Views

6 Answers

one month ago