Hey All, When Using Gcp Autoscaler I Always Get An Error That No Gpus Are Recognized The First Time The Autoscaler Boots Up The Vm And Runs The Code. After That First Experiment Fails I Again Run It, Autoscaler Picks Up The Same Vm And Runs The Code And

Answered

Hey all,

When using GCP Autoscaler I always get an error that no gpus are recognized the first time the autoscaler boots up the VM and runs the code. After that first experiment fails I again run it, autoscaler picks up the same VM and runs the code and now the GPUs are detected. Anyone had this problem?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					NastyBear13
				
					0
					 × 1

Votes Newest

Answers

Hi @<1837300695921856512:profile|NastyBear13> , can you provide logs from the machine itself? Are you certain its the same VM? Can you also provide logs from the tasks themselves? \

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

490 Views

1 Answer

3 months ago

Hi, I Am Trying To Set Up The Gcp Autoscaler. I Created A Gcp Service Account And Granted It The Following Roles: Compute Admin, Service Account User, And Logs Writer. I Then Added Its Credentials Under The Gcp Credentials Section In Clearml. Vm Instanc

Hi, I'M Using Aws Ec2 Instance To Trian My Models With Clearml Autoscaler, But It Says Cuda Device Is Not Avaliable. The Code Runs Well On My Local Pc And It Runs Well On Clearml With Ec2 Yesterday, But It Suddenly Doesn'T Work Today. Is There Anyway To S

Hi, I Have A Question Regarding The Autoscaler. I Implemented A Custom Driver For Gcp And I Manager To Launch The Clearml.Automation.Auto_Scaler.Autoscaler Which Runs Smoothly (Kudos!!). I Can See Instance Being Created/Destroyed On Demand As Expected. Th

Hi Everyone, I Am Running A Pipeline Using The Autoscaler, I Am Able To Spin Up The Vm Instance Using The Autoscaler And The Docker Is Also Getting Installed In There Perfectly. The Issue I Am Facing Is That During Executing A Pipeline Task While Cloning

Hi, Thanks For Making This Awesome Mlops Solution! I'M Trying To Get Gcp Autoscaler To Work In Docker Mode But The Vm Instances Keep Crashing After Running The Agent. Pretty Sure The Service Account Grants Sufficient Permissions To Do Whatever Needed. An

Back To Autoscaler; Is There Any Way To Ensure The Environment Variables On The Services Queue (Where The Scaler Runs) Will Be Automatically Exposed To New Ec2 Instance? Some Bash Hack Or Similar Would Be Nice, Really

Hey Everyone, I Have An Autoscaler Configuration That Runs Different Ec2 Instances. I Want The Ec2 Worker Launched By The Autoscaler Could Handle A Bucket With Different Aws Keys. The Configuration I Am Passing Is As Follows: