Another Question About The Aws Auto_Scaler, I Managed To Get The Scaler Service Running, It Listens To A Queue Called "Aws" (Yes, Very Imaginative), When I Queue A Simple Task I See That It'S Being Handled But The Task Is Stuck As Pending. When Checking T

Answered

another question about the aws auto_scaler, i managed to get the scaler service running, it listens to a queue called "aws" (yes, very imaginative), when i queue a simple task i see that it's being handled but the task is stuck as pending. when checking the scaler service log, i see this error message:

2023-11-14 14:14:49,692 - clearml.auto_scaler - ERROR - Failed to start new instance (resource 'vai-clearml-runner'), Error: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials

so basically a credentials issue, but i'm sure i gave the correct amazon key and secret to the autoscaler.yaml file. am i missing a place where i need to give the aws credentials?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

Votes Newest

Answers 3

@<1523701070390366208:profile|CostlyOstrich36>
hey john,
yeah this is what i'm running, im running it on a local machine (where the web server is also running if thats relevant), the aws credentials i've given it are with an IAM role that gives access to EC2:

{                  
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:TerminateInstances",
                "ec2:RequestSpotInstances",
                "ec2:DeleteTags",
                "ec2:CreateTags",
                "ec2:RunInstances",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:GetConsoleOutput"
            ],
            "Resource": "*"
        }
    ]
}

along with S3 access, then i put the key and secret into the autoscaler.yaml so it can run

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

@<1523701070390366208:profile|CostlyOstrich36>
ok from what im seeing, for some reason my the updated credentials don't get to the actual task, when i spin up the task and go to the configurations tab in the webserver, it shows an old key and secret. any idea why this is happening?

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

Hi @<1612982606469533696:profile|ZealousFlamingo93> , how exactly are you running the autoscaler?
Is this what you're running?
None

  				
Posted 
	5 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

291 Views

3 Answers

5 months ago