Another Question About The Aws Auto_Scaler, I Managed To Get The Scaler Service Running, It Listens To A Queue Called "Aws" (Yes, Very Imaginative), When I Queue A Simple Task I See That It'S Being Handled But The Task Is Stuck As Pending. When Checking T

Answered

another question about the aws auto_scaler, i managed to get the scaler service running, it listens to a queue called "aws" (yes, very imaginative), when i queue a simple task i see that it's being handled but the task is stuck as pending. when checking the scaler service log, i see this error message:

2023-11-14 14:14:49,692 - clearml.auto_scaler - ERROR - Failed to start new instance (resource 'vai-clearml-runner'), Error: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials

so basically a credentials issue, but i'm sure i gave the correct amazon key and secret to the autoscaler.yaml file. am i missing a place where i need to give the aws credentials?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

Votes Newest

Answers 3

Hi @<1612982606469533696:profile|ZealousFlamingo93> , how exactly are you running the autoscaler?
Is this what you're running?
None

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

@<1523701070390366208:profile|CostlyOstrich36>
hey john,
yeah this is what i'm running, im running it on a local machine (where the web server is also running if thats relevant), the aws credentials i've given it are with an IAM role that gives access to EC2:

{                  
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:TerminateInstances",
                "ec2:RequestSpotInstances",
                "ec2:DeleteTags",
                "ec2:CreateTags",
                "ec2:RunInstances",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:GetConsoleOutput"
            ],
            "Resource": "*"
        }
    ]
}

along with S3 access, then i put the key and secret into the autoscaler.yaml so it can run

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

@<1523701070390366208:profile|CostlyOstrich36>
ok from what im seeing, for some reason my the updated credentials don't get to the actual task, when i spin up the task and go to the configurations tab in the webserver, it shows an old key and secret. any idea why this is happening?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ZealousFlamingo93
				
					0
					 × 1

Write your answer

845 Views

3 Answers

one year ago