Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Question About The Auto Scaling Service Under

Question about the auto scaling service

Under extra_trains_conf when I supply a configuration file path, should it be a path on the trains server running the autoscale task, or should it be embedded in the AMI that will be launched?

  
  
Posted 3 years ago
Votes Newest

Answers 33


If the configurations and hyper params still appear properly in the task there's no need to rerun the wizard. just make sure you're using the updated trains repo

  
  
Posted 3 years ago

I ran the wizard liek a milion times

  
  
Posted 3 years ago

can you tell me which API call exactly are you using for spinning up? I would like to debug and try to use boto3 myself in order to spin up an instance, so I can understand where the problem is coming from

  
  
Posted 3 years ago

image

  
  
Posted 3 years ago

the trains ones

  
  
Posted 3 years ago

image

  
  
Posted 3 years ago

So what's our next move FriendlySquid61 ? O_O

  
  
Posted 3 years ago

I doubled checked the credentials in the configurations, and they have full EC2 access

  
  
Posted 3 years ago

Okay, I'll try that

  
  
Posted 3 years ago

now I get this error in my Auto Scaler task
Warning! exception occurred: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials Retry in 15 seconds

  
  
Posted 3 years ago

Now I remind you that using the same credentials exactly, the auto scaler task could launch instances before

  
  
Posted 3 years ago

Update - New AWS credentials solved this issue.

  
  
Posted 3 years ago

I'll try to debug this and updat

  
  
Posted 3 years ago

Searching this error it seems it could be many things.
Either wrong credentials or a wrong region (different than the one for your key-pair).
It could also be that your computer clock is wrong (see example https://github.com/mitchellh/vagrant-aws/issues/372#issuecomment-87429450 ).
I suggest you search it online and see if it solves the issue, I think it requires some debugging on your end.

  
  
Posted 3 years ago

FriendlySquid61

Just updating, I still haven't touched this.... I did not consider the time it would take me to set up the auto scaling, so I must attend other issues now, I hope to get back to this soon and make it work

  
  
Posted 3 years ago

Make sure you're testing it on the same computer the autoscaler is running on

  
  
Posted 3 years ago

and also in the extra_vm_bash_script variables, I ahve them under export TRAINS_API_ACCESS_KEY and export TRAINS_API_SECRET_KEY

  
  
Posted 3 years ago

the ones in the export are like you said

  
  
Posted 3 years ago

I have them only in one place

  
  
Posted 3 years ago

Hi FriendlySquid61 I did all the changes you said

  
  
Posted 3 years ago

no need to do it again, I ahve all the settings in place, I'm sure it's not a settings thing

So just to correct myself and sum up, the credentials for AWS are only in the cloud_credentials_*

  
  
Posted 3 years ago

Actually I removed the key pair, as you said it wasn't a must in the newer versions

It isn't a must, but if you are using one, it should be in the same region

  
  
Posted 3 years ago

Sure, we're using RunInstances , you can see the call itself https://github.com/allegroai/trains/blob/master/trains/automation/aws_auto_scaler.py#L163

  
  
Posted 3 years ago

Those are different credentials.
You should have the aws info under:
cloud_credentials_key , cloud_credentials_secret and cloud_credentials_region
And the stuff added to the extra_vm_bash_script are the trains key and secret from your profile page in the UI.
I suggest you use the wizard again to run the task, this will make sure all the data is where it should be.

  
  
Posted 3 years ago

Actually I removed the key pair, as you said it wasn't a must in the newer versions

  
  
Posted 3 years ago

I have them in two different places, once under Hyperparameters -> General

  
  
Posted 3 years ago

I'll try to put it again

  
  
Posted 3 years ago

Hey WackyRabbit7 ,
Is this the only error you have there?
Can you verify the credentials in the task seem ok and that it didn't disappear as before?
Also, I understand that the Failed parsing task parameter ... warnings no longer appear, correct?

  
  
Posted 3 years ago

and when looking at the running task, I still see the credentials

  
  
Posted 3 years ago

Yep, the warnings no longer appear

  
  
Posted 3 years ago