Okay, so if my python script imports some other scripts I've written - I must use git?

now I get this error in my Auto Scaler task
Warning! exception occurred: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials Retry in 15 seconds

the level of configurability in this thing is one of the best I've seen

pgrep -af trains shows that there is nothing running with that name

I'll try to put it again

and I'm using the same credentials

It's working! 😄

And yes, it makes perfect sense, thanks for the answer

I'll try it

Do you have any idea as to why does that happen SuccessfulKoala55

I have a single IAM, my question is what kind of permissions I should associate with the IAM so that the autoscaler task will work

?

yep

SuccessfulKoala55 AppetizingMouse58

[ec2-user@ip-10-0-0-95 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 880K 3.9G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/nvme0n1p1 8.0G 6.5G 1.5G 82% / tmpfs 790M 0 790M 0% /run/user/1000

that is because my own machine has 10.2 (not the docker, the machine the agent is on)

e.g. docker-compose restart

Let's try

But does it disable the agent? or will the tasks still wait for the agent to dequeue?

Okay, I'll try that

Actually I removed the key pair, as you said it wasn't a must in the newer versions

So prior to doing any work on the trains autoscaler servcice, I should first create a auto scaling group in AWS?

😅

doesn't contain the number 4

Hi FriendlySquid61 I did all the changes you said

What about permissions to the machines that are being spun up? For exampel if I want the instances to have specific permissions to read/write to S3 for example, how do I mange those?

same result

Gotcha

AgitatedDove14 I really don't know how is this possible... I tried upgrading the server, tried whatever I could

About small toy code to reproduce I just don't have the time for that, but I will paste the callback I am using to this explanation. This is the overall logic so you can replicate and use my callback

From the pipeline task, launch some sub tasks, and put in their post_execute_callback the .collect_description_tables method from my callback class (attached below) Run t...

this is the full one TimelyPenguin76