edited the aws_auto_scaler.py, actually I think it’s just a typo, I just need to double the brackets
The running task in the UI for it
AMI ami-08e9a0e4210f38cb6 , region: eu-west-1a
Now it starts, I’ll see if this solves the issue
so what worked for me was the following startup userscript:#!/bin/bash sleep 120 while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done sudo apt-get update while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done sudo apt-get install -y python3-dev python3-pip gcc git build-essential python3 -m pip install -U pip ...
As you can see, more hard waiting (initial sleep), and then before each apt action, make sure there is no lock
there is no error from this side, I think the aws autoscaler just waits for the agent to connect, which will never happen since the agent won’t start because the userdata script fails
agree
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)Another process is using the lock, can you specify the ami (and region) so I can try to reproduce it?
can you attach the full log of the instance? Did the aws scalar output any logs?
the deep learning AMI from nvidia (Ubuntu 18.04)
I think waiting for the apt locks to be released with something like this would workstartup_bash_script = [ "#!/bin/bash", "while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done", "sudo apt-get update", ...Weirdly this throws an error in the autoscaler:Spinning new instance type=v100_spot Error: Failed to start new instance, unexpected '{' in field name
How did you add it? Just edited the configuration part of the task or with the wizard?
Hi JitteryCoyote63 , which ec2 type and AMI are you using?
the instances takes so much time to start, like 5 mins