so what worked for me was the following startup userscript:#!/bin/bash sleep 120 while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done sudo apt-get update while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done sudo apt-get install -y python3-dev python3-pip gcc git build-essential python3 -m pip install -U pip ...
Now it starts, I’ll see if this solves the issue
the instances takes so much time to start, like 5 mins
there is no error from this side, I think the aws autoscaler just waits for the agent to connect, which will never happen since the agent won’t start because the userdata script fails
can you attach the full log of the instance? Did the aws scalar output any logs?
Hi JitteryCoyote63 , which ec2 type and AMI are you using?
The running task in the UI for it
edited the aws_auto_scaler.py, actually I think it’s just a typo, I just need to double the brackets
How did you add it? Just edited the configuration part of the task or with the wizard?
AMI ami-08e9a0e4210f38cb6 , region: eu-west-1a
As you can see, more hard waiting (initial sleep), and then before each apt action, make sure there is no lock
the deep learning AMI from nvidia (Ubuntu 18.04)
agree
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)Another process is using the lock, can you specify the ami (and region) so I can try to reproduce it?
I think waiting for the apt locks to be released with something like this would workstartup_bash_script = [ "#!/bin/bash", "while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do echo 'Waiting for other instances of apt to complete...'; sleep 5; done", "sudo apt-get update", ...Weirdly this throws an error in the autoscaler:Spinning new instance type=v100_spot Error: Failed to start new instance, unexpected '{' in field name