This looks to me like a permission issue on GCP side. Do your GCP credentials have the compute.images.useReadOnly permission set? It looks like the worker needs that permission to be able to pull the images correctly 🙂
Should I use machine image for autoscaler or image (not sure what is the difference)?
Indeed that should be the case. By default debian is used, but it's good that you ran with a custom image, so now we know it's not clear that more permissions are needed
I'm using image and machine image interchangeably here. It is quite weird that it is still giving the same error, the error clearly asked for "Required 'compute.images.useReadOnly' permission for 'projects/image-processing/global/images/image-for-clearml'" 🤔
Also, now I see your credentials even have the role of compute admin, which I would expect to be sufficient.
I see 2 ways forward:
- Try running the autoscaler with the default machine image and see if it launches correctly
- Double check that the json credentials you added in the GCP wizard are from the same account that you changed the permissions on. Possibly re-add the json to the wizard?
It looks like you need to add the compute.imageUser role to your credentials: None
Did you by any chance set up the autoscaler to use a custom image? It's trying to use ‘projects/image-processing/global/images/image-for-clearml’ which is a path I don't recognise. Is this your own, custom image? If so, we can add this role to the documentation as required when using a custom image 🙂
Great! Please let me know if it works when adding this permission, we'll update the docs in a jiffy!
ExasperatedCrab78 Thanks. I tried to check it. Can you refer me to the right place to check/ add permission? (I followed the instruction on clearml site with no luck)
Using app.clear.ml
Sorry- my mistake- when I open a new autoscaler, the field is not empty
Are you running a self-hosted/enterprise server or on app.clear.ml? Can you confirm that the field in the screenshot is empty for you?
Or are you using the SDK to create an autoscaler script?
BTW- I understand that usually we should be able to use standard images for the autoscaler?
can you please write the full path to the default machine image?
see attached after added permission. I’m getting the same error (I cloned the autoscaler and task)
It is not filled in by default?
projects/debian-cloud/global/images/debian-10-buster-v20210721