BTW- I understand that usually we should be able to use standard images for the autoscaler?
This looks to me like a permission issue on GCP side. Do your GCP credentials have the compute.images.useReadOnly permission set? It looks like the worker needs that permission to be able to pull the images correctly 🙂
Using app.clear.ml
Sorry- my mistake- when I open a new autoscaler, the field is not empty
It is not filled in by default?
projects/debian-cloud/global/images/debian-10-buster-v20210721
can you please write the full path to the default machine image?
Great! Please let me know if it works when adding this permission, we'll update the docs in a jiffy!
I'm using image and machine image interchangeably here. It is quite weird that it is still giving the same error, the error clearly asked for "Required 'compute.images.useReadOnly' permission for 'projects/image-processing/global/images/image-for-clearml'" 🤔
Also, now I see your credentials even have the role of compute admin, which I would expect to be sufficient.
I see 2 ways forward:
- Try running the autoscaler with the default machine image and see if it launches correctly
- Double check that the json credentials you added in the GCP wizard are from the same account that you changed the permissions on. Possibly re-add the json to the wizard?
see attached after added permission. I’m getting the same error (I cloned the autoscaler and task)
Should I use machine image for autoscaler or image (not sure what is the difference)?
@<1523701118159294464:profile|ExasperatedCrab78> Thanks. I tried to check it. Can you refer me to the right place to check/ add permission? (I followed the instruction on clearml site with no luck)
It looks like you need to add the compute.imageUser role to your credentials: None
Did you by any chance set up the autoscaler to use a custom image? It's trying to use ‘projects/image-processing/global/images/image-for-clearml’ which is a path I don't recognise. Is this your own, custom image? If so, we can add this role to the documentation as required when using a custom image 🙂
Indeed that should be the case. By default debian is used, but it's good that you ran with a custom image, so now we know it's not clear that more permissions are needed
Are you running a self-hosted/enterprise server or on app.clear.ml? Can you confirm that the field in the screenshot is empty for you?
Or are you using the SDK to create an autoscaler script?
