Right, basically someone needs to configure the “regular” environment variables for boto to use the IAM role,

clearml

will basically uses boto, so it should be transparent. does that make sense ? How do you spin the job on the k8s cluster and how do you configure it?

Yep I was thinking the same that the design choice must have been inspired by transparency. At the moment we just use the sdk to log training runs, model eartifacts etc and upload the model. We dont use the clearml agent. The pod has an annotation with a AWS role which has write access to the s3 bucket.

Hmm and were you able to get a token that will last for the entire “running time” of the Task? When are you calling

saml2aws

and are you using the output to configure the OS environments ? (edited)

Yes. The token lasts 1 hour. And my task is takes less than 5 mins. Yes saml2aws configures the default variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. I was expecting clearml to pick them by default from the environment. From what I understand from https://github.com/allegroai/clearml/blob/master/clearml/storage/helper.py only the access_key_id and secrete access_key are used. Isnt this a bit restricting since use of temporary credentials is quite common?