There is no need to add creds on the machine, since the EC2 instance has an attached IAM profile that grants access to s3. Boto3 is able retrieve the files from the s3 bucket
I will go for lunch actually 😄 back in ~1h
Why is it required in the case where boto3 can figure them out itself within the ec2 instance?
This seems to be the same issue like in https://clearml.slack.com/archives/CTK20V944/p1633599511350600
Whats the pyjwt version you are using?
JitteryCoyote63 this is currently not supported in the ClearML Storage driver (only key/secret are)
you should have access and secret too
do you have all your AWS credentials in your ~/clearml.conf file?
I am confused now because I see in the master branch, the clearml.conf file has the following section:# Or enable credentials chain to let Boto3 pick the right credentials. # This includes picking credentials from environment variables, # credential file and IAM role using metadata service. # Refer to the latest Boto3 docs use_credentials_chain: falseSo it states that IAM role using metadata service should be supported, right?
JitteryCoyote63 so you don’t need to use creds anymore?
You can look around there, and you can also open a GitHub issue for us 🙂
Yea I really need that feature, I need to move away from key/secrets to iam roles
If you can look around and maybe help with a PR that would be awesome 🙂
ClearML uses the access and secret for creating the storage object, you can have those as env params too
SuccessfulKoala55 Could you please point me to where I could quickly patch that in the code?
SuccessfulKoala55 I was able to make it work with use_credentials_chain: true in the clearml.conf and the following patch: https://github.com/allegroai/clearml/pull/478
Well, I'm not sure it's that simple. It's basically in the _Boto3Driver class, but the issue is with what the code required to verify the URL
TimelyPenguin76 , no, I’ve only set the sdk.aws.s3.region = eu-central-1 param