well, I can get this exact object from my machine, from the same script through boto3, so I’m not sure where the disconnect is AgitatedDove14 SuccessfulKoala55
Can you perhaps use the AWS_PROFILE environment variable to specify the profile?
SuccessfulKoala55 and AgitatedDove14 So setting my environment variable worked for my local runs, but it doesn’t work when running it from a container (in AWS ECS, for example)
so a little more clarity. I get nearly this same error when downloading the same object from boto3 if I try to get at it from an s3 resource like so:
` bad_s3_resource = boto3.resource("s3")
obj = bad_s3_resource.Object(bucket, key)
obj.get()["Body"]
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Only when I set the boto3 session with the appropriate profile does it work:sess = boto3.session.Session(profile_name="foo-profile")
s3_resource = sess.resource("s3")
obj = s3_resource.Object(bucket, key)
obj.get()["Body"]
<botocore.response.StreamingBody at 0x1463120d0> `
So this makes me wonder if there is a similar mechanism that’s giving me this error.
I still think the issue is getting boto3 credentials
It might be the case
Are you using clearml-agent or are you running it manually ?
BeefyHippopotamus73 are you saying that on a remote machine you cannot set AWS_PROFILE ? or is it the clearml.conf is missing ? (not sure I follow how / who spins the remote machine)
I think the issue is we do not pass a profile parameter to boto
Don’t know why I didn’t think of it earlier, but I set my env variable AWS_PROFILE=<foo-profile> and it copied it successfully.
yes I did finally resolve this. If my memory serves, I think I needed to do some double checking that the clearml.conf file was in the proper location. It wasn’t where I thought I put it and I had to set all the proper environment variables in the container.
BeefyHippopotamus73 indeed it seems like this is boto3 announcing it can't locate credentials on your machine
I think I can set AWS_PROFILE . I have theoretically placed the clearml.conf in the correct place in the container: $HOME/clearml.conf . But I get an error that’s actually not very helpful, I still think the issue is getting boto3 credentials but I’m not sure I can prove it at this point. I posted the error in the community channel https://clearml.slack.com/archives/CTK20V944/p1653507863528969
BeefyHippopotamus73 are you running this using a ClearML Agent?
BeefyHippopotamus73 this error seems like it is coming from boto3, are you sure the credentials are properly configured and that you have read permission ?
Hi BeefyHippopotamus73
did you managed to get rid of your issue ?
