I will check the aws token, just to verify, you imported the StorageManager after the os.environ calls?
We actually want to set the output_uri in Task.init to be a folder in AWS S3. And run the script on sagemaker.
TimelyPenguin76 Unfortunately, no. Is the error because of token?
Hi TimelyPenguin76 , I tried and got the same error as before. One thing was different from yours, our company is using SSO to access AWS, so I added os.environ['AWS_SESSION_TOKEN'] = "***". It worked for boto3 to upload the file to S3, so I think probably SSO is not the reason.
Hi TimelyPenguin76 , Thank you for your reply, I've tried env vars. The problem happened when I tried to upload the files to S3 using ClearML. os.environ['AWS_ACCESS_KEY_ID'] = "***"
os.environ['AWS_SECRET_ACCESS_KEY'] ="***"
os.environ['AWS_SESSION_TOKEN'] = "***"
I suspect that
I will try to generate a new token for myself and reproduce it with it
SquareFish25 Will try to reproduce it
Hi TimelyPenguin76 , I added the AWS_DEFAULT_REGION as well, and it did not work. The version of ClearML is '0.17.4'.
SquareFish25 do you have a way trying with access and secret without token? just for the checking
Hi SquareFish25 , what about AWS_DEFAULT_REGION , did you add it too? Can you try with it if not?
What version of ClearML are you using?
I used boto3 to upload and it worked means the key should be right. But it did not work when using clearml.
Hi SquareFish25 ,
I tried the follow and succeed to upload the file:
` import os
os.environ['AWS_ACCESS_KEY_ID'] = "***"
os.environ['AWS_SECRET_ACCESS_KEY'] = "***"
os.environ['AWS_DEFAULT_REGION'] = "***"
from clearml import StorageManager
remote_file = StorageManager.upload_file(<file to upload>, 's3://bucket_name/inner_folder/file_name') `Can you try it and update if it works for you?
Hi SquareFish25 ,
Which section would you like to modify? Can env vars do the trick? e.g.
os.environ["CLEARML_API_HOST"] = "***" os.environ["CLEARML_WEB_HOST"] = "***" os.environ["CLEARML_FILES_HOST"] = "***" os.environ["CLEARML_API_ACCESS_KEY"] = "***" os.environ["CLEARML_API_SECRET_KEY"] = "***"