Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Attached An Iam Role To An Ec2 Instance To Grant Access To An S3 Bucket. The Ec2 Instance Is Running A Clearml-Agent (V1.1.0). I Didn’T Specify Any Key/Secret For Clearml. The Tasks Fail With The Following Error:

Hi, I attached an IAM role to an ec2 instance to grant access to an s3 bucket. The ec2 instance is running a clearml-agent (v1.1.0). I didn’t specify any key/secret for clearml. The tasks fail with the following error:
2021-10-18 08:21:10,853 - clearml.storage - ERROR - Failed creating storage object Reason: Missing key and secret for S3 storage access ( ) Traceback (most recent call last): File "devops/train.py", line 73, in <module> train(parse_args) File "devops/train.py", line 37, in train train_task.get_logger().set_default_upload_destination(args.artifacts + '/clearml_debug_images/') File "/root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/logger.py", line 1038, in set_default_upload_destination uri = storage.verify_upload(folder_uri=uri) AttributeError: 'NoneType' object has no attribute 'verify_upload'If I connect to the ec2 instance and just do aws s3 ls customs-artefacts , I can see all the files, so the CLI is able to access the s3 bucket, but not clearml. How can I make this setup work?
Note: In the clearml.conf, I’ve set sdk.aws.s3.region = eu-central-1 , so the logs for the failing task is as follows:
... sdk.aws.s3.key = sdk.aws.s3.region = eu-central-1 ...

  
  
Posted 2 years ago
Votes Newest

Answers 26


SuccessfulKoala55 I was able to make it work with use_credentials_chain: true in the clearml.conf and the following patch: https://github.com/allegroai/clearml/pull/478

  
  
Posted 2 years ago

If you can look around and maybe help with a PR that would be awesome πŸ™‚

  
  
Posted 2 years ago

Why?

  
  
Posted 2 years ago

Well, I'm not sure it's that simple. It's basically in the _Boto3Driver class, but the issue is with what the code required to verify the URL

  
  
Posted 2 years ago

Why is it required in the case where boto3 can figure them out itself within the ec2 instance?

  
  
Posted 2 years ago

SuccessfulKoala55 Could you please point me to where I could quickly patch that in the code?

  
  
Posted 2 years ago

JitteryCoyote63 so you don’t need to use creds anymore?

  
  
Posted 2 years ago

I am confused now because I see in the master branch, the clearml.conf file has the following section:
# Or enable credentials chain to let Boto3 pick the right credentials. # This includes picking credentials from environment variables, # credential file and IAM role using metadata service. # Refer to the latest Boto3 docs use_credentials_chain: falseSo it states that IAM role using metadata service should be supported, right?

  
  
Posted 2 years ago

JitteryCoyote63 this is currently not supported in the ClearML Storage driver (only key/secret are)

  
  
Posted 2 years ago

TimelyPenguin76 , no, I’ve only set the sdk.aws.s3.region = eu-central-1 param

  
  
Posted 2 years ago

do you have all your AWS credentials in your ~/clearml.conf file?

  
  
Posted 2 years ago

I will go for lunch actually πŸ˜„ back in ~1h

  
  
Posted 2 years ago

I am doing so

  
  
Posted 2 years ago

yes SparklingHedgehong28 πŸ™‚

  
  
Posted 2 years ago

Can you see if it works?

  
  
Posted 2 years ago

This seems to be the same issue like in https://clearml.slack.com/archives/CTK20V944/p1633599511350600

Whats the pyjwt version you are using?

  
  
Posted 2 years ago

Oh, so I might have forgot that 😞

  
  
Posted 2 years ago

Cool! πŸ™‚

  
  
Posted 2 years ago

There is no need to add creds on the machine, since the EC2 instance has an attached IAM profile that grants access to s3. Boto3 is able retrieve the files from the s3 bucket

  
  
Posted 2 years ago

Yes, I will check now

  
  
Posted 2 years ago

πŸ‘

  
  
Posted 2 years ago

you should have access and secret too

  
  
Posted 2 years ago

ClearML uses the access and secret for creating the storage object, you can have those as env params too

  
  
Posted 2 years ago

Bon appetite 🍎

  
  
Posted 2 years ago

You can look around there, and you can also open a GitHub issue for us πŸ™‚

  
  
Posted 2 years ago

Yea I really need that feature, I need to move away from key/secrets to iam roles

  
  
Posted 2 years ago
823 Views
26 Answers
2 years ago
one year ago
Tags
aws
Similar posts