Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone! I’M Trying To Setup Non Aws S3 To Store Datasets And Get And Error:

Hello everyone!
I’m trying to setup non aws s3 to store datasets and get and error:

2024-05-27 16:15:52,241 - clearml.storage - ERROR - Failed uploading: Failed to upload /tmp/state.pewntfqw.json to clearml-data/NAME/.datasets/NAME_1/NAME_1.9537e7b56828492a83acae01ab2bf4/artifacts/state/state.json: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

My config looks like:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 
  
  
Posted 26 days ago
Votes Newest

Answers 15


Hi @<1706116294329241600:profile|MinuteMouse44> , this section is only for aws s3 configuration. Please see the minio example - None

This is the relevant section:

sdk {
    aws {
            s3 {
                # default, used for any bucket not specified below
                key: ""
                secret: ""
                region: ""
    
                credentials: [
                    {
                        # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                        host: "my-minio-host:9000"
                        key: ""
                        secret: ""
                        multipart: false
                        secure: false
                    }
                ]
            } 
    }
}
  
  
Posted 26 days ago

I tried None :port/bucket
But result still the same 🥲

  
  
Posted 25 days ago

I tried:

1. output_url = "
"
2. output_url = "
"
3. output_url = "s3://"
  
  
Posted 25 days ago

Try removing the key/secret from the top section in s3

  
  
Posted 26 days ago

Now I use same config, as in original question:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 

And pass output_url as:

dataset.upload(show_progress=True, verbose=True,
               output_url = "
:port/bucket_name/")

And get an error:

2024-05-28 16:31:40,818 - clearml.storage - INFO - Uploading: 0.00MB / 0.00MB @ 0.00MBs to /tmp/state.8qhb6dfu.json
2024-05-28 16:31:40,819 - clearml.storage - ERROR - Failed uploading: Connection was closed before we received a valid response from endpoint URL: "
:port/bucket/NAME/.datasets/NAME_/NAME__.4feaeb4d8285adbc778f61d262/artifacts/state/state.json".
  
  
Posted 25 days ago

Get the same result
But then I will have an empty credentials inside S3BucketConfigurations , here

Upd: if I delete keys ‘key’ and ‘secret_key’ from top section I got

2024-05-27 16:29:51,597 - clearml.storage - ERROR - Failed creating storage object 
 Reason: Missing key and secret for S3 storage access (
)
  
  
Posted 26 days ago

I think the sdk is aware that this is not amazon, because when I specify the ‘region’ field then it ignores my host and uses the amazon host

  
  
Posted 25 days ago

Try with " None :port/bucket" or something along those lines, the SDK needs to be aware this is not AWS s3 from the output_uri

  
  
Posted 25 days ago

How did you set the output URI?

  
  
Posted 25 days ago

@<1523701070390366208:profile|CostlyOstrich36> looks the same to me, I use it this example to setup my confign
Or is there any diff?

  
  
Posted 26 days ago

Can you add full configuration of both and add what the error you get with latest configuration?

  
  
Posted 25 days ago

@<1523701070390366208:profile|CostlyOstrich36> mb any other ideas, how to fix? 🫠

  
  
Posted 26 days ago

Non-AWS S3-like services (e.g. MinIO):

:port/bucket

  
  
Posted 25 days ago

Do try with the port through

  
  
Posted 25 days ago