Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Everyone! I’M Trying To Setup Non Aws S3 To Store Datasets And Get And Error:

Hello everyone!
I’m trying to setup non aws s3 to store datasets and get and error:

2024-05-27 16:15:52,241 - clearml.storage - ERROR - Failed uploading: Failed to upload /tmp/state.pewntfqw.json to clearml-data/NAME/.datasets/NAME_1/NAME_1.9537e7b56828492a83acae01ab2bf4/artifacts/state/state.json: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

My config looks like:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 
  
  
Posted 5 months ago
Votes Newest

Answers 15


Get the same result
But then I will have an empty credentials inside S3BucketConfigurations , here

Upd: if I delete keys ‘key’ and ‘secret_key’ from top section I got

2024-05-27 16:29:51,597 - clearml.storage - ERROR - Failed creating storage object 
 Reason: Missing key and secret for S3 storage access (
)
  
  
Posted 5 months ago

Try with " None :port/bucket" or something along those lines, the SDK needs to be aware this is not AWS s3 from the output_uri

  
  
Posted 5 months ago

Hi @<1706116294329241600:profile|MinuteMouse44> , this section is only for aws s3 configuration. Please see the minio example - None

This is the relevant section:

sdk {
    aws {
            s3 {
                # default, used for any bucket not specified below
                key: ""
                secret: ""
                region: ""
    
                credentials: [
                    {
                        # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                        host: "my-minio-host:9000"
                        key: ""
                        secret: ""
                        multipart: false
                        secure: false
                    }
                ]
            } 
    }
}
  
  
Posted 5 months ago

@<1523701070390366208:profile|CostlyOstrich36> mb any other ideas, how to fix? 🫠

  
  
Posted 5 months ago

@<1523701070390366208:profile|CostlyOstrich36> looks the same to me, I use it this example to setup my confign
Or is there any diff?

  
  
Posted 5 months ago

I think the sdk is aware that this is not amazon, because when I specify the ‘region’ field then it ignores my host and uses the amazon host

  
  
Posted 5 months ago

How did you set the output URI?

  
  
Posted 5 months ago

Non-AWS S3-like services (e.g. MinIO):

:port/bucket

  
  
Posted 5 months ago

I tried:

1. output_url = "
"
2. output_url = "
"
3. output_url = "s3://"
  
  
Posted 5 months ago

Try removing the key/secret from the top section in s3

  
  
Posted 5 months ago

Do try with the port through

  
  
Posted 5 months ago

Now I use same config, as in original question:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 

And pass output_url as:

dataset.upload(show_progress=True, verbose=True,
               output_url = "
:port/bucket_name/")

And get an error:

2024-05-28 16:31:40,818 - clearml.storage - INFO - Uploading: 0.00MB / 0.00MB @ 0.00MBs to /tmp/state.8qhb6dfu.json
2024-05-28 16:31:40,819 - clearml.storage - ERROR - Failed uploading: Connection was closed before we received a valid response from endpoint URL: "
:port/bucket/NAME/.datasets/NAME_/NAME__.4feaeb4d8285adbc778f61d262/artifacts/state/state.json".
  
  
Posted 5 months ago

Can you add full configuration of both and add what the error you get with latest configuration?

  
  
Posted 5 months ago

I tried None :port/bucket
But result still the same 🥲

  
  
Posted 5 months ago