Hi @<1706116294329241600:profile|MinuteMouse44> , this section is only for aws s3 configuration. Please see the minio example - None

This is the relevant section:

sdk {
    aws {
            s3 {
                # default, used for any bucket not specified below
                key: ""
                secret: ""
                region: ""
    
                credentials: [
                    {
                        # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                        host: "my-minio-host:9000"
                        key: ""
                        secret: ""
                        multipart: false
                        secure: false
                    }
                ]
            } 
    }
}

@<1523701070390366208:profile|CostlyOstrich36> looks the same to me, I use it this example to setup my confign
Or is there any diff?

Try removing the key/secret from the top section in s3

Get the same result
But then I will have an empty credentials inside S3BucketConfigurations , here

Upd: if I delete keys ‘key’ and ‘secret_key’ from top section I got

2024-05-27 16:29:51,597 - clearml.storage - ERROR - Failed creating storage object 

 Reason: Missing key and secret for S3 storage access (

)

@<1523701070390366208:profile|CostlyOstrich36> mb any other ideas, how to fix? 🫠

How did you set the output URI?

I tried:

1. output_url = "

"
2. output_url = "

"
3. output_url = "s3://"

Try with " None :port/bucket" or something along those lines, the SDK needs to be aware this is not AWS s3 from the output_uri

I think the sdk is aware that this is not amazon, because when I specify the ‘region’ field then it ignores my host and uses the amazon host

Do try with the port through

None

Non-AWS S3-like services (e.g. MinIO):

:port/bucket

I tried None :port/bucket
But result still the same 🥲

Can you add full configuration of both and add what the error you get with latest configuration?

Now I use same config, as in original question:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 

And pass output_url as:

dataset.upload(show_progress=True, verbose=True,
               output_url = "

:port/bucket_name/")

And get an error:

2024-05-28 16:31:40,818 - clearml.storage - INFO - Uploading: 0.00MB / 0.00MB @ 0.00MBs to /tmp/state.8qhb6dfu.json
2024-05-28 16:31:40,819 - clearml.storage - ERROR - Failed uploading: Connection was closed before we received a valid response from endpoint URL: "

:port/bucket/NAME/.datasets/NAME_/NAME__.4feaeb4d8285adbc778f61d262/artifacts/state/state.json".