I think the sdk is aware that this is not amazon, because when I specify the ‘region’ field then it ignores my host and uses the amazon host

Can you add full configuration of both and add what the error you get with latest configuration?

Try removing the key/secret from the top section in s3

Do try with the port through

@<1523701070390366208:profile|CostlyOstrich36> looks the same to me, I use it this example to setup my confign
Or is there any diff?

I tried:

1. output_url = "

"
2. output_url = "

"
3. output_url = "s3://"

How did you set the output URI?

Try with " None :port/bucket" or something along those lines, the SDK needs to be aware this is not AWS s3 from the output_uri

Non-AWS S3-like services (e.g. MinIO):

:port/bucket

Hi @<1706116294329241600:profile|MinuteMouse44> , this section is only for aws s3 configuration. Please see the minio example - None

This is the relevant section:

sdk {
    aws {
            s3 {
                # default, used for any bucket not specified below
                key: ""
                secret: ""
                region: ""
    
                credentials: [
                    {
                        # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                        host: "my-minio-host:9000"
                        key: ""
                        secret: ""
                        multipart: false
                        secure: false
                    }
                ]
            } 
    }
}

None

I tried None :port/bucket
But result still the same 🥲

@<1523701070390366208:profile|CostlyOstrich36> mb any other ideas, how to fix? 🫠

Now I use same config, as in original question:

s3 {
                key: "key"
                secret: "sec_key"
    
                credentials: [
                    {
                        host: "host:port"
                        bucket: "bucket_name"
                        key: "key"
                        secret: "sec_key"
                        multipart: false
                        secure: false
                    }
                ]
            } 

And pass output_url as:

dataset.upload(show_progress=True, verbose=True,
               output_url = "

:port/bucket_name/")

And get an error:

2024-05-28 16:31:40,818 - clearml.storage - INFO - Uploading: 0.00MB / 0.00MB @ 0.00MBs to /tmp/state.8qhb6dfu.json
2024-05-28 16:31:40,819 - clearml.storage - ERROR - Failed uploading: Connection was closed before we received a valid response from endpoint URL: "

:port/bucket/NAME/.datasets/NAME_/NAME__.4feaeb4d8285adbc778f61d262/artifacts/state/state.json".

Get the same result
But then I will have an empty credentials inside S3BucketConfigurations , here

Upd: if I delete keys ‘key’ and ‘secret_key’ from top section I got

2024-05-27 16:29:51,597 - clearml.storage - ERROR - Failed creating storage object 

 Reason: Missing key and secret for S3 storage access (

)