Hi @<1706116294329241600:profile|MinuteMouse44> , this section is only for aws s3 configuration. Please see the minio example - None
This is the relevant section:
sdk {
aws {
s3 {
# default, used for any bucket not specified below
key: ""
secret: ""
region: ""
credentials: [
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "my-minio-host:9000"
key: ""
secret: ""
multipart: false
secure: false
}
]
}
}
}
Get the same result
But then I will have an empty credentials inside S3BucketConfigurations
, here
Upd: if I delete keys ‘key’ and ‘secret_key’ from top section I got
2024-05-27 16:29:51,597 - clearml.storage - ERROR - Failed creating storage object
Reason: Missing key and secret for S3 storage access (
)
@<1523701070390366208:profile|CostlyOstrich36> looks the same to me, I use it this example to setup my confign
Or is there any diff?
Try removing the key/secret from the top section in s3
@<1523701070390366208:profile|CostlyOstrich36> mb any other ideas, how to fix? 🫠
Try with " None :port/bucket" or something along those lines, the SDK needs to be aware this is not AWS s3 from the output_uri
I tried:
1. output_url = "
"
2. output_url = "
"
3. output_url = "s3://"
I think the sdk is aware that this is not amazon, because when I specify the ‘region’ field then it ignores my host and uses the amazon host
Non-AWS S3-like services (e.g. MinIO):
:port/bucket
I tried None :port/bucket
But result still the same 🥲
Now I use same config, as in original question:
s3 {
key: "key"
secret: "sec_key"
credentials: [
{
host: "host:port"
bucket: "bucket_name"
key: "key"
secret: "sec_key"
multipart: false
secure: false
}
]
}
And pass output_url as:
dataset.upload(show_progress=True, verbose=True,
output_url = "
:port/bucket_name/")
And get an error:
2024-05-28 16:31:40,818 - clearml.storage - INFO - Uploading: 0.00MB / 0.00MB @ 0.00MBs to /tmp/state.8qhb6dfu.json
2024-05-28 16:31:40,819 - clearml.storage - ERROR - Failed uploading: Connection was closed before we received a valid response from endpoint URL: "
:port/bucket/NAME/.datasets/NAME_/NAME__.4feaeb4d8285adbc778f61d262/artifacts/state/state.json".
Can you add full configuration of both and add what the error you get with latest configuration?