If I set it to False I get another error:Failed creating storage object
Reason: Missing key and secret for S3 storage access (
)
BTW, is it correct to set the
files_server
in the
api
section?
files_server: "
"
yes, should be fine
containing the correct on-premises s3 settings
Do you mean like an example for minio?
Wait, the aws
configuration section should be under sdk
I think it would be good to know since it might be an issue related to some specific integration with this storage solution (for example, there were some things to figure out when we first started working with minio 🙂 )
Oh, I see... Well, good to know - I'm not sure there's anything we can do to provide more info in this case if it happens to anyone else 😞
Well. what’s for sure is that I have the required permissions to write to the bucket, as I manage to upload files into it through s3cmd
and boto3
Do you mean like an example for minio?
Yeah, but with the output_uri
in task initialisation as well. Am I right that in that case it would be like that?output_uri='
s3://my-minio-host:9000/bucket_name '
The error you're getting looks like some kind of permissions issue perhaps?
I’m also not exactly an expert here, but it must be Ceph if it’s possible to be so
I was just wondering if there’s some valid example of a clearml.conf
containing the correct on-premises s3 settings so that I could use them as a basis?
I know, but which is it exactly? minio?
Did that and still have the same error:Failed creating storage object
Reason: Missing key and secret for S3 storage access (
)
So it’s Ceph (RADOS) Object Gateway in my case
BTW, is it correct to set the files_server
in the api
section?files_server: "
s3://s3.kontur.host:443/srs-clearml "
With this variant of clearml.config I’m now getting a new error:ERROR - Exception encountered while uploading Failed uploading object s3.kontur.host:443/srs-clearml/SpeechLab/ASR/data_logging/test1.1be56a53647646208ffd665908056d49/artifacts/data/valset_2021_02_01_sb_manifest_true_micro.json (405): <?xml version="1.0" encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx00000000000000000fc69-0062781afb-eba8e9-default</RequestId><HostId>eba8e9-default-default</HostId></Error>
the
secure
flag is
false
I played with this setting as well - didn’t make it work
Yeah, it holds. I just sent an extract from the config for it to be concise. Here’s the full version
I think Ceph should work as well, from past experience
Probably so, but not sure:( I’ll have to figure it out with our DevOps engineer
Finally solved it. Turned out it was an authentication issue. In my case, I had to use values for ACCESS_KEY/SECRET other than those which I used with boto3 client
I assume you have actual values for key
and secret
in:credentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: "s3.kontur.host:443" secure: false key: "" secret: "" multipart: false }
Right?
SillySealion58 I think your issue might be with use_credentials_chain: true
- any specific reason to set it to true
?
It’s a self-hosted one. Its address is s3.kontur.host, port 443
clearml 1.3.2
boto3==1.22.7
botocore==1.25.7
I didn’t deploy the server myself but I verified that it works with s3cmd
SillySealion58
something look odd with the bucket settings.
url has port 443
which usually means ssl secure, but the secure
flag is false
OK, I get it. So the discrepancy is between this:output_uri='
s3://s3.kontur.host/srs-clearml '
and this:credentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: "s3.kontur.host:443" secure: false key: "" secret: "" multipart: false }
Since clearml can't match the host (it is expected to be exactly the same).
I think you need to do:output_uri='
s3://s3.kontur.host:443/srs-clearml '
I assume you have actual values for
key
and
secret
in:
That’s right, I use the same values which work for that bucket with s3cmd
What's the exact S3 server you're running?