Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! I’M Having Trouble With Connecting My On-Premises S3 Storage To Clearml. Could Anyone Please Tell Me How To Fix It?

Hello!
I’m having trouble with connecting my on-premises S3 storage to ClearML. Could anyone please tell me how to fix it?

  
  
Posted 2 years ago
Votes Newest

Answers 30


the

secure

flag is

false

I played with this setting as well - didn’t make it work

  
  
Posted 2 years ago

So it’s Ceph (RADOS) Object Gateway in my case

  
  
Posted 2 years ago

Yeah, it holds. I just sent an extract from the config for it to be concise. Here’s the full version

  
  
Posted 2 years ago

I think Ceph should work as well, from past experience

  
  
Posted 2 years ago

SillySealion58 I think your issue might be with use_credentials_chain: true - any specific reason to set it to true ?

  
  
Posted 2 years ago

The error you're getting looks like some kind of permissions issue perhaps?

  
  
Posted 2 years ago

Wait, the aws configuration section should be under sdk

  
  
Posted 2 years ago

clearml 1.3.2
boto3==1.22.7
botocore==1.25.7
I didn’t deploy the server myself but I verified that it works with s3cmd

  
  
Posted 2 years ago

Do you mean like an example for minio?

Yeah, but with the output_uri in task initialisation as well. Am I right that in that case it would be like that?
output_uri=' s3://my-minio-host:9000/bucket_name '

  
  
Posted 2 years ago

I was just wondering if there’s some valid example of a clearml.conf containing the correct on-premises s3 settings so that I could use them as a basis?

  
  
Posted 2 years ago

containing the correct on-premises s3 settings

Do you mean like an example for minio?

  
  
Posted 2 years ago

Well. what’s for sure is that I have the required permissions to write to the bucket, as I manage to upload files into it through s3cmd and boto3

  
  
Posted 2 years ago

I assume you have actual values for key and secret in:
credentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: "s3.kontur.host:443" secure: false key: "" secret: "" multipart: false }Right?

  
  
Posted 2 years ago

I assume you have actual values for

key

and

secret

in:

That’s right, I use the same values which work for that bucket with s3cmd

  
  
Posted 2 years ago

BTW, is it correct to set the

files_server

in the

api

section?

files_server: "

"

yes, should be fine

  
  
Posted 2 years ago

Finally solved it. Turned out it was an authentication issue. In my case, I had to use values for ACCESS_KEY/SECRET other than those which I used with boto3 client

  
  
Posted 2 years ago

It’s a self-hosted one. Its address is s3.kontur.host, port 443

  
  
Posted 2 years ago

With this variant of clearml.config I’m now getting a new error:
ERROR - Exception encountered while uploading Failed uploading object s3.kontur.host:443/srs-clearml/SpeechLab/ASR/data_logging/test1.1be56a53647646208ffd665908056d49/artifacts/data/valset_2021_02_01_sb_manifest_true_micro.json (405): <?xml version="1.0" encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx00000000000000000fc69-0062781afb-eba8e9-default</RequestId><HostId>eba8e9-default-default</HostId></Error>

  
  
Posted 2 years ago

I know, but which is it exactly? minio?

  
  
Posted 2 years ago

I’m also not exactly an expert here, but it must be Ceph if it’s possible to be so

  
  
Posted 2 years ago

BTW, is it correct to set the files_server in the api section?
files_server: " s3://s3.kontur.host:443/srs-clearml "

  
  
Posted 2 years ago

Probably so, but not sure:( I’ll have to figure it out with our DevOps engineer

  
  
Posted 2 years ago

What's the exact S3 server you're running?

  
  
Posted 2 years ago

OK, I get it. So the discrepancy is between this:
output_uri=' s3://s3.kontur.host/srs-clearml '
and this:
credentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: "s3.kontur.host:443" secure: false key: "" secret: "" multipart: false }Since clearml can't match the host (it is expected to be exactly the same).
I think you need to do:
output_uri=' s3://s3.kontur.host:443/srs-clearml '

  
  
Posted 2 years ago

SillySealion58
something look odd with the bucket settings.
url has port 443 which usually means ssl secure, but the secure flag is false

  
  
Posted 2 years ago

If I set it to False I get another error:
Failed creating storage object Reason: Missing key and secret for S3 storage access ( )

  
  
Posted 2 years ago

Yes, that's right

  
  
Posted 2 years ago

Oh, I see... Well, good to know - I'm not sure there's anything we can do to provide more info in this case if it happens to anyone else 😞

  
  
Posted 2 years ago

Did that and still have the same error:
Failed creating storage object Reason: Missing key and secret for S3 storage access ( )

  
  
Posted 2 years ago

I think it would be good to know since it might be an issue related to some specific integration with this storage solution (for example, there were some things to figure out when we first started working with minio 🙂 )

  
  
Posted 2 years ago