My current setup is:
sdk.development.default_output_uri=< None > # no port, no bucket
sdk.aws.s3.key=<my-access-key>
sdk.aws.s3.secret=<my-secret-key>
sdk.aws.s3.region=<my-region> # I think it can be skipped but somewhere in the clearml code it says that it must be specified if it's not default like us-east-1 or something
sdk.aws.s3.credentials.bucket=<my-bucket> # just a bucket name
sdk.aws.s3.credentials.host=< None : 443> # the same as output_uri and plugged 443 port in
sdk.aws.s3.credentials.key=<my-access-key>
sdk.aws.s3.credentials.secret=<my-secret-key>
sdk.aws.s3.credentials.region=<my-region>
sdk.aws.s3.credentials.secure=true
@<1523701070390366208:profile|CostlyOstrich36>
My question could be this: what's get plugged into endpoint_url in boto3 client inside ClearML?
@<1523701087100473344:profile|SuccessfulKoala55> So I have to provide a host for it to work and no other way around it?
Maybe @<1523701087100473344:profile|SuccessfulKoala55> has more insight into this 🙂
Docstring from inside the boto3 lib says:
:param endpoint_url: The complete URL to use for the constructed
client. Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service. You
can specify a complete URL (including the "http/https" scheme)
to override this behavior. If this value is provided,
then ``use_ssl`` is ignored.
I want ClearML to use my endpoint
@<1526734383564722176:profile|BoredBat47> if your S3 server is using https (which I assume it does) the port will be 443
@<1523701070390366208:profile|CostlyOstrich36> You mean using port in credentials.host ?
This is what it does when you specify a port...
@<1523701087100473344:profile|SuccessfulKoala55> It's the URL I use when creating boto3 session from Python like this fro example
s3 = self.session.client(
service_name='s3',
endpoint_url=endpoint,
verify=False
)
Hi @<1526734383564722176:profile|BoredBat47> , it should be very easy and I've done it multiple times. For the quickest fix you can use api.files_server in clearml.conf
If it points to your own S3 server, it must have a port
@<1526734383564722176:profile|BoredBat47> , what happens if you configure it like @<1523701087100473344:profile|SuccessfulKoala55> is suggesting?
@<1523701087100473344:profile|SuccessfulKoala55> Right
After I run my experiment I have a console error that says I am missing security headers. This is a custom XML response. The same behaviour could be achieved when just trying to curl the endpoint or plug it in the browser. When I run e.g. boto3 client where I explicitly specify endpoint, ak, sk and bucket I could do whatever I want. So it seems to me ClearML is trying to get to this endpoint in some incorrect way
No port needed when accessing this URL from things like boto3 or s3-client CLI
A port is needed when using it through ClearML
sdk.development.default_output_uri=<
no port, no bucket
@<1526734383564722176:profile|BoredBat47> this points to the s3 server?
session = boto3.Session(
aws_access_key_id=self.access_key,
aws_secret_access_key=self.secret_key)
He tried to help me in another thread but I still couldn't make things work
Sorry, guys, maybe I am not expressing myself clear or it's something I am missing, I am not a native speaker so I'll try to reformulate. What we have is enterprise solution built on S3 technology, I don't have an access to servers on where it's run, I don't have a port. All I have been provided with are: secret key, access key, endpoint that looks like a regular web URL and a bucket name. Using these creds I can access this cloud storage just fine by any means except ClearML
@<1523701087100473344:profile|SuccessfulKoala55> No port needed when accessing this URL from things like boto3 or s3-client CLI
Thanks for the reply! We have a custom S3 server, it has an URL — endpoint like https://<some-domain>.<sub-domain>. I've read in docs that when you provide credentials.host — port must be specified. @<1523701070390366208:profile|CostlyOstrich36>