@<1523701435869433856:profile|SmugDolphin23> I didn't use a region at first and that was not working. Now I use a region and it still doesn't work.
From the boto3 inside a Python I could create a session where I specify ak and sk, and create a client from the session where I pass service_name and endpoint_url. It works just fine
@<1523701087100473344:profile|SuccessfulKoala55> I figured where to find a region but we don't have an AWS dashboard. We have a custom S3 solution for our own enterprise servers like many companies do, data is not stored on amazon servers. That is why we have and endpoint which is an URL starting with http://
If I would connect to our bucket via boto3 I would pass endpoint to a client session with endpoint_url
SmugDolphin23 Thank you very much!
That's clearml.conf for ClearML end users right?
@<1526734383564722176:profile|BoredBat47> Just to check if u need to do update-ca-certificates or equivalent?
May I know where to set the cert to in env variable?
The code is run from another machine where clearml.conf configured to connect to ClearML server, no other configurations are provided
@<1523701435869433856:profile|SmugDolphin23> Thanks a lot, that actually worked! It was very difficult to figure out you have to plug those exact values given you have https endpoint:
- Using s3 protocol instead of https together with bucket name in output URI
- Not providing a bucket name in credentials section where it is by default
- Providing default secure port for both host and output URI
- Disabling credentials chainI think a common use case for many people that they get S3 storage with integrated Amazon solution where they are provided with region and a bucket name. Together with access key it's sufficient to connect to their cloud. But a lot of people, especially in enterprise have a case like mine where they have https endpoint to their company hosted S3 solution so I think it would be great to reflect that case in documentation so other people would have easier time to configure https endpoints for clearml-agent. Another thing would be nice to have is to support endpoint parameter under S3 section of clearml.conf which if provided as is (with https and no port) is sufficient to connect to S3 bucket. That would require some coding and rewriting URL constructing methods and maybe boto3 calls (I peeked inside a code and would say some places regarding this issue were questionable e.g. init method in _Container class in helper.py). I would try to fix it myself and make a pull request if working schedule lets me but I can't make a promise on that.
@<1523701070390366208:profile|CostlyOstrich36> @<1523701087100473344:profile|SuccessfulKoala55> H. Thank you too for helping! Would be great if you'd try to look at the issue I discussed in this message.
Good luck, guys!
Could you try adding region
under credentials
as well?
I think that will work, but I'm not sure actually. I know for sure that something like us-east-2
is supported
@<1523701304709353472:profile|OddShrimp85> I fixed my SSL error by putting REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
in .bashrc
file
@<1523701087100473344:profile|SuccessfulKoala55> Hey, Jake, getting back to you. I couldn't be able to resolve my issue. I can access my bucket by any means just fine, e.g. by S3 CLI client. All the tools I use require 4 params: AK, SK, endpoint, bucket. I wonder why ClearML doesn't have explicit endpoint
parameter and you have to use output_uri
for it and why is there a region
when other tools don't require it.