yup - this is handled by the secure: true part
so you should have something like:credentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: " " key: "REMOVED" secret: "REMOVED" multipart: false secure: true region: "us-east-1"
Wait - adding the output_uri seems to work.
Wait, can you try calling boto3 like you did without the bucket?
Just to make sure that's the source of the error
Ok - it's the URL in the files_server that was wrong. It needs to be s3 and not https.
Nope - bucket_name in clearml.conf didn't work. Maybe default_uri somewhere?
Is that because I didn't list the bucket name in the clearml.conf?
Same response. Should I change that in the fileserver section too?
No worries. I probably should have revisited the examples. Too much cutting/pasting on my part. Thanks so much for helping!
I think you shouldn't have the scheme in the host definitioncredentials: [ { # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket) host: "STORAGE_GRID_URL:443" key: "REMOVED" secret: "REMOVED" multipart: false secure: true region: "us-east-1"
Great! Now to tell our IT that I need more space on S3 🙂
Now to get clearml-data to use S3... 🙂
you can add output_uri= https://STORAGE_GRID_URL:443/ , but I don't think that's it
Well, the nice thing is, if the SDK works with it, clearml-data basically uses the SDK, so... 😄
I added secure and region - didn't change the behavior.
I do see, looking at the code, that we're not passing the region_name to boto, maybe that's it
Also - I'm not specifying the URI when I create the Task
Oh, right, that's totally my bad, sorry - too late 😞