Hi @<1590514584836378624:profile|AmiableSeaturtle81> ! To help us debug this: are you able to simply use the boto3 python package to interact with your cluster?
If so, how does that code look like? This would give us some insight on how the config should actually look like or what changes need to be made.

Can you actually add the bucket to the credentials just to try it out?
Also, can you check that this snippet works for you (with your creds):

import boto3
import json
import six

key = ""
secret = ""
host = "our_host.com"
bucket_name = "bucket"
profile = None
filename = "test"
data = {"test": "data"}

boto_session = boto3.Session(aws_access_key_id=key, aws_secret_access_key=secret, profile_name=profile)
endpoint = "https://" + host
boto_resource = boto_session.resource("s3", region_name=None, endpoint_url=endpoint)
bucket = boto_resource.Bucket(bucket_name)
bucket.put_object(Key=filename, Body=six.b(json.dumps(data)))

Hi @<1590514584836378624:profile|AmiableSeaturtle81> , you need to setup your s3 key/secret in clearml.conf
I suggest following this documentation - None

Or whatever port you use

Adding bucket in clearml.conf causes the same error: clearml.storage - ERROR - Failed uploading: Could not connect to the endpoint URL: " None "

@<1590514584836378624:profile|AmiableSeaturtle81> , please see the section regarding minio in the documentation - None

Hi, ok im really close now to working system
Debug image is uploading to s3, im seeing the files, all ok there

Problem now is viewing these images in web UI
Going to Debug Samples panel in Task drops me a popup to fill in s3 credentials

I cant figure out what the right setup is for the creds to work
This is what I have now (Note that we dont have region)

digging deeper it seems like a parsing issue

@<1523701435869433856:profile|SmugDolphin23> Any ideas how to fix this?

we do, yes. Changing it to https in settings doesnt help

@<1590514584836378624:profile|AmiableSeaturtle81> weren't you using https for the s3 host? maybe the issue has something to do with that?

As I wrote, you need to remove the s3 from the start of the host section..

@<1523701070390366208:profile|CostlyOstrich36> Any news on this? We are currently stuck without this fix, cant finish up clearml setup

removing it doesnt fix the problem

This is the link generated

Meaning that you should configure your host as follows host: "somehost.com:9000"

i need clearml.conf on my clearml server (in config folder which is mounted in docker-compose) or user PC? Or Both?
Its self hosted S3 thats all I know, i dont think it s Minio

In which ui? Because there are two ways to do it. When clicking on artifacti url there is a popup (but has no way to change host url)
Our s3 host doesnt have port (didnt specify port in clearml.conf anywhere and upload works)

Hi @<1590514584836378624:profile|AmiableSeaturtle81> , any non-AWS S3-like storage must have a port in this setup, how did you configure the SDK?
Also the two ways you're showing are the same - the popup will fill in the details in the settings page

• Here is how client side clearml.conf looks like together with the script im using to create the tasks. Uploads seems to work and is fixed thanks to you guys 🙌
  
  
  

I think that the problem is with missing region definition. You need to set region in the config file.

But it looks like that for the existing version it will not work since there still appears to be a bug related to this. The hotfix is already on the way from my understanding

So, in short, you need to set the region in the config file + wait for the hotfix that is pending for 1.14

Hi @<1590514584836378624:profile|AmiableSeaturtle81> , the hotfix should right around the corner 🙂

Bump, still waiting, closing in on a month since we are unable to deploy. We have team of 10+ people

2024-02-08 11:23:52,150 - clearml.storage - ERROR - Failed creating storage object

Reason: Missing key and secret for S3 storage access (

)

(edited)

This looks unrelated, to the hotfix, it looks like you misconfigured something and therefor failing to write to s3

I tried it with port, but still having the same issue
Tried it with/without secure and multipart

I know these keys work, url and everything else works because I use these creds daily

host: "my-minio-host:9000"

host: "my-minio-host:9000"

The port should be whatever port that is used by your S3 solution

will it be appended in clearml?
"s3" is part of domain to the host