ClearML FAQ | Hello! I Have A Hard Time Connecting To Non-Aws S3 Bucket To Use It As A Storage To Clearml Dataset. Even Though I Add Access And Secret Key In ~/Clearml.Conf File, When I Trying To Create A New Clearml Dataset:

Answered

Hello! I Have A Hard Time Connecting To Non-Aws S3 Bucket To Use It As A Storage To Clearml Dataset. Even Though I Add Access And Secret Key In ~/Clearml.Conf File, When I Trying To Create A New Clearml Dataset:

Hello! I have a hard time connecting to non-aws s3 bucket to use it as a storage to clearml Dataset. Even though I add access and secret key in ~/clearml.conf file, when i trying to create a new clearml dataset:

from clearml import Dataset
cdataset = Dataset.create(
dataset_name='synthetica_train',
dataset_project='Seismic',
dataset_version="1.0",
description='Synthetic data for training',
output_uri = "s3://backet"
)
I get an error:

2023-04-07 16:38:34,629 - clearml.storage - ERROR - Failed uploading: Could not connect to the endpoint URL: "https://clearml-data-test.s3.ru-central1.amazonaws.com/.clearml.test"
I really can’t find out where this amazonaws.com/.clearml.test is coming from and how to set endpoint url of my non amazon aws s3 bucket, so it looks like

https://<my_endpoint>/<bucket>

Can some please help me?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AdventurousElephant3
				
					0
					 × 1

Votes Newest

Answers

Hi @<1555362936292118528:profile|AdventurousElephant3>

hard time connecting to non-aws s3 bucket

if this is a non-s3 the output_uri should look something like:
output_uri = " None :port/backet"
Then make sure you have the correct credentials in your clearml.conf:
None

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

693 Views

1 Answer

one year ago

one year ago

Tags

aws

Similar posts

Hi All, Quick Question About Clearml Datasets. Does Anyone Know If It Is Possible To Access (Could Just Be Paths To The Data In A Bucket) A Dataset Directly From S3, Instead Of Downloading A Local Copy? We Typically Store And Access Large Quantities Of

Hi! How Can I Delete Dataset From Ui And From S3 Bucket? I Tried To Delete From Ui And Then Checking But I Still Have It ... Api Client Doesn'T Have Methods To Work With Datasets ...

Hi Guys! Is There Any Way To Get Full State Of Dataset From Somewhere, Except My S3 Bucket? I Need A Mapping Of Files And Batches That Were Uploaded As A Dataset. Maybe This Information Is Also Available In One Of The Clearml Databases?

Hi, I’M Trying To Create A Dataset On Clearml Server From My Aws S3 Bucket Via:

Hello, For Datasets, Is There An Explanation Somewhere As To How The Debug Samples Are Created? I'M Not Entirely Sure Which Permission It Uses If The Dataset Is Stored In A Private Bucket

Hi Everyone, I'M Currently Trying To Add A Csv-File That Is Located In An S3-Bucket To An Existing Clearml Dataset Using The Following Code:

Hi Guys, I Have Data ( For Folder Tree Image Classification) And We Stored That On S3 Bucket Minio Can I Register Dataset To Clearml Without Downloading (Using Storage Manager) Data On S3 Bucket To My Local Then Upload To Clearml?

Does Anyone Know How To Setup “Web App Cloud Access” For Non-Aws Cloud Providers? Setting All The Fields “Bucket”, “Key”, “Secret”, “Aws Region” And “Host Endpoint” Has No Effect. Thus, Everytime An Experiment Is Deleted, Clearml Says That Artifacts Were

Hi! Im Trying To Log 150Gb Dataset With Clearml To S3 Bucket. Whats The Reason For This Problem?

Hi, Newbe Question In Clearml Data Assuming My Files Are Already In A Bucket On The Cloud - What Would Be The Best Practice To Create A Dataset And Register It In Clearml?