Hello, Is There A Way To Specify The Default Storage To Be S3 (Minio) And Not The Local File System For The Self Hosted (Docker Compose) Version Of Clearml? Is There A Setting For This In The Configuration File? Should The Path Substitutions Be Used For T

Answered

Hello, is there a way to specify the default storage to be s3 (minio) and not the local file system for the self hosted (docker compose) version of clearml? is there a setting for this in the configuration file? should the path substitutions be used for this? if yes, what to substitute?

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					EmaciatedSeal35
				
					0
					 × 1

Votes Newest

Answers 2

This way the agent has direct access to s3, i may case I wan it to go though the ClearML server and the ClearML server to fethc the data for the agent

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					EmaciatedSeal35
				
					0
					 × 1

When I was setting up the ClearML agent it asked for a default storage path, which I pointed to s3 and it works as expected.

clearml-agent init

# Default Task output_uri. if output_uri is not provided to Task.init, default_output_uri will be used instead.
sdk.development.default_output_uri="s3:..."

Is this what you have in mind?

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					WhimsicalChimpanzee6
				
					0

Write your answer

67 Views

2 Answers

14 days ago

13 days ago

I Want The Location To Be Not From A Tmp File

But For Example The Pytorch Versions Would Be Different For The Two, Macbook Is Running A Different Version And My Gpu(An Nvidia Machine) An Another Version, How Can I Make Sure They Can Both Run Seamlessly Without Having Any Compatibility Issue?

Hi Everyone Can Clearml Session Be Somehow Saved, If The Host Machine For Docker Container, Where Agents Are Running, Was Rebooted?

Hi All, I Am Working With An Instance Segmentation-Type Dataset. I Wanted To Know If There Is A Way To Visualize These Images And Their Webui In The Free Tier. I Have Seen Some Samples, But They Seem To Be Only For Hyper-Datasets. Could Some Correct Me I

Hi All, My Organisation Wants To Be Able To Trace The Changes Of All Dataset Versions From The Latest Version To The Root (Aka We Would Like To Be Able To Query And Display The Entire Version Tree). However, Clearml From My Knowledge Does Not Allow For Th

Hi Everyone, I'M Currently Trying To Set Up S3 As A File Server For Storing All My Data (Artifacts, Datasets, Etc.) Using Clearml, But I'M Encountering Some Issues With The Configuration. I Am Getting This Error

Hi There, Our. Self-Hosted Server Is Periodically Very Slow To React In The Web Ui. We'Ve Been Debugging For Quite Some Time, And It Would Seem That Elastisearch Might Be The Culprit. Looking At The Elastisearch Index, We Have An Index Of Around 80G Of Tr

Hi. I Am Using Clearml Server On Premises To Manage Ml Data And Tasks/Experiments. Is It Possible To Re-Create/Upload From Disk A Final Dataset? One Dataset Part Is Corrupted On Disk At The File Server And Cannot Longer Be Downloaded. Thanks In Advance!

Hi, I'M Trying To Setup The Storage (Aws S3) For All Components Of Clearml. I'M Deploying Clearml On Kubernetes With The Official Helm Charts. I Can'T Find A Way To Inject The Configuration File In The Webserver / Apiserver / Fileserver. With The Agent,