Hello, I Look Up To The Clearml Data Documentation, The Dataset.Add_External_Files Method Seems Like To Perform A Download From The Source Url. What If The Case I Already Have My Dataset In The Output_Uri That I Specify In The Dataset.Create? Btw I Use A

Answered

Hello, I look up to the clearml data documentation, the Dataset.add_external_files method seems like to perform a download from the source url. What if the case I already have my dataset in the output_uri that I specify in the Dataset.create? Btw I use a network storage to store the dataset

  				
Posted 
	2 days ago

					More
				  		
  Report
		
					ConfusedSealion46
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1784754456546512896:profile|ConfusedSealion46> , in that case you can simply use add_external_files to the files that are already in your storage. Or am I missing something?

  				
Posted 
	one day ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

I think it is an issue of confusion here. I thought that using the add_external_files causes the clearml server to download the dataset to the server. After investigating through the file system, it turns out that the server save it as a link in state.json

  				
Posted 
	one day ago

					More
				  		
  Report
		
					ConfusedSealion46
				
					0
					 × 1

Write your answer

24 Views

2 Answers

2 days ago

7 hours ago

Hi All! I’M Still New To This And Would Be Very Grateful If You Could Answer My Question Or Point Me To A Topic Where This Is Discussed. I’M Working On My First Pet Project, And I Managed To Create A Dataset And Train A Model. However, After That The Mode

Hello, Is There A Way To Add Info To A Dataset Task (The One That Is Created Automatically When Creating The Dataset) Without Causing A Disconnection Between The Dataset And The Task? I T Tried Using Dataset._Task, But It Disrupt The Connection Between Th

Hello! I Am Running Self-Hosted Clearml. Does Deleting Dataset From The Ui Also Remove It From Disk? I'Ve Just Deleted One (Send To Archive Then Delete) And Can Still See It On The Desk (

Hi All, I Am Working With An Instance Segmentation-Type Dataset. I Wanted To Know If There Is A Way To Visualize These Images And Their Webui In The Free Tier. I Have Seen Some Samples, But They Seem To Be Only For Hyper-Datasets. Could Some Correct Me I

When I'M Looking At A Frames Of A Dataset Version That Is Stored In Private Blob Storage I Can'T See The Photos, It First Prompts Me To Enter A Sas Token. I Entered A Sas Token, But I Still Can'T See The Photos. I Assume This Is Because The Sas Token I En

Hi. I Am Using Clearml Server On Premises To Manage Ml Data And Tasks/Experiments. Is It Possible To Re-Create/Upload From Disk A Final Dataset? One Dataset Part Is Corrupted On Disk At The File Server And Cannot Longer Be Downloaded. Thanks In Advance!

Hi!! I Have A Question, I Have A Dataset Saved In S3 Using Clearml, Is There A Way Of Getting The Full Path Of This Dataset In S3?? Something Like

Hi Community, I'M Installing Multiple Clearml-Agent On Some Gpu Workstations, So I Have A Question: How Can I Use The Cache Of Dataset As When I Run Only 1 Clearml-Agent? I Don'T Want To Download Full Dataset Everytime When A Task Run Because Now Since I

Hi Abit Of A Crazy Question... But Is It Possible To Use Clearml In Rust, Without Writing A Wrapper. I Noticed The Api Doesnt Cover Dataset Operations But The Cli Can.