Hi, I Know That If You Have A Child Dataset Of A Dataset With Zips, And If The Parent Has Been Cached Locally, The Files In The Zips Would Be Symlinked To The Parent'S In

Answered

Hi, I know that if you have a child dataset of a dataset with zips, and if the parent has been cached locally, the files in the zips would be symlinked to the parent's in child.get_local_copy() . But now I have a child dataset of a dataset with external files. When I call child.get_local_copy() , even though I have the parent already downloaded in the cache, the child.get_local_copy() redownloads all external files that it inherited from the parent. Is this intended behavior?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

Votes Newest

Answers 9

@<1523701070390366208:profile|CostlyOstrich36> 👀

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

child-ext local cache (without a symlink to the parent):

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

Hi @<1709015393701466112:profile|ScatteredPeacock14> ! I think you are right. We are going to look into fixing this

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

Hi @<1709015393701466112:profile|ScatteredPeacock14> , can you provide a sample script that reproduces this behavior? What version of the SDK are you using?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Here's a minimal example running on ClearML 1.16.1:

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

@<1523701070390366208:profile|CostlyOstrich36> Any update on this?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

child-zip local cache:

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

@<1523701070390366208:profile|CostlyOstrich36> 👀

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					AmiableSeaturtle81
				
					0
					 × 1

None

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					ScatteredPeacock14
				
					0
					 × 1

Write your answer

511 Views

9 Answers

4 months ago