Hi, Currently Running

Answered

Hi, currently running Dataset.list_datasets() on a giant list of datasets takes a very long time to return the full datasets list. Similarly running Dataset.get() also takes a fairly long time to run and retrieve a single dataset object. Does anyone know if there is a way to retrieve a whole list of datasets and getting a single dataset faster?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1747428509627715584:profile|CumbersomeDuck6> , are you using a self hosted server?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

currently not, but the aim is to host it locally. Would that make retrieval much faster?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

So you're using the community server? Response time really depends on the resources of the machine that is running the server and amount of data to filter

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hm I see. Because I am wondering how come when I view clearml datasets in the clearml GUI, it can just show all my data almost instantly, but when using ClearML SDK, it is so slow?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

Write your answer

86 Views

4 Answers

2 months ago