Hi, Currently Running

Answered

Hi, currently running Dataset.list_datasets() on a giant list of datasets takes a very long time to return the full datasets list. Similarly running Dataset.get() also takes a fairly long time to run and retrieve a single dataset object. Does anyone know if there is a way to retrieve a whole list of datasets and getting a single dataset faster?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

Votes Newest

Answers 4

Hi @<1747428509627715584:profile|CumbersomeDuck6> , are you using a self hosted server?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

So you're using the community server? Response time really depends on the resources of the machine that is running the server and amount of data to filter

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

currently not, but the aim is to host it locally. Would that make retrieval much faster?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

Hm I see. Because I am wondering how come when I view clearml datasets in the clearml GUI, it can just show all my data almost instantly, but when using ClearML SDK, it is so slow?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CumbersomeDuck6
				
					0
					 × 1

Write your answer

644 Views

4 Answers

one year ago