Hi Community, I'M Installing Multiple Clearml-Agent On Some Gpu Workstations, So I Have A Question: How Can I Use The Cache Of Dataset As When I Run Only 1 Clearml-Agent? I Don'T Want To Download Full Dataset Everytime When A Task Run Because Now Since I

Answered

Hi community, I'm installing multiple clearml-agent on some gpu workstations, so I have a question: How can I use the cache of dataset as when I run only 1 clearml-agent? I don't want to download full dataset everytime when a task run because now since I have some agents, each one can execute 1 task => doesn't have cache from previous tasks. How can I solve this? Perhaps using nfs mount of cache folder on clearml agent nodes to a nfs server?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					UnevenDeer21
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1749965229388730368:profile|UnevenDeer21> , an NFS is one good option. You can also point all agents on the same machine to the same cache folder as well. Or just like you suggested, point all workers to the same cache on a mounted NFS

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Thanks, but I wonder about if using NFS, when a agent need to read a dataset from cache folder, it need to copy dataset from NFS server => local machine, so it's basically same as downloading from clearml server unless NFS has a better bandwidth, faster or smth like that. Do you know if Is there any faster way to do that?

  				
Posted 
	2 months ago

					More
				  		
  Report
		
					UnevenDeer21
				
					0
					 × 1

Write your answer

204 Views

2 Answers

2 months ago