ClearML FAQ | Hi Everyone! Is There A Way To Force An Agent To Clear The Clearml Cache Folder Every Time It’S Done With A Task? I Have A Machine Running Multiple Agents For Training, But The Disk Space Gets Filled Up Quickly Since Every New Task Uses A New Dataset. I S

Answered

Hi Everyone! Is There A Way To Force An Agent To Clear The Clearml Cache Folder Every Time It’S Done With A Task? I Have A Machine Running Multiple Agents For Training, But The Disk Space Gets Filled Up Quickly Since Every New Task Uses A New Dataset. I S

Hi everyone! Is there a way to force an agent to clear the clearml cache folder every time it’s done with a task? I have a machine running multiple agents for training, but the disk space gets filled up quickly since every new task uses a new dataset. I saw there is a sdk.storage.cache.default_cache_manager_size to limit the number of files kept in cache, by default is set to 100 files but the datasets are not deleted even if they contains 50k images, so I guess a dataset counts as 1?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SlipperySheep79
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1570220858075516928:profile|SlipperySheep79> , you can set various cache limitations in clearml.conf . The issue you encountered is specifically regarding Datasets? If that is the case, I think this is the section that you're looking for - None

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1523701070390366208:profile|CostlyOstrich36> , yes it's specifically with datasets. Probably the option I need is size.max_used_bytes but it looks like it's available only for the enterprise plan? Is there any other way to clean the cache after each task?

  				
Posted 
	10 months ago

					More
				  		
  Report
		
					SlipperySheep79
				
					0
					 × 1

Write your answer

684 Views

2 Answers

10 months ago

10 months ago

Tags

Similar posts

Hello All! Please Tell Me How To Make It So That Clearml Does Not Write New Tqdm Updates Every 20 Seconds From A New Line? As A Result, The Logs Of One Launch Start All Over 5-10 Megabytes ...

Clearml Agent Cant Download Packages With Recent Version Numbers. I'Ve Already Had To Deprecate Many Packages Like Matplotlib. And I'M Still Not Done. Every Time I Deprecate A Package, Theres A New Version Issue With A New Package. Please Help. Am I Missi

Hi Again Everybody, Is There A Way To Cache The Docker Containers Used By The Agents? As Far As I Understood Every Time It Spawns A New Container With A Shared Apt Cache (And Venv Cache If Configured)

Hi All, I'M Setting Up A Pipeline To Auto Train New Model Every Month. In Each Run, I Also Need To Get The Model Of The Previous Run And Compare The Old Model And The New Model That Just Being Generated. How Should I Access The Model Of The Previous Run?

Hi Guys, I Have A Recurring Problem : Every Few New Commits - I Try To Upload A New Task And I Have The Following Error:

Hello, I’Ve Enabled The Web Authentication For My Clearml Server And I’M Wondering Whether There’S A Way To Avoid Restarting All The Clearml Services Every Time A New User Is Added. For Example If Some Experiments Are Ongoing A New User Can’T Be Added Wit

Hi, I Have A File On Azure Blob, Which Will Be A Parent For Some Experiments, Which In Every One Of Them I Will Manipulate The Orig File. Now I Want To Create A Dataset, Define The Orig File As The Parent, And Then, While Creating Each Of The New Files, D

Hi Everyone, I Wanna Implement Some Sort Of A Tool That Checks Every N Minutes For A New Data In Db Or Maybe S3 Bucket And Whenever The New Data Appears It Runs Some Preprocessing And Then Starts Training. Is It A Good Idea To Use Clearml Agent Services F

Hi! I Deployed Clearml Server Along With Jupyterhub On Azure K8S (Aks). The Way It Works Is That Every User Is Assigned A New Pod That Is Spawned With A Docker Image Of A Choice (One Of Them With Clearml Sdk Installed). I Managed To Configure Most Of The