ClearML FAQ | Hi Everyone! Is There A Way To Force An Agent To Clear The Clearml Cache Folder Every Time It’S Done With A Task? I Have A Machine Running Multiple Agents For Training, But The Disk Space Gets Filled Up Quickly Since Every New Task Uses A New Dataset. I S

Answered

Hi Everyone! Is There A Way To Force An Agent To Clear The Clearml Cache Folder Every Time It’S Done With A Task? I Have A Machine Running Multiple Agents For Training, But The Disk Space Gets Filled Up Quickly Since Every New Task Uses A New Dataset. I S

Hi everyone! Is there a way to force an agent to clear the clearml cache folder every time it’s done with a task? I have a machine running multiple agents for training, but the disk space gets filled up quickly since every new task uses a new dataset. I saw there is a sdk.storage.cache.default_cache_manager_size to limit the number of files kept in cache, by default is set to 100 files but the datasets are not deleted even if they contains 50k images, so I guess a dataset counts as 1?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SlipperySheep79
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1570220858075516928:profile|SlipperySheep79> , you can set various cache limitations in clearml.conf . The issue you encountered is specifically regarding Datasets? If that is the case, I think this is the section that you're looking for - None

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi @<1523701070390366208:profile|CostlyOstrich36> , yes it's specifically with datasets. Probably the option I need is size.max_used_bytes but it looks like it's available only for the enterprise plan? Is there any other way to clean the cache after each task?

  				
Posted 
	9 months ago

					More
				  		
  Report
		
					SlipperySheep79
				
					0
					 × 1

Write your answer

598 Views

2 Answers

9 months ago

9 months ago

Tags

Similar posts

Hello All! Please Tell Me How To Make It So That Clearml Does Not Write New Tqdm Updates Every 20 Seconds From A New Line? As A Result, The Logs Of One Launch Start All Over 5-10 Megabytes ...

Clearml Agent Cant Download Packages With Recent Version Numbers. I'Ve Already Had To Deprecate Many Packages Like Matplotlib. And I'M Still Not Done. Every Time I Deprecate A Package, Theres A New Version Issue With A New Package. Please Help. Am I Missi

Hi Again Everybody, Is There A Way To Cache The Docker Containers Used By The Agents? As Far As I Understood Every Time It Spawns A New Container With A Shared Apt Cache (And Venv Cache If Configured)

Hi All, I'M Setting Up A Pipeline To Auto Train New Model Every Month. In Each Run, I Also Need To Get The Model Of The Previous Run And Compare The Old Model And The New Model That Just Being Generated. How Should I Access The Model Of The Previous Run?

Hi Guys, I Have A Recurring Problem : Every Few New Commits - I Try To Upload A New Task And I Have The Following Error:

Hello, I’Ve Enabled The Web Authentication For My Clearml Server And I’M Wondering Whether There’S A Way To Avoid Restarting All The Clearml Services Every Time A New User Is Added. For Example If Some Experiments Are Ongoing A New User Can’T Be Added Wit

Hi, I Have A File On Azure Blob, Which Will Be A Parent For Some Experiments, Which In Every One Of Them I Will Manipulate The Orig File. Now I Want To Create A Dataset, Define The Orig File As The Parent, And Then, While Creating Each Of The New Files, D

Hi Everyone, I Wanna Implement Some Sort Of A Tool That Checks Every N Minutes For A New Data In Db Or Maybe S3 Bucket And Whenever The New Data Appears It Runs Some Preprocessing And Then Starts Training. Is It A Good Idea To Use Clearml Agent Services F

Hi! I Deployed Clearml Server Along With Jupyterhub On Azure K8S (Aks). The Way It Works Is That Every User Is Assigned A New Pod That Is Spawned With A Docker Image Of A Choice (One Of Them With Clearml Sdk Installed). I Managed To Configure Most Of The