I'M Having Trouble With Storagemanager Leaving Files In The Local Cache On The Clearml Agent Node. Somehow, These Files Are Not Cleaned Up By Storagemanager When I Try To Run New Experiments, Which Then Fail Due To Lack Of Local Storage. Archiving Old E

Answered

I'm having trouble with StorageManager leaving files in the local cache on the clearml agent node. Somehow, these files are not cleaned up by StorageManager when I try to run new experiments, which then fail due to lack of local storage. Archiving old experiments and running the cleanup task doesn't affect them. When I search ClearML tasks, datasets and models for their ids, they aren't found.

I assume I can just delete them manually without causing problems -- is this correct?
And how did I end up with these zombie files in the first place?

Any help much appreciated.

  				
Posted 
	one month ago

					More
				  		
  Report
		
					UptightMoth89
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1760474471606521856:profile|UptightMoth89> , I'm not sure that Task.get_task is related to the cache. Also the cache should be auto cleared by the SDK. Can you provide the full log of the experiment? What exactly are you trying to do in the code?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Clarification -- Task.get_task fails with the following:
clearml.Task - DEBUG - Failed reloading task 22304c030bd30b7a58889b7d5ad7a560: Action failed <400/101: tasks.get_by_id/v1.0 (Invalid task id: ids=(22304c030bd30b7a58889b7d5ad7a560))> (task=22304c030bd30b7a58889b7d5ad7a560)

So it's not that they aren't found, exactly -- it's that the task couldn't be reloaded.

  				
Posted 
	one month ago

					More
				  		
  Report
		
					UptightMoth89
				
					0
					 × 1

Write your answer

172 Views

2 Answers

one month ago