Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I'M Having Trouble With Storagemanager Leaving Files In The Local Cache On The Clearml Agent Node. Somehow, These Files Are Not Cleaned Up By Storagemanager When I Try To Run New Experiments, Which Then Fail Due To Lack Of Local Storage. Archiving Old E

I'm having trouble with StorageManager leaving files in the local cache on the clearml agent node. Somehow, these files are not cleaned up by StorageManager when I try to run new experiments, which then fail due to lack of local storage. Archiving old experiments and running the cleanup task doesn't affect them. When I search ClearML tasks, datasets and models for their ids, they aren't found.

I assume I can just delete them manually without causing problems -- is this correct?
And how did I end up with these zombie files in the first place?

Any help much appreciated.

  
  
Posted 5 days ago
Votes Newest

Answers 2


Clarification -- Task.get_task fails with the following:
clearml.Task - DEBUG - Failed reloading task 22304c030bd30b7a58889b7d5ad7a560: Action failed <400/101: tasks.get_by_id/v1.0 (Invalid task id: ids=(22304c030bd30b7a58889b7d5ad7a560))> (task=22304c030bd30b7a58889b7d5ad7a560)

So it's not that they aren't found, exactly -- it's that the task couldn't be reloaded.

  
  
Posted 5 days ago

Hi @<1760474471606521856:profile|UptightMoth89> , I'm not sure that Task.get_task is related to the cache. Also the cache should be auto cleared by the SDK. Can you provide the full log of the experiment? What exactly are you trying to do in the code?

  
  
Posted 2 days ago
39 Views
2 Answers
5 days ago
2 days ago
Tags