Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, Is There A Way To Increase The Cache Size Of Each Clearml Task? I'M Running An Experiment And Many Artifacts Are Downloaded. My Dataloader Fails To Load Some Of The File Since They Are Missing, Although They Were Downloaded. I Guess There Is

Hi everyone,
Is there a way to increase the cache size of each ClearML task?
I'm running an experiment and many artifacts are downloaded. My dataloader fails to load some of the file since they are missing, although they were downloaded. I guess there is a limit for caching artifacts for each task, so some the downloaded files are deleted after exceeding the limit.
Thank you in advance.

Here is the error:
FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/.clearml/cache/storage_manager/global/94fc7259e0247a7781982a95a8a9fc5a.xxx_xxx_xx_x_features.npz'
before reaching the data loader, I checked it and the file was available under this path.

  
  
Posted 2 years ago
Votes Newest

Answers 12


I resolved it by adding this line:
StorageManager.set_cache_file_limit(10000, cache_context='global')

  
  
Posted 2 years ago

Hi ScaryKoala63
Sure, add the following to your clearml.conf:
sdk.storage.cache.default_cache_manager_size = 400I think you are correct, it seems like for some reason you hit the cache limit, and a previous entry was deleted

  
  
Posted 2 years ago

Is this consistent on the same file? can you provide a code snippet to reproduce (or understand the flow) ?
Could it be two machines are accessing the same cache folder ?

  
  
Posted 2 years ago

ScaryKoala63
When it fails what's the number of files you have in:
/home/developer/.clearml/cache/storage_manager/global/ ?

  
  
Posted 2 years ago

Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.

  
  
Posted 2 years ago

Thanks 🙂

  
  
Posted 2 years ago

Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global' resolved the issue.

  
  
Posted 2 years ago

ScaryKoala63 nice!!!!!

  
  
Posted 2 years ago

And when retrieve just this file? is it working ?
(Maybe for some reason the file is corrupted) ?

  
  
Posted 2 years ago

Are you suggesting the conf file did not set the default size? It sounds like a bug, can you verify?

  
  
Posted 2 years ago

I increased the cache size a lot but still I get the same error.

  
  
Posted 2 years ago

AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.

  
  
Posted 2 years ago
1K Views
12 Answers
2 years ago
one year ago
Tags
Similar posts