Is this consistent on the same file? can you provide a code snippet to reproduce (or understand the flow) ?
Could it be two machines are accessing the same cache folder ?

ScaryKoala63 nice!!!!!

Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global' resolved the issue.

Are you suggesting the conf file did not set the default size? It sounds like a bug, can you verify?

I resolved it by adding this line:
StorageManager.set_cache_file_limit(10000, cache_context='global')

I increased the cache size a lot but still I get the same error.

Thanks 🙂

AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.

And when retrieve just this file? is it working ?
(Maybe for some reason the file is corrupted) ?

Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.

ScaryKoala63
When it fails what's the number of files you have in:
/home/developer/.clearml/cache/storage_manager/global/ ?

Hi ScaryKoala63
Sure, add the following to your clearml.conf:
sdk.storage.cache.default_cache_manager_size = 400I think you are correct, it seems like for some reason you hit the cache limit, and a previous entry was deleted