Hi ScaryKoala63
Sure, add the following to your clearml.conf:sdk.storage.cache.default_cache_manager_size = 400
I think you are correct, it seems like for some reason you hit the cache limit, and a previous entry was deleted
I increased the cache size a lot but still I get the same error.
ScaryKoala63
When it fails what's the number of files you have in:/home/developer/.clearml/cache/storage_manager/global/
?
AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.
Is this consistent on the same file? can you provide a code snippet to reproduce (or understand the flow) ?
Could it be two machines are accessing the same cache folder ?
Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.
And when retrieve just this file? is it working ?
(Maybe for some reason the file is corrupted) ?
I resolved it by adding this line:StorageManager.set_cache_file_limit(10000, cache_context='global')
Are you suggesting the conf file did not set the default size? It sounds like a bug, can you verify?
Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global'
resolved the issue.