I increased the cache size a lot but still I get the same error.

AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.

Thanks, I'll try it in the next training round and will write the result here.

Hi AgitatedDove14
I'm using ClearML 1.1.2 and Pytorch lightning 1.5.9.

Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.

AgitatedDove14 Thanks it works now, after removing ClearmlLogger.

Thanks 🙂

I resolved it by adding this line:
StorageManager.set_cache_file_limit(10000, cache_context='global')

I define the ClearmlLogger like this:
logger = ClearmlLogger(project_name=PROJECT_NAME, task_name=out_task_name, output_uri=True, finalize_close_clearml_task_default=False, tags=task_params['tags'])Then pass it as an argument to Pytorch lightning trainer.

Thank you @<1523701768532267008:profile|TimelyPenguin76> @<1523701205467926528:profile|AgitatedDove14> it works. I'm using Openvino models (pytorch models converted to openvino).

Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global' resolved the issue.