Hi AgitatedDove14
I'm using ClearML 1.1.2 and Pytorch lightning 1.5.9.

AgitatedDove14 Thanks it works now, after removing ClearmlLogger.

I increased the cache size a lot but still I get the same error.

Thanks 🙂

Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.

I resolved it by adding this line:
StorageManager.set_cache_file_limit(10000, cache_context='global')

Thank you @<1523701768532267008:profile|TimelyPenguin76> @<1523701205467926528:profile|AgitatedDove14> it works. I'm using Openvino models (pytorch models converted to openvino).

Thanks, I'll try it in the next training round and will write the result here.

Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global' resolved the issue.

AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.

I define the ClearmlLogger like this:
logger = ClearmlLogger(project_name=PROJECT_NAME, task_name=out_task_name, output_uri=True, finalize_close_clearml_task_default=False, tags=task_params['tags'])Then pass it as an argument to Pytorch lightning trainer.