Thanks 🙂

Yes, it happens on the same file.
I try to provide a code snippet. No, it's a single docker container.

I resolved it by adding this line:
StorageManager.set_cache_file_limit(10000, cache_context='global')

I define the ClearmlLogger like this:
logger = ClearmlLogger(project_name=PROJECT_NAME, task_name=out_task_name, output_uri=True, finalize_close_clearml_task_default=False, tags=task_params['tags'])Then pass it as an argument to Pytorch lightning trainer.

Thank you @<1523701768532267008:profile|TimelyPenguin76> @<1523701205467926528:profile|AgitatedDove14> it works. I'm using Openvino models (pytorch models converted to openvino).

I increased the cache size a lot but still I get the same error.

AgitatedDove14 Thanks it works now, after removing ClearmlLogger.

Thanks, I'll try it in the next training round and will write the result here.

AgitatedDove14 The number is 1001 and I set the default_cache_manager_size to 20000.

Yes, I tried updating clearml.conf file but didn't help. finally adding the line I mentioned with cache_contex='global' resolved the issue.

Hi AgitatedDove14
I'm using ClearML 1.1.2 and Pytorch lightning 1.5.9.