Thanks, I think I could identify the issue. I opened a bug here: None

The problem is with the keras BackupAndRestore callback, where clearml overwrites the local backup storage with a storage to the clearml server. In this case, however, the local storage is sufficient as this is only for continuing an interruption.

Depending on the framework you're using it'll just hook into the save model operation. Every time you save a model, which will probably happen every epoch for some subset of the training. If you want to do it with the existing framework you could change the checkpoint so that it only clones the best model in memory and saves the write operation for last. The risk with this is if the training crashes, you'll lose your best model.

Optionally, you could also disable the ClearML integration with your framework and manually specify when to sync everything to the server.

I'm still a bit new to the platform, I'd love to hear from others if there's another solution.

Hi @<1547390464557060096:profile|NuttyKoala57> ! You can use wildcards in auto_connect_framework to filter your models. Check the docs under init: None . You might also want to check out this GH thread for an another way to do this: None

Yeah, it's because it's just hooking into the save operation and capturing the output, regardless of the parent call.