Hi everyone!
I am currently using ClearML to track my Datasets. At each run I am producing some datasets that are somehow "heavy". As I want to track any diff change every time, I am using the parent_datasets parameter ( clearml.Dataset.create ) when creating my Dataset before uploading anything.
But with time, my workspace will eventually be full and I want to use a policy where I can clean up the old parents to free some spaces.
I have archived and deleted the old parents manually. But, it seems that when I try to upload my data and finalize my Dataset during a new run, ClearML is looking for all parents and try to find the first one which has been deleted, so an error is raised.
Does someone ever had something similar ?
Thanks a lot in advance if you have any suggestion 😄
Hi @<1787653566505160704:profile|EnchantingOctopus35> , I don't think you can clean up parents without damaging the children since they rely on that. I would suggest taking any data you don't want scrubbed and then creating a new version with it. Then delete the unrelated older datasets. What do you think?
Hi John, thanks for your answer. Indeed, this is what I did. It forces me to develop some custom archiving process but it sounds acceptable for the moment 👍
