Hi Fawad!
You should be able to get a local mutable copy using Dataset.get_mutable_local_copy
and then creating a new dataset.
But personally I prefer this workflow:
dataset = Dataset.get(dataset_project=CLEARML_PROJECT, dataset_name=CLEARML_DATASET_NAME, auto_create=True, writable_copy=True) dataset.add_files(path=save_path, dataset_path=save_path) dataset.finalize(auto_upload=True)
The writable_copy
argument gets a dataset and creates a child of it (a new dataset with your selected one as parent). In this way you can just add some files and upload the whole thing. It will now contain everything your previous dataset did + the files you added AND keep track of the previous dataset. In this way clearml knows not to upload the data that was already there, it will only upload your newly added files.
auto_create
will create a dataset is none exist yetauto_upload=True
is basically the same as first uploading and then finalizing
These 3 lines use functionality that's only just available, so make sure to have the latest clearml version :)