Let's see if I got how it works on the CLI.
So if I execute:
clearml-data create --name <improved_dataset> --parents <existing_dataset_id>
Where the parent dataset was updated with sync,
I just need to run:
clearml-data upload --id <created_dataset_id>
And the delta will be automatically uploaded to the new dataset?
Every "commit" is a new version, so sync changes you need to either create a new version (with parent version as the previous one), and sync the local folder (or manually add/remove files).
If you do not need to actually store the "current" version, you can just reset the Task, and sync it again.
By your description it seems to make no difference whether I added the files via sync or add, since I will have to create a new dataset either way.
Sync is design to take a local folder/s and add/remove files from a dataset based on the local changes (it does that automatically based on file existence / content)
The changes (i.e. added files) are uploaded as delta changes relative to the parent version, this means we are not always uploading all files.
Add on the other hand means you know already which files are added to the dataset and these files only will be added to the datasets (again relative to the parent version). Notice that here also if you are adding files with the same content as files in the parent version, they will not be uploaded twice.
Make sense ?