@<1545216070686609408:profile|EnthusiasticCow4>
This:

            parent = self.clearml_dataset = Dataset.get(
                dataset_name="[LTV] Dataset",
                dataset_project="[LTV] Lifetime Value Model",
            )
            # generate the local dataset
            dataset = Dataset.create(
                dataset_name=f"[LTV] Dataset",
                parent_datasets=[parent],
                dataset_project="[LTV] Lifetime Value Model",
            )

should likely be this:

            parent =  Dataset.get(
                dataset_name="[LTV] Dataset",
                dataset_project="[LTV] Lifetime Value Model",
            )
            # generate the local dataset
            dataset = self.clearml_dataset = Dataset.create(
                dataset_name=f"[LTV] Dataset",
                parent_datasets=[parent],
                dataset_project="[LTV] Lifetime Value Model",
            )

your new dataset would be the child, which then you will be able to upload and finalize.
Here, you should likely revert this:

            if not any(synced):
                Dataset.delete(dataset.id)
                self.task.connect(parent)
                logger.info(f"Data already exists on ClearML remote. Skipping upload.")
                self.clearml_dataset = parent  # notice this line