Unanswered
In My Current Project I Generate The Data From An Sql Query. Is The Only Way To Register The Dataset With Clearml To Write The Files To Disk First Or Is There Another Method?
This Leads Into The Second Issue I Have, Which Is What Happens When I Store The
Alright, I tried testing it out by commenting out the code for generating new csv's, so for successive runs the CSVs are identical. However, when I use dataset.add_files() it still generated a new version of the dataset.
# log the data to ClearML if a task is passed
if self.task:
self.clearml_dataset = Dataset.create(dataset_name="[LTV] Dataset")
self.clearml_dataset.add_files(path=save_path, verbose=True)
if self.tags is not None:
self.clearml_dataset.add_tags(self.tags)
self.clearml_dataset.upload(
show_progress=True,
verbose=True,
)
self.task.connect(self.clearml_dataset)
self.clearml_dataset.finalize()
logger.info(f"Saved the data to ClearML.")
259 Views
0
Answers
2 years ago
2 years ago