Hi all,

Juts learning the ropes of ClearML atm. And am doing a really simple ETL pipeline: raw data -> clean data

My current approach is in one script, I add the raw data file to a dataset in the project:

register_raw.ipynb

ds = Dataset.create(
    dataset_name="raw",
    dataset_project="example",
)

ds.add_files(path=local_file_path)

ds.finalize(auto_upload=True)

Then, for the ETL section, I have this approach

clean_data.ipynb

# Make task
task = Task.init(project_name="example", task_name="clean-raw")

# 1. pull raw
raw = Dataset.get(dataset_name="raw")
raw_data = pd.read_parquet(next(Path(raw.get_local_copy()).glob("*.parquet")))

# 2. Clean data
clean_data = raw_data[...]

# 3. Save data
with TemporaryDirectory() as tmp:
    out = Path(tmp) / "cleaned_data.parquet"
    clean_data.to_parquet(out, index=False)

    clean_ds = Dataset.create(
        dataset_name="clean-data",
        dataset_project="example",
        parent_datasets=[raw],
    )
    clean.add_files(out)
    clean.finalize(auto_upload=True) 

But this seems wrong to me to do it this way? It makes two dataset objects (I guess that makes sense) but the new dataset "clean-data" contains both original, and new file. Which was not my intention.

Ideally, what I wanted was a pipeline that saved certain intermediate steps of the process. Is this the canonical way to achieve that?