Hi all, Juts learning the ropes of ClearML atm. And am doing a really simple ETL pipeline: raw data -> clean data My current approach is in one script, I add...
4 months ago
WackyDolphin95
1
Question,
2
Answers
Active since 02 May 2025
Last activity
4 months ago
Reputation
0
Badges 1
2 × Eureka!
0
Hi All,
Juts Learning The Ropes Of Clearml Atm. And Am Doing A Really Simple Etl Pipeline: Raw Data -> Clean Data
My Current Approach Is In One Script, I Add The Raw Data File To A Dataset In The Project:
# Register_Raw.Ipynb
Hi @<1523701070390366208:profile|CostlyOstrich36> - Cheers for your time
I thought about that, but I think the lineage feature is really valuable.
I've opted for this as a go to pattern now to achieve what I wanted. I literally just remove all files in the new dataset before finalizing it
with TemporaryDirectory() as tmp:
out = Path(tmp) / "df_clean.parquet"
result.to_parquet(out, index=False)
clean = Dataset.create(
dataset_name="clean-data",
dataset_p...
4 months ago
0
Hi All,
Juts Learning The Ropes Of Clearml Atm. And Am Doing A Really Simple Etl Pipeline: Raw Data -> Clean Data
My Current Approach Is In One Script, I Add The Raw Data File To A Dataset In The Project:
# Register_Raw.Ipynb
I wonder, is this stye of data set handling trying to square the circle with ClearML? Is it built for this type of stuff
4 months ago