Hi there,
I am working on an audio analysis project and I am interested in using CLEARML to manage our data versions and models. However, I have some questions about dataset versioning and reproducibility that I hope you can help me with.
Specifically, I have a large amount of audio data (hundreds of gigabytes) that I extract features from in a pipeline. As the pipeline changes several times, I need to switch between versions of datasets. How can I do this with CLEARML? Does it require a rerun of the pipeline, and is the data saved in the different versions?
Furthermore, if I want to reproduce an experiment that I did in the past with manipulations on the data, is the dataset saved together with the experiment's results? How can I ensure that the experiment is reproducible?
I would greatly appreciate any insights or advice you can provide on these topics.
Thank you in advance for your help.
Best regards
Hi @<1523713932588486656:profile|PerplexedWalrus3> , I'm not sure about the exact configuration of your setup but I'm quite sure you could do this fairly easily with pipelines and datasets in ClearML. Have you tried playing with Datasets to get the feeling of how it works?
A big part of the way Datasets work is to turn the data into a parameter rather than be part of the code. You will be able to easily reproduce experiments 🙂
