Could I get some feedback from people with experience using ClearML pipelines on the best way to handle caching? My team is working on configuring ClearML Pipelines for our data processing workflow.

We currently have an experimental pipeline configured for batch data processing. It runs a basic algorithm on each item provided as input, essentially just mapping each input piece of data to a new, processed output. However the algorithm we run is somewhat expensive, and we want to be able to cache as much computation as possible. If we run the pipeline with 1000 items from our ClearML Data dataset, and then add another item, when we re-run the pipeline with those 1001 items as input, we want to be able to cache all the previous computation and only have to process the single new item.

As far as I can tell, the built-in ClearML pipeline cache features will re-run the entire pipeline step if the input changes at all, so when the new item is added the entire batch pipeline step will re-run with all 1001 items.

What’re the best practices for handling this with ClearML? I’d really appreciate any information anyone can share about their experiences with this. Thank you :)