The same can be said for ClearML, each of these steps is a clearml Task (with it's own repo/environment)
I think, the tasks are too small to merit a separate repo/environment. One example is a node that resizes the images, this node receives as input a Dataset and iterates over each image, resizes it an outputs a new Dataset, which is used in the next node downstream in the Pipeline.
the main use for Kedro is the nice web UI of the pipeline
To be frank, we have not even seen the UI yet 🙂 . The main benefit of Kedro is in the "authoring of the pipeline". You can quickly browse the organization structure https://github.com/noklam/allegro_test/tree/main/src/allegro_test/pipelines/data_science and see for yourself (this pipeline has two nodes train_model
and predict
)
You nicely described the features of ClearML and that is why we are inclined in using it. Its just that we would like to use Kedro's structure with ClearML.