The same can be said for ClearML, each of these steps is a clearml Task (with it's own repo/environment)
I think, the tasks are too small to merit a separate repo/environment. One example is a node that resizes the images, this node receives as input a Dataset and iterates over each image, resizes it an outputs a new Dataset, which is used in the next node downstream in the Pipeline.
the main use for Kedro is the nice web UI of the pipeline
To be frank, we have not even seen t...
AgitatedDove14 , we are also in the same boat. We tried Kedro and found the organizational aspect to be really clean and would love to stick to it. We also like how each node of the pipeline are independent re-usable blocks.
ClearML is definitely more comprehensive (specially the concept of Tasks, Data and agents) and has its special place in our project. Now, we are trying to figure out how to run our Kedro pipelines in ClearML.
After playing with both for few days, we still cant wrap our h...
AgitatedDove14 , we did not do more experimentation on clearml/kedro later and moved to Dagster.. but am still keeping an eye out for clearML :)