I know I drop a very general question, so I am sorry for the confusion.
However, I was reading online how other companies were dealing with data pipeline for handling data both in training & inference in a very standard, and monitorable way.
I saw people refer to something called feature store , the first was probably uber ( source https://eng.uber.com/michelangelo-machine-learning-platform/ ) ... I know tecton is one provider of it (source https://www.tecton.ai/ ), but there might be others.
Does this clarify a bit more?
I think it would be amazing Martin 😃 . You guys are providing an amazing service at clearml!
However if I could give a feedback for you to even improve it, I would suggest to start think of implementing some feature store API. you know better then me that for productionization of ML data pipelines are very critical.
And I really hope I don't sound arrogant by giving this feedback 🙂
Hi SmoggyGoat53
What do you mean by "feature store" ? (These days the definition is quite broad, hence my question)
I am completely honest, in the sense I am starting to get familiar with feature store in general.
For use case I have something specific to my data pipeline, but I think it can be easily generalized. I could start to look into feast and see from there what we need!
Actually it would be interesting to combine the two, feast is fully open-source and supported by the linux foundation, so I cannot see the harm in that.
wdyt?
Exactly! so do you guys have any sort of API for anything like that?
Sure, something like : https://github.com/feast-dev/feast
Not at all, we love ideas on improving ClearML.
I do not think there is a need to replace feast, it seems to do a lot, I'm just thinking on integrating it into the ClearML workflow. Do you have a specific use case we can start to work on? Or maybe a workflow that would make sense to implment?
Actually AgitatedDove14 I think that Feast is great, but is missing the part of really transform the data with feature engineering ... so you may want either to develop that part on your side or look at other provider ...
is a feature-store-as-a-service. A big difference between Feast and Tecton is that Tecton supports transformations, so feature pipelines can be managed end-to-end within Tecton. Tecton is a managed offering, and a great feature store choice if you need production SLAs, hosting, advanced collaboration, managed transformations (batch/streaming/real-time), and/or enterprise capabilities.
from