Unanswered
Heyo,
After Building Some Custom Pipelining Functionality On Mlflow, I
Started Looking For Better Software That Can Beat What I Created - With A
Similar Amount Of Effort. Problem Has Been That Up Till Now, All I
Found Could Make Things Way Better But Al
That makes sense to me, what do you think about the following:
` from clearml import PipelineDecorator
class AbstractPipeline(object):
def init():
pass
@PipelineDecorator.pipeline(...)
def run(self, run_arg):
data = self.step1(run_arg)
final_model = self.step2(data)
self.upload_model(final_model)
@PipelineDecorator.component(...)
def step1(self, arg_a):
# do something
return value
@PipelineDecorator.component(...)
def step2(self, arg_b):
# do something
return value This would mean steps 1/2 are executed on different machines, where the data passed between them is automatically serialized. It also allows you to build the actual logic in
def run ` that drives the different components.
wdyt?
175 Views
0
Answers
2 years ago
one year ago