Unanswered
Hi Team, How Would You, Or What Is The Clearml Recommended Practice For Organising Sub-Steps Inside Each Pipeline Step?
For Example I Have Several Sources Of Data To Retrieve, Each Of Which Is Potentially A Sub Step As I Wanted To Group Them Into One Big
Hi Team, how would you, or what is the ClearML recommended practice for organising sub-steps inside each pipeline step?
For example I have several sources of data to retrieve, each of which is potentially a sub step as I wanted to group them into one big pipeline step called "ingestion" ( I don't think make those sub step a pipeline step a good idea anyway).
What would be a good practice you recommend? Having class gathering them all will not record all the dependencies in each sub step correctly, right? (local imports are in each sub step's function code)
@PipelineDecorator.Component()
def ingestion_step():
from somewhere.ingestion import Ingestion
ingestion = Ingestion()
ingestion.run() # Runs all the sub step inside this class
class Ingestion:
def __init__(self):
self.state1 = 'Ah'
self.a_data = None
self.b_data = None
def retrieve_a(self):
# This import is not visible to ClearML?
from sources.a import ASource
self.a_data = ASource.retrieve()
7 Views
0
Answers
5 hours ago
5 hours ago
Tags