Unanswered
How Can I Send A Composed Chunk Of Code For Remote Execution
Consider e.g:
# steps.py
class DataFetchingStep:
def __init__(self, source, query, locations, timestamps):
# ...
def run(self, queue=None, **kwargs):
# ...
class DataTransformationStep:
def __init__(self, inputs, transformations):
# inputs can include instances of DataFetchingStep, or local files, for example
# ...
def run(self, queue=None, **kwargs):
# ...
And then the following SDK usage in a notebook:
from steps import DataFetchingStep, DataTransformationStep
data_step = DataFetchingStep(source='some_db', query='...', locations='...', timestamps='...')
# Run options:
data_step.run(queue=None) # Runs locally, hurray
data_step.run(queue='default') # Runs remotely, polls until the Task is done, stores the saved artifact somewhere locally in the class
transform_step = DataTransformationStep(inputs=[data_step, "my_data.csv"], transformations=[func1, func2, func3])
transform_step.run() # Runs locally; if `data_step` was not run before, it will run now; will also poll until the Task is done, etc...
105 Views
0
Answers
10 months ago
10 months ago