Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Unanswered
How Can I Send A Composed Chunk Of Code For Remote Execution


Consider e.g:

# steps.py

class DataFetchingStep:
    def __init__(self, source, query, locations, timestamps):
        # ...
    def run(self, queue=None, **kwargs):
        # ...

class DataTransformationStep:
    def __init__(self, inputs, transformations):
        # inputs can include instances of DataFetchingStep, or local files, for example
        # ...
    def run(self, queue=None, **kwargs):
        # ...

And then the following SDK usage in a notebook:

from steps import DataFetchingStep, DataTransformationStep

data_step = DataFetchingStep(source='some_db', query='...', locations='...', timestamps='...')
# Run options:
data_step.run(queue=None)  # Runs locally, hurray
data_step.run(queue='default')  # Runs remotely, polls until the Task is done, stores the saved artifact somewhere locally in the class

transform_step = DataTransformationStep(inputs=[data_step, "my_data.csv"], transformations=[func1, func2, func3])
transform_step.run()  # Runs locally; if `data_step` was not run before, it will run now; will also poll until the Task is done, etc...
  
  
Posted 11 months ago
116 Views
0 Answers
11 months ago
11 months ago