Unanswered
How Can I Run A New Version Of A Pipeline, Wait For It To Finish And Then Check Its Completion/Failure Status?
I Want To Kick Off The Pipeline And Then Check Completion
- At its simplest, this could just mean checking that all of the steps and the pipeline itself have completed successfully (by checking their “Task status”).If a pipeline step ends with "failed" status in the pipeline execution function an exception will be raised, if the exception is not caught, the pipeline itself will also fail
run
pipeline_script.py
which contains the pipeline code as decorators.
So in theory the following should actually work.
Let's assume you have pipeline_script.py` with:
...
@PipelineDecorator.pipeline(pipeline_execution_queue=None, name='custom pipeline logic', project='examples', version='0.0.5')
def executing_pipeline(model_ids, mock_parameter='mock'):
accs = [test_model(i) for i in model_ids]
# the casting will wait for the model comparison to complete, before we just launched the components
accs = [int(c) for a in accs]
print("best model is", max(accs))
return Task.current_task()
def run_pipeline(model_ids):
PipelineDecorator.set_default_execution_queue('default')
return executing_pipeline(model_ids=model_ids)
Then from the git action you could do:
from pipeline_script import run_pipeline
task_id = run_pipeline(model_ids=['aa', 'bb', 'cc])
pipeline_task = Task.get_task(task_id)
# do some stuff here
Without getting into too much details this seems totally doable.
I suggest you play around with it, and if you feel something is missing, I would love to help solve / add new features to the pipeline. wdyt?
167 Views
0
Answers
one year ago
one year ago