Hi @<1523704198338711552:profile|RoughTiger69> ,

there is a pipeline example that gets the input from previous tasks in the pipeline, can this help?

None

I want the task of human tagging a model to be “just another step in the pipeline”

That makes total sense.
Quick question, would you prefer the pipeline controller to "wait" for the tagging and then continue, or would it make more sense to create a trigger on the tagging ?

Awesome! let me know how/if it works 🙂

Hi @<1523704198338711552:profile|RoughTiger69>
From this scenario can we assume the "selection" will be tagging the model manually?
Also, how would an human operator decide on the best model, that is what is the input to base the decision on?

could work! is there a way to visualize the pipeline such that this step is “stuck” in executing?

sounds workable, will start with that

yes and no.
This is a pseudo flow:
Data download -> pre-processing -> model training (e.g. HPT) - > model evaluation (per variant) -> model comparison dashboard -> human selects the best model using a heuristic and the status of the weather -> model packaging -> inference tests etc.

I could divide it into two pipelines:
Data download --> dashboard
Packaging --> …

Where packaging takes a parameter which is the human selected ID of the model.
However, this way, I lose the context of the entire flow.
I also don’t have a way to “push” the human to make a decision or reflect it back to them.

So I am looking for something that is similar to manual QA approval step in CI/CD.

is there a way to visualize the pipeline such that this step is “stuck” in executing?

Yes there is, the pipelline plot (see plots section on the Pipeline Task, will show the current state of the pipeline.
But I have a feeling you have something else in mind?
Maybe add Tag on the pipeline Task itself (then remove it when it continues) ?
I'm assuming you need something that is quite prominent in the UI, so someone knows ?
(BTW I would think of integrating it with the slack monitor, to push a p=message to users that the pipeline is waiting for human interaction), wdyt ?

yeah, its a tradeoff that is dependent on parameters that lie outside the realm of human comprehension.
Let’s call if voodoo.
Yes, the manual selection can be done via tagging a model.
The main thing is that I want the selection to be part of the overall flow.
I want the task of human tagging a model to be “just another step in the pipeline”

So “wait” is a better metaphore for me

So I would do something like (I might have a few typos but that's the gist):

def post_execute_callback_example(a_pipeline, a_node):
    # type (PipelineController, PipelineController.Node) -> None
    print('Completed Task id={}'.format(a_node.executed))
    # wait until model is tagged, then pass it as argument
    while True:
        found = Moodel.query_models(...) # model filter here, inlucing tag and project
        if found:
            break
        print('waiting for model taggging')
        sleep(60)
    # find the actual next step node and pipe the model model ID
    next_step_node= a_pipeline.get_pipeline_dag()['next_step_name_here']
    next_step_node.parameters['Args/model_id'] = found[0].id
    return

pipe.add_step(name='stage_process', parents=['stage_data', ],
              base_task_project='examples', base_task_name='pipeline step 2 process dataset',
              parameter_override={'General/dataset_url': '${stage_data.artifacts.dataset.url}',
                                  'General/test_size': 0.25},
              post_execute_callback=post_execute_callback_example
              )

WDYT?

that’s the thing. I want to it to appear like one long pipeline, vs. trigger a new set of steps after the approval. So “wait” is a better metaphore for me

thanks so much for taking the time by the way!