I want pipeline / task dispatch to be reported and monitored outside of clearml. For example, I might want to log the dispatch event in some non-clearml system and then monitor the health of the pipeline and alert if if it is pending for too long.Hmm interesting, so like a callback?!
I'm thinking a callback is being executed after the Pipelines is sent, but once the callback is done, the pipeline process leaves?
Does that make sense ?
I might want to dispatch other jobs from within the same process.This is actually something that you should not do with decorators as pipelines. Since the logic of the entire pipeline, including execution is the entire script, not just the pipeline decorated function. We might be able to change that, but the current implementation assumes the entire script is the entrypoint for the remote pipeline execution (not just the pipeline decorated function). The idea is that you might need to have some preconfiguration before. But I'm not actually sure we still need to support that.

pipeline dispatch might be part of a ci/cd task that is triggered on code changes related to the pipeline. It dispatches the pipeline and verifies that expected artifacts were produced within some time limits.Then it should be clone-pipeline->enqueue into "services" queue -> wait for result?