first, thanks for having these discussions. I appreciate this kind of support is an effort 🙏
Yes. i perfectly understand that once a pipeline job (or a task) is sent off in this manner, it executes separately (and, most likely in a different machine) from the process that instantiated it.
I still feel strongly that such a command should not be thought of as a fire and exit operation. I can think of several scenarios where continued execution of the instantiating process is desired:
I want pipeline / task dispatch to be reported and monitored outside of clearml. For example, I might want to log the dispatch event in some non-clearml system and then monitor the health of the pipeline and alert if if it is pending for too long. I might want to dispatch other jobs from within the same process. pipeline dispatch might be part of a ci/cd task that is triggered on code changes related to the pipeline. It dispatches the pipeline and verifies that expected artifacts were produced within some time limits.I realise that there may be ways around this exit call or existing clearml mechanisms that support performing some of the example use-cases I mentioned here but IMO - leave the decision up to the developer and don't kill their process.