Hey @<1537605940121964544:profile|EnthusiasticShrimp49> ! You’re mostly correct. The Step classes will be predefined (of course developers are encouraged to add/modify as needed), but as in the DataTransformationStep , there may be user-defined functions specified. That’s not a problem though, I can provide these functions with the helper_functions argument.

• The .add_function_step is indeed a failing point. I can’t really create a task from the notebook because calling Task.init(…) and then task.execute_remotely() closes the notebook process.- In theory it would be nice to even just create a “Draft Task” and copy that, but that, too, requires an entry point, the code to be prepared, etc. It does not take it from a class, and I cannot simply create it from within a notebook.- I was just thinking about the clutter it will create. Such experimentations can easily create many (hundreds?) of pipeline runs. Some examples would be:- We have a DAG of “A -> B”. The user wants to run Step A first, so they can examine the output in the notebook. It creates a pipeline with a single step, and the results are polled in the notebook. After they examined it, they now want to run Step B. I can add Step B to the previous pipeline, and let it run (assuming the cache would work), but I’d rather keep the logic internal and basically only run Step B. That entails recreating “the notebook pipeline” with only a single step, B.
• Same DAG, but after examining results from A, the user may change that step. I cannot rely on ClearML’s caching, because this definition might be out-of-scope for the caching (i.e. a user may have installed a package in doing so). I’d like to recreate the pipeline so that both A and B are run.
• Overall, this is not an issue, just a clutter problem, that can be resolved by automatically nesting pipelines in one location and archiving as needed.
  Closing thoughts, is that the ideal case would be just creating a task. I don’t need the pipeline, really, I just need to be able to create a task from a class function, define inputs/outputs, and execute it remotely. That’s why I originally tried task.create_function_task , but it cannot be executed remotely and/or it will close the running process…