Hi @<1619505588100665344:profile|GrievingHare27>
My understanding is that initiating a task with
Task.init()
captures the code for the entire notebook. I'm facing difficulties when attempting to build a final training pipeline (in a separate notebook) that uses only certain functions from the other notebooks/tasks as pipeline steps.
Well this is is kind of the limit of working with jupyter notebooks, referencing code from one to another is not really feasible (of course you can start a git repo and write python script, but that is for another discussion)
That said I think that what would work for you is the pipeline decorator flow.
Basically you can have one notebook with all your functions, then when you want to build a pipeline you just create a pipeline form the functions and they will be running, if you need functions to be available to different pipeline components, just add them to the helper_functions argument
wdyt?
fyi: the pipeline Logic itself (if created from a notebook) is the notebook Main execution, so I would clean it up a bit before making it a pipeline, basically it should look like the script linked below
