My bad, I worded my question wrong I see, I meant the tasks of the pipeline’s components. (it shows that I’m a newbie 😅 )
This does make perfect sense though! The problem seems to just be that the components themselves are ran on the same queue as the pipeline logic, even though I configured it differently
However, all of the pipelines tasks are ran on the same queue. Could I be missing something?
The pipeline Task itself is running on a dedicated queue (meaning agent/s) usually because the pipeline logic is mostly idling, where as the components themselves are doing the actual compute.
Specifically you can control the pipeline logic queue with
Does that make sense ?
This workflow however is the only way I have found to easily fix my previous ‘Module not found’ errors
Hmm okay make sense,
Did you try to set these ?
or even hack the sys.path with something like
import sys, os sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)+"/../")
My bad, I worded my question wrong I see,
LOL no worries 🙂
Any chance you have some "debug" leftover in the Pipeline code:
Maybe we should show a warning when we it is being called, or ignore it when running via an agent ...