Unanswered
Thread Re: Pipelines And How They'Re Meant To Be Used / How Long They Take To Orchestrate.
mind-blowing... but somehow just later in the same day I got the same pipeline to create its DAG and start running in under a minute.
I don't know what exactly I changed. The pipeline task was run locally (which I've never done before), then cloned to run remotely in my services queue. And then it just flew through the experiment at the pace I expected.
so there's hope. i'll keep stress-testing it and see what causes differences. I was right to suspect that such a simple DAG should not take upwards of 30-60 minutes to compute.
46 Views
0
Answers
4 months ago
3 months ago