N/A (still shows as running despite Abort being sent)

I really can't provide a script that matches exactly (though I do plan to publish something like this soon enough), but here's one that's quite close / similar in style:
None where I tried function-steps out instead, but it's a similar architecture for the pipeline (the point of the example was to show how to do a dynamic pipeline)

it happens consistently with this one task that really should be all cache.
I disabled cache in the final step and it seems to run now.

it's pretty reliably happening but the logs are just not informative. just stops midway

odd bc I thought I was controlling this... maybe I'm wrong and the env is mis-set.

ugh. again. it launched all these tasks and then just died. logs go silent.

damn, it just happened again... "queued" steps in the viz are actually complete. the pipeline task disappeared again without completion, logs mid-stream.

yeah, it just shows what I see in the Console, but then immediately goes back to polling for more work (so... instead of running backtest, it exits, no completion message)

ah. a clue! it came right below that but i guess out of order...
that id is the pipeline that failed

when i run the pipe locally, im using the same connect.sh script as the workers are in order to poll the apiserver via the ssh tunnel.

yeah locally it did run. I then ran another via UI spawned from the successful one, it showed cached steps and then refused to run the bottom one, disappearing again. No status message, no status reason. (not running... actually dead)

I have tried other queues, they're all running the same container.
so far the only thing reliable is pipe.start_locally()

that's the final screenshot. it just shows a bunch of normal "launching ..." steps, and then stops all the sudden.