odd bc I thought I was controlling this... maybe I'm wrong and the env is mis-set.
would it be on the pipeline task itself then, since that's what's disappearing? that likely the case
hoping this really is a 1.16.2 issue. fingers crossed. at this point more pipes are failing than not.
ugh. again. it launched all these tasks and then just died. logs go silent.
worker thinks its in venv mode but is containerized .
apiserver is docker compose stack
ill check logs next time i see it .
currently rushing to ship a model out, so I've just been running smaller experiments slowly hoping to avoid the situation . fingers crossed .
yeah this problem seems to happen on 1.15.1 and 1.16.2 as well, prior runs were on the same version even. It just feels like it happens absolutely randomly (but often).
just happened again to me.
The pipeline is constructed from tasks, it basically does map/reduce. prepare data -> model training + evaluation -> backtesting performance summary.
It figures out how wide to go by parsing the date range supplied as input parameter. Been running stuff like this for months but only recently did things just start... vanishing like this.
Would appreciate any help. Really need this to be more robust to make the case for company-wide adoption.
that's the final screenshot. it just shows a bunch of normal "launching ..." steps, and then stops all the sudden.
None here's how I'm establishing worker-server (and client-server) comms fwiw
it's pretty reliably happening but the logs are just not informative. just stops midway
when i run the pipe locally, im using the same connect.sh script as the workers are in order to poll the apiserver via the ssh tunnel.