the workers connect to the clearml server via ssh-tunnels, so they all talk to "localhost" despite being deployed in different places. each task creates artifacts and metrics that are used downstream
N/A (still shows as running despite Abort being sent)
ugh. again. it launched all these tasks and then just died. logs go silent.
Hi @<1689446563463565312:profile|SmallTurkey79> ! I will take a look at this and try to replicate the issue. In the meantime, I suggest you look into other dependencies you are using. Maybe some dependency got upgraded and the upgrade now triggers this behaviour in clearml.
yeah this problem seems to happen on 1.15.1 and 1.16.2 as well, prior runs were on the same version even. It just feels like it happens absolutely randomly (but often).
just happened again to me.
The pipeline is constructed from tasks, it basically does map/reduce. prepare data -> model training + evaluation -> backtesting performance summary.
It figures out how wide to go by parsing the date range supplied as input parameter. Been running stuff like this for months but only recently did things just start... vanishing like this.
Would appreciate any help. Really need this to be more robust to make the case for company-wide adoption.
ah. a clue! it came right below that but i guess out of order...
that id is the pipeline that failed
let me downgrade my install of clearml and try again.
I really can't provide a script that matches exactly (though I do plan to publish something like this soon enough), but here's one that's quite close / similar in style:
None where I tried function-steps out instead, but it's a similar architecture for the pipeline (the point of the example was to show how to do a dynamic pipeline)
do you have the agent logs that is supposed to run your pipeline? Maybe there is a clue there. I would also suggest to try enqueuing the pipeline to some other queue, maybe even run the agent on your on machine if you do not already and see what happens
damn. I can't believe it. It disappeared again despite having 1.15.1 be the task's clearml version.
I'm going to try running the pipeline locally.
enqueuing. pipe.start("default") but I think it's picking up on my local clearml install instead of what I told it to use.
my tasks have this in them... what's the equivalent for pipeline controllers?
but maybe here's a clue. after hanging like that for a while... it seems like the agent restarts (the container it runs in does not)
would it be on the pipeline task itself then, since that's what's disappearing? that likely the case
would it be on the pipeline task itself then, since that's what's disappearing?
I will do some experiment comparisons and see if there are package diffs. thanks for the tip.
yeah locally it did run. I then ran another via UI spawned from the successful one, it showed cached steps and then refused to run the bottom one, disappearing again. No status message, no status reason. (not running... actually dead)
I have tried other queues, they're all running the same container.
so far the only thing reliable is pipe.start_locally()
hoping this really is a 1.16.2 issue. fingers crossed. at this point more pipes are failing than not.
are you running this locally or are you enqueueing the task (controller)?
damn, it just happened again... "queued" steps in the viz are actually complete. the pipeline task disappeared again without completion, logs mid-stream.
(the "magic" of the env detection is nice but man... it has its surprises)
it happens consistently with this one task that really should be all cache.
I disabled cache in the final step and it seems to run now.
that's the final screenshot. it just shows a bunch of normal "launching ..." steps, and then stops all the sudden.
trying to run the experiment that kept failing right now, watching logs (they go by fast)... will try to spot anything anamolous
None here's how I'm establishing worker-server (and client-server) comms fwiw
it's pretty reliably happening but the logs are just not informative. just stops midway
when i run the pipe locally, im using the same connect.sh script as the workers are in order to poll the apiserver via the ssh tunnel.