ah. a clue! it came right below that but i guess out of order...
that id is the pipeline that failed

enqueuing. pipe.start("default") but I think it's picking up on my local clearml install instead of what I told it to use.

my tasks have this in them... what's the equivalent for pipeline controllers?

trying to run the experiment that kept failing right now, watching logs (they go by fast)... will try to spot anything anamolous

clearml-server-1.15.1, clearml-1.16.2

default queue is served with (containerized + custom entrypoint) venv workers (agent services just wasn't working great for me, gave up)

are you running this locally or are you enqueueing the task (controller)?

but maybe here's a clue. after hanging like that for a while... it seems like the agent restarts (the container it runs in does not)

I think i've narrowed this down to the ssh connection approach.

regarding the container that runs the pipeline:

• when I made it stop using autossh tunnels and instead put it on the same machine as the clearml server + used docker network host mode, suddenly the problematic pipeline started completing.
  it's just so odd that the pipeline controller task is the only one with an issue. the modeling / data-creation tasks really all seem to complete consistently just fine.

so yeah, best guess now is that its unrelated to clearml verison but rather to the connectivity of the pipeline controller task to the api server.

when I run this pipeline controller locally (also using the same ssh tunnel approach for comms), the pipeline completes just fine. so it's something specific about how its working inside the container vs on my machine, it seems.

do you have any STATUS REASON under the INFO section of the controller task?

thank you

it's pretty reliably happening but the logs are just not informative. just stops midway

that's the final screenshot. it just shows a bunch of normal "launching ..." steps, and then stops all the sudden.

damn. I can't believe it. It disappeared again despite having 1.15.1 be the task's clearml version.
I'm going to try running the pipeline locally.

I really can't provide a script that matches exactly (though I do plan to publish something like this soon enough), but here's one that's quite close / similar in style:
None where I tried function-steps out instead, but it's a similar architecture for the pipeline (the point of the example was to show how to do a dynamic pipeline)

I have tried other queues, they're all running the same container.
so far the only thing reliable is pipe.start_locally()

hoping this really is a 1.16.2 issue. fingers crossed. at this point more pipes are failing than not.

can you share the logs of the controller?

do you have the agent logs that is supposed to run your pipeline? Maybe there is a clue there. I would also suggest to try enqueuing the pipeline to some other queue, maybe even run the agent on your on machine if you do not already and see what happens

let me downgrade my install of clearml and try again.

(the "magic" of the env detection is nice but man... it has its surprises)

Hi @<1689446563463565312:profile|SmallTurkey79> !
Prior runs of this pipeline worked just fine What SDK version were you using for the prior runs? Does this still happen if you revert to that version?
Can you provide a script that imitates what you are doing?
In the pipeline you are running, are you creating new tasks/pipelines/datasets?

would it be on the pipeline task itself then, since that's what's disappearing? that likely the case

nothing came up in the logs. all 200's

odd bc I thought I was controlling this... maybe I'm wrong and the env is mis-set.

None here's how I'm establishing worker-server (and client-server) comms fwiw

damn, it just happened again... "queued" steps in the viz are actually complete. the pipeline task disappeared again without completion, logs mid-stream.

clearml_agent v1.8.1

did you take a look at my connect.sh script? I dont think it's a problem since only the controller task is the problem.

Is there some sort of culling procedure that kills tasks by any chance? the lack of logs makes me think it's something like that.

I can also try different agent versions.

would it be on the pipeline task itself then, since that's what's disappearing?
I will do some experiment comparisons and see if there are package diffs. thanks for the tip.