@<1523701868901961728:profile|ReassuredTiger98> id suggest trying to set up diff queues for diff repos then - each with a read only token . The issue is really only when you want a single token to grant access to many repos . Little inconvenient but definitely possible .

Ill also say ive had an easy time forking and modifying the agent code for custom logic changes, so you can always consider that option as well . It’s easy enough to read to be honest .

did you take a look at my connect.sh script? I dont think it's a problem since only the controller task is the problem.

Is there some sort of culling procedure that kills tasks by any chance? the lack of logs makes me think it's something like that.

I can also try different agent versions.

would it be on the pipeline task itself then, since that's what's disappearing?
I will do some experiment comparisons and see if there are package diffs. thanks for the tip.

oooh thank you, i was hoping for some sort of debugging tips like that. will do.

from a speed-of-clearing-a-queue perspective, is a services-mode queue better or worse than having many workers "always up"?

ah. a clue! it came right below that but i guess out of order...
that id is the pipeline that failed

I really can't provide a script that matches exactly (though I do plan to publish something like this soon enough), but here's one that's quite close / similar in style:
None where I tried function-steps out instead, but it's a similar architecture for the pipeline (the point of the example was to show how to do a dynamic pipeline)

oh it's there, before running task.

from task pick-up to "git clone" is now ~30s, much better.

though as far as I understand, the recommendation is still to not run workers-in-docker like this:

export CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1
  export CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=$(which python)

(and fwiw I have this in my entrypoint.sh )

cat <<EOF > ~/clearml.conf
agent {
    vcs_cache {
        enabled: true
    }

    package_manager: {
        type: pip,
      ...

of what task? i'm running lots of them and benchmarking execution times. would you like to see a best case or worst case scenario? (ive kept some experiments for each).

and yeah, in those docs you just linked, "boolean" vars like CLEARML_AGENT_GIT_CLONE_VERBOSE explicitly say true so I ended up trying that pattern. but originally i did try 1. let me go back to that now. thank you.

overall I've seen some improvements in execution time using the suggestions in this thread (tysm!) - th...

hoping this really is a 1.16.2 issue. fingers crossed. at this point more pipes are failing than not.

I'm just working on speeding up the time from "queue experiment" to "my code actually runs remotely" - as of yesterday things would sit for many minutes at a time. trying to see if venv is the culprit .

odd bc I thought I was controlling this... maybe I'm wrong and the env is mis-set.

default queue is served with (containerized + custom entrypoint) venv workers (agent services just wasn't working great for me, gave up)

when i run the pipe locally, im using the same connect.sh script as the workers are in order to poll the apiserver via the ssh tunnel.

worker thinks its in venv mode but is containerized .
apiserver is docker compose stack

ill check logs next time i see it .

currently rushing to ship a model out, so I've just been running smaller experiments slowly hoping to avoid the situation . fingers crossed .

its odd... I really dont see tasks except the controller one dying

let me downgrade my install of clearml and try again.

not quite seeing that one. hoping these views help

yeah, it just shows what I see in the Console, but then immediately goes back to polling for more work (so... instead of running backtest, it exits, no completion message)

still no graphs showing up, and still seeing this error in the console logs.
(deployment is localhost)

clearml_agent v1.8.1

hm. yeah i do see something like what you have in the screenshot.

{"meta":{"id":"d7d059b69fc14cba9ba6ff52307c9f67","trx":"d7d059b69fc14cba9ba6ff52307c9f67","endpoint":{"name":"queues.get_queue_metrics","requested_version":"2.30","actual_version":"2.4"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"queues":[{"avg_waiting_times":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0...

None here's how I'm establishing worker-server (and client-server) comms fwiw

this is not about storage access tokens . its about the App Credentials .
those things you set as CLEARML_API_KEY and SECRET so that clients can talk to the api

yup! that's what I was wondering if you'd help me find a way to change the timings of. Is there an option I can override to make the retry more aggressive?

I've definitely narrowed it down to the reverse proxy I'm behind. when I switch to a cloudflare tunnel, the overhead of the network is <1s compared to localhost, everything feels snappy!

But for security reasons, I need to keep using the reverse proxy, hence my question about configuring the silent clearml retries.

damn, it just happened again... "queued" steps in the viz are actually complete. the pipeline task disappeared again without completion, logs mid-stream.

trying to run the experiment that kept failing right now, watching logs (they go by fast)... will try to spot anything anamolous

https://github.com/clearml/clearml-session/issues/37

Pipeline step caching matches on inputs and task status. If your task points to latest commit, clearml can’t know what that is until runtime and cant cache. On a fixed tag or commit, it sees no code has changed, and so if inputs match (hashable, all parameters are serializable), then it caches.

minute of silence between first two msgs and then two more mins until a flood of logs. Basically 3 mins total before this task (which does almost nothing - just using it for testing) starts.

i would love some advice on that though - should I be using services mode + docker and some max # of instances to be spinning up multiple tasks instead?

my thinking was to avoid some of the docker overhead. but i did try this approach previously and found that the container limit wasn't exactly respected.