BroadMole98 as one can expect long answer as well 🙂

I have a workflow with 19000 job nodes in it.

wow, 19k job nodes? as in a single pipeline 19k steps?

The main idea of the trains-agent is to allow multi-node workloads, and creating pipelines on top of a scheduler without worrying about docker packaging (done automatically for you), and to have a proper scheduler with priority (that is missing from k8s)

If the first step is just "logging" all the steps, you can easily add "Task.init" at the beginning of any script and have the ability to upload artifacts, or access other tasks artifacts, create graphs etc.

The trains automl is all about multi-node automl (single node is actually relatively easy to do with optuna and the like), setting up the environment on multiple machines reading back performance metrics in realtime and controlling the flow is the real challenge, which trains covers, as this is all part of the fact everything is logged into the trains-server with full pythonic interface.

regrading invoking jobs, think of the trains-server as a server that holds all the configurations for all the jobs. Then trains-agent execute basically pulls the configuration and sets the environment (whether inside a docker or as virtual environment). On top of it, trains-agent daemon pulls jobs form the execution queue and runs the trains-agent execute to actually launch the job.
Makes sense?