A few implementation / design details:
When you run code with Trains (and call init) it will record your environment (python packages, git code, uncommitted changes etc) Everything is stored on the Task object in the trains-server, when you clone a task you literally create a copy of the Task object (i.e. a second experiment). on the cloned experiment, you can edit everything (parameters, git, base docker image etc) When you enqueue a Task you add its ID to the execution queue list a trains-agent listens on the execution pops the Task ID and sets up the environment as written on the task, then it will launch and monitor the code Multiple trains-agent workers can listen on the same queue, or on multiple queues in a priority fashion (i.e. first try to pop from the first, then the seconds etc.)