AgitatedDove14 sorry if that wasn't clear. I think the issue is that when trains-agent runs the script, none of the flag values are set until the Task object is initialized. For that to happen, the task object needs to know what project/task to connect to, which I presume is via the project_name and task_name parameters.
If those parameters are themselves dependent on flags, then they will be uninitialized when trains-agent runs the script, as it does not run it with any command line flags.
That's great to know. Thank you AgitatedDove14 . I might have gone wrong somewhere else, so I'll double-check.
I presume is via the
project_name
and
task_name
parameters.
You are correct in your assumption, it only happens when you call Task.init but two distinctions:
ArgParser arguments are overridden (with trains-agent) even before Task.init is called Task.init when running under trains-agent will totally ignore the project/task name, it receives a pre-made task id, and uses it. So the project name and experiment are meaningless if you are running the task with trains-agent
A few implementation details to complete the image for you. Task.init creates a new Task when running manually, and puts data into the created task (e.g. ArgParser dictionary params etc)
When called under trains-agent, the Task.init will do the opposite. It will not create a Task but it receives a Task id, from it it will pull all the parameters into the code, including argparser etc.
The exception is ArgParser, since it can be called before Task.init is called. As long as someone imported the trains package, the auto-magic will make sure that in trains-agent mode, the argparser will get the correct arguments before the call for Task.init
Hi SillyPuppy19
I think I lost you half way through.
I have a single script that launches training jobs for various models.
Is this like the automation example on the Github, i.e. cloning/enqueue experiments?
flag which is the model name, and dynamically loading the module to train it.
a Model has a UUID in the system as well, so you can use that instead of name (which is not unique), would that solve the problem?
This didn't mesh well with Trains, because the project and task names were dynamically generated based on the flag.
Not sure I follow, but you can always change a Task name, even after initialization of cloning. Is this what you mean?