SuccessfulKoala55 this seems to work, but I would have preferred a continue_last_task version of enqueue, which would handle things for me, instead of introducing another level of hierarchy
This is what's working for me.for task in tasks: subprocess.run(['python', './test_task.py', '--task_id', task.id])where in test_task.py I have the following:
` parser = ArgumentParser()
if Task.running_locally():
parser.add_argument('--task_id',type=str)
args=parser.parse_args()
task = Task.init(
continue_last_task=args.task_id
)
task.execute_remotely(queue_name='default') `and the rest is the inference code, which is only run on the remote, and includes populating the argparaser instance with the necessary arguments
the only solution I see is to have another script that would run the test task using (for example) popen in a separate process.
can you give me a snippet? when I tried it didn't work, since execute_Remotes terminates the local task, unless you tell it to clone, which is not what I need here.
SuccessfulKoala55 on a different question that came up in the context of this use case, I want to use Task.init(continue_last_task=id) in order to add the inference results to the original task that ran the inference.
I was able to do this for a single task. I locally fetch the task ids, then for the first task I run Task.init with continue_last_task and subsequently I run task.execute_remotely() and the task is then run remotely with the outputs appended to the original training task.
When doing this to a single task, I see that the task's script is being replaced with the current inference script and that's working fine. How can I do this for multiple tasks? I tried runnitn Task.enqueue after the task.init with continue but that doesn't work.
args_string = ['--{} {}'.format(k, v) for k, v in task.get_parameters_as_dict().get('Args', {}).items()] args_strings = [a.replace("'", '').replace("[", '').replace("]", '').replace(",", '') for a in reduce(lambda l, s: l + s.split(' '), args_string, []) if a]and then just parser.parse_args(ars_strings)
It's not even very clean, I could have replaced the multiple replace calls with a regex etc. just a quick hack to work around it.
Actually, I found out that if I use execute_remotely I don't even need this, since the parser arguments are apparently reconstructed. so just instantiating the parser and running parser.parse_args() is enough
SuccessfulKoala55 I managed to find a workaround, by instantiating a new parser and feeding it the string values. if there'd be a cleaner solution I'll be happy to hear about it.
SuccessfulKoala55 I ran a training task. I now wish to run inference on some data. My model's init function expects the parsed args Namespace as an argument. In order to load the weights of the saved model I need to instantiate the class for which I need the args variable.
ThickDove42 what is the use-case, exactly?