Thanks for releasing this awesome experiment manager! I was logging a single training session on multiple GPUs (using Detectron2), and torch.mp is called for...
4 years ago
VexedKangaroo32
1
Question,
2
Answers
Active since 10 January 2023
Last activity
one year ago
Reputation
0
Badges 1
2 × Eureka!
0
Thanks For Releasing This Awesome Experiment Manager! I Was Logging A Single Training Session On Multiple Gpus (Using Detectron2), And Torch.Mp Is Called For Each Gpu. This Creates A Separate Task In Trains For Each Gpu, And Only One Of The Tasks Has The
I'm not using torch launch, but the launch function in https://github.com/facebookresearch/detectron2/blob/master/detectron2/engine/launch.py I placed Task.init(...) inside the "main_func" that gets called in mp.spawn.
4 years ago
0
Thanks For Releasing This Awesome Experiment Manager! I Was Logging A Single Training Session On Multiple Gpus (Using Detectron2), And Torch.Mp Is Called For Each Gpu. This Creates A Separate Task In Trains For Each Gpu, And Only One Of The Tasks Has The
Meant to get back to you a bit sooner, but I can report that I no longer have duplicate tasks after updating to 0.13.4rc0 and putting Task.init in those two places. The job hasn't run to completion, so I can't report if it ends cleanly or not.
No manual logging attempted Tensorboard, terminal outputs, and argparser log properly No longer need the "-W ignore" arguments
4 years ago