How does ClearML handle running jobs while server is down?
We saw some mismatches between what ClearML logged with the actual iteration, in time periods while clearml-server was down
We run distributed training using pytorch, It looks the jobs died with watchdog timer and did not continue to next epoch while server was down (for ~3 hours for backup purposes)
Is there a chance when job failed to report to ClearML - it does continue to next epoch, does not consume any utilization which result in pytorch killing the job?
Hi @<1523701842515595264:profile|PleasantOwl46> , I think that is what happening. If server is down, code continues running as if nothing happened and ClearML will simply cache all results and flush them once server is back up
@<1523701070390366208:profile|CostlyOstrich36> unfortunately, this is not the behavior we are seeing
same exact issue happen tonight
on epoch number 53 ClearML were shut down, the job did not continue to epoch 54 and eventually got killed with watchdog timer
