OK, I will look into agents and think about this. One pain we have is that tensorboard logs are stuck on the machine used for training, and we can’t compare models training on two different machines in one tensorboard (unless they mount the same network filesystem). But it is also important to be able to see TB both during training and after it is finished (and even though the log files are large, storage is cheap, so maybe it would be OK to keep them around). I need to think about the best way to organize this though. For instance, maybe we should log logs directly to S3? We would still need some system for keeping track of where exactly they are and for launching tensorboard instances to show a given set of logs.