Unanswered
Hi Everyone,
Looking For Ml Management Tools I Stumbled Upon Trains, I Must Say It Has Been Awesome So Far. I Just Have A (Probably Stupid) Question: I'M Trying To Setup A Multi-Node Training Environment And I Thought I Could Solve This With Agents, But A
Hi AgitatedDove14 , thanks for the quick reply! Well, not sure if this answers your question, but what I'd like to have is a scheduler that is aware of the resources that each agent has access to, in some way. E.g. I launch an agent in machine1, then an agent in machine2, but when I enqueue a task I'd like to say "this requires 2 GPUs" and some sort of scheduling machanism assigns it to any available agent with those resources available. Of course, once I turn off the machine1, agent 1 also dies, so the scheduler should handle new tasks accordingly. This is basically how I imagined it to work, I apologize if it's a weird mechanism and/or the Trains stack was not supposed to work this way.
151 Views
0
Answers
4 years ago
2 years ago
Tags