I am looking for a way to gracefully stop the task (clean up artifacts, shutdown backend service) on the agent
The clean up service is awesome, but it would require to have another agent running in services mode in the same machine, which I would rather avoid
I mean kill a detached subprocess on the machine executing the agent
Oh, I see. Now I understand why you want it on the same remote machine...
Also maybe we are not on the same page - by clean up, I mean kill a detached subprocess on the machine executing the agent
Hi JitteryCoyote63 ,
Will this task be executed in development mode (i.e. locally) or using Trains Agent?
Just to make sure we're on the same page, the cleanup code won't actually listen for a signal, but will sample the tasks periodically and look for the above-mentioned status reason in tasks whose
status field is
stopped (when getting tasks from the server in the cleanup code, you can't query on the
status_reason field, but you can query on the
In that case, you can monitor the Task's
status_reason field which should contain
"task was stopped by tasks.stop"
using Trains Agent 🙂
The Trains Agent that is actually running your experiment already detects the user abort selected in the UI and stops executing. The Cleanup Server picks up on that and can clean up whatever you desire (i.e. by adapting the cleanup service code)
Ok, but that means this cleanup code should live somewhere else than inside the task itself right? Otherwise it won't be executed since the task will be killed
The task requires this service, so the task starts it on the machine - Then I want to make sure the service is closed by the task upon completion/failure/abortion
Well, how would you imagine it running? Did you intend for it to run as part of the executed task itself?
mmh it looks like what I was looking for, I will give it a try 🙂
Can you make do with
yes, exactly 🙂
That's the purpose of the Trains Agent Services mode - in a nutshell, you run an agent who's purpose if to run cpu-only maintenance tasks that take care of things like cleanup
Which uses this: https://allegro.ai/docs/concepts_arch/concepts_arch/#trains-agent-services-container