Well, how would you imagine it running? Did you intend for it to run as part of the executed task itself?

Ok, but that means this cleanup code should live somewhere else than inside the task itself right? Otherwise it won't be executed since the task will be killed

Hi JitteryCoyote63 ,
Will this task be executed in development mode (i.e. locally) or using Trains Agent?

I mean kill a detached subprocess on the machine executing the agent

Oh, I see. Now I understand why you want it on the same remote machine...

The clean up service is awesome, but it would require to have another agent running in services mode in the same machine, which I would rather avoid

See https://allegro.ai/docs/examples/services/cleanup/cleanup_service/

That's the purpose of the Trains Agent Services mode - in a nutshell, you run an agent who's purpose if to run cpu-only maintenance tasks that take care of things like cleanup

The task requires this service, so the task starts it on the machine - Then I want to make sure the service is closed by the task upon completion/failure/abortion

In that case, you can monitor the Task's status_reason field which should contain "task was stopped by tasks.stop"

mmh it looks like what I was looking for, I will give it a try 🙂

yes, exactly 🙂

Also maybe we are not on the same page - by clean up, I mean kill a detached subprocess on the machine executing the agent

I am looking for a way to gracefully stop the task (clean up artifacts, shutdown backend service) on the agent

Which uses this: https://allegro.ai/docs/concepts_arch/concepts_arch/#trains-agent-services-container

Can you make do with atexit.register ?

using Trains Agent 🙂

The Trains Agent that is actually running your experiment already detects the user abort selected in the UI and stops executing. The Cleanup Server picks up on that and can clean up whatever you desire (i.e. by adapting the cleanup service code)

Just to make sure we're on the same page, the cleanup code won't actually listen for a signal, but will sample the tasks periodically and look for the above-mentioned status reason in tasks whose status field is stopped (when getting tasks from the server in the cleanup code, you can't query on the status_reason field, but you can query on the status field)