Ah, the 2 second grace period answers a question I had. I tried to hijack the Tasks's signal handler to see if I can do my exit cleanup then run the Task's handler, but it didn't seem to work. I think I must have triggered the 2s cooldown and had my task terminated.
I think I can work around this right now by running my tasks manually without trains-agent, but I'd love a way to do something on exit. AgitatedDove14 I'd be happy to create an issue. I think the solution might be a bit more involved as a callback because the signal handler might be called in the same thread that also handles the cleanup. As an example, I'm using ignite and in the signal handler calling the terminate()
function on the engine. Whatever graceful exit handler that's implemented would need to be able to handle the asynchronicity between the signal handler returning and the script terminating some time after.