Graceful shutdown on kubernetes

[mattbooks]

After reading #71, testing and reading the code, my understanding is that falcon expects a SIGINT to trigger a graceful shutdown. However, in kubernetes, SIGTERM is sent to trigger shutdown. When this happens, falcon immediately shuts down children, exits with code 1 and logs several errors. I'm not totally sure what the impact is on code running in the children.

Is it possible to enable a clean shutdown with SIGTERM?

SIGINT behaviour:

❯ bundle exec falcon host
  0.0s     info: Falcon::Command::Host [oid=0x9d0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | Falcon Host v0.52.4 taking flight!
               | - Configuration: falcon.rb
               | - To terminate: Ctrl-C or kill 92915
               | - To reload: kill -HUP 92915
 0.01s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | {status: "Initializing controller..."}
 0.01s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:45 -0800]
               | Preloading preload.rb...
  1.6s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Starting my-app on #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {protocol: Async::HTTP::Protocol::HTTP11}>
  1.6s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Controller starting...
  1.6s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Starting container...
 1.64s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Waiting for startup...
 1.65s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Finished startup.
 1.65s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | {ready: true, size: 11}
 1.65s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=92915] [2025-12-01 10:08:47 -0800]
               | Controller started...
34.39s     info: Async::Container::Forked [oid=0x2ea8] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Stopping container...
               | {
               |   "timeout": true,
               |   "caller": [...]
               | }
34.39s     info: Async::Container::Group [oid=0x2eb0] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Sending interrupt to 11 running processes...
34.53s     info: Async::Container::Forked [oid=0x2ea8] [ec=0x9d8] [pid=92915] [2025-12-01 10:09:20 -0800]
               | Group has stopped.

SIGTERM behaviour:

❯ bundle exec falcon host
  0.0s     info: Falcon::Command::Host [oid=0x9d0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | Falcon Host v0.52.4 taking flight!
               | - Configuration: falcon.rb
               | - To terminate: Ctrl-C or kill 99553
               | - To reload: kill -HUP 99553
 0.01s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | {status: "Initializing controller..."}
 0.01s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:22 -0800]
               | Preloading preload.rb...
 2.08s     info: Falcon::Service::Server [oid=0x9f0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Starting my-app on #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {protocol: Async::HTTP::Protocol::HTTP11}>
 2.08s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Controller starting...
 2.08s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Starting container...
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Waiting for startup...
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Finished startup.
 2.09s     info: Async::Container::Notify::Console [oid=0x9e0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | {ready: true, size: 2}
 2.09s     info: Async::Service::Controller [oid=0x2df0] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:24 -0800]
               | Controller started...
10.37s     info: Async::Container::Forked [oid=0x2e18] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Stopping container...
               | {
               |   "timeout": false,
               |   "caller": [...]
               | }
10.37s     info: Async::Container::Group [oid=0x2e20] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Sending terminate to 2 running processes...
10.37s    error: Async::Container::Forked::Child [ec=0x2df8] [pid=99556] [2025-12-01 10:44:33 -0800]
               | SIGTERM
               |   Async::Container::Terminate: SIGTERM
               |   → /Users/matt.books/.local/share/mise/installs/ruby/3.4.2/lib/ruby/gems/3.4.0/gems/async-2.34.0/lib/async/scheduler.rb:525 in 'Thread.handle_interrupt' [...]
10.37s    error: Async::Container::Forked::Child [ec=0x2e08] [pid=99557] [2025-12-01 10:44:33 -0800]
               | SIGTERM
               |   Async::Container::Terminate: SIGTERM
               |   → /Users/matt.books/.local/share/mise/installs/ruby/3.4.2/lib/ruby/gems/3.4.0/gems/async-2.34.0/lib/async/scheduler.rb:525 in 'Thread.handle_interrupt' [...]
10.38s    error: Async::Container::Forked [oid=0x2e18] [ec=0x2e28] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Child exited with error!
               | {
               |   "status": "pid 99557 exit 1",
               |   "running": false
               | }
10.38s    error: Async::Container::Forked [oid=0x2e18] [ec=0x2e30] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Child exited with error!
               | {
               |   "status": "pid 99556 exit 1",
               |   "running": false
               | }
10.38s     info: Async::Container::Forked [oid=0x2e18] [ec=0x9d8] [pid=99553] [2025-12-01 10:44:33 -0800]
               | Group has stopped.

[ioquatix]

Thanks for your detailed report. I need to think about the best approach.

[roger-mo-gusto]

Hi @ioquatix!

I'm working with Matt and wanted to chime in with a temporary solution I've added while he's OOO.

We updated our Helm chart to accept a custom preStopCommand variable that we're inserting into the preStop lifecycle hook. Here's what we're passing in preStopCommand: [ "sh", "-c", "kill -INT 1 && sleep 65" ]

Customizing the stopSignal hook would be more applicable, but that's in alpha (link).

Before

After

Didn't go with this, but I also stumbled upon another solution to intercept the TERM signal by adding the following to preload.rb:
Signal.trap("TERM") { Process.kill("INT", $$) }

Let me know if you have any thoughts on these approaches.

Thanks & happy holidays!

[ioquatix]

Thanks for working on this. I don't have a strong opinion yet, but INT-TERM-KILL is a more elaborate shutdown mechanism.

• SIGINT -> all tasks receive Async::Stop exception, except defer_stop tasks.
• SIGTERM -> all tasks are receiving Async::Stop until they exit.
• SIGKILL -> uncatchable / process dead.

A simple solution might be to shim a process around async-service to deal with this, in other words, map SIGTERM to SIGINT to the child process, which is probably more reliable, but similar to your proposed fix.