Process Management: Daemons, systemd, and Graceful Shutdown

Daemons and Services

A multi-channel AI gateway is not a script you run once and walk away from — it is a service that must be running at 3 a.m., come back automatically after a reboot, and not garble a user's request during a redeploy. The central mindset shift is that your application should not manage its own lifecycle. It runs in the foreground, logs to standard output, responds to a couple of signals, and lets a dedicated supervisor own everything else.

Historically a program made itself a daemon through an elaborate ritual: fork, have the parent exit, call setsid() to detach from the terminal, fork again, close the standard file descriptors, and write a PID file. This double-forking dance is fiddly, error-prone, and hard to observe — if the process died, nothing noticed. The modern approach is the opposite: the app runs in the foreground and a supervisor starts it, restarts it on crash, captures its logs, enforces limits, and orders it relative to other units. You write no "go into the background" logic at all.

The dominant supervisor is systemd, which runs as PID 1 and manages the machine's services. But the concept is portable. In a container, the runtime plays supervisor: Docker restart policies (--restart on-failure, unless-stopped, always) restart a crashed container much as Restart=on-failure restarts a crashed service. In Kubernetes, the kubelet restarts crashed containers per the pod's restartPolicy and uses liveness probes to detect hangs.

One of the strongest habits is treating logs as event streams to stdout. A well-behaved service does not open its own log files or rotate them; it writes each event as a line to stdout/stderr and lets the environment capture it. Under systemd this is automatic — anything on stdout/stderr is captured by the journal, timestamped, and tagged with the unit name. A minimal setup is just:

import logging, sys
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(message)s',
    stream=sys.stdout,      # let the supervisor own the stream
)

The payoff is uniformity: the same log line is captured identically under systemd, Docker, or Kubernetes, and each environment forwards the stream wherever it likes without the application knowing.

systemd Units in Practice

A systemd unit is a plain-text, .ini-style file with three sections: [Unit] (metadata and ordering), [Service] (how to run the process), and [Install] (how it hooks into boot). Here is a production-shaped unit for a gateway worker:

[Unit]
Description=AI Gateway Worker
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=simple
User=gateway
WorkingDirectory=/opt/gateway
Environment=GATEWAY_ENV=production
EnvironmentFile=/etc/gateway/worker.env
ExecStart=/opt/gateway/.venv/bin/python -m gateway.worker
Restart=on-failure
RestartSec=5
RestartSteps=4
RestartMaxDelaySec=60
TimeoutStopSec=30
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Several fields deserve attention. ExecStart= must use an absolute path to the interpreter — ideally the virtualenv's Python — because systemd does not inherit your login shell's PATH and does not activate virtualenvs; the process must run in the foreground (no fork, no &) because systemd tracks the PID it spawned. User= runs the service as an unprivileged dedicated account, limiting blast radius. Environment= sets inline variables; EnvironmentFile= loads KEY=value pairs from a file — the natural home for secrets. After=/Wants=network-online.target order the unit after the network is up.

The Type= field tells systemd how to decide the service has started. Type=simple (default) considers it started the instant the process is forked; Type=exec waits for execve(); the most robust is Type=notify, where the app calls sd_notify with READY=1 once it has bound sockets and opened connections. Avoid Type=forking, which exists for legacy double-forking daemons.

The Restart= directive is the heart of self-healing. The recommended value for a long-running service is Restart=on-failure: it restarts on a non-zero exit, a fatal signal, a timeout, or a watchdog expiry — but not after a clean exit you requested, so it never fights systemctl stop.

RestartSec= sets the delay before restarting (default is a mere 100 ms); a value like 5 prevents a tight crash-loop from pegging the CPU. For exponential backoff, add RestartSteps= and RestartMaxDelaySec=. But backoff alone will not stop an unrecoverable failure from looping forever — that is the job of start-rate limiting: StartLimitIntervalSec=60 and StartLimitBurst=5 mean "if the service is (re)started more than 5 times within any 60-second window, give up." The unit enters failed; run systemctl reset-failed <name> once fixed.

Figure 12.2: systemd service state lifecycle under Restart=on-failure.

Because the service logs to stdout/stderr, systemd routes everything into the journal, read with journalctl:

journalctl -u gateway-worker -f                    # follow live, like tail -f
journalctl -u gateway-worker --since "10 min ago"  # recent window
journalctl -u gateway-worker -p err                 # only error-priority lines
journalctl -u gateway-worker -b                      # only since last boot

Managing the unit follows a small set of commands. The critical pairing is enable versus start: start runs it now but is not persistent; enable wires it into a boot target so it comes up on reboot. enable --now does both — almost always what you want.

systemctl daemon-reload                 # re-read unit files after editing
systemctl enable --now gateway-worker   # start now AND start on every boot
systemctl status gateway-worker         # state + a few recent log lines
systemctl restart gateway-worker        # stop then start

Graceful Shutdown

When systemd (or Kubernetes, or Docker) stops a service it does not kill it instantly. It first sends SIGTERM — the polite request and the default KillSignal= — then waits up to TimeoutStopSec=, and only if the process is still alive at the end of that window does it send the uncatchable SIGKILL. That window is precisely the application's opportunity to perform a graceful shutdown: stop accepting new work, drain in-flight work, flush buffers, close connections, acknowledge messages, and exit cleanly.

Figure 12.1: The service stop lifecycle.

The default behavior of a bare python worker.py is that SIGTERM terminates the process abruptly with no cleanup — the worst outcome for a message processor. In asyncio the correct fix is cooperative signal handlers on the event loop. The Python docs recommend loop.add_signal_handler(signum, callback) over the low-level signal.signal(), because the callback runs inside the event loop and can safely touch loop state and set events. It is Unix-only and must be registered from the main thread:

import asyncio, signal

async def main():
    loop = asyncio.get_running_loop()
    stop = asyncio.Event()
    for sig in (signal.SIGINT, signal.SIGTERM):
        # Handler does the MINIMUM: just flip the event. No I/O here.
        loop.add_signal_handler(sig, stop.set)
    await run_until_shutdown(stop)

The single most important rule: the signal handler must do nothing but flip a flag. Cleanup happens in normal coroutine code that observes the event, never inside the signal-handling context, where async I/O is fragile.

Draining means: stop pulling new work, let running tasks finish (or be safely cancelled), and only then exit. A robust worker combines two strategies: signal-an-event and let workers finish (await completion of in-flight tasks); and cancel-and-cleanup for stragglers still running when the grace period expires (call .cancel(), then await asyncio.gather(*tasks, return_exceptions=True); each task catches CancelledError in try/finally to release resources — roll back a transaction, nack a message, flush a metric).

Figure 12.3: The two-phase drain — finish in-flight tasks, then force-cancel stragglers.

Here is a complete worker that catches SIGTERM, stops accepting new work, drains within a bounded timeout, and force-cancels stragglers. Notice the ordering: stop accepting work first, drain with a bounded asyncio.wait(..., timeout=GRACE_PERIOD), force-cancel, and only then close the queue.

GRACE_PERIOD = 25  # seconds; MUST be < unit's TimeoutStopSec (30)

async def handle_message(msg, inflight: set):
    try:
        await do_work(msg)      # the real side effect
        await msg.ack()         # ack AFTER the work is durably done
    except asyncio.CancelledError:
        await msg.nack()        # drain expired: let it be redelivered
        raise
    finally:
        inflight.discard(asyncio.current_task())

async def main():
    loop = asyncio.get_running_loop()
    stop = asyncio.Event()
    for sig in (signal.SIGINT, signal.SIGTERM):
        loop.add_signal_handler(sig, stop.set)
    inflight = set()
    queue = await connect_queue()
    while not stop.is_set():
        try:
            msg = await asyncio.wait_for(queue.get(), timeout=1.0)
        except asyncio.TimeoutError:
            continue            # re-check stop periodically even when idle
        task = asyncio.create_task(handle_message(msg, inflight))
        inflight.add(task)
    if inflight:
        done, pending = await asyncio.wait(inflight, timeout=GRACE_PERIOD)
        for task in pending:
            task.cancel()
        if pending:
            await asyncio.gather(*pending, return_exceptions=True)
    await queue.close()          # close external resources last

The grace period is a hard budget set by TimeoutStopSec=. Keep the app's internal grace period shorter than it (25 vs 30 here, leaving margin for connection teardown); if a legitimate drain needs 45 seconds, raise TimeoutStopSec to match. Make shutdown idempotent and re-entrant: a second SIGTERM should not restart or crash the drain (the asyncio runner even escalates a second Ctrl+C to KeyboardInterrupt). Note that asyncio.run() already best-effort-cancels remaining tasks on exit, but it is signal-blind, so you still need explicit SIGTERM handling to trigger and bound your drain.

Surviving Crashes

Graceful shutdown handles the orderly stop, but a supervised worker will also be killed at inconvenient times — SIGKILL after a hung drain, a segfault, an OOM kill, a power loss. A crash-safe design assumes the process can die at any point, including after doing the work but before recording that it did. Two failure modes pull in opposite directions: message loss (work vanishes) and double-processing (the same work runs twice). Virtually every production broker resolves this with at-least-once delivery.

At-least-once means a message stays owned by the broker — invisible and unacknowledged — until the consumer explicitly acknowledges it; if the ack does not arrive within a visibility/lease timeout, the broker redelivers. This is the default across SQS standard queues, Celery, Sidekiq, BullMQ/Redis, RabbitMQ, and Kafka consumer groups. It guarantees no loss even across crashes — but explicitly permits duplicates. The classic window: a worker pulls a message, performs the side effect, then crashes before the ack lands; the broker cannot tell "succeeded but didn't ack" from "never processed," so it redelivers and the effect happens twice.

Exactly-once delivery is impractical end-to-end — the acknowledgement that would confirm a single delivery can itself be lost (the Two Generals problem). So the engineering stance is to achieve exactly-once effect by making the consumer idempotent: re-running a message is a safe no-op yielding the same outcome. The core mechanism is the idempotency key — a stable identifier (message ID, payload hash, client request ID). Before the side effect, the worker atomically checks whether that key was already processed; if so, it short-circuits and re-acks.

Beyond dedup, three rules make workers crash-safe: ack after, never before (ack early is at-most-once — a crash between ack and completion silently loses the message); make the effect itself idempotent (prefer upserts, SET status='done', conditional writes); and handle poison messages (a message that always crashes the worker redelivers forever — cap attempts and route exhausted messages to a dead-letter queue).

Crash-and-restart handles a process that dies, but not one that hangs. systemd's WatchdogSec= establishes a heartbeat contract: the app periodically calls sd_notify with WATCHDOG=1, and if the heartbeat stops (a deadlock), systemd restarts the unit per Restart=on-watchdog or on-failure. In clusters the same idea appears as liveness probes (restart on failure) and readiness probes (stop routing traffic until ready).

Assembling the full recovery story: when the worker crashes, Restart=on-failure restarts it; because in-flight messages were never acknowledged, the broker never removed them, so the restarted process picks them back up (nothing lost); the idempotency layer then absorbs the inevitable redelivery (nothing duplicated).

Figure 12.4: Crash recovery — at-least-once redelivery absorbed by an idempotency key.

The two safety nets play distinct roles. Graceful shutdown reduces the number of redeliveries in the common, orderly-stop case — fewer messages are in flight when the process exits. But the idempotency layer is what makes correctness unconditional, even when the process is SIGKILL'd with no chance to drain. Graceful shutdown is the optimization; idempotency is the guarantee. Supervised restart + at-least-once + ack-after-commit + an idempotency key is the four-part recipe that lets a gateway worker crash — or be killed outright — and recover without a single user ever seeing a lost or duplicated response.