ADR-0012: Health Check Architecture
| Field | Value |
|---|---|
| Status | Accepted |
| Date | 2026-04-23 |
| Author | Subhash Bhushan |
Context
Before this ADR, Protean had no built-in health probes. Operators
deploying to Kubernetes wrote their own liveness and readiness checks. A
typical check was a Python script that imported the domain and called
broker.ping(), and operators exposed it through a sidecar or a manual HTTP
server. The pattern in docs/guides/server/production-deployment.md showed
this exact workaround. Three problems made this approach untenable at scale:
- No uniform probe contract: Each team invented its own probe
semantics. Some checked only broker connectivity; some checked every
adapter; some returned
200even when the domain was mid-shutdown. There was no standard for what "ready" meant. - FastAPI vs engine duplication: Teams running both an API tier
(FastAPI) and the async engine (
protean server) wrote two separate probes. The async engine had no HTTP surface at all. - Shutdown invisibility: A pod mid-drain looked identical to a healthy pod from outside. Load balancers and service meshes had no way to know a pod was winding down until TCP connections started failing.
The question was: should Protean ship a standard health-probe contract, and if so, how should it be delivered?
Decision
We will ship a standard health-probe contract with two implementations that share their readiness logic, and we will embed a probe server in the async engine by default.
Probe endpoints (uniform across implementations):
GET /healthz: Liveness. Proves the process is alive and able to respond. Returns200with{"status": "ok", "checks": {"event_loop": "responsive"}}on the async engine;{"application": "running"}on the FastAPI router.GET /livez: Alias for/healthz.GET /readyz: Readiness. Inspects every provider, broker, event store, and cache; on the async engine it also reports asubscriptionsblock (see "Per-subscription readiness" below). Returns200when every check passes,503when any check fails (statusdegraded), and503with{"shutting_down": true}when shutdown is in progress (statusunavailable).
Two implementations, one readiness logic:
- Async engine:
HealthServerinsrc/protean/server/health.py, built directly onasyncio.start_serverwith minimal HTTP/1.1 parsing. Enabled by default, listening on127.0.0.1:8080(loopback; set[server.health] host = "0.0.0.0"to expose off-host), disabled via[server.health] enabled = false. - FastAPI applications:
create_health_router(domain)insrc/protean/integrations/fastapi/health.pyreturns anAPIRouterthat mounts the same three paths on an existing FastAPI app. Pods that already serve HTTP traffic do not need a separate probe server.
Both implementations share readiness logic through protean.utils.health,
which exposes check_providers, check_brokers, check_event_store,
and check_caches. The async-engine probe also reports the subscriptions
block, but the FastAPI router does not. The FastAPI router could report the
block, since collect_subscription_statuses() walks the registry and works
without a running engine. The router leaves it out because an API pod's
readiness should not depend on how far behind a worker's consumers are. Those
are separate deployments with separate failure modes, and a web pod that can
serve requests is ready even when a worker is backed up. Only circuit_state
and total are genuinely engine-only.
Transport choice: asyncio over aiohttp/ASGI.
The engine's probe server uses asyncio.start_server directly, not aiohttp
or an ASGI framework. Health probes are the simplest possible HTTP workload: a
few fixed routes that return small JSON payloads. An HTTP framework for a probe
server would add to the dependency graph of a package that is
deliberately minimal.
Readiness during shutdown:
/readyz returns 503 the moment Engine.shutdown() sets
shutting_down = True, well before subscriptions stop or handlers
drain. This 503 is the "graceful drain" signal to load balancers. A load
balancer pulls the pod out of rotation first, and in-flight work then completes
on the old pod.
Liveness during shutdown:
/healthz keeps returning 200 while the engine drains. The asymmetry is
deliberate: a liveness failure triggers a container restart, which would stop
the drain before it finishes. Only the event loop itself being unresponsive
should fail liveness.
Consequences
Positive:
- Every Protean deployment that runs the async engine gets
Kubernetes-compatible probes without user code. Operators no longer need the
docs/guides/server/production-deployment.mdworkaround. - The async engine is visible to service meshes and load balancers for the first time. Rolling deploys drain cleanly.
- Liveness and readiness have correct asymmetric behaviour during shutdown (readiness fails first, liveness holds until the event loop is compromised).
- FastAPI users get probes with one
app.include_routercall. The shared readiness logic means API probes and engine probes report consistent health. - No new dependencies. The implementation uses only
asyncio(stdlib) and, for FastAPI, the user's existing FastAPI install.
Negative:
- Port
8080is opened by default. Operators running other services on8080(local Jenkins, another app's admin port) must either move Protean via[server.health] port = ...or disable the server. This port conflict is the most common problem adopters will hit. - We wrote the engine probe's HTTP implementation ourselves, without an HTTP library. It is deliberately minimal (no keep-alive, no chunked transfer, no compression) but it is our responsibility to maintain. If we later need HTTPS or HTTP/2 on the probe port, we will have to rewrite the probe server.
- Readiness checks call
provider.ping(),broker.ping(),cache.ping(), and the event store's equivalent on everyGET /readyz. Probes every 5 seconds add ping load to every adapter. For high-throughput systems this is negligible; for rate-limited upstreams (e.g., managed Redis with per-second caps) it is worth sizing. - The
subscriptionsblock queries infrastructure once per subscription (RedisXLEN/XINFO, event-store reads, outbox counts). These queries cost far more than the adapter pings above, and unlike the pings they can hang. So the block is not collected on the probe path at all: a background task refreshes it every two seconds, and the probe reads the result from memory. Answering/readyztherefore stays as cheap as it was when the field was a bare integer. A deployment with very many subscriptions pays the collection cost once per interval regardless of probe frequency. This is deliberately the opposite trade from the adapter pings, which still run inline: those decide the verdict, so they must be current; the subscription block only informs, so it may be stale. - Reporting lag without acting on it is deliberate (see
"Per-subscription readiness" below), so
/readyzalone will not page anyone about a stuck subscription. Alerts still have to come from the metrics. - The block reports what the engine actually runs. A follow-up fix (#1288)
closed three earlier gaps in
collect_subscription_statuses(). Outbox processors foroutbox.external_brokersnow get a row, providers withmanaged = falseno longer get one for a processor the engine never started, and a partitioned (sequential_by) category sums lag across its per-key partition streams instead of reading an empty base stream.totalandlen(details)can still differ, since one partitioned process manager is a single engine subscription but one row per stream category.
Alternatives Considered
Unix domain socket instead of TCP. Rejected. Kubernetes probes
use HTTP over TCP; a Unix socket would force an exec probe, which
is heavier and less portable across orchestrators (Nomad, ECS, etc.).
aiohttp or Starlette. Rejected. Both would have satisfied the
probe requirements, but each carries a non-trivial dependency tree.
Protean's server surface is deliberately minimal to keep the base
install small. The probe server's footprint in src/protean/server/health.py
is under 250 lines.
TCP-only probe (no HTTP). Rejected. TCP SYN-ACK is not a strong
signal of readiness. The engine's event loop could be unresponsive while the
socket layer still accepts connections. HTTP with a response body is the
standard Kubernetes probe contract and signals something meaningful.
Mounting probes on an existing API app. Considered. create_health_router
does exactly this for FastAPI users. The async engine serves no other HTTP
endpoints, so a standalone probe server was simpler than making FastAPI a
required runtime dependency.
One combined probe with query-parameter liveness vs readiness.
Rejected. Kubernetes expects distinct paths with distinct behaviours
(livenessProbe restarts the container; readinessProbe pulls it
from rotation). If one endpoint serves both probes, operators can miss the
difference between them and configure the probes wrongly.
Per-subscription readiness. Originally deferred in favour of a bare
count; adopted in 0.17 (#832) as a reported-but-not-enforced block.
/readyz now carries per-subscription lag, pending count, DLQ depth, status,
and circuit-breaker state. These values come from the same
collect_subscription_statuses() that feeds the
protean.subscription.consumer_lag and protean.subscription.pending_messages
gauges and the protean subscriptions status CLI. No new metric names
were introduced.
The block is informational: it never changes the probe's verdict. We
considered letting lag switch readiness to 503, and we rejected it. A backlog
is a normal, self-correcting condition (a burst of traffic, a replay, a slow
downstream), and it is precisely when a consumer is behind that you least want
Kubernetes to pull it out of rotation and stop it draining. An open circuit
breaker is treated the same way: one handler is paused, and the engine is still
healthy. Readiness means "this pod can process messages", which stays true
under lag. Operators should alert on lag from the metrics. The block lets an
operator debugging a pod see which subscription is behind from the probe
response alone.
References
docs/guides/server/hardening.md: Operational guidance for probe wiring and KubernetesterminationGracePeriodSeconds.docs/reference/server/hardening.md: Probe response bodies, status codes, and configuration reference.- ADR-0011, Engine shutdown and resource lifecycle contract: The shutdown
sequence during which
/readyzreturns503.