ADR-0017: Consume-Side Idempotency for Projectors
Status: Accepted
Date: July 2026
Amended November 2026 (#1932), see Amendment below.
Context
Event delivery to projectors is at-least-once. Two independent paths replay
the same event (same metadata.headers.id):
- the outbox reclaims a row whose broker publish succeeded but whose
mark_publisheddid not commit before a crash, and republishes it; - a broker redelivers a message whose projector UnitOfWork already committed but which was not yet acked.
Protean had no consume-side de-duplication, so a non-idempotent projector
(the classic total_reviews += 1 accumulator) double-applies a redelivered
event and corrupts its read model. A naive "projection == fold(events)"
convergence check misses the failure, because that check folds the same
duplicated stream the projector saw. Only a check that computes its expected
value independently of the stream catches the failure (#1042).
Three facts shape the design:
- A stable key exists:
metadata.headers.idis deterministic (<stream>-<aggregate-id>-<version>.<n>) and identical on every redelivery. - Each projector method already runs in its own UnitOfWork (the
@handlewrapper), so a dedup marker written inside that method's transaction commits or rolls back atomically with the read-model write. This holds only when a transactional provider backs the projection. - Exactly-once cannot span two stores: A projection can be cache-backed
(Redis); a relational marker and a Redis read-model write are two systems
with no shared transaction, the same two-store limit as ADR-0015. The
existing command
IdempotencyStore(Redis, caller-key, records success after the handler commits) is likewise non-atomic and does not fit.
Decision
Provide opt-in, atomic consume-side idempotency for projectors.
- Opt in per projector:
@domain.projector(idempotent=True). - Per-provider marker: A framework
ProcessedMessageaggregate is synthesized per managed provider (mirroring the outbox), with a unique index on(message_id, handler).handleris the fully-qualified handler method (module.Class.method) so distinct projectors/methods dedupe independently. - Atomic check-and-mark: In the
@handlewrapper's UnitOfWork, an idempotent projector first checks the marker for(message_id, handler); if present it skips (the redelivery is a no-op), otherwise it runs the method and writes the marker in the same transaction as the read-model write. The marker repository is resolved for the projection's provider so the two writes share one transaction. - On a relational provider the unique index is the concurrency guarantee.
Two concurrent redeliveries both miss the check and both attempt to write the
marker; the index rejects the second commit, so exactly one applies. The
in-memory provider materializes neither the unique index nor real
transactions, so there it degrades to the sequential
is_processedskip only, which fixes ordinary redelivery but leaves concurrent redelivery unguarded. The in-memory provider is for development, and production idempotency should use a relational projection. - Off by default, zero-cost when unused: The marker table is created only
when a projector opts in (
Domain.has_idempotent_consumers). - Scope: projectors: Event handlers that write to arbitrary aggregates are out of scope for this decision (a projector has a single, known projection provider, which is what makes the marker atomic). Broker subscribers opt in too, under the Amendment.
Consequences
Positive:
- A redelivered event no longer corrupts an
idempotent=Trueprojector's read model on a transactional provider, exactly-once at the projection boundary. - Concurrency-safe via the unique index on a relational provider (see Decision point 4 for the in-memory caveat).
- Opt-in and gated, so domains that don't use it carry no extra cost.
Negative / boundaries (documented, not hidden):
- Not universal: When the projection is cache-backed, or writes span a provider different from the marker's, the marker is not atomic with the read-model write. In that case the option resolves to a no-op and the projector must be written as an idempotent upsert. The guide describes this case, and Protean logs it at debug level.
- Concurrent-redelivery loser surfaces a conflict: The losing writer's commit hits the unique index and raises, so its delivery fails and reports an error. The next redelivery finds the marker and skips, which gives eventual correctness within the at-least-once contract.
- Throughput cost on opt-in projectors: one extra indexed read + write per processed message.
- Marker growth is bounded by retention, not automatic: The marker table
gains one row per
(message_id, handler), andis_processedreads it on every delivery.protean idempotency cleanupprunes markers older than[consume_idempotency.cleanup].retention_hours(default 7 days) in bounded batches; operators run it periodically (e.g. cron). It is not auto-scheduled today; wiring it into an Engine maintenance tick (asDLQMaintenanceTaskdoes for the DLQ) is a possible future enhancement. A marker is only useful while its event can still be redelivered, so pruning past that window is safe. Becausehandlerismodule.Class.method, renaming a projector method orphans its old markers (the cleanup eventually removes them). The rename also opens a brief double-apply window for in-flight redeliveries across that deploy. - This does not make Protean exactly-once across stores; it is exactly-once only within one transactional provider.
Alternatives Considered
Redis seen-store mirroring command idempotency. Rejected: a Redis marker is not atomic with the read-model write, so a crash between the projector commit and the marker write re-opens the double-apply window. It would give false exactly-once confidence.
Documentation only, "projectors must be idempotent." Retained as the guidance for the non-transactional case, but insufficient on its own: some projectors genuinely need dedup and cannot be expressed as pure upserts.
Dedup at the subscription / ack layer. Rejected: the broker ack and the read-model write are in different systems, so the marker would again be non-atomic, the same gap this decision closes by putting the marker in the projection's transaction.
Amendment (November 2026): broker subscribers
Context: A broker subscriber has the same at-least-once delivery as a projector. A subscriber that issues a command or writes an aggregate applies a redelivered message twice. Its only defence was a hand-written marker table.
Amended decision:
- Opt in per subscriber:
@domain.subscriber(idempotent=True). The option is off by default, and a subscriber without it runs as before, with no unit of work added. - The subscriber names the marker's provider:
provider(default"default"). A subscriber has no projection to read a provider from, so it states one. The marker goes to that provider in the same unit of work as the subscriber's writes.domain.init()raises aConfigurationErrorwhen an idempotent subscriber names a provider that is not configured or not managed, because such a provider has no marker table. - The key is the payload's id: the marker is
(metadata.headers.id, subscriber fully-qualified class name). The broker-assigned delivery id is never the key, because a redelivery can carry a new one. - A missing id is an error: an idempotent subscriber that receives a
payload with no non-blank string
metadata.headers.id, or one longer than the marker's 255-character column, logs an error and raisesInsufficientDataErrorwithout calling the subscriber. The Engine treats it as a failed message and callshandle_error; a synchronouspublishraises it. Running without deduplication would ignore the opt-in. - Same atomicity rules: the check, the subscriber call and the marker
write share one unit of work, and the marker is written only after the
subscriber returns. The guarantees and limits of Decision point 4 and the
Consequences apply unchanged, keyed on the subscriber's provider. They cover
writes made through repositories on that provider. A command the subscriber
issues with
domain.processis stored in the event store at once, and an event store that writes outside the unit of work, such as Message DB, keeps it after a rollback, so a redelivery after a raise issues it again. Side effects outside the database repeat the same way.
Consequences: Domain.has_idempotent_consumers also counts subscribers, so
a domain whose only idempotent consumer is a subscriber gets the marker table
and the cleanup. Renaming or moving a subscriber class orphans its markers, as
renaming a projector method does.