Skip to content

Migrating to 0.18

From 0.17 to 0.18

Protean 0.18 right-sizes the runtime dependency surface. A plain pip install protean no longer forces a web server, an ASGI stack, an interactive REPL, and a project scaffolder onto every install. Those concerns now live behind install extras, matching how the database and broker adapters have always worked. See ADR-0029 for the boundary and the rationale.

This is a Tier-1 breaking change, but a small one: nothing in your code changes, and the fix when you hit it is a one-word extra. If a feature you use now needs an extra, the CLI or the import tells you exactly which one.

On this page:

What moved behind extras

Five packages that used to be hard runtime dependencies are now install extras:

Package(s) Extra Feature it gates
fastapi, uvicorn, jinja2 protean[server] protean observatory, protean.integrations.fastapi
ipython protean[shell] protean shell
copier protean[scaffold] protean new

Two convenience bundles are also available:

  • protean[cli]: The full interactive CLI experience (shell + scaffold + mcp). The mcp extra is new in 0.18. It installs mcp, the package behind protean mcp, the MCP (Model Context Protocol) server that coding agents connect to.
  • protean[all]: server + cli. It installs every package the commands and integrations above need.

import protean and defining a domain, persisting through the memory adapter, and running protean server (the async engine) need none of these: they were already lazy, and they stay in the lean core.

The one-line fix

If your project relied on the old fat install and you would rather not think about which extras you need, install all:

pip install "protean[all]"

That brings back every feature the 0.17 core install supported, and adds mcp. It does not bring back werkzeug, cffi or greenlet, which also left the core install (see cffi and greenlet left the core install and werkzeug left the core install). From there you can trim to the specific extras you actually use whenever you like.

What each feature needs now

You only need to act if you use one of these features and you install a trimmed set of dependencies (for example in a production image that installs protean without [all]):

  • Running the Observatory (protean observatory) or using the FastAPI integration (from protean.integrations.fastapi import ...):
pip install "protean[server]"
  • Opening the domain shell (protean shell):
pip install "protean[shell]"
  • Scaffolding a project (protean new):
pip install "protean[scaffold]"

If you run one of these without its extra, the command fails with a clear message naming the extra to install, not a raw ModuleNotFoundError:

Error: 'protean observatory' requires the 'fastapi' package. Install it with 'pip install "protean[server]"'.

bleach stays in the core install. String and Text fields no longer sanitize by default (see String and Text no longer sanitize by default), but bleach still backs the opt-in sanitize=True path.

cffi and greenlet left the core install

0.17 installed cffi and greenlet as hard dependencies, but nothing in Protean imports either package. They only ever arrived as transitive dependencies: SQLAlchemy 2.0 (the postgresql, postgresql-psycopg2, sqlite, mssql and mysql extras) requires greenlet on common platforms, and cryptography requires cffi (sendgrid pulls it directly, and mysql through pymysql[rsa]).

In 0.18 both are out of the core install. Their version floors moved onto the extras that pull them (greenlet>=3.2.3,<4 on the SQLAlchemy extras, cffi>=2.0.0 on sendgrid and mysql), so installing one of those extras still resolves a version with wheels for the newest supported Python. SQLAlchemy 2.1 no longer requires greenlet. The SQL extras keep it so that an install at the SQLAlchemy 2.0 floor still works.

You only need to act if your own code imports cffi or greenlet and relied on Protean to provide it. Declare the package in your own project instead.

PostgreSQL defaults to psycopg 3

0.17 capped SQLAlchemy at <2.1, and protean[postgresql] installed psycopg2-binary. In 0.18:

  • The SQL extras accept SQLAlchemy >=2.0.36,<2.2, so an application that needs SQLAlchemy 2.1 can install them. The 2.0.36 floor is unchanged.
  • protean[postgresql] installs psycopg 3 (psycopg[binary]).
  • The new protean[postgresql-psycopg2] extra installs psycopg2-binary.

The driver follows database_uri:

database_uri Driver
postgresql+psycopg://... psycopg 3
postgresql+psycopg2://... psycopg2
postgresql://... psycopg 3 if it is installed, otherwise psycopg2

Protean resolves the plain postgresql:// form itself. SQLAlchemy 2.0 reads it as psycopg2 and SQLAlchemy 2.1 as psycopg 3, so leaving it to SQLAlchemy would make the driver depend on which SQLAlchemy is installed. When the driver a URL needs cannot be imported, domain.init() raises a ConfigurationError that names the extra to install.

Who is affected

  • You install protean[postgresql] and use a plain postgresql:// URL. A fresh install (a new Docker image, a CI run) now gets psycopg 3, and your application runs on it. Protean's own queries behave the same on both drivers. Raw SQL you run yourself through the session can see psycopg 3's differences, for example in how some Python types are adapted. See the psycopg 3 differences guide.
  • You want to stay on psycopg2. Install protean[postgresql-psycopg2] and write postgresql+psycopg2:// in database_uri. The explicit URL keeps psycopg2 even when psycopg 3 is also installed.
  • You upgrade in place with both drivers installed. A plain URL moves to psycopg 3, because psycopg 3 comes first. Pin the driver in the URL if you want to control when that happens.

Why it was not put behind a flag

The postgresql-psycopg2 extra and the postgresql+psycopg2:// URL already select the old behaviour, so a flag would be a third way to say the same thing.

werkzeug left the core install

0.17 used Werkzeug's LocalProxy/LocalStack to back current_domain, current_uow, and g. In 0.18 these context locals are backed by stdlib contextvars, and werkzeug is no longer a core dependency.

For nearly all code this is transparent: the same push/pop nesting, the same proxy behavior, and the same async support (in fact, contextvars follows asyncio tasks and await boundaries natively). The public surface of current_domain, current_uow, and g is unchanged.

You only need to act if your own code depends on Werkzeug-specific details of the proxy or stack, such as inspecting the underlying LocalStack directly. Internal helpers like _domain_context_stack and _uow_context_stack remain private and have changed their implementation type.

check and verify machine output changed shape and exit codes

protean check --format json and protean verify --json now emit one shared, versioned result envelope and adopt one exit-code convention (that other commands can converge onto later). check's JSON output has shipped before (0.15.0; the diagnostic schema and [lint].level gating landed in 0.17.0) and verify is new in this cycle, but Protean is still pre-1.0 and neither has a stability guarantee yet, so this is a direct correction rather than a deprecation-managed break. You only need to act if you already parse check --format json or verify --json output, or branch on their exit codes.

The JSON shape. check --format json was the raw Domain.check() document (domain, status, errors, diagnostics, counts at the top level). It is now the envelope:

{
  "version": "0.1.0",
  "status": "fail",
  "data": { "domain": "...", "status": "warn", "errors": [], "counts": {} },
  "diagnostics": [ "..." ]
}

The check report moved under data; diagnostics is at the top level. A consumer reading counts now reads data.counts; one reading diagnostics reads the top-level diagnostics (unchanged path). verify --json gained the same frame: its old {verdict, stages} is now under data.

The sarif and github-annotations formats are unchanged: they are external standard schemas and are not wrapped.

--level no longer filters the JSON envelope. --level is now a display filter for the human rich/--quiet views only; the JSON, SARIF, and github-annotations machine formats always carry the full, unfiltered set of findings (a machine consumer can filter for itself). diagnostics can still be empty on a status="fail" envelope. A validation error, such as two handlers for one command, skips the diagnostics; check data.errors for those. A malformed [lint] config or an unresolved reference stops the run before any check. It gives status="error", the message under data.error, and exit 2. If you relied on check --format json --level error to pre-filter diagnostics, filter the output with jq instead.

The exit codes.

check:

Case Old New
Clean 0 0
Validator errors 1 1
Gating warning/info at the floor 2 1
Bad option, unloadable domain, malformed [lint] config 1 2

The 1-error-vs-2-warning split collapsed into a single findings-failure code (1); the severity now lives in the envelope. Load and config errors became usage errors (2).

verify shifted its codes up so 2 is reserved for usage:

Case Old New
All green 0 0
Usage (bad/missing --domain, --path not a directory) 1 2
Init failed 2 3
Check failed 3 4
Tests failed 4 5

Clean stdout. Under machine output, stdout now carries only the envelope, exactly one JSON object. Logs and error lines go to stderr, so protean check --format json | jq and protean verify --json | jq stay parseable.

events catalog, subscriptions status, and projection status JSON moved onto the envelope

protean events catalog --json, protean subscriptions status --json, and protean projection status --json each printed a bare JSON array. They now emit the same result envelope that check and verify use, so one shape parses every command. The array moves under data, keyed by the command:

Command Old top level New location
events catalog --json array of events data.events
subscriptions status --json array of subscriptions data.subscriptions
projection status --json array of projections data.projections

A run that read the array at the top level now reads it under its data key:

{
  "version": "0.1.0",
  "status": "pass",
  "data": { "events": [] },
  "diagnostics": []
}

All three are pre-1.0 with no stability guarantee, so this is a direct correction with no deprecation window. You only need to act if you already parse one of these --json outputs: read data.events, data.subscriptions, or data.projections in place of the top-level array.

All three also adopt the shared exit codes and clean stdout on their error paths under --json. Under --json, any usage or load failure (an unloadable --domain, a missing/unreadable --ir file, or, for events catalog, neither --domain nor --ir or both at once) is now the error envelope (status="error", the message under data.error) on stdout and exits 2, so | jq stays parseable.

Two things change without --json as well. The events catalog usage errors (neither --domain nor --ir, or both at once) now print to stderr and exit 2, where they exited 1 before. A domain or --ir load failure without --json is unchanged: it prints a human message and exits 1.

The memory cache no longer raises on an absent key

Behavioural break, memory cache only. remove(projection), remove_by_key(key) and set_ttl(key, ttl) raised KeyError when the key was not in the cache. They now do nothing, which is what the Redis cache has always done: Redis' DEL and PEXPIRE are no-ops on a missing key. The port documents the silent behaviour as the contract, so the two adapters answer alike.

An expired entry counts as absent. A key whose TTL has run out is treated the same as one that was never written, on both adapters.

Who is affected

Anyone who used the KeyError as a presence check on a memory-backed cache. That code no longer takes its except branch, and nothing tells you: the call succeeds quietly. It is the only shape that breaks.

# fragment
# Before: worked on the memory cache, never fired on Redis
try:
    cache.remove_by_key(key)
except KeyError:
    log.info("nothing cached for %s", key)

After:

def remove_cached(cache, key):
    # Ask before removing, and it works on either adapter
    if cache.get(key) is None:
        log.info("nothing cached for %s", key)
    cache.remove_by_key(key)

Nothing inside Protean relied on the KeyError, so this only affects your own code.

What did not change

set_ttl still rejects an invalid TTL when the key is absent. A TTL that is not a positive, finite number of seconds raises ConfigurationError whether or not there is a key to apply it to, so a malformed TTL cannot hide behind a missing key.

get_ttl now answers alike on both adapters too; see get_ttl returns None or math.inf instead of a raw sentinel.

Why it was not put behind a flag

The behaviour was a divergence between two adapters of the same port, not an intentional contract, so no program that runs on both could have depended on it. A flag preserving the KeyError would have kept the divergence alive on the adapter that is documented for development and testing, which is the one place the inconsistency is least worth paying for.

get_ttl returns None or math.inf instead of a raw sentinel

Behavioural break, both cache adapters. get_ttl(key) used to answer differently depending on the adapter, and neither answer was easy to use. The memory cache raised KeyError for a missing key and could never report a never-expiring key. Redis returned its raw PTTL sentinels: -2 for a missing key, -1 for a key with no expiry. A caller that wanted to work on both had to handle an exception and two magic negatives.

Both adapters now answer the same three cases:

  • None when there is no such key.
  • math.inf when the key exists and never expires.
  • the seconds remaining otherwise (unchanged).

An expired entry counts as absent, so get_ttl returns None for it, the same way get does. math.inf only ever comes from Redis: the memory cache writes every entry with a concrete TTL, so it has no never-expiring keys to report.

Who is affected

Anyone who read get_ttl's old return value directly.

# fragment
# Before: two shapes, one per adapter
try:
    ttl = cache.get_ttl(key)          # memory: raises KeyError if absent
except KeyError:
    ttl = None
if ttl is not None and ttl < 0:       # redis: -2 absent, -1 no expiry
    ttl = None

After:

import math

def describe_ttl(cache, key):
    # One shape on both adapters
    ttl = cache.get_ttl(key)
    if ttl is None:
        return "no such key"
    elif ttl == math.inf:
        return "never expires"
    else:
        return f"{ttl:.0f} seconds remaining"

A ttl < 0 check on the old Redis return no longer fires, because the negative sentinels are gone. A ttl < math.inf check is the way to ask "does this key expire".

Why it was not put behind a flag

Like the absent-key change above, this was a divergence between two adapters of the same port, not an intentional contract. BaseCache is a Provisional API, so its signatures can still change without a deprecation period. Nothing inside Protean read get_ttl, so only your own callers are affected.

Cache key patterns are a glob on every adapter

Behavioural break, memory cache. _get_all, count, and remove_by_key_pattern take a key_pattern. The memory cache used to compile it as a Python regular expression, while Redis passed it to SCAN ... MATCH as a glob. The same string selected different keys depending on which adapter you ran, and a pattern that leaned on regex behaviour matched nothing once you switched to Redis.

The memory cache now matches with fnmatchcase, the same glob language Redis uses. Like Redis, it is case-sensitive on every platform. * matches any run of characters, ? matches one, [...] is a character class, and other characters, including ., are literal. Adapters agree on *, ?, and literal characters; bracket negation and escaping can differ, so keep patterns to the name:::* shape.

Who is affected

Anyone who passed a regex-flavoured key_pattern to the memory cache. Cache keys are name:::identifier, so most patterns are already a name and a *, which is a valid glob and needs no change. A pattern that used regex syntax needs rewriting:

# fragment
# Before, on the memory cache: a regex (and the method was public then)
cache.get_all("user_profile:::.*")

After:

def all_user_profiles(cache):
    # A glob on every adapter, through the private `_get_all`
    return cache._get_all("user_profile:::*")

A literal . in a pattern used to match any character on the memory cache and now matches only a .. BaseCache is a Provisional API, so this lands in a minor release with no deprecation period.

Cache get_all is now the private, unpaginated _get_all

Breaking, both cache adapters. The public cache.get_all(key_pattern, last_position, size) is gone. It is now cache._get_all(key_pattern): private (the leading underscore), and it returns every matching entry in key order with no pagination.

A cache is for point reads by key. Enumerating a match set, and paging through it, is not what a cache is for, and offering offset pagination over it invited use it cannot support: on a store with no native key order (Redis) every call scans the whole keyspace, and paging a large set is quadratic. So the method is marked private and scoped to what it is actually good for, a convenience for tests and small, bounded stores.

It returns at most 1000 entries (GET_ALL_MAX). Past that it truncates to the first 1000 in key order and logs a warning, so a store that outgrew the cache surfaces as a warning rather than a silent partial read.

Who is affected

Anyone calling cache.get_all(...). The method no longer exists under that name, and it no longer takes last_position or size.

# fragment
# Before
for start in range(0, total, 50):
    for entry in cache.get_all(pattern, last_position=start, size=50):
        process(entry)

After:

def process_all(cache, pattern, process):
    # A bounded read of the whole match set (small stores, tests)
    for entry in cache._get_all(pattern):
        process(entry)

If you need to page a large projection set, that belongs in the projection's repository, not the cache. Query repository_for(Projection).query, which has real indexes and native pagination. The cache stays for hot point lookups by key.

Why it was renamed rather than deprecated

BaseCache is a Provisional API, so it can change without a deprecation period, like the other cache changes above. _get_all is private for the same reason it is unpaginated: it is not a surface to build on. count and remove_by_key_pattern stay public and take the same key_pattern glob; note that count also scans the whole keyspace on Redis.

flush_all() on a Redis cache clears only its database

Behavioural break, Redis cache. RedisCache.flush_all() used to send FLUSHALL, which empties every database on the Redis server. It now sends FLUSHDB, which empties only the database in the cache's URI (the /4 in redis://localhost:6379/4). The Redis and Redis PubSub brokers' test reset, _data_reset(), changed the same way.

Keys in that one database still all go: RedisCache adds no cache-specific namespace to its keys, so flush_all() removes anything else stored in the cache's database as well. Give the cache a database of its own.

Who is affected

Anyone who relied on flush_all() on a Redis cache, or on a Redis broker's reset between tests, to also clear other databases on the same server. Clear those databases yourself now, for example with FLUSHDB on a client connected to each one. BaseCache is a Provisional API, so this lands in a minor release with no deprecation period.

The IR schema is now v0.2.0

The materialized IR carries a new method_edges key on the elements that own methods, so the schema version moved from 0.1.0 to 0.2.0. Protean 0.18 then moved it again, to 0.3.0 (see The IR schema is now v0.3.0). A baseline written by Protean 0.17 is at 0.1.0, so it picks up the changes in both sections when you regenerate it. Documents and schemas validate both ways: a 0.1.0 document still passes the 0.2.0 schema (method_edges is optional), and a 0.2.0 document still passes the 0.1.0 schema. The v0.1.0 schema stays published at https://docs.proteanhq.com/assets/ir/v0.1.0/schema.json. The $schema URL that Protean 0.18 writes into the IR, https://protean.dev/ir/v{X}/schema.json, does not resolve. 0.19 writes the docs site URL instead (see The IR's $schema URL moved).

What does change is the checksum. Adding a key changes the canonical bytes, so a baseline built under 0.1.0 cannot be compared against a live IR built by Protean 0.18.

Who is affected

Anyone with a committed .protean/ir.json. protean ir check now reports a version mismatch and exits 3 instead of comparing checksums. The protean-check-staleness pre-commit hook prints the same message and fails (it exits 1 for every failure), so a CI job that gates on the hook goes red until the baseline is regenerated.

protean ir show --domain <module> --canonical > .protean/ir.json

The pre-commit hook rewrites it for you if you run it with --fix.

Your domain code does not change. For this section, the regenerated document differs only in $schema, ir_version, the checksum, the new method_edges entries, and the sanitize key on String and Text fields. That key now appears only for a field that passes an explicit sanitize=True, so it drops off every field that left sanitize unset (see String and Text no longer sanitize by default). A domain with a Decimal field also picks up the 0.3.0 changes to that field, and the commit that adds the regenerated baseline needs the step in The IR schema is now v0.3.0.

What a cross-version diff looks like

protean ir diff against a 0.1.0 baseline reports method_edges as added on every element that has one, and sanitize as removed on every String/Text field declared without an explicit sanitize=. Compatibility classification does not read method_edges or sanitize, so protean ir diff and the protean-check-compat hook classify neither as breaking. Regenerate the baseline and diff two documents built by the same Protean version if you want the noise gone.

The IR schema is now v0.3.0

The IR used to record a Decimal field as "type": "String" and drop its precision and scale. It now records "type": "Decimal", plus precision and scale when the field sets them:

"total": {"kind": "standard", "type": "Decimal", "precision": 19, "scale": 4}

The type list in the v0.2.0 schema is closed, so the new type needed a new schema version: ir_version moved from 0.2.0 to 0.3.0. Protean 0.17 wrote 0.1.0, so a baseline from 0.17 also picks up the 0.2.0 changes in The IR schema is now v0.2.0. A 0.2.0 document still passes the 0.3.0 schema. A 0.3.0 document that holds a Decimal field fails the 0.2.0 schema. The v0.2.0 schema stays published.

Who is affected

Anyone with a committed .protean/ir.json. protean ir check reports a version mismatch and exits 3, and the protean-check-staleness hook fails, until you regenerate the baseline:

protean ir show --domain <module> --canonical > .protean/ir.json

The 0.3.0 IR also records what a save writes. A Date or DateTime field declared with auto_now=True or auto_now_add=True carries that flag, and a domain that registers enrichers gets a top-level enrichers section that lists them by qualified name. protean ir diff reports both as changes, and neither is breaking. If your domain has no Decimal field, no Custom(decimal.Decimal) field, no auto_now or auto_now_add field, no enricher and no Integer or Float field with a bound written as a string, the regenerated baseline differs only in $schema and ir_version. For a string bound, see How the IR writes min_value and max_value. A domain that raises one of the three diagnostics in Diagnostic and CLI messages no longer contain em-dashes also sees that diagnostic's new text.

If it has one, commit the regenerated baseline. Until you do, protean ir diff compares the old String entry with the new Decimal entry and reports a breaking field_type_changed for each such field. Your data and your payloads have not changed. The old baseline recorded each of these fields as String, and the new one records it as Decimal.

The protean-check-compat hook compares the live IR with the baseline in the HEAD commit, not with the staged file. So it also fails on the commit that adds the regenerated baseline. Skip the hook for that one commit:

SKIP=protean-check-compat git commit -m "Regenerate the IR baseline"

What changes in the generated contracts

The contract generators read the IR type, so the output for a Decimal field changes:

Generator Before After
JSON Schema {"type": "string"}, plus minimum and maximum when the field set min_value or max_value {"type": "string", "pattern": ...}, plus x-precision and x-scale when set. minimum and maximum are left off, since they only apply to numbers.
Avro "string" {"type": "bytes", "logicalType": "decimal", "precision": p, "scale": s} when precision is set (scale defaults to 0); "string" when it is not, or when precision is 0 or smaller than scale
Protobuf string string (no change)

A Decimal value is still written as a string in a JSON payload, so the JSON Schema output matches what Protean sends, and Protobuf keeps string. In Avro, a Decimal field with a precision moves from string to bytes with the decimal logical type, so a consumer that registered the old schema needs the new one.

How the IR writes min_value and max_value

A decimal.Decimal bound used to stop the IR build with a TypeError. It is now written in a form JSON can hold:

Field Declared bound Written as
Decimal, or Custom(decimal.Decimal) decimal.Decimal("0.010") "0.010", a string that keeps every digit
Integer decimal.Decimal("10"), or the string "10" 10
Float decimal.Decimal("0.5") 0.5
any field an int or a float the same number, as before

Integer(min_value="5") used to write "5", which failed the schema. It now writes 5. The 0.3.0 schema accepts a number or a string for min_value and max_value, but accepts a string only when the field type is Decimal. If a field in your domain declares a bound in one of these forms, regenerate the baseline as shown above.

New in ir diff: field_precision_widened

On a Decimal field that stays Decimal, ir diff now compares precision and scale. Raising precision, or removing it, with scale unchanged is reported as a safe field_precision_widened. Lowering or adding precision, or any change to scale, is a breaking field_type_changed. See Decimal precision and scale.

diff_ir() checks every field of a published event

diff_ir() lists the breaking changes to published events in contracts.breaking_changes, and sets summary.has_breaking_changes from that list. It used to compare a published event's fields only when a field was removed or renamed. A field that kept its name and changed its type, or changed its Decimal precision or scale in a breaking way, was missed. So was a required field added without a default. The compatibility report from classify_changes() flagged all three, so the two disagreed.

The published-contract check now applies the same field rules as the compatibility report. A published event where amount goes from Decimal(precision=19, scale=2) to scale=4 now gives:

{
  "type": "contract_field_type_changed",
  "fqn": "shop.Placed",
  "field": "amount",
  "left": "Decimal",
  "right": "Decimal",
  "message": "Field 'amount' Decimal shape changed (scale from 2 to 4) in published event 'Shop.Placed.v1'"
}

and summary.has_breaking_changes is true. A field that keeps its name has no renamed_to key. A required field added without a default gives a new contract_required_field_added entry with fqn, field and message. Every other contracts entry keeps its shape and message. See Published event contracts.

Who is affected

  • Code that calls diff_ir() directly and reads contracts.breaking_changes or summary.has_breaking_changes. A diff that reported no break can now report one.
  • Readers of protean ir diff --format json that look at contracts.breaking_changes or summary.has_breaking_changes.

The exit codes of protean ir diff and the protean-check-compat hook do not change, with one exception. Both already exited 1 on these changes, because the compatibility report flagged them. An event version bump that an upcaster covers is not an exception: the report marks the field change safe, but the bump changes the event's type string, so contract_type_changed already made the run breaking. The exception is a published event listed in exclude in .protean/config.toml. exclude filters the compatibility report, and it has never filtered the contract breaks. Take a field that keeps its name on an excluded published event, and change its type or its Decimal shape in a breaking way, or add a required field without a default. That change used to exit 2 from protean ir diff and pass the hook. Under strictness = "strict" it now exits 1 and fails the hook, as a removed field on the same event already did. Under strictness = "warn", protean ir diff now exits 0 instead of 2, and the hook still passes.

Three smaller changes come with this:

  • In the JSON compatibility block, field_name is now filled for field_type_changed and required_field_added. For a rename it holds the old name. It used to be null for these two change types.
  • The text output of protean ir diff prints a field break once. It used to print a break twice when both lists reported it with different wording, for example a renamed Decimal field whose scale changed.
  • The protean-check-compat hook prints the contract breaks as well as the report's. A break on an excluded published event used to fail the hook with no break named.

Why it was not put behind a flag

diff_ir() gave the wrong answer: it reported no breaking change on a published event that had one. protean ir diff and the hook already failed on these changes, so a flag would only preserve the wrong answer for direct diff_ir() callers.

Diagnostic and CLI messages no longer contain em-dashes

Each em-dash in a message Protean prints, raises or logs is now a colon, a comma or a full stop. The meaning of each message is the same. Three diagnostics change the text the IR records:

Code What changed
AGGREGATE_WITHOUT_COMMAND_HANDLER message and rationale
PROJECTOR_HANDLES_ORPHANED_EVENT rationale
QUERY_HANDLER_WITHOUT_QUERY rationale

The rest are CLI output (protean dx, protean ir, the git hooks and the table titles of subscriptions, projection and eventstore dlq), some exception messages, and some log lines.

Who is affected

Anyone with a committed .protean/ir.json whose domain raises one of the three codes. The IR checksum covers each diagnostic's text, so protean ir check reports the baseline as stale even though the domain did not change. A baseline from 0.17 needs regenerating anyway (see The IR schema is now v0.3.0), and the same command picks up the new text:

protean ir show --domain <module> --canonical > .protean/ir.json

protean ir diff matches diagnostics by code and element and does not compare their text, so it reports no change for these three.

Anyone whose script or test matches one of these messages by its exact text needs the new wording. A match on the diagnostic code, the exit code or the JSON shape is unaffected.

protean ir rejects an unknown format value

protean ir show, protean ir diff, and protean ir check check --format before they load anything. An unknown value prints the accepted ones and aborts:

$ protean ir diff --left baseline.json --right current.json --format=summary
Error: invalid --format: 'summary'. Choose from: text, json, event-model
Aborted.

Each command takes its own set:

Command Accepted --format values Default
protean ir show json, summary json
protean ir diff text, json, event-model text
protean ir check text, json text

Who is affected

Anyone whose script passes a value outside that command's set. It used to fall through to the command's default: protean ir diff --format=summary printed the text diff, and protean ir show --format=text printed the full JSON. That was silent, so a typo in a CI job or a value borrowed from a sibling command read as success for as long as nobody compared the output to the flag.

The failure exits 1, the same code as the other argument checks that protean ir show and protean ir diff make themselves. Other usage errors exit 2: an option the command does not have, and, under protean ir check, an invalid .protean/config.toml. On protean ir diff, 1 is also the breaking-changes code, so a CI gate that branches on the exit code sees a usage error as a break until the flag is fixed; the invalid --format line on stdout is what tells them apart.

The fix is to pass a value from the table. Nothing else about the commands changed, and a script that already passed a valid value is unaffected.

protean new --pretend is now --dry-run, and it writes nothing

Flag rename and behavioural break, the protean new dry run only. The flag is --dry-run now, and --pretend (with its -p short form) is gone. Passing --pretend is a usage error. --pretend was copier's name for the option; the rest of the CLI already says --dry-run, and so does the create_project core it calls (dry_run=).

The dry run also behaves differently. It used to render through copier's own pretend flag, and everything around that render ran for real. So a dry run reached the target directory before it printed anything: with --force it cleared the target, and without --force on a non-empty target it raised FileExistsError. Neither is something a preview should do.

--dry-run renders into a system temp directory and prints the project-relative path of each file it would create, one per line, sorted. The target directory is left alone whether or not it already has files in it, and --force no longer clears anything under a dry run. The list it prints is the same set of files an apply produces from the same inputs.

The preview is also complete now. .protean/project.json, AGENTS.md, the CLAUDE.md bridge, and .protean/dx-state.json are written after the copier render, so the old dry run never listed them even though an apply creates all four.

Who is affected

Anyone passing --pretend or -p to protean new. Pass --dry-run instead.

Anyone parsing the output of the dry run. It used to be copier's coloured create <path> log, in copier's render order; it is now a plain list of relative paths, sorted.

# Before: copier's own log, no manifest, no AGENTS.md
$ protean new myproject --pretend
Copying from template version None
    create  .gitignore
    create  docker-compose.override.yml
    ...

# After: one project-relative path per line, sorted, and the full set
$ protean new myproject --dry-run
.copier-answers.yml
.dockerignore
.env.example
...
.protean/dx-state.json
.protean/project.json
AGENTS.md
CLAUDE.md
...
src/myproject/domain.py
...

Anyone who used --pretend --force to clear a directory is also affected. That was a side effect of how the dry run was wired, not a documented feature, but it did delete files. Use protean new <name> --force without --dry-run if clearing the target is what you want.

Why it was not put behind a flag

A flag defaulting to the old behaviour would default to a preview that deletes your files. There is no reading of a dry run under which that is the contract a user asked for, so there is nothing to preserve.

protean new . and protean new .. are rejected

Input break. A project name has to be a single directory segment now: a non-empty string, not . or .., and with no <>:"/\|?* or whitespace. The first two used to pass validation. Passing one raises ValueError before any file is touched.

Who is affected

Anyone calling protean new . expecting it to scaffold in place. It never did that. project_name feeds the template, so . produced a project with no package directory (src/domain.py instead of src/<name>/domain.py) and name = "." in pyproject.toml, which no installer accepts. The generated tree did not build.

# Before: rendered a tree that could not be installed
mkdir myproject && cd myproject && protean new .

# After: name the project, then work inside it
protean new myproject && cd myproject

The rule also closes a hole in --force. The clear removes the contents of <output_folder>/<project_name>, so a name that escaped its segment (. or ..) pointed the clear at the output folder or its parent. Requiring a single segment means the clear can only ever reach the project directory.

protean new refuses a target symlinked out of the output directory

Behavioural break, symlinked targets only. protean new <name> writes to <output-dir>/<name>. If something already at that path was a symlink, every step followed it: --force cleared the directory the link pointed at, and the render wrote its files there. A link pointing outside the output directory meant files outside the output directory were deleted and written.

The command now resolves the target first. If it does not sit directly inside the resolved output directory, the command stops with a ValueError before it clears or writes anything. protean.scaffold.create_project raises the same error, on apply and under dry_run.

Who is affected

Anyone who symlinked a project directory to another disk or another checkout and then re-ran protean new over it. Render into the real location and symlink to it afterwards, or point --output-dir at the directory that actually holds the project:

# Before: followed the link and wrote (or cleared) at /mnt/work/myproject
ln -s /mnt/work/myproject myproject
protean new myproject --force

# After: name the real location
protean new myproject --output-dir /mnt/work --force

A symlink is still fine when it resolves to a path directly inside the output directory, say <output-dir>/other-name. One that resolves deeper, like <output-dir>/nested/other-name, is refused too: the check is that the target's resolved parent is the resolved output directory.

DAO.update() and QuerySet.update() are deprecated

A deprecation, not a break. Both methods still work in 0.18. They now emit a RemovedInProtean020Warning (a DeprecationWarning subclass) and are removed in 0.20.0. Nothing else about their behaviour changes.

DAO.update() and QuerySet.update() take a field patch and write it straight to the store. That skips the aggregate: the change does not go through a behaviour method, so the rule that owns the change and the events it should raise never run. The replacement is to load the aggregate, call a method on it that applies the change, and persist it with repository.add().

Single aggregate

# fragment
# Before: patch-and-persist through the DAO
repo = domain.repository_for(Order)
dao = repo._dao
order = dao.get(order_id)
dao.update(order, status="shipped")

After:

def ship_order(order_id):
    # Load, invoke a behaviour method, add
    repo = domain.repository_for(Order)
    order = repo.get(order_id)
    order.ship()  # the aggregate applies the change and raises OrderShipped
    repo.add(order)

A child entity has no repository of its own. Load the aggregate root, change the child through the root, and repo.add() the root.

Many aggregates

QuerySet.update() patched every matched row in one call, but it was never one transaction: it looped and persisted each row on its own. Load the matched aggregates, invoke the behaviour method on each, and repo.add() each one.

# fragment
# Before: bulk patch through the query set
repo = domain.repository_for(Order)
repo._dao.query.filter(status="pending").update(status="cancelled")

After:

def cancel_pending_orders():
    # Load each match and invoke the behaviour method on it
    repo = domain.repository_for(Order)
    for order in repo.query.filter(status="pending").all():
        order.cancel()
        repo.add(order)

Each add() commits in its own transaction, which is the same granularity QuerySet.update() gave you. Do not wrap the loop in one UnitOfWork to make the batch atomic: every Order is its own consistency boundary, and one transaction, one aggregate is the rule. If the whole sweep has to land or fail together, it is a backfill, not domain behaviour.

Large backfills

A one-off migration that rewrites a column across a whole table is an infrastructure job, not domain behaviour, the same way a hard delete is. Do it at the adapter or database level (a SQL UPDATE, an adapter-specific bulk write) rather than reaching for the deprecated domain-level path.

Supervisor(debug=...) and Reloader(debug=...) are deprecated

A deprecation, not a break. debug=True still works in 0.18. It now emits a RemovedInProtean020Warning (a DeprecationWarning subclass), and the argument is removed in 0.20.0.

Both classes now take log_level, log_format and log_config, which match the CLI's --log-level, --log-format and --log-config flags. debug=True is the same as log_level="DEBUG". If you pass both, log_level wins.

debug=True no longer builds each worker's Engine with debug=True. That setting forced the protean.server.engine logger to DEBUG. Now a [logging.per_logger] level for that logger applies to it like any other.

# fragment
# Before
supervisor = Supervisor("my_package.domain", num_workers=4, debug=True)

After:

from protean.server.supervisor import Supervisor

supervisor = Supervisor("my_package.domain", num_workers=4, log_level="DEBUG")

Only direct Python callers are affected. The protean server command does not pass debug.

Deprecations that named 1.0.0 now name 0.19.0 or 0.20.0

A deprecation schedule change, not a break. Nothing stops working in 0.18. Ten deprecations shipped in 0.17 saying they would be removed in 1.0.0. No 1.0 release is planned, so that gave you no date to plan against. Each one now names a planned release.

These ten are removed in 0.19.0 and emit RemovedInProtean019Warning:

Deprecated Use instead
is_event_sourced= option alias event_sourced=
published option on a command Remove it; it has no effect
@domain.email element Notify from an event handler or subscriber
email_providers config block Notify from an event handler or subscriber
Domain.get_email_provider() Notify from an event handler or subscriber
Domain.send_email() Notify from an event handler or subscriber
Method field type Remove it; serializer fields are not supported
Nested field type Remove it; serializer fields are not supported
pickled= argument on List Remove it; it has no effect
protean.utils.* import shims No public replacement; it is internal plumbing

Five deprecations are new in 0.18. They are removed in 0.20.0 and emit RemovedInProtean020Warning. Three have their own sections: Supervisor(debug=...) and Reloader(debug=...), and a non-boolean sequential_by on a process manager. These are the other two:

Deprecated Use instead
DAO.update() Load the aggregate, call a behaviour method, repository.add()
QuerySet.update() The same, once per matched aggregate

See DAO.update() and QuerySet.update() are deprecated for the replacement code.

How to find the code you need to change

protean check reports the other eight, with the new removal version in the message. It does not report the four method calls (get_email_provider(), send_email(), DAO.update() and QuerySet.update()):

protean check --domain=my_package.domain

The Method and Nested fields and the protean.utils.* imports are found by reading source files, and protean check reads only the modules that define a domain element. A use in a helper module with no domain element is not reported. A method call has no declaration for protean check to read at all. For these, rely on the runtime warning. Turn Protean's deprecation warnings into errors in your test run and the suite fails at each use:

# fragment
# pyproject.toml
[tool.pytest.ini_options]
filterwarnings = [
    "error::protean.exceptions.ProteanDeprecationWarning",
]

Use pytest's filterwarnings. Python's -W flag ignores a filter that names a Protean warning class, because Python reads -W before Protean can be imported.

A committed IR baseline records the old version

The diagnostics protean check raises for these deprecations name the removal version in their text: DEPRECATED_FIELD, DEPRECATED_OPTION, DEPRECATED_EMAIL, DEPRECATED_CONFIG and DEPRECATED_IMPORT. The IR checksum covers that text. If your domain raises one of them and you commit a .protean/ir.json, protean ir check reports the baseline as stale. Regenerate it:

protean ir show --domain <module> --canonical > .protean/ir.json

A filter on RemovedInProtean10Warning stops matching

RemovedInProtean10Warning still exists, but no Protean deprecation emits it now. RemovedInProtean019Warning and RemovedInProtean020Warning are not subclasses of it. If a filterwarnings entry names RemovedInProtean10Warning, it no longer matches these warnings. Point it at the new class, protean._deprecation.RemovedInProtean019Warning or protean._deprecation.RemovedInProtean020Warning. Or point it at the base class protean.exceptions.ProteanDeprecationWarning, which matches every Protean deprecation.

String and Text no longer sanitize by default

Behavioural break, String and Text fields. An ordinary String or Text field that left sanitize unset used to run bleach.clean() on every value, so the stored value was HTML-escaped. The default is now False: an unset field stores the raw input. sanitize=True and sanitize=False still mean exactly what they did.

Two shapes were never sanitized and so do not change: a field with a real choices= value, and a String or Text used as a container content spec such as List(String(max_length=50)). upgrade-check skips both, as listed below.

This is a fail-open change: where a field silently escaped its input before, it now stores it verbatim. That matters because sanitization is a display-layer concern. Escaping on write bakes one encoding (HTML) into stored data, corrupts values rendered in any other context (a plain-text email, a JSON API, a CSV export), and is not a substitute for encoding output where it is actually rendered. The default now matches that: store the real value, encode at the point of display.

Who is affected

Anyone who relied on the old default to escape stored values, on a String or Text field with no explicit sanitize=. After the upgrade that field stores the raw input, and nothing tells you at runtime: the write succeeds.

protean upgrade-check --opportunities finds these sites for you once you are on 0.18.0. It reports each String/Text field that leaves sanitize unset, by module:line, listing every one so the report works as a checklist. Unset means either not passing sanitize at all or passing None, which the field layer reads the same way.

What it skips, because none of these changed behaviour:

  • A field that gives sanitize a real value either way, by keyword (sanitize=True) or positionally (Text(True), String(255, None, True)). It declared its intent.
  • A field with a real choices= value. Choice values were never sanitized. (choices=None is not a choices field, so it is still reported.)
  • A String/Text used as a container content spec, as in List(String(max_length=50)). A container reads the inner spec's type and choices and never applies its sanitization, so the inner field was not sanitized before the flip either.
  • A String or Text that is not Protean's. The scan resolves the name back to a protean.fields import, so from sqlalchemy import String is not reported and from protean.fields import String as StringField still is.
  • Every field, if the domain sets [field_defaults] sanitize = true. That opts the whole domain back into the old behaviour, so there is nothing to migrate.

A field whose arguments are hidden behind a splat (String(**opts)) is also skipped, because sanitize could be inside and a static scan cannot see it. Check those by hand.

The fix

Encode output where these values are rendered as HTML. That is the durable fix, and it protects the values a write-time escape never could (a template that renders the same field in an email or a CSV).

To restore the old behaviour instead, opt back in. For one field:

# fragment
name = String(max_length=100, sanitize=True)

For a whole domain, set the domain-level default:

[field_defaults]
sanitize = true

Precedence is field kwarg > domain default > framework default, so a domain default of true sanitizes every unset String/Text field, and a field can still opt out with sanitize=False.

Why it was not put behind a flag defaulting to the old behaviour

This is a deliberate deviation from the standard Tier-2 transition in ADR-0004, which is opt-in flag → warning window → default flip, with the flag defaulting to the old behaviour. Here the default flips in a single release, there is no per-field runtime warning, and the [field_defaults] sanitize flag defaults to the new behaviour (you set it to true to opt back in). It is silent at runtime: the write succeeds and nothing warns.

That deviation is intentional. Storing an HTML-escaped value is the wrong default for a domain model, and bleach on write is not the security control it looks like, so the old default was not a contract worth a multi-release transition to preserve. A per-field runtime warning was rejected as noise on a path that is usually harmless. The migration net is static instead: protean upgrade-check --opportunities enumerates the exact sites (see above), so the flip is discoverable at upgrade time even though it is silent at runtime. This is not one of the two narrow single-release exceptions ADR-0004 already lists (operational defaults, silent correctness bugs); it is a security-relevant default flip settled on its own merits.

Reloading data written under a different default

The sanitize decision is read from the active domain at validation time, and a DB load, event-sourced replay, or serialization round-trip re-runs the validator. So a value written while sanitization was off, then reloaded after [field_defaults] sanitize is flipped to true, is cleaned on the way back in. Because bleach can lengthen a string (& becomes &amp;), a value that fit max_length on write can grow past it on reload and fail to load with ... characters after sanitization, exceeding max_length. Flip the domain default with this in mind: it changes how already-stored values validate, not just new writes. An explicit sanitize=True field does not have this hazard, because it cleaned on write too, so its stored value already satisfies the bound (see ADR-0026).

choices fields map to their declared column type

The SQLAlchemy adapter builds a different column for a field with choices. Before 0.18 such a field got the adapter's fallback column, an unsized string column. The adapter now builds the column from the type the choice values share, the same column a field of that Python type gets without choices:

  • String choices give VARCHAR(n), where n is the field's max_length. This covers a String with choices, a Status over an Enum of strings, and a String whose choices is such an Enum.
  • Integer choices give INTEGER. This covers an Integer with choices, an IntEnum class passed as choices, and a Status over an IntEnum.
  • Float, boolean, Decimal, date and datetime choices give the float, boolean, numeric, date and date-time column of that database.

The rule follows the values, not the field. A Float(choices=[1, 2]) has integer values, so its column is INTEGER.

The old column was TEXT on MySQL and MariaDB, VARCHAR(max) on SQL Server, and an unsized VARCHAR on PostgreSQL and SQLite. With a non-string column there, an Integer with choices saved 2 and loaded back the string "2", which the field then rejected.

Two kinds of choices keep the old column: choices of more than one type, such as ["a", 1] or [True, 1], and choices of one type that has no column mapping, such as bytes values or a list of Enum members. An annotation that starts with a Literal and joins another type, such as Literal[1] | str, keeps the old column too. A union that starts with another type, such as int | Literal[1, 2], gets that type's column, as it did before 0.18. The Elasticsearch adapter does not change.

# fragment
class Article(BaseAggregate):
    status: String(max_length=20, choices=["draft", "published"])  # VARCHAR(20)
    level: Integer(choices=[1, 2, 3], required=True)  # INTEGER NOT NULL

Who is affected

New tables get the new columns. Protean never alters an existing table, so a table created before 0.18 keeps its old columns until you change them. The string columns work as they are. A non-string choices column (integer, float, boolean, decimal, date or date-time values) holding text does not load back, so change it.

The MySQL and MariaDB index check now measures these columns. A declared Index over a String with choices counts its max_length:

  • When Protean builds the table, the column used to be TEXT, so the index was rejected as unindexable. It now builds when the key fits.
  • protean schema render --indexes used to count the field as zero bytes. An index that only fitted because of that is now rejected with IncorrectUsageError.

A String(choices=...) gets max_length=255 by default, so its column is VARCHAR(255). Three string choices fields have no length: a String declared with max_length=None, a Text with choices, and a Status that sets no max_length. Their column is TEXT on MySQL and MariaDB, which MySQL cannot index. An index over any of them is rejected when Protean builds the table, and is now also rejected by protean schema render --indexes, which passed it before.

How to change an existing table

For a table named article with the two fields above, where level is required:

=== "PostgreSQL"

```sql
ALTER TABLE article ALTER COLUMN status TYPE VARCHAR(20);
ALTER TABLE article ALTER COLUMN level TYPE INTEGER USING level::integer;
```

PostgreSQL keeps `NOT NULL` when it changes the type.

=== "MySQL and MariaDB"

```sql
ALTER TABLE article MODIFY status VARCHAR(20);
ALTER TABLE article MODIFY level INTEGER NOT NULL;
```

`MODIFY` replaces the whole column definition. Leave out `NOT NULL` and the
column becomes nullable without an error.

=== "SQL Server"

```sql
ALTER TABLE article ALTER COLUMN status VARCHAR(20);
ALTER TABLE article ALTER COLUMN level INTEGER NOT NULL;
```

`ALTER COLUMN` replaces the nullability too. Leave out `NOT NULL` and the
column becomes nullable without an error.

=== "SQLite"

SQLite ignores the length of a `VARCHAR`, so `status` needs no change.
SQLite cannot change a column's type in place, so rebuild the table for
`level`:

1. Create `article_new` with the new column types.
2. Copy the rows across with `INSERT INTO article_new SELECT ... FROM article`,
   using `CAST(level AS INTEGER)` for the integer column.
3. Drop `article` and rename `article_new` to `article`.
4. Create the old table's indexes again. Dropping a table drops its indexes.

For any other type, use the column the field's Python type gets without choices, and add NOT NULL when the field is required=True.

Check first that every stored value fits the new type. A status longer than 20 characters makes the ALTER TABLE fail. A level that is not a whole number makes it fail on PostgreSQL, MariaDB and SQL Server. MySQL 8 does not fail: it rounds '2.5' to 3.

The InlineBroker no longer dead-letters a message its subscription owns

On the InlineBroker, a stream consumed by a BrokerSubscription or a StreamSubscription now has one dead-letter destination: the {stream}:dlq stream the subscription publishes. The broker's own nack ceiling and its native dead-letter queue no longer apply to that stream. A nacked message is held and redelivered instead, which is what Redis Streams already did. See ADR-0042.

Who is affected

Almost nobody, and there is nothing to change in your code. The old behaviour needed a {stream}:dlq publish that kept failing inside the same process, on an in-memory test double. If you had that, a message the subscription was holding eventually landed in the broker's native DLQ and showed up under the {stream}:dlq name in protean dlq list. It no longer does: it stays held and is redelivered until the DLQ publish succeeds.

A test that asserted such a message reaches protean dlq list (or broker.get_dlq_messages) has to assert the hold instead. Drive the DLQ outage with retry_delay=0 so the held message comes back on the next read.

A stream consumed straight off the broker with no subscription is unchanged: it still hits the InlineBroker's ceiling and lands in the native DLQ. Production adapters are unchanged too. Redis Streams holds a nacked message pending and Redis PubSub does not support nack, so neither had a second dead-letter destination to reconcile.

Why it was not put behind a flag

Two layers counting the same failure was the bug, and a flag that kept the old path alive would keep the broker dead-lettering underneath the subscription. This takes the silent-correctness-bug exception in ADR-0004.

Replay hazards on an event-sourced aggregate are now breaking

protean ir diff now classes four changes as breaking. It used to list them as ordinary changes. Each one stops an existing event-sourced aggregate being rebuilt from its events, and none of them is a field change, so the checker did not class them before:

Change Reported as
is_event_sourced turned on or off event_sourcing_changed
An aggregate's stream_category moved stream_category_changed
The aggregate's identity moved to another field identity_field_changed
An @apply handler dropped while its event survives apply_handler_removed

Who is affected

A domain that makes one of these changes and runs protean ir diff in a check or a pre-commit hook. The command used to exit 2, the code for changes with none breaking, and now exits 1. A check that fails on any non-zero exit was already failing. A check that fails only on exit 1 now fails too.

If your domains make none of these changes, nothing about your diff output moves.

What to do

Treat the report as correct and roll the change back, because each one strands data that already exists:

  • An aggregate writes to f"{stream_category}-{identifier}", so moving the stream category leaves the whole history under the old name and a load of an existing aggregate finds an empty stream. Keep the old category. If the rename is genuinely wanted, copy or replay the existing streams under the new name first, as a data migration, outside the compatibility checker.
  • Moving the identity to another field asks for a stream keyed by a value no writer ever used, with the same result. Keep the identity where it is.
  • Turning event sourcing on leaves the aggregate's existing state as table rows that replay cannot rebuild; turning it off leaves it as a stream the table-backed loader never reads. Either direction needs a data migration first.
  • Dropping an @apply handler leaves stored events of that type with nothing to apply them, and BaseAggregate._apply_handler raises on one with no handler. Keep the handler for as long as events of that type are in the store, even after you stop raising the event.

Where the change is deliberate and the data has been migrated by hand, the config-level exclude list is the escape hatch, as it is for any break the checker cannot verify:

[compatibility]
exclude = ["ordering.order.order.Order"]

Why it was not put behind a flag

The checker's purpose is to report a change that strands stored data, and these four do. Passing them clean was the bug. A flag that kept the old behaviour alive would keep the checker silent about a domain whose aggregates can no longer load, which is the case it exists to catch. This takes the silent-correctness-bug exception in ADR-0004.

Removing a field from an event-sourced aggregate needs reserved

protean ir diff reports removing a field from an event-sourced aggregate as breaking, the same as for any persisted element. For an event-sourced aggregate that removal is usually safe: its authoritative state is a stream of events, and the snapshot is a rebuildable cache Protean discards and replays past when it no longer constructs. But the removal is not safe on its own. A retained @apply handler kept for a retired event can still assign the removed field, or call its add_<name>/remove_<name> helper, and replay would raise on it.

Declare the removed name in reserved on the aggregate to earn the downgrade:

@domain.aggregate(event_sourced=True, reserved=["note"])
class Order:
    order_id = Identifier(identifier=True)
    # `note` removed; its name is reserved so it is never reused

reserved is a declaration the author adds to control how the removal is handled, like renamed_from and deprecated. Like renamed_from, it also carries a runtime effect; deprecated only changes the checker verdict:

  • During replay only, an assignment to a reserved name is dropped instead of raising, so a retained handler for a retired event still rebuilds the aggregate. The live raise_ path never takes this relaxation, so a write to a removed field on a new event still raises.
  • For a removed association such as a HasMany, a retained handler may call the generated add_<name> or remove_<name> helper instead of assigning. During replay those calls are dropped too and return None. The child entities they would have added are dropped with them: the rebuilt aggregate does not have them. get_one_from_<name> and filter_<name> still raise, because a handler that reads the removed collection depends on it. On the live raise_ path the helpers still raise AttributeError.

  • A retained handler that calls such a helper has two more needs. The child entity class it builds has to stay in the domain, because deleting it breaks the handler and is itself a breaking element_removed. And the helper call needs # type: ignore[attr-defined]: type checkers do not know about reserved, so mypy --strict reports the helper as missing once the association is gone:

    # fragment
@apply
def item_added(self, event: ItemAdded):
    self.add_items(OrderItem(name=event.name))  # type: ignore[attr-defined]

  • The drop keys on the name alone. With a plain field such as total reserved, a call to add_total is dropped during replay too.

  • protean ir diff downgrades the field_removed change to safe, cited reserved, when the aggregate is event-sourced on both sides and the removed name is reserved in the new snapshot. The rating assumes the retained handlers only write the removed name. A handler that reads the removed collection still breaks replay, and the diff cannot see that.
  • The declaration has to stay. Taking a name back out of reserved on an event-sourced aggregate is reported as reservation_removed, breaking: replay stops dropping an assignment to the name and a call to its helpers, so a retained handler that still makes either raises again.
  • Reusing a reserved name for a live field raises IncorrectUsageError at registration.
  • A reserved name has to be a field name. An empty or private (_-prefixed) name raises IncorrectUsageError at registration.

Only the field removal is earned this way. A type change or a newly required field on an event-sourced aggregate stays breaking, and the replay hazards above (a moved stream category, a moved identity, a dropped handler whose event survives) stay their own breaking changes.

A RawIndex with an unknown dialect now raises

Behavioural break, Index.from_sql and index DDL rendering. Index.from_sql(dialect, ddl) takes the dialect as a free string. A value the framework does not render index DDL for used to be accepted at registration and then dropped at every call site that matched the dialect by name. The aggregate looked indexed and never got the index, on any dialect, with nothing said.

Domain.init() now rejects it. A RawIndex whose dialect is not one the framework renders for raises IncorrectUsageError, naming the element, the index (its name, or the DDL when it has no name), and the bad dialect. The dialect is matched by exact string, so "postgres", "PostgreSQL", and " postgresql" are all rejected: only "postgresql" renders.

The dialects the framework renders for are postgresql, sqlite, mssql, mysql, and mariadb.

Who is affected

Anyone who called Index.from_sql with a dialect name outside that set, most often a typo. That declaration never produced an index, so the fix is to correct the name:

# fragment
# Before: accepted, then silently dropped, so no index on any dialect
Index.from_sql("postgres", "CREATE INDEX ix_data_gin ON outbox USING gin (data)")

After:

gin_index = Index.from_sql(
    "postgresql", "CREATE INDEX ix_data_gin ON outbox USING gin (data)"
)

If you targeted a dialect Protean has no provider for, that DDL was never emitted either. Remove the declaration, or wait for the provider.

Why it was not put behind a flag

The old behaviour was a silent failure: the index the author declared was never created. There is no reading under which accepting a name that renders nothing is a contract worth preserving. The change moves the existing failure to the declaration point, where the author finds it, and introduces no new one. This takes the silent-correctness-bug exception in ADR-0004.

protean schema render --indexes rejects an unknown dialect

--dialects takes a comma-separated list, and an unrecognised name used to fall through to the SQLite dialect. The command wrote a .sql file that looked right, said the requested dialect in its header comment, and held DDL compiled for a different database. It now raises:

$ protean schema render --domain my_package.domain --indexes --dialects postgres
Error: Unknown index DDL dialect 'postgres'. Supported: mariadb, mssql, mysql, postgresql, sqlite.

postgres is the spelling that catches people: the dialect is postgresql. The accepted set is the same one Domain.init() checks a RawIndex dialect against, so a dialect you can declare is always one you can render.

Who is affected

Anyone passing --dialects explicitly with a name outside that list. The default (postgresql,sqlite,mssql,mysql) is unaffected, and so is every name that was already valid. If a script has been silently producing SQLite DDL under another name, it now fails instead, which is the point.

Why it was not put behind a flag

The old behaviour had no correct use: there is no case where rendering SQLite DDL under another dialect's filename is what the caller wanted. This takes the silent-correctness-bug exception in ADR-0004.

The test-mode engine stops as soon as it is idle

Engine(domain, test_mode=True).run() and protean server --test-mode used to run eleven 0.1-second cycles every time, so each run took at least 1.1 seconds. Test mode now stops as soon as every subscription has run out of work. A run with little to process finishes in a few milliseconds.

Test mode also pauses at most 10 milliseconds between a subscription's ticks, whatever its tick_interval. A broker subscription used to wait a full second before it saw the messages a handler had just raised.

The limits are unchanged. Test mode still stops after about a second when a subscription keeps finding work, and it still waits up to five seconds for a subscription to start. An event handler that failed counts as work while its position waits for a retry, so a test with a failing handler still runs for the full second.

Who is affected

A test that relied on the engine running for a fixed time while nothing was left to process. For example, a test that counted on an outbox message whose publish failed being retried at its scheduled time within that second. Test mode now stops before that time arrives. Assert on what the engine processed, and run the retry path directly if the test is about retries.

Why it was not put behind a flag

Test mode is for tests, and its documented job is to process the pending messages and stop. How long it runs was never part of that contract, and nothing a production engine does has changed.

assert_valid and assert_invalid are removed

The protean.testing.assert_valid and protean.testing.assert_invalid helpers were deprecated in 0.16.1 and are now gone. Importing either one raises ImportError.

Replace assert_invalid with pytest.raises:

# fragment
# Before
from protean.testing import assert_invalid

assert_invalid(lambda: Order(customer_id="c1", items=[]), message="at least one item")

After:

import pytest

from protean.exceptions import ValidationError

def test_an_order_needs_an_item():
    with pytest.raises(ValidationError) as exc_info:
        Order(customer_id="c1", items=[])

    assert "at least one item" in exc_info.value.messages["items"][0]

assert_invalid returned the exception. exc_info.value gives you the same object. Its messages attribute maps each field name to a list of messages, so you can check the messages of one field. The old message argument checked for a plain substring in any field's messages.

You can also pass match to pytest.raises. Know how it differs from message. match is a regular expression, and pytest searches it in str(exc), which is the repr of the whole messages dictionary. That text holds the field names too, and the repr escapes quotes and backslashes. So a pattern can match a field name, and a message with a quote or a backslash may not match even after re.escape. For plain messages, match works fine:

def test_an_order_needs_an_item_with_match():
    with pytest.raises(ValidationError, match="at least one item"):
        Order(customer_id="c1", items=[])

Replace assert_valid with a plain call. A ValidationError fails the test on its own:

# fragment
# Before
from protean.testing import assert_valid

order = assert_valid(lambda: Order(customer_id="c1", items=[item]))

After:

def test_an_order_with_an_item_is_valid():
    item = OrderItem(sku="SKU-1")
    order = Order(customer_id="c1", items=[item])

    assert order.items == [item]

The RemovedInProtean018Warning category is gone too

These two helpers were the last APIs scheduled for removal in 0.18.0, so the protean._deprecation.RemovedInProtean018Warning warning class is removed with them. If a pytest filterwarnings entry or a -W flag names it, delete that entry. pytest stops with an error on a warning category it cannot import, and Python ignores such a -W flag with a warning. To keep turning Protean deprecations into errors, filter on protean.exceptions.ProteanDeprecationWarning, which covers every removal version.

protean server and protean observatory apply [logging]

Single-worker protean server and protean observatory used to ignore the [logging] section of domain.toml. The commands set up console logging before the domain loaded, at the PROTEAN_LOG_LEVEL level or INFO. Domain.init() then saw handlers on the root logger and skipped its own setup. Both commands now call Domain.configure_logging() once the domain loads, the way --reload and each multi-worker process already did. Your level, format, redact, per_logger, log_dir, and sampling settings now take effect, and engine logs carry correlation_id.

--log-level and --log-format now override only the level and the format. The rest of [logging] still applies. --log-config still replaces the section.

With several --domain values, protean observatory applies the first domain's [logging] section.

Who is affected

Anyone who runs single-worker protean server or protean observatory. Log output changes on upgrade in these cases:

  • You have a [logging] section. Its settings now apply: a JSON format, a different level, redacted fields, and rotating log files when log_dir is set.
  • You have no level setting. With no [logging].level, no PROTEAN_LOG_LEVEL, and no --log-level, the level now comes from the environment: PROTEAN_ENV, then ENV, then ENVIRONMENT. When none is set, a plain protean server or protean observatory logs at INFO with colored console output, as it did in 0.17. Under PROTEAN_ENV=development it now logs at DEBUG, and under PROTEAN_ENV=test at WARNING. See A domain with no environment set logs at INFO.
  • You have no redact setting. The default redaction keys (password, token, secret, api_key, and others) now mask those fields in log records.
  • Your domain module sets up logging when it is imported. The [logging] setup now replaces it. Set PROTEAN_NO_AUTO_LOGGING=1 to keep your own setup.
  • A [logging] value is invalid. A bad value, such as a numeric level or an unwritable log_dir, now stops the command with an error. It used to be ignored, because Domain.init() catches errors in its own logging setup. Multi-worker and --reload workers already failed this way.

A multi-worker run without --log-level also changes: the supervisor now forwards worker records at the [logging] level, so level = "DEBUG" shows DEBUG records it used to drop.

Why it was not put behind a flag

Applying [logging] is the documented behaviour, and --reload and the workers already did it. The old single-worker result was a bug.

protean server workers apply the logging flags

--log-level, --log-format and --log-config now reach the worker processes of protean server --workers N and protean server --reload. Each worker applies them the way the parent does: --log-level and --log-format override the level and the format of [logging], and --log-config replaces [logging]. A worker started after a reload gets the same flags.

Before this, the workers ignored the flags. Once the domain loaded, each worker applied [logging] and PROTEAN_LOG_LEVEL only.

Who is affected

Anyone who passes a logging flag to a multi-worker or --reload run, or sets PROTEAN_LOG_LEVEL for one. The workers' log output now follows the flags. For example, protean --log-level DEBUG server --workers 4 used to drop worker DEBUG records when [logging] set a higher level, and now shows them. With --reload, the worker's level, format and dictConfig all follow the flags now.

Before the domain loads, a worker now sets up its logging the same way the parent does. With no logging flag, it logs at PROTEAN_LOG_LEVEL from the moment it starts. Before, it logged at INFO until the domain loaded, so it dropped the DEBUG lines it wrote while loading the domain. With no logging flag and no PROTEAN_LOG_LEVEL, the worker starts at INFO. With --log-format alone, the worker uses that format from the start, and takes its level from PROTEAN_LOG_LEVEL and then PROTEAN_ENV, as the parent does. With --log-config, the worker applies only that dictConfig, and PROTEAN_LOG_LEVEL does not override it.

Why it was not put behind a flag

The flags are documented to set the log level, the format, and the configuration of protean server. Ignoring them in the workers was a bug.

[lint] and [observatory] tables now load

The config loader keeps only the top-level sections listed in its defaults, and lint and observatory were not in that list. A top-level [lint] or [observatory] table in domain.toml, a [tool.protean.lint] or [tool.protean.observatory] table in pyproject.toml, or a lint or observatory key in a Domain(config=...) dict was dropped without a warning. Both sections are now loaded, so their settings take effect.

Who is affected

  • You have a [lint] table. level, suppressions, rules and the other [lint] keys now apply to protean check and protean verify, and every key except level applies to the IR builder too. A project that passed protean check can start failing, for example when level = "info" makes info findings gate, or when a custom rule listed in rules reports findings. Run protean check and adjust [lint].

  • Your [lint] table has a malformed value. A value that used to be ignored is now checked. These values are rejected:

    • a [lint] that is not a table
    • a bad level
    • a suppressions value that is not a table, or a suppression count that is not a non-negative integer
    • a rules value that is not a list of strings
    • a limit that is not a non-negative integer
    • a check_* flag that is not true or false

protean check exits 2 and protean verify exits 4. Building the IR raises ConfigurationError for the same values, except a bad level: the IR builder does not use level, so it does not check it. protean ir show, the MCP check, validate and introspect tools and the pre-commit hooks each report that error. Domain.check() raises it, even when the domain has validation errors. Code that calls Domain.check() directly and expects a result dict should catch ConfigurationError. Fix the value the error message names. - You commit a materialized IR. [lint] settings change the IR's diagnostics, and the diagnostics are part of the IR checksum. protean ir check can report the committed IR as stale. Regenerate it. - You have an [observatory] table. trace_retention_days now applies to trace persistence. The value goes through Python's int(): 3.9 becomes 3 days, and 0 or a negative number turns trace persistence off. An [observatory] that is not a table, or a retention int() cannot convert (such as "abc" or inf), falls back to the default of 7 days.

Environment overlays such as [test.lint] were already loaded when PROTEAN_ENV named that environment. The base table now loads too, and the overlay is merged over it. If you have both [lint] and [test.lint], the effective settings under PROTEAN_ENV=test now include the [lint] keys the overlay does not set.

Why it was not put behind a flag

These tables are the documented way to configure protean check and trace retention. Ignoring them was a bug.

Logging filters now run for records from child loggers

Protean attached ProteanCorrelationFilter, OTelTraceContextFilter and ProteanRedactionFilter to the root logger only. A logger's filters run only for records logged on that logger. They do not run for records that propagate up from a child logger such as logging.getLogger("myapp.orders"). So those records did not get correlation_id or causation_id, did not get the trace ids when telemetry was on, and were not masked by [logging].redact.

Protean now also attaches each filter to every handler the root logger holds when logging is configured. Handler filters run for every record the handler writes, so records from child loggers that reach those handlers now get the same fields and the same masking as records logged on the root logger. With protean server --workers N, each worker's queue handler gets the filters too, and the supervisor keeps the ids a worker set on its records.

Who is affected

  • You log through child loggers. Their records now carry correlation_id and causation_id, and the trace ids when telemetry is on. A format string that uses %(correlation_id)s now works for them.
  • You rely on [logging].redact for stdlib records. Values of the listed keys are now masked on records from child loggers too. Log output that used to show those values now shows the mask.
  • You list correlation_id, causation_id, trace_id or span_id in [logging].redact and read them from the log record. The redaction filter now runs after the correlation and trace-context filters on the root logger and each of its handlers. Before, those filters wrote the real ids onto the record after redaction had run. Protean's JSON and console output masked them anyway, but a formatter or filter that reads record attributes (a %(correlation_id)s format string, caplog) saw the real ids. It now sees the mask.
  • Your dict_config puts ProteanRedactionFilter or OTelTraceContextFilter on the root logger only. Protean now copies each to every root handler, so records from child loggers are masked, or carry trace ids, too.
  • You set correlation_id or causation_id yourself through extra=. The filter now overwrites them on records from child loggers, as it already did on records logged on the root logger. Outside a domain context both become "". Set g.correlation_id and g.causation_id in the domain context instead, and the filter picks them up.
  • You add a handler to the root logger after logging is configured. That handler does not get the filters. Attach them to it yourself, for example handler.addFilter(ProteanCorrelationFilter()) from protean.integrations.logging. The same applies to handlers on named loggers and to child loggers that set propagate = False, because their records never reach the root logger's handlers.
  • You call configure_logging() more than once with different redact keys. Each logger and handler keeps the first ProteanRedactionFilter it got. The root logger keeps the keys from the first call, while handlers created by a later call get the keys from that call. With dict_config, those handlers also get the root logger's redaction filters, so they mask the keys from both calls. Under protean server --workers N, each worker sends its records through one queue handler. That handler masks every key set the worker's root logger or its handlers held, so a worker can mask more keys than a single-process run does. This covers stdlib record attributes. Fields logged through get_logger() are masked by the structlog processor that the last call built, so they use only that call's keys. Pass the full list of keys on every call.

Why it was not put behind a flag

The fields and the redaction are documented to apply to every stdlib record. Skipping records from child loggers was a bug.

A zero read timeout on Redis no longer waits

On the Redis broker, read_blocking(timeout_ms=0) sent XREADGROUP ... BLOCK 0, which Redis reads as "wait until a message arrives". The subscription engine calls it with 0 to check the primary stream without waiting when priority lanes are on. That check hung on an idle stream, logged broker.redis.read_blocking_failed when the socket timed out, and held up the backfill lane.

read_blocking(timeout_ms=0) now returns at once, as the broker contract describes. A negative timeout_ms is rejected and logged as broker.redis.read_blocking_failed, and the call returns [].

The blocking_timeout_ms subscription setting must now be positive. A value of 0 raises ConfigurationError when the server starts. With the new meaning, 0 would make the poll loop query Redis with no pause.

Who is affected

  • You set blocking_timeout_ms = 0 on a stream subscription. The server now refuses to start. Before this release the setting logged broker.redis.read_blocking_failed every few seconds. Remove the setting to get the default, or set a positive value such as 1000.
  • You call broker.read_blocking(..., timeout_ms=0) yourself to wait for a message. Call it in a loop with a positive timeout instead. Values above 1000 ms are capped at 1000 ms.
  • You run with priority lanes on. An idle subscription now waits on the primary and the backfill stream in one read. Before, it waited on the primary stream only, which delivered production messages at once and held up the backfill lane. Now a message on either stream ends the wait, so a production message that arrives while the subscription is idle is picked up at once, and backfill messages are no longer held up. No setting change is needed.

Why it was not put behind a flag

The broker contract gives 0 no wait, and the engine's priority-lane check depends on that. Waiting forever on 0 was a bug. There is no setting that waits forever. The read runs in a worker thread that shutdown cannot interrupt, and a client socket_timeout cuts the wait short with an error.

The memory provider's raw() raises DatabaseError, and some error logs drop the exception text

Two small changes came with the upgrade to ruff 0.16.

The memory provider's raw() raises DatabaseError when the query is not valid JSON. It used to raise a bare Exception. The json.JSONDecodeError is kept as the cause. raw() replaces every ' with " before it parses the query, so a query written with single quotes still parses. Until something has been saved to the store, raw() returns [] without parsing the query.

Error logs in the observatory API, the server subscriptions and the worker supervisor no longer put the exception text in the message. They are logged with logger.exception, so the record still carries the exception and its traceback in exc_info. For example, a worker that fails on start now logs Worker 3 failed with the traceback, where it logged Worker 3 failed: db unreachable. The subscription retry log is now Error in subscription <name> (attempt <n>), and the DLQ log is Failed to move message <id> to DLQ.

Who is affected

  • You catch the error from the memory provider's raw(). An except Exception handler still catches it. An except DatabaseError or except ProteanException handler now catches it too, where it did not before. Check that such a handler is meant to see a malformed query.
  • You match exception text in log messages. A log filter, an alert rule or a search that looks for the exception text in the message line of these records no longer finds it. Match on the exception in exc_info instead, or on the traceback in the formatted output. A JSON formatter writes the traceback to its own field, so search that field.

Why it was not put behind a flag

DatabaseError is a subclass of Exception, so no handler that caught the error before stops catching it. The exception text was printed twice before: once in the message and once in the traceback. It is still printed once.

Some internal fallbacks let unexpected errors through

A few internal paths used to catch every exception. Each one now handles only the case it was written for. Any other error reaches the caller, unless the table names an outer handler that still catches it.

Path Case still handled What happens in that case
Unit of work commit, recording event names for the access log No domain context is active The step is skipped
Resetting the access log counters before a handler runs No domain context is active The step is skipped
Message CloudEvents source, read from source_uri or the domain name No domain context is active The source comes from the stream category, or is urn:protean:unknown
get_logging_config_value() [logging] is not a table The default is returned
server.version_retry settings for handler retries [server] or [server.version_retry] is not a table, or a value is not a number, such as max_retries = "three" or max_retries = inf (TypeError, ValueError, OverflowError) The defaults are used, and a warning is logged once for each distinct [server] table
Health-check server closing a client connection The peer reset the connection or the pipe broke (OSError) The close error is ignored
IR diff comparing a removal version to the current version The two versions mix numbers and text, such as "1.0" and "1.x" (TypeError) The removal is classed as premature
Redis broker dlq_list() A DLQ stream that does not exist The stream yields no entries. Any ResponseError, such as WRONGTYPE for a key that is not a stream, reaches the caller
Redis broker dlq_inspect(), dlq_replay(), dlq_replay_all(), dlq_purge() and dlq_depth() A DLQ stream that does not exist, or a dlq_id that is not a Redis stream ID (<ms> or <ms>-<seq>) A missing stream gives no entry, False or 0. A malformed ID gives None or False without calling Redis. Any ResponseError, such as WRONGTYPE for a key that is not a stream, reaches the caller. The observatory DLQ endpoints then return a 500, the protean dlq commands log it and exit with an error, and the DLQ maintenance task logs it and moves on to the next stream
Redis brokers' stream and message counts in health_stats() A key that does not exist or holds another type, found with TYPE The key is left out of the counts
Redis brokers' ensure_connection() A Redis error, a socket error, or a malformed reply from the server (RedisError, OSError, ValueError) The attempt counts as failed, and the error is logged
Redis brokers' ping() A Redis error, a socket error, or a malformed reply from the server (RedisError, OSError, ValueError) ping() returns False, and the error is logged. The base broker's ping() still catches any other error and returns False too
Redis brokers' counts and stream details in health_stats() A Redis error (RedisError) The counts or the stream list read as empty, and the error is logged. Any other error is caught by the broker's own health check, which logs it with the traceback. health_stats() then reports the broker as degraded if ping() still succeeds, with the error text in details
Redis Streams broker's info(), reading one consumer group A Redis error, or a group or consumer name that is not valid UTF-8 (RedisError, UnicodeDecodeError) That group is left out, and the error is logged at debug level. The other groups are still listed
Redis PubSub broker's reconnect in ensure_connection() A connection URI that redis.Redis.from_url() cannot parse (ValueError) The attempt counts as failed, and the error is logged. A missing URI (None) raises AttributeError to the caller
Redis Streams broker's stale-consumer cleanup when a subscription starts A Redis error (RedisError), or a consumer name that is not valid UTF-8 (UnicodeDecodeError) If listing the group's consumers fails, the cleanup is skipped. If removing one consumer fails, or its name cannot be decoded, that consumer is skipped and the others are still removed. A Redis error is logged as a warning, and an undecodable name at debug level. Any other error reaches the subscription, which logs a warning with the traceback and starts anyway
TraceParent.build() A value that is not a string, or a header without four hyphen-separated parts, with a version other than 00, or with flags that are not hex (ValueError). The trace ID and parent ID are not checked for length or hex digits None is returned, and the error is logged
server.transient_retry settings for handler retries [server.transient_retry] is not a table, or a value is not a number, or max_retries is infinite (max_retries = inf) (AttributeError, TypeError, ValueError, OverflowError). An infinite delay setting is accepted as it is The defaults are used, and a warning is logged
Command-deadline check before a handler retry No domain context is active (AttributeError) The retry is not limited by a deadline
Domain() guessing the path of the file that created it The call stack is too shallow, or the path cannot be resolved (ValueError, TypeError, RuntimeError) The current working directory is used
Source parsing for protean check The file cannot be read or parsed (OSError, SyntaxError, ValueError, RecursionError, MemoryError) The checks that need the source are skipped
SQLAlchemy connection-failure log, hiding the password in the URI The URI cannot be parsed (ArgumentError, ValueError) The URI is logged as <unparseable>
Teaching-skill hints on a diagnostic The DX pack cannot be imported (ImportError) No hints are shown
Observatory trace listings, trace stats, trace overview, worker throughput, handler and process-manager metrics, recent messages, the correlation timeline and the live trace stream A trace entry that is not valid UTF-8, is not valid JSON, or is not a JSON object. Not valid JSON includes an integer longer than 4300 digits and very deep nesting. In the views that count by handler or event, also an entry whose handler or event is not a string. In worker throughput, also an entry whose worker_id is not a string The entry is skipped. A duration_ms counts when it is a finite number or a numeric string such as "12.5". Any other value, including infinity and NaN, is left out of the latency figures, and the entry still counts. The correlation timeline shows such a duration as null. An error raised after the entry is decoded reaches the caller. The single-trace endpoint, /traces/{stream_id}, answers an entry it cannot decode with a 500 Failed to parse trace data response. It returns any other JSON object with a _stream_id key added. An object that holds NaN or Infinity still raises an error, as in 0.17, because a JSON response cannot carry those values

The IR diff also accepts a removal version written as a number, such as removal: 0.18, and compares it as text. A number cannot keep a trailing zero, so removal: 0.20 is read as "0.2", which sorts before 0.18. diff_ir(..., current_version="0.18") then classes a field due for removal in 0.20 as safe to remove. Write both versions in deprecated= as strings. 0.19 refuses a number (see Deprecation versions must be strings).

Who is affected

Only code where one of these paths raised some other exception. In 0.17 that error was hidden. The path skipped its step or used its fallback value. In 0.18 the error reaches the caller, for example as a failed handler call. A domain.config mapping whose get() raises is one way to hit this. If you see a new error from one of these paths after upgrading, look for its cause in your configuration or your code.

Why it was not put behind a flag

Each fallback was written for one known case. When it also caught every other error, a real bug looked the same as the known case, and nobody saw it.

A failed post-invariant rolls back the change

Behavioural fix. When an @invariant.post check failed after a change, Protean raised the error but left the new value on the object. Now the change is undone: the object keeps its previous state, and its changed flag goes back to what it was before the change.

The rollback covers:

  • field assignments of every kind (plain, value object, Reference, HasOne and HasMany), on the aggregate and on its child entities
  • add_<field>() and remove_<field>() calls
  • atomic_change blocks: when the check at the end of the block fails, every change made in the block is undone, including child entities and associations, and the events raised inside the block are discarded
  • events raised on event-sourced aggregates: when a check fails after the @apply handler, raise_() undoes the handler's field changes and leaves the event out of _events

It also runs when the invariant fails with an error other than ValidationError.

def rejected_total():
    order = Order(customer_id="c1", total=10)
    try:
        order.total = -5  # breaks a post-invariant
    except ValidationError as exc:
        print("rejected:", exc.messages)

    return order.total  # 0.17: -5.0. 0.18: 10.0

Who is affected

Code or tests that caught the ValidationError and then read the rejected value from the object. Such code now sees the previous value. Code that discarded the object after the error is not affected.

A ValueError at commit is no longer reported as a version conflict

The unit of work turned almost every ValueError raised during commit into ExpectedVersionError, with the outbox on or off. The one exception was a ValueError from saving the outbox rows, which raised TransactionError. A handler's version retry then ran the handler again, even when the failure had nothing to do with a version conflict.

Now only a real version conflict raises ExpectedVersionError at commit:

  • Message DB reports a wrong expected version as an error with the P0001 code. The Message DB event store's append and _write now raise ExpectedVersionError for that error itself, inside or outside a unit of work. It raised ValueError before. Other Message DB errors still raise ValueError.
  • The memory event store raises ExpectedVersionError itself on a wrong expected version. It raised ValueError before. The message text is the same: Wrong expected version: 0 (Stream: ..., Stream Version: 1).
  • Any other ValueError raised during commit becomes TransactionError, with or without the outbox. Its extra_info["original_exception"] is "ValueError", its __cause__ is the ValueError, and the version retry does not run the handler again.

Who is affected

  • Code under event_processing = "sync" where an event handler raises a ValueError (a pydantic ValidationError is one). The handler runs inside the commit, so the caller used to see ExpectedVersionError, and the enclosing command handler's version retry ran a command whose events were already stored. The caller now sees TransactionError with the ValueError as its __cause__, and the command runs once. The 0.17 guide describes the old behavior under A failing handler method no longer stops its siblings and Sibling handler classes are independent in failure under synchronous dispatch. A single failing handler raising ValueError now surfaces the same way as any other single failure.
  • Code that catches ExpectedVersionError around a commit and expects it for a ValueError that is not a version conflict. Catch TransactionError for those failures.
  • Code that catches ValueError from the memory or Message DB event store's append or _write on a wrong expected version. Catch ExpectedVersionError instead.

At commit, version conflicts on the commit's own writes (Message DB, the memory event store, relational providers) still raise ExpectedVersionError and are still retried. A conflict raised by a synchronous event handler is reported differently; see the next section.

Why it was not put behind a flag

Both changes are corrections. Retrying a failure that is not a version conflict runs the handler again for no reason. The memory and Message DB event stores now report a conflict with the error the unit of work reports for the relational providers.

A synchronous event handler's version conflict no longer reruns the command

Under event_processing = "sync", event handlers run inside the command's commit, after the command's writes are saved. When a handler raised ExpectedVersionError, the commit passed it to the caller unchanged. The command handler's version retry then ran the command again, and the command saved its writes a second time. On an event-sourced aggregate the second run appended the command's events again.

Now any Exception from a synchronous event handler, ExpectedVersionError included, reaches the caller as TransactionError. A BaseException that is not an Exception, such as KeyboardInterrupt, still passes through unchanged.

  • The TransactionError's __cause__ is the handler's ExpectedVersionError. When only this handler fails, its extra_info["original_exception"] is "ExpectedVersionError".
  • Under command_processing = "sync", the command runs once. Under command_processing = "async", the command's subscription redelivers the failed command and it runs again, as it does for any handler failure.
  • When version retry is enabled, an event handler's or projector's own retry, inside @handle, still retries it. A handler that conflicts once and then succeeds lets the command succeed. A process manager has no such retry, so its conflict is reported at once.
  • A conflict in one handler class no longer stops the other handler classes for the same event. They all run, and when more than one fails the __cause__ is an ExceptionGroup holding every failure.
  • A conflict on the command's own writes still raises ExpectedVersionError, and the command's version retry still retries it.

Who is affected

Code under event_processing = "sync" that catches ExpectedVersionError from domain.process() and expects it for a conflict in an event handler. Catch TransactionError and read its __cause__:

# fragment
from protean.exceptions import ExpectedVersionError, TransactionError

try:
    domain.process(PlaceOrder(order_id=order_id))
except TransactionError as exc:
    if isinstance(exc.__cause__, ExpectedVersionError):
        ...  # an event handler conflicted; the order is already placed
    raise

The same holds for a ConfigurationError raised by a synchronous event handler. It used to reach the caller unchanged and now arrives as the __cause__ of a TransactionError. A ConfigurationError raised by the commit itself, outside the event handlers, still reaches the caller unchanged.

The protean.testing helpers that run handlers synchronously (given(Aggregate, *events), given(event, ...).then(), given(event, ...).results_in() and given(ProcessManager, *events)) call the same dispatch outside any commit, so they raise no TransactionError. A lone failing handler still raises its own exception, ExpectedVersionError included. What changed there is the conflict no longer stops the other handler classes. When another one also fails, the test gets an ExceptionGroup holding both, where it used to get the ExpectedVersionError.

Code under event_processing = "async" is not affected. There, each handler runs in its own subscription and retries on its own.

Why it was not put behind a flag

Running the command twice saves its writes twice, so the old behaviour was a bug.

An omitted value object argument gets the value object's default

Behavioural break, flattened value object arguments only. When you build an aggregate, entity or projection from some of a value object's flattened arguments, such as Customer(billing_address_street="1 Main St"), Protean used to pass None for every argument you left out. It now passes only the arguments you gave, and the value object's own defaults fill the rest.

Who is affected

Code that leaves out a flattened argument for a value object field that accepts None but defaults to something else. That field now gets its default where it used to get None:

from protean.core.aggregate import BaseAggregate
from protean.core.value_object import BaseValueObject
from protean.fields import ValueObject

class Venue(BaseValueObject):
    street: str = ""
    city: str | None = "Unknown"


class Event(BaseAggregate):
    venue = ValueObject(Venue)


# Before: Event(venue_street="1 Main St").venue.city is None
# After:  Event(venue_street="1 Main St").venue.city == "Unknown"
def event_without_a_city():
    # To keep None, pass it
    return Event(venue_street="1 Main St", venue_city=None)

Loading from a repository is unchanged, because every stored column is passed back in. Building the value object yourself and passing it as an object is unchanged too.

Why it was not put behind a flag

Passing None for a missing argument ignored the defaults the value object declares. It failed outright on any field that does not accept None, such as city: str = "". The new behaviour builds the value object the same way as Venue(street="1 Main St").

A domain with no environment set logs at INFO

Behavioural break, log output only. In 0.17, Domain.init() treated an unset environment as development and logged at DEBUG. With PROTEAN_ENV, ENV and ENVIRONMENT all unset, a script or a REPL session that called domain.init() printed a DEBUG line for every provider and broker it registered. Protean now logs at INFO when no environment is set. The output stays colored console. Only an explicit PROTEAN_ENV=development (or ENV or ENVIRONMENT) selects DEBUG.

Three related changes go with it:

  • The Exception:: debug line is gone. Creating any Protean exception with messages, such as a ValidationError your code catches, used to log Exception:: {messages} at DEBUG. It now logs nothing.
  • A broken plugin logs a WARNING. A provider or broker plugin whose load or registration raises an error other than ImportError now logs Failed to load provider plugin (or broker plugin) at WARNING, so you see it at the default level.
  • An ImportError is a DEBUG skip. A plugin whose load or registration raises any ImportError logs Skipping provider plugin (or broker plugin) at DEBUG with the error text. The usual cause is a missing optional extra, such as the Redis brokers without protean[redis], and the text then names the missing module. An ImportError from a bug inside a plugin, such as cannot import name ..., takes the same path, so set the level to DEBUG when a plugin you expect does not register.

Who is affected

Anyone who runs a domain with no environment variable set and reads its DEBUG output, or the INFO lines from protean.core and protean.adapters: a script, a REPL, protean shell, or a FastAPI app that calls domain.init(). A plain protean server or protean observatory logged at INFO in 0.17 and still does. To get the DEBUG lines back, set one of these:

export PROTEAN_ENV=development
# or
export PROTEAN_LOG_LEVEL=DEBUG

protean --log-level DEBUG <command> and level = "DEBUG" under [logging] in domain.toml work too. The Exception:: line does not come back at any level.

Why it was not put behind a flag

The change lowers what a default run prints. At INFO, Protean holds protean.core and protean.adapters at WARNING, as it already did for an explicit INFO level. Their INFO lines, such as Executing use case and the broker INFO lines, stop printing along with the DEBUG lines. The one line added at the default level is the WARNING for a plugin that fails with an error other than ImportError, and that line reports a real fault. One environment variable brings back all of these lines except Exception::, which carried the same messages the exception already holds.

A non-dict positional argument raises TypeError

Aggregates, entities, value objects, commands, events, queries, projections, process managers and emails accept dicts as positional arguments. Each dict is a template of common values, and keyword arguments override it:

def placed_order():
    order = Order({"customer_id": "c-1", "status": "DRAFT"}, status="PLACED")
    return order

Passing anything other than a dict used to raise AssertionError. It now raises TypeError. The message still says the argument must be a dict. The entity helper _update_data changed the same way.

A field's renamed_from changed the same way. A value that is not a string, a list or a tuple used to raise ValueError. It now raises TypeError. An alias inside the list that is empty or not a string still raises ValueError.

Who is affected

Code that catches AssertionError around element construction, or ValueError around a field's renamed_from. Catch TypeError instead:

# fragment
# Before
try:
    order = Order(payload)
except AssertionError:
    ...

After:

def build_order(payload):
    try:
        order = Order(payload)
    except TypeError:
        order = None
    return order

Why it was not put behind a flag

Both errors report a programming mistake. It shows up as a traceback at the call site, and code rarely catches it, so a transition period would protect almost nothing. A wrong argument type is what TypeError is for. Catching AssertionError also catches any failed assert in the same block, and it was never a documented pattern around element construction.

An event or command rebuilt from a stored message keeps its trace IDs and origin stream

Message.to_domain_object() rebuilds an event from a stored message. The rebuilt event used to drop the stored correlation_id and causation_id. Outside a handler both came back as None. Inside a handler they were replaced with the IDs of the message being handled. The rebuilt event now keeps the IDs it was stored with. A stored ID that is empty stays empty, even inside a handler. A rebuilt command works the same way: an empty stored causation_id is no longer filled with the ID of the message being handled.

origin_stream follows the same rule for both. The stored value used to be ignored. A rebuilt event took the origin_stream of the command being handled, if that command had one. A rebuilt command took the stream of the event being handled. In every other case, including an event or command delivered to its own handler, it came back as None. The rebuilt event or command now keeps the origin_stream it was stored with, and an empty one stays empty.

Who is affected

Code that reads _metadata.domain.correlation_id, causation_id or origin_stream on a rebuilt event or command and relies on the old values. Most code read these values from the Message instead, and that still works. Nothing needs to change unless your code expected None or the values of the current handler's message.

Why it was not put behind a flag

The stored values are the correct ones. The old values were wrong for anyone who read them from the event or command, and no code was meant to depend on them.

An event or command rebuilt from a stored message keeps its store positions

Message.to_domain_object() used to drop the stored message's event_store metadata, so _metadata.event_store was always None on a rebuilt event or command. The rebuilt event or command now keeps it, with position (the position in its stream) and global_position (the position across the whole store).

A handler sees the positions only when it is given a message read back from the event store. That happens with event store subscriptions and during a projection rebuild. A handler on a stream subscription, or one that runs synchronously, is given the event as it was raised, before the store assigned positions, so _metadata.event_store is None there. The exception is an event the outbox reconciler sends again after a crash: it rebuilds the outbox row from the stored message, so that event arrives with its positions. A new event or command built from another one's metadata was never stored, and it has no positions either.

Writing a rebuilt event or command again does not copy its old positions. The store assigns new ones on the write.

Who is affected

Code that reads _metadata.event_store on a rebuilt event or command and expects None. Nothing needs to change unless your code relied on that.

Why it was not put behind a flag

The positions describe where the message is stored, and they were already on the Message. Dropping them on the rebuilt object was a bug, and no code was meant to depend on it.

sequential_by on a handler or projector must be a field name

On an event handler, command handler or projector, sequential_by names a field on the handled messages. domain.init() now raises IncorrectUsageError when the value is not a non-empty string. The message says these handlers take a field name and that the boolean form is for process managers.

Before, on an event handler or command handler, sequential_by=True failed with an error about a missing field named True. sequential_by=False and sequential_by="" were silently ignored, so the handler ran without partitioning. Projectors did not accept sequential_by at all before 0.18.

Who is affected

A domain that sets sequential_by=False or sequential_by="" on an event handler or command handler. Remove the option, or set it to None, to keep the handler unpartitioned:

# fragment
# Before: accepted and ignored
@domain.event_handler(part_of=Order, sequential_by=False)
class OrderEventHandler(BaseEventHandler): ...


# After
@domain.event_handler(part_of=Order)
class OrderEventHandler(BaseEventHandler): ...

A process manager still opts in with sequential_by=True. Any other non-boolean value is deprecated; see the next section.

A non-boolean sequential_by on a process manager is deprecated

A deprecation, not a break. On a process manager, sequential_by is a boolean opt-in. The partition key comes from each handler's correlate setting. Before, any truthy value turned partitioning on and any falsy value left it off. A field name such as sequential_by="order_id", copied from an event handler, worked like True, and the IR carried the string.

That still works in 0.18. domain.init() now emits a RemovedInProtean020Warning (a DeprecationWarning subclass) for any value other than True, False or None, and stores the value back as a boolean, so the IR carries true or false. In 0.20.0 a non-boolean value raises IncorrectUsageError.

Who is affected

A domain that sets sequential_by on a process manager to anything other than a boolean. Replace a truthy value with True and a falsy one with nothing:

# fragment
# Before: treated as True; warns in 0.18
@domain.process_manager(stream_categories=["shop::order"], sequential_by="order_id")
class OrderFulfillment(BaseProcessManager): ...


# After
@domain.process_manager(stream_categories=["shop::order"], sequential_by=True)
class OrderFulfillment(BaseProcessManager): ...

An integer identity under the uuid strategy is refused on SQL

The uuid identity strategy makes an integer identity from uuid4().int, a 128-bit number. No SQL integer column holds it. Domain.init() now raises ConfigurationError when an aggregate, entity or projection on a SQL provider (SQLite, PostgreSQL, MySQL/MariaDB or MSSQL) gets an integer identity under that strategy. The message names the element and the provider.

The check reads the identifier's own identity_type and identity_strategy arguments first, then the domain configuration. It skips an element whose custom database model declares the identity column itself, because that column's type is the model's choice.

Who is affected

On a domain whose identity_type is "integer", a SQL model for an element with a generated identifier could not be built before this release: building it raised TypeError: Integer() takes no arguments. That now builds, and the uuid case fails at Domain.init() with a clear message. An identifier that sets identity_type="string" itself on such a domain still does not build: Domain.init() raises ConfigurationError, because its column is an integer column. Drop identity_type from the field. Two cases built before and are unchanged: an identifier declared with Auto(increment=True), and an event-sourced aggregate, which has no SQL table.

The change that breaks working code is narrower. An identifier that sets identity_type="integer" itself, on a domain whose identity_type is "string", used to save. Its column follows the domain's identity_type, so it was a text column, and the 128-bit number went into it as text. That identifier is now refused:

# fragment
# Before: saved the 128-bit number into the text column
@domain.aggregate
class Order:
    order_id: Auto(identifier=True, identity_type="integer")

Pick one of the two paths the error names:

  • Set identity_strategy="function" with an identity_function that returns integers. The id is set when the aggregate is created, and the column stays a text column.
  • Declare the identifier as Auto(increment=True). The database assigns the id on the first save, and the column becomes an integer column, so the table needs a schema change. Existing rows keep their 128-bit ids, which do not fit the integer column, so this path suits a new table. Columns in other tables that reference the aggregate follow the domain's identity_type and stay text columns.

The memory provider holds the 128-bit value. Elasticsearch stores it as a string. Neither is affected.

Saving the same aggregate instance again writes a new version

Saving an aggregate instance, changing it, and saving it again in a later transaction without reloading it now writes a new version. Before, the second save wrote the version the instance already had. Its events reused the ids of the first save's events (<stream>-0.1 twice), so an event handler that deduplicates by event id dropped the second event. A copy loaded before the second save could also overwrite it without an ExpectedVersionError.

Now each committed transaction advances the instance the way a reload would. After the commit, _version is the version just written and _next_version is one more. Events raised after it are numbered from the next version (<stream>-1.1). If the transaction rolls back, _version goes back to the version the instance had before the transaction. On a provider without transactions, such as Elasticsearch, the rollback does not undo the write, so the instance keeps the version it wrote. Unless another writer saved the aggregate in the meantime, the instance can be changed and saved again without a reload.

Event-sourced aggregates track their versions from their events, so none of this applies to them.

Adding one instance several times inside a single UnitOfWork still writes one version for that transaction. The events raised between those adds are numbered within it (<stream>-0.1, <stream>-0.2).

Inside a UnitOfWork, a get() or query of an aggregate that the transaction has already loaded or saved returns that same instance, with any changes made to it since. Before, each read built a new copy. Each copy counted its version and event position from the store, which had not seen the other copies' pending changes, so events raised on an earlier copy were lost and saving a later copy could raise ExpectedVersionError. This holds for event-sourced aggregates too: before, a get() returned the tracked instance only after add() had saved it with new events.

Who is affected

Code that keeps an aggregate instance across transactions and saves it more than once. Two things change for it:

  • _version on the stored record and on the instance goes up by one for each committed transaction that saves it.
  • A copy loaded before one of those saves now raises ExpectedVersionError when you save it. Before, it silently overwrote the newer record. Reload the aggregate and apply the change again, as you would for any version conflict.

Code that reads the same aggregate more than once inside one UnitOfWork now gets one instance back. This is how the identity map in SQLAlchemy's session and in Hibernate behaves. Look for code that changes an aggregate, decides to abandon the change, and calls get() again to start over. The second get() returns the changed instance. If the code then saves it with add(), the abandoned change is written too. Events raised before the second get() are saved at commit even without add() on a state-based aggregate. On an event-sourced aggregate they are saved only through add(). Before, the second get() returned a copy of the stored record, and the events raised on the first copy were dropped.

To abandon a change, raise an exception so the UnitOfWork rolls back. To read the stored state, read the aggregate in a new UnitOfWork. A change that breaks an invariant needs neither, because Protean undoes the assignment when the check fails.

A QuerySet evaluated in one UnitOfWork transaction, or outside any, runs its query again when it is read in a different transaction. Before, it handed out copies of what it had cached: a copy of an aggregate the transaction tracked lost the events raised on it, and the cached values could be older than what was stored.

Code that loads an aggregate once per transaction sees no difference.

Why it was not put behind a flag

The old behaviour lost events for handlers that deduplicate by id and let a stale copy overwrite newer data. Reading a new copy inside a transaction lost events or raised a conflict with no other writer. No correct program could rely on any of these.

A failed IR build now fails check

Domain.check() used to log a failure to build the IR, with its traceback, and return the result without IR diagnostics. The status could be pass, and protean check printed "All checks passed".

Now the failure is one entry in errors. Its element is "to_ir" and its code is the exception class name, such as AttributeError. The status is fail. protean check exits 1, protean verify fails its check stage and exits 4, and the MCP check tool reports the error. The log keeps one warning line without the traceback.

Who is affected

  • Code that calls Domain.check() directly and treats a pass status as success. A domain whose IR cannot be built now returns fail.
  • A CI job that runs protean check or protean verify on such a domain. It passed before and fails now. Fix the element the error message names.

Why it was not put behind a flag

A check that could not build the IR did not run the IR rules, so a pass was not true. Reporting the failure is a correction.

Domain traversal loads modules in name order

When Domain.init() traverses the domain's folder, it now loads the root directory's files first, then each subdirectory in name order, with the files in each directory sorted by name. It used to load them in the order os.listdir returned, which depends on the filesystem.

Who is affected

A domain that registers enrichers from more than one traversed module, when the result depends on the order they run in. Enrichers run in registration order, and a module's enrichers register when the module loads, so the new load order can change that order. Two cases depend on it:

  • Two enrichers write the same field or the same metadata key. The one that runs last wins.
  • An aggregate enricher reads a value that another one sets. Aggregate enrichers change the same aggregate one after another, so swapping their order can change what is saved even when they write different fields.

Enrichers that each set their own fields and read nothing another one sets are not affected.

To keep a fixed order of your own, register the enrichers from one module, such as the module that creates the Domain, in the order you want.

Why it was not put behind a flag

The old order was not a contract. Two machines with different filesystems could already run the same enrichers in different orders. The IR now records enricher order, so it has to be the same on every machine.

A missing MySQL or MSSQL driver fails domain.init() with ConfigurationError

The MySQL and MSSQL providers now check for their driver when domain.init() builds them. When the driver cannot be imported, domain.init() raises a ConfigurationError. The message names the database's configured name, the driver and the extra to install:

Database 'default' uses a 'mysql+pymysql://' URL, but the 'pymysql' driver cannot be imported (No module named 'pymysql'). Install it with `pip install "protean[mysql]"`.

It used to raise the driver's own ModuleNotFoundError, from inside SQLAlchemy.

Which driver is checked depends on the database_uri:

  • mysql+pymysql:// and mariadb+pymysql:// are checked for PyMySQL, the driver the protean[mysql] extra installs.
  • A plain mysql:// or mariadb:// is checked for mysqlclient (MySQLdb), because SQLAlchemy connects through mysqlclient for those URLs. When mysqlclient is missing, the message asks you to add +pymysql to the URL and install protean[mysql]. When mysqlclient is installed, the URL works as before.
  • mssql+pyodbc:// and a plain mssql:// are checked for pyodbc. pyodbc installed without the unixODBC system library fails to import, so it raises ConfigurationError the same way.
  • A database_uri that names another driver, such as mysql+mysqldb:// or mssql+pymssql://, is not checked. It goes to SQLAlchemy as written, and a missing driver there still raises SQLAlchemy's error.
  • A database config with a module key is not checked either. Protean passes it to SQLAlchemy's create_engine(), which uses that module as the driver.

Who is affected

Code that catches ModuleNotFoundError or ImportError around domain.init() to detect a missing MySQL or MSSQL driver. Catch ConfigurationError instead. The original import error is on its __cause__.

If the driver SQLAlchemy uses for your URL is installed, nothing changes.

Why it was not put behind a flag

The old error came from deep inside SQLAlchemy and did not say which extra installs the driver. The MySQL and MSSQL providers now report a missing driver the way the PostgreSQL provider does.

The database_model element option is removed

@domain.aggregate, @domain.entity and @domain.projection no longer take a database_model option. The same goes for domain.register() on those elements. The option never had any effect. Protean accepted it, stored it and then built its own database model anyway, so a custom model passed this way was never used.

Passing the option now fails at registration:

# fragment
# Before
@domain.aggregate(database_model=UserModel)
class User:
    email: String(max_length=255, required=True)

ConfigurationError: Unknown option(s) {'database_model'}

Register the custom model with @domain.database_model instead, and name the element it belongs to with part_of:

from protean import Domain
from protean.core.database_model import BaseDatabaseModel
from protean.fields import String

domain = Domain(name="Accounts")


@domain.aggregate
class User:
    email: String(max_length=255, required=True)


@domain.database_model(part_of=User)
class UserModel(BaseDatabaseModel):
    pass

See Database Models for how a custom model maps columns to fields.

Who is affected

Code that passes database_model= to @domain.aggregate, @domain.entity, @domain.projection, or to domain.register() for one of those elements. Code that reads meta_.database_model on an element also breaks, since the key is gone from meta_.

If you already register custom models with @domain.database_model, nothing changes.

Why it was not put behind a flag

The option did nothing, so there is no old behaviour to keep. A flag would only keep a setting that looks like it works and does not.

mypy sees decorated elements as subclasses of their base

Under the mypy plugin, a class decorated with @domain.aggregate, @domain.entity or another element decorator is a subclass of its Protean base class, as it is at runtime. Before, the plugin copied the base's methods onto the class, so mypy rejected a decorated element wherever a signature named the base class.

Nothing changes at runtime.

Who is affected

Projects that run mypy with plugins = protean.ext.mypy_plugin. What can change in a mypy run:

  • A field or method that reuses the name of a base-class attribute with a different type is reported as an assignment error. It was a no-redef error before, so a # type: ignore[no-redef] on that line no longer covers it. pydantic's BaseModel defines methods such as json, copy and schema, and a field with one of those names already shadows the method at runtime. Rename the field.
  • A mixin listed as a base of a decorated class is dropped, because the decorator rebuilds the class on its Protean base alone. mypy now reports the mixin's methods as missing, which is what happens at runtime.
  • A decorated aggregate, entity, value object, command, event, projection or query with no fields now has an __init__ that takes no keywords, so passing one is a call-arg error.
  • A field declared in the annotation-only form (name: String(...)) is now typed Any and no longer reports valid-type. A # type: ignore[valid-type] or # type: ignore[call-arg] on such a line is no longer needed, and warn_unused_ignores reports it.

A value object's string part_of must name an aggregate

@domain.value_object(part_of="Product") now resolves the string to the Product aggregate class at domain.init(), as it already did for entities, events, commands, command and event handlers, application services and repositories. Before, the value object's meta_.part_of stayed the string "Product". For a value object that no ValueObject or ValueObjectList field embeds, protean ir then failed with an AttributeError, and protean check reported that AttributeError as an error.

The name is looked up among aggregates only. When no aggregate has that name, domain.init() raises ConfigurationError:

Unresolved references in domain `Shop`: `TierPrice` references `Prodct` via part_of

Who is affected

Domains where a value object's string part_of is misspelt, or names an entity instead of an aggregate. Those domains used to initialize and failed later, or never. Name the aggregate the value object belongs to, or pass the aggregate class:

# fragment
# Before: initialized, then failed inside the IR builder
@domain.value_object(part_of="Prodct")
class TierPrice:
    amount: Float(required=True)

# After: name the aggregate
@domain.value_object(part_of="Product")
class TierPrice:
    amount: Float(required=True)

One case worked before and now fails domain.init(). A value object that an entity embeds through a ValueObject field, with a string part_of naming that entity, built without errors because the IR placed it through the field. Name the entity's aggregate instead:

# fragment
# Before: initialized and built, though `Line` is an entity
@domain.value_object(part_of="Line")
class Discount:
    percent: Float()

# After: name the aggregate that owns `Line`
@domain.value_object(part_of="Order")
class Discount:
    percent: Float()

A string part_of that names an aggregate keeps working, and the IR build no longer fails on it.

Why it was not put behind a flag

The old behaviour left an unresolved name in place without an error or warning, and the IR build crashed on it. Entities, events, commands, command and event handlers, application services and repositories already reject a part_of they cannot resolve at domain.init(). Value objects now do the same.

DomainFixture.setup() warns about a broker that does not answer

DomainFixture.setup() now pings every configured broker once, after domain.init() and before it creates the database schema. For each broker that does not answer, it emits a UserWarning that names the broker and its URI, with any password shown as ***:

UserWarning: Broker 'default' at redis://127.0.0.1:6379/15 did not answer a
ping. Tests that publish to it will fail, and the per-test reset cannot clear it.

When the broker has no URI, or its URI cannot be shown without risking part of the password, the warning names the broker's provider instead, as in Broker 'default' (redis provider).

Before, a suite whose tests never published to a down broker passed with no visible sign of it. A wrong password or a denied command also makes the ping fail.

Who is affected

Suites that turn warnings into errors, with pytest -W error::UserWarning, -W error, or filterwarnings = ["error"] in the pytest configuration. In those suites, a broker that does not answer now makes setup() raise the warning, so the session fixture fails before any test runs. The raise comes before the schema is created, so no tables are left behind.

Start the broker before the run, or point the domain at a broker that answers. To keep the old behaviour in a suite that runs without a broker, ignore this one warning:

[tool.pytest.ini_options]
filterwarnings = [
    "error",
    "ignore:Broker .* did not answer a ping:UserWarning",
]

Suites that leave warnings as warnings see one entry per down broker in pytest's warnings summary, and nothing else changes.

Outbox processors run only for providers an aggregate is stored in

When the outbox is enabled, the engine used to run an outbox processor, and one external processor per broker in outbox.external_brokers, for every managed database provider. That included the built-in memory provider, which the default configuration always defines. A domain that stores everything in default on Postgres ran these four:

outbox-processor-default-to-default
outbox-processor-memory-to-default
outbox-processor-default-to-global-external
outbox-processor-memory-to-global-external

Nothing writes to the memory outbox in that domain, so the two memory processors had nothing to publish. The engine now runs processors only for a managed provider that at least one concrete aggregate names in its provider option. Event-sourced aggregates count. Abstract aggregates, projections and internal elements do not.

Who is affected

Anyone who reads outbox rows in /readyz checks.subscriptions or in protean subscriptions status. Rows for providers that no aggregate uses are gone, so total drops by that many. The response shape does not change. A dashboard or alert that expects a fixed row count, or names a memory processor, needs updating.

If you move the last aggregate off a provider, rows already pending in that provider's outbox have no processor to publish them. The engine logs an engine.outbox_rows_without_processor warning at startup that names the provider and the number of unpublished rows. Publish or clear those rows before the move, or keep an aggregate on that provider until they drain.

The multi-worker Supervisor waits for the drain window

protean server --workers N runs a Supervisor that stops its workers on shutdown. It used to send SIGKILL to any worker still running 30 seconds after SIGTERM, whatever server.drain_timeout said. A worker with a drain window above 25 seconds could be killed before its in-flight handlers finished.

The Supervisor now reads server.drain_timeout from the project config and waits for the drain window plus 5 seconds, and never less than 30 seconds. With the default window of 10, nothing changes. The engine's engine.drain_timeout_at_or_above_supervisor_kill_timeout startup warning is removed, because no fixed 30-second limit is left to warn about.

Who is affected

  • Anyone with drain_timeout above 25 seconds and more than one worker. A worker that does not stop now gets SIGKILL later: at 50 seconds for a window of 45, instead of 30.
  • Anyone whose process manager has its own stop deadline, such as systemd's TimeoutStopSec or Kubernetes' terminationGracePeriodSeconds. Set it above the Supervisor's wait, or the orchestrator kills the server first. See server.drain_timeout.
  • Anyone who alerts on the removed warning. Nothing logs it any more.

part_of must name the right kind of element

Protean now checks the kind of element each part_of names. A class target of the wrong kind raises IncorrectUsageError when the element registers. A string target resolves only against the allowed kinds, so a name of the wrong kind fails domain.init() with the unresolved-reference ConfigurationError. In 0.18 a database model takes only a class target. From 0.19 it also accepts a string target, resolved the same way.

Element Valid part_of target
value object, application service, repository, event-sourced repository, command handler, event handler an aggregate
entity an aggregate or an entity (a nested entity)
event, command an aggregate or a process manager
database model an aggregate, an entity or a projection
query, query handler a projection
domain service a list of two or more aggregates (unchanged)

For every element except a repository, the message names the element, the target and the kinds it accepts. A repository keeps its existing message, "Repository X can only be associated with an Aggregate". For example:

Event `OrderPlaced` can only be associated with an Aggregate or a Process Manager; `Line` is an Entity

An entity whose part_of is an abstract aggregate gets the same error, ending in "is abstract". This holds for a class and for a string name, and the string form fails at domain.init(). An entity whose part_of is a base class such as BaseAggregate gets the same error. These cases used to fail with a bare AssertionError. Other elements may still belong to an abstract aggregate.

Three string forms that failed before now work. An entity's string part_of can name a parent entity, as the class form already allowed. An event's or command's string part_of can name a process manager. An event-sourced repository's string part_of resolves to its aggregate, where it used to raise AttributeError.

When a name belongs to both an aggregate and a process manager, an event's or command's string part_of resolves to the aggregate, as before. An entity chain that loops back on itself (an entity naming itself, or two entities naming each other) fails domain.init() with a ConfigurationError that names the loop. Before, the string form failed as unresolved and the class form made domain.init() hang.

Who is affected

Domains that registered an element with an owner of the wrong kind. These registered before, and most of them did not do what the configuration said:

  • An application service tied to an entity, a value object or a projection.
  • A database model tied to a value object.
  • An event or command tied to an entity. raise_ on both the entity and its aggregate rejected such an event, so it could never be raised.
  • A value object tied to anything other than an aggregate. The value object sat in no aggregate's cluster.

Some setups that ran before are now rejected too:

  • An event handler tied to a process manager. It subscribed to the process manager's stream category.
  • A command handler tied to a process manager. It handled commands tied to that process manager.
  • An application service or a database model tied to a process manager.

Most other wrong targets used to fail later with an AttributeError, a TypeError or a bare AssertionError that did not mention part_of. They now raise the IncorrectUsageError above. A query or query handler tied to a class that is not a projection already failed cleanly at domain.init(). It now fails earlier, at registration, with the message above.

An event belongs to the aggregate, which owns the stream, or to the process manager that raises it. To say which entity an event concerns, tie the event to the aggregate and carry the entity's identity in the payload:

# fragment
# Before: registered, but the event could never be raised
@domain.event(part_of=Line)
class LineDiscounted:
    percent: Float()

# After: the aggregate owns the event; the payload names the entity
@domain.event(part_of=Order)
class LineDiscounted:
    line_id: Identifier(required=True)
    percent: Float()

For the other cases, point part_of at the aggregate (or, for a database model, the aggregate, entity or projection) the element works on.

Why it was not put behind a flag

Most of these configurations were accepted without doing what they said, or failed later with an error that did not name the cause. The error at registration names the element and the target, so you know which line to fix.

The audit-domain skill reports every finding

The audit-domain teaching skill used to tell the agent to drop some signals as harmless: an aggregate with many cohesive fields, or a short handler that orchestrates one aggregate. It now tells the agent to report every signal at its severity. A case it used to drop appears as a LOW finding, with the context in its description.

Who is affected

Anyone who compares an audit report from 0.17 with one from 0.18. The newer report can list more LOW findings for the same code. Those come from the reporting change, not from a new problem in the domain. The CRITICAL, HIGH and MEDIUM findings are unchanged.

The teaching skills store money as Decimal

The teaching skills and the pack's rules used to declare most money fields as Float. They now declare a money field as Decimal(precision=19, scale=4), because a float cannot hold most cents exactly. The examples also pass money as a decimal, so a method takes amount: Decimal and a test compares with Decimal("45.00").

The refactor-extract-value-object skill also changes the field's type when it moves a Float amount into a Money value object. The Money it generates multiplies by an int or a Decimal, so a factor such as a tax rate is written as Decimal("0.08").

Who is affected

Anyone who has an agent add money fields with the skills. Code the skills wrote before 0.18 is unchanged. New code declares money as Decimal, so:

  • On a relational database, a field that changes from Float to Decimal needs a migration that changes the column to a numeric type.
  • Python does not combine a Decimal with a float, so amount * 0.08 and balance += 1.5 raise TypeError. Use Decimal("0.08") and Decimal("1.5").
  • A Decimal field rejects NaN and infinity, and a value with more than four decimal places. A rule that used to catch NaN in an invariant now sees the field's ValidationError first.
  • A Decimal does not equal most float literals, so a test that checks total == 59.98 fails. Compare with Decimal("59.98").

Changes in 0.18.1

eventstore verify exits 2 when it cannot connect

protean eventstore verify used to exit 1 with a logged traceback when it could not connect to the event store. Under --json, stdout stayed empty.

It now exits 2, the code for a usage or environment error:

  • Under --json, stdout holds one result envelope with status set to error and the connection error under data.error.
  • Without --json, the connection error is printed as one line on stderr, with no traceback.

A script that checked for exit 1 no longer gets it when the command cannot connect. Other errors during the scan still exit 1 with a traceback. If a CI job treated any non-zero exit as a failure, nothing changes for it.

Value objects compare their field values

A value object used to compare and hash its to_dict() output, where a Decimal becomes its string. Two value objects holding Decimal("9.99") and Decimal("9.9900") were unequal, hashed differently, and took two entries in a set.

== and hash() now use the field values as the instance holds them. Those two value objects are now equal and hash the same, and so is a value object that nests either of them. The same holds for any field whose values are equal in Python but render differently, such as aware datetimes at the same instant with different UTC offsets. A value object of another class, including a subclass, is still never equal.

to_dict() is unchanged and keeps each amount's scale, so the two value objects above still render as "9.99" and "9.9900".

A field's default now goes through the same validation as an assigned value. Integer(default="5") stores 5, not "5", and a default that breaks its own field's rules raises ValidationError when the value object is built.

No action is needed unless your code relied on the old inequality. Code that quantized amounts only so that value objects would compare equal can drop the quantize step. Keep it where the stored or serialized form matters.

eventstore verify reports a message whose body is not a JSON object

Message DB allows a NULL data column. protean eventstore verify used to raise TypeError on such a row against Message DB. The memory store passed the same row without a report.

Both stores now report the row as a malformed_message violation with its stream and position, and verify exits 1. Message DB also reports a row whose data holds a JSON array, string or number, which it used to pass. A snapshot row keeps its own check and is still reported as malformed_snapshot. A message whose body is an empty object is not reported.

The message-db extra now requires message-db-py 0.3.5 or later. Upgrade the package if your lock file pins an earlier version.

A CI job that ran verify against a store holding such a row now fails. Find the rows from the stream and position in the report, then repair or remove them.

A nested handler's conflict reaches the enclosing handler or the caller

@handle retries a handler on ExpectedVersionError, and on a transient error when transient retry is on. Before 0.18.1 it did so even when the handler ran inside a Unit of Work that was already active. That happens when another handler or a @use_case processes a command with domain.process(..., asynchronous=False), or when your code opens a UnitOfWork around the call. The handler's Unit of Work joins the outer one, so the failed attempt had already marked the whole transaction for rollback. The retry then succeeded, the outer commit rolled the transaction back, and the caller got a normal return with no error.

A handler that runs inside an active Unit of Work now runs once and lets the exception propagate:

  • When the outermost caller is a @handle handler whose own policy covers the exception (version retry on, or a transient policy that lists it), that handler runs the whole chain again in a fresh Unit of Work. The outer handler body runs again too, so it must be safe to re-run.
  • When a @use_case, a process manager or your own UnitOfWork encloses the call, the caller receives ExpectedVersionError, or the transient exception, and the transaction is rolled back.
  • The nested handler's own retries and retry_exceptions are not used. Only the outermost handler's policy applies.
  • A command processed synchronously with an idempotency_key inside an outer Unit of Work records its key when the outer transaction commits. If the transaction rolls back, the key is not recorded, so the outer retry runs the command again.
# fragment
@use_case
def record_result(self, notification_id: str) -> None:
    current_domain.process(
        RecordDispatchResult(notification_id=notification_id),
        asynchronous=False,
    )
    # Before 0.18.1, a version conflict in the RecordDispatchResult handler
    # was retried inside this use case's transaction and then lost.
    # It now reaches the caller of record_result().

A caller of such a use case that wants a retry calls the use case again. Handlers that run with no Unit of Work active keep both retries, as do event handlers run synchronously after a command commits.

Event-store subscriptions report their exhausted positions as DLQ depth

dlq_depth used to be zero for every event-store subscription, even when its failed-* stream held exhausted positions. protean subscriptions status, /readyz, the observatory and the protean_subscription_dlq_depth gauge all reported it that way.

dlq_depth now counts the positions whose latest record in the subscription's failed-* stream is Exhausted, the same positions protean eventstore dlq list shows. A position stops being counted once a replay or purge writes a later record for it. A replay that finds an expired command writes no record, so that position stays counted. An alert on dlq_depth > 0 can start to fire for an event-store subscription. Clear the positions with protean eventstore dlq replay or purge.

On the observatory overview, the health banner adds these positions to its count of DLQ messages, and the subscription summary counts an event-store subscription with exhausted positions under DLQ. The DLQ tab the banner links to and the protean dlq commands list only broker {stream}:dlq streams. Use protean eventstore dlq list to see and clear the positions behind an event-store subscription's count.

protean eventstore dlq inspect used to show a position that was replayed or purged after it was exhausted. It now reports such a position as not found, the same as replay and purge.

Outbox reconciliation repairs only event-sourced aggregates' events

The Unit of Work appends events to the event store before it commits the aggregate's row and its outbox rows. When that commit rolled back, a state-based aggregate's event stayed in the event store while the aggregate's change and its outbox rows were not saved. protean outbox reconcile and the reconcile that runs when the server starts rebuilt the outbox rows for such an event, so consumers and external brokers received an event for a change that was never saved.

Reconcile now rebuilds outbox rows only for events of event-sourced aggregates. It skips every state-based aggregate's event, whether or not the aggregate's row exists. A provider with no event-sourced aggregate is not read at all.

On a provider that writes without a transaction, such as Elasticsearch, the aggregate row can be saved while its outbox row is lost. Reconcile used to rebuild that outbox row. It no longer does, so the event is not published. No code change is needed for this. A script or runbook that relied on reconcile to recover such rows after a crash no longer gets them back.

See Recover from a crash: reconciliation.

An outer commit after a nested failure raises

A Unit of Work started inside another one joins it. When the nested one fails, the whole transaction is marked for rollback. Before 0.18.1, if your code caught the error and let the outer block end normally, the outer commit rolled the transaction back, logged a warning and returned. The caller saw a normal return, and nothing was saved.

The outer commit now rolls back and raises:

  • When the nested Unit of Work failed with an exception, the commit raises that same exception object.
  • When a nested Unit of Work called rollback() with no exception, the commit raises TransactionError, with a message that the transaction was marked rollback-only.

Who is affected

  • A handler or a @use_case that calls domain.process(..., asynchronous=False), catches the exception the inner handler raised, and carries on.
  • Code that opens a UnitOfWork inside another one, catches the error the inner block raised, and carries on.
  • Code that calls rollback() on a nested UnitOfWork and expects the outer block to return normally.
# fragment
@handle(PlaceOrder)
def place(self, command: PlaceOrder) -> None:
    current_domain.repository_for(Order).add(Order(id=command.order_id))
    try:
        current_domain.process(
            ReserveStock(order_id=command.order_id), asynchronous=False
        )
    except OutOfStock:
        pass
    # Before 0.18.1: the handler returned, and the order was silently not saved.
    # Now: OutOfStock reaches the caller, and the order is not saved.

A @use_case that catches a nested ExpectedVersionError now gets it again when its Unit of Work commits. A @handle handler retries the whole chain in a fresh Unit of Work when its own retry policy covers the exception, as described in A nested handler's conflict reaches the enclosing handler or the caller. An exception outside that policy propagates after one run.

How to fix it

Let the exception propagate out of the outer block, so the caller handles it. When the outer work must be saved even if the inner step fails, run the step that can fail outside the outer Unit of Work, for example in an event handler that reacts to an event the outer aggregate raises.

A shutdown finishes the batch in hand, and a start claims what stopped workers left

On SIGTERM, the engine used to refuse every message left in a batch it had already read. Those messages stayed pending in the Redis consumer group under the stopped worker's consumer name, which no later worker reads, so they were lost.

The engine now keeps handling a batch it has read until server.drain_timeout ends, and cancels it then. A partitioned stream subscription (sequential_by) is the exception: it cancels its partition workers as soon as shutdown starts. A message the worker did not handle stays pending. On startup, each stream subscription and broker subscription on Redis claims the group's entries that have stayed unacknowledged for at least server.reclaim_idle_seconds (120 by default), then handles them. A claimed message counts its earlier deliveries toward max_retries.

What to check when you upgrade:

  • Handlers can now run after SIGTERM, inside the drain window. A handler must not assume that no message arrives once shutdown has begun.
  • Keep server.reclaim_idle_seconds above circuit_breaker_reset_seconds and above your longest batch. A running worker whose entries stay unread longer than the threshold can lose them to a worker that starts, and both may handle them.
  • A worker that starts claims only the entries last delivered at least server.reclaim_idle_seconds earlier. A later start claims the rest. A partitioned subscription (sequential_by) does not use this threshold: the partition's next owner claims its entries through the lease handover.
  • POST /drainz now returns "drained": true once the worker has no batch in flight. To stop a worker without cutting a batch off, repeat the POST until drained is true, then send SIGTERM. See Draining: POST /drainz.
  • To keep the old behaviour of never claiming, set server.reclaim_idle_seconds higher than any entry can wait, such as 31536000 (one year). A subscription that claims messages on start logs how many it claimed.

A process manager's dates, datetimes and decimals keep their type after a reload

A process manager stores its fields as JSON in each transition event, and is rebuilt from those events when the next event arrives. Two things changed.

  • Saving. A Decimal field used to fail the save with TypeError: Object of type Decimal is not JSON serializable, which rolled back the unit of work. It is now stored as a string, such as "100.50".
  • Reloading. A Date, DateTime or Decimal field used to come back as the stored string, such as "2025-06-15T10:30:00". It now comes back as a date, datetime or Decimal. A stored value that no longer fits the field's type, for example one written before the field's type changed, is kept as the stored value so the process manager still loads.

Who is affected

A handler that read a reloaded date or datetime field as a string, for example by parsing it with datetime.fromisoformat(), now receives a datetime and the parse raises TypeError. Use the value directly. A process manager with a Decimal field works for the first time.

Outbox reconciliation repairs every lost row in its window

protean outbox reconcile and the reconcile that runs when the server starts rebuild the outbox rows of an event-sourced aggregate's event when the event is in the event store and its internal outbox row is missing. They used to check only the newest such event in the window. When that event had its row, they repaired nothing. So an event whose row was lost in a crash stayed unpublished once another process in the same domain committed a later event.

They now rebuild the rows of every such event in the window that is newer than the shorter of published_retention_hours and abandoned_retention_hours. A published event also gets back one row per broker in [outbox].external_brokers.

No action is needed for most domains. After the upgrade:

  • The first restart, or the first protean outbox reconcile, can publish events whose rows were lost before the upgrade and that the old check missed. Consumers can receive these events after the events committed after them.
  • For a sequential_by key, a rebuilt event is delivered after the events of later commits for the same key when their rows are in the same priority lane. A rebuilt row has priority 0.
  • If you raised the shorter retention after cleanup ran with a lower value, the next run rebuilds the rows cleanup deleted in between, and those events are published again. Raise the shorter retention only when your consumers can take a repeated event.
  • When an event-sourced aggregate is on the provider, every run reads the whole --limit window, 1000 messages by default, and looks up the outbox rows of its events in batches. Before, a run with nothing to repair read one message.

See Recover from a crash: reconciliation.

Datetimes are compared and stored as UTC

A query that filtered a DateTime field with a value in another UTC offset could return the wrong rows. A row saved at 12:00+00:00 and filtered with lte=13:00+02:00 (11:00 UTC) was returned, because the provider compared the wall-clock digits. The memory provider compared the string form of the two values. SQLite, MySQL, MariaDB and SQL Server store a datetime with no offset, so they wrote the caller's wall-clock time and compared it the same way. PostgreSQL gave the right answer.

Every provider now compares the instant:

  • Memory. exact, in, lt, lte, gt, gte, any and overlap compare two datetimes as UTC instants. A naive datetime on either side is read as UTC. contains, startswith and endswith still compare the string form, and so does a datetime filtered with a string.
  • SQLite, MySQL, MariaDB and SQL Server. A DateTime column converts each value to UTC before it writes or compares it, and reads a naive value as UTC. A load returns a UTC-aware datetime, where it used to return a naive one. The column definition is unchanged, so existing tables need no schema change.

PostgreSQL and Elasticsearch are unchanged. A custom SQLAlchemy model keeps the column types it declares.

Who is affected

Code that compares a loaded datetime with a naive one, such as datetime.now(), changes on SQLite, MySQL, MariaDB and SQL Server. An ordering comparison (<, <=, >, >=) now raises TypeError. An equality check raises nothing: == always returns False and != always returns True. Compare with an aware value, such as datetime.now(UTC).

Rows written before the upgrade keep the wall-clock time they were saved with. A row saved with the +00:00 offset, or as a naive UTC value, already holds UTC and needs nothing. Two kinds of row are now read as UTC and are off by their offset:

  • a row saved with any other offset, such as +02:00;
  • a row saved as a naive local time, such as a field with default=datetime.now, which is off by the server's offset.

How to fix it

If your application wrote one known offset, such as +02:00, convert the column once. Stop every writer before the upgrade and keep it stopped until the conversion finishes: a row the upgraded version writes already holds UTC, and the statement would shift it too. Each statement below converts placed_at on an orders table from +02:00 to UTC:

-- MySQL and MariaDB
UPDATE orders SET placed_at = CONVERT_TZ(placed_at, '+02:00', '+00:00');

-- SQL Server
UPDATE orders SET placed_at = DATEADD(hour, -2, placed_at);

-- SQLite; datetime() gets only the whole seconds, because some SQLite
-- versions round the fraction into the seconds, and substr keeps the fraction
UPDATE orders
SET placed_at = datetime(substr(placed_at, 1, 19), '-2 hours') || substr(placed_at, 20)
WHERE placed_at IS NOT NULL;

If the offset changed over time, for example with daylight saving time, run one statement per period and limit each with a WHERE clause on the column.