Skip to content

Migrating to 0.19

From 0.18 to 0.19

This page lists the changes in 0.19 that may need action when you upgrade.

On this page:

Outbox claim order

The outbox processor used to claim rows by priority alone. Rows of the same priority came back in whatever order the database returned them, so on PostgreSQL one aggregate's events could be published out of commit order.

claim_batch and find_unprocessed now order rows by:

  1. priority, highest first.
  2. created_at, oldest first.
  3. sequence_number, lowest first.
  4. id, lowest first.

The last key, id, says nothing about commit order. It only makes the order the same on every run when the other three keys tie.

The Unit of Work now writes sequence_number on every outbox row. It holds the event's position within its commit: 0 for the first event, 1 for the next, and so on. Each database provider has its own outbox table, and the count starts at 0 for each provider in the commit. Within a provider, the Unit of Work numbers events one aggregate at a time, each aggregate's events in the order it raised them. When two rows share a created_at, the row with the lower number is claimed first. Every outbox row written for one event (the internal row and one row per external broker) carries the same number. The startup sweep that rebuilds lost outbox rows from the event store numbers the rows it rebuilds in event store order.

When can the claim order differ from commit order?

created_at is the time the Unit of Work built the row, just before the commit. One aggregate's events go out in commit order, because a second commit on that aggregate has to load it after the first commit finished. Rows of different aggregates go out in the order their rows were built, which can differ from the order their commits finished.

The claim order can also differ from commit order in these cases:

  • A higher-priority message is claimed ahead of an earlier lower-priority one.
  • A retried row, or a second worker, publishes events out of order. This does not apply to a category partitioned with sequential_by: a later row of a key waits until the key's earlier rows have published (see below).
  • created_at comes from the clock of the host that wrote the row. If two hosts' clocks disagree, or a clock steps back, a later commit can get an earlier created_at and be claimed first.
  • On a SQL Server outbox table created before 0.19 and not converted, created_at is a DATETIME, which rounds to about 3 milliseconds. Rows of two commits that land in the same tick share a created_at, and since each commit numbers its events from 0, their rows can interleave. See SQL Server stores DateTime as DATETIME2.

Rows written before the upgrade

Rows already in the outbox have a NULL sequence_number. They are ordered by created_at first, like every other row. Among rows with the same created_at, a NULL sequence_number sorts after every number, on the in-memory adapter and every SQL database.

To place NULLs the same way everywhere, ordering on a nullable column now puts NULLs last when ascending and first when descending on SQLite, MySQL, MariaDB and SQL Server. PostgreSQL and the in-memory adapter already did this. The rule applies to every repository query that orders by a nullable field, not only to the outbox. If a query relied on NULLs coming first in an ascending sort on one of those databases, it now gets them last.

Rebuild ix_outbox_active

The partial polling index ix_outbox_active now covers the claim order: status, priority DESC, created_at, sequence_number.

  • New deployments: Nothing to do. Tables Protean creates on 0.19 get the new index. A schema you manage yourself, with Alembic or by hand, needs the rebuild below.
  • Existing deployments: Protean does not change an existing table's indexes. Claims work without the rebuild, but the old index does not cover the new sort. Drop and recreate the index once.

The statements below use the default table name outbox.

=== "PostgreSQL"

```sql
DROP INDEX IF EXISTS ix_outbox_active;
CREATE INDEX ix_outbox_active
    ON outbox (status, priority DESC, created_at, sequence_number)
    WHERE status IN ('pending', 'failed');
```

On a busy table, use `DROP INDEX CONCURRENTLY` and
`CREATE INDEX CONCURRENTLY` to avoid blocking writes. Neither runs inside
a transaction block.

=== "SQLite"

```sql
DROP INDEX IF EXISTS ix_outbox_active;
CREATE INDEX ix_outbox_active
    ON outbox (status, priority DESC, created_at, sequence_number)
    WHERE status IN ('pending', 'failed');
```

=== "SQL Server (full index)"

Protean builds a full index on SQL Server, without the `WHERE` filter.

```sql
DROP INDEX IF EXISTS ix_outbox_active ON outbox;
CREATE INDEX ix_outbox_active
    ON outbox (status, priority DESC, created_at, sequence_number);
```

=== "MySQL / MariaDB (full index)"

MySQL and MariaDB have no partial indexes, so Protean builds a full
index there.

```sql
DROP INDEX ix_outbox_active ON outbox;
CREATE INDEX ix_outbox_active
    ON outbox (status, priority DESC, created_at, sequence_number);
```

Outbox rows wait for their partition key's earlier rows

A handler category that declares sequential_by gives each outbox row a partition key. The outbox processor used to claim and publish each row on its own. If a key's first row failed, its second row could still publish, and the consumer saw the two events out of commit order.

The claim now groups partitioned rows by target broker, stream category, partition key and priority lane. A row is claimed only when every earlier row of its group has been published, or is claimed in the same call. An earlier row holds the later ones while it is pending, processing, failed or abandoned. Rows without a partition key are claimed as before.

Add the stream_category column

The outbox table has a new stream_category column. The Unit of Work copies it from the event's metadata.domain.stream_category when it writes a row. A new index, ix_outbox_hold_back, backs the lookup of a row's earlier group rows.

  • New deployments: Nothing to do. Tables Protean creates on 0.19 have the column and the index.
  • Existing deployments: Protean does not change an existing table. Add the column before you start 0.19, because every outbox query reads it. Then create the index.

The statements below use the default table name outbox.

=== "PostgreSQL"

```sql
ALTER TABLE outbox ADD COLUMN stream_category VARCHAR(255);
CREATE INDEX ix_outbox_hold_back
    ON outbox (target_broker, stream_category, partition_key, status);
```

On a busy table, use `CREATE INDEX CONCURRENTLY` to avoid blocking
writes. It does not run inside a transaction block.

=== "SQLite"

```sql
ALTER TABLE outbox ADD COLUMN stream_category VARCHAR(255);
CREATE INDEX ix_outbox_hold_back
    ON outbox (target_broker, stream_category, partition_key, status);
```

=== "SQL Server"

```sql
ALTER TABLE outbox ADD stream_category VARCHAR(255);
CREATE INDEX ix_outbox_hold_back
    ON outbox (target_broker, stream_category, partition_key, status);
```

=== "MySQL / MariaDB"

```sql
ALTER TABLE outbox ADD COLUMN stream_category VARCHAR(255);
CREATE INDEX ix_outbox_hold_back
    ON outbox (target_broker, stream_category, partition_key, status);
```

Rows already in the outbox

Rows already in the outbox have a NULL stream_category. The claim treats them as rows without a partition key: they are not held, and they do not hold later rows.

To keep a key in order across the upgrade, fill the column on the rows that are not yet published. Protean stores each row's metadata pickled, so plain SQL cannot read the stream category out of it. Run this function once, with 0.19 installed, after the ALTER TABLE and before you start the server:

from protean.utils.outbox import OutboxStatus

UNPUBLISHED = [
    OutboxStatus.PENDING.value,
    OutboxStatus.PROCESSING.value,
    OutboxStatus.FAILED.value,
    OutboxStatus.ABANDONED.value,
]


def fill_stream_category(provider: str = "default", page_size: int = 500) -> int:
    filled = 0
    skipped = 0
    with domain.domain_context():
        repo = domain._get_outbox_repo(provider)
        while True:
            rows = (
                repo.query.filter(
                    partition_key__isnull=False,
                    stream_category__isnull=True,
                    status__in=UNPUBLISHED,
                )
                .order_by(["created_at", "id"])
                .offset(skipped)
                .limit(page_size)
                .all()
                .items
            )
            if not rows:
                return filled
            for row in rows:
                meta = row.metadata_.domain
                if meta is None or meta.stream_category is None:
                    skipped += 1
                    continue
                row.stream_category = meta.stream_category
                repo.add(row)
                filled += 1

It returns the number of rows it filled. Pass the provider name when the outbox lives on a provider other than default. A row whose metadata has no stream category is left NULL.

The fill also sets the column on abandoned rows, and a filled abandoned row halts its key as soon as 0.19 starts (see the next section). That includes every row the key gets after the upgrade. Before you run the fill, list the abandoned rows that have a partition key, and retry or skip the ones you do not want to halt their key:

SELECT id, partition_key, created_at, last_error
FROM outbox
WHERE status = 'abandoned' AND partition_key IS NOT NULL;

A partition key can halt

A key whose earliest row is abandoned stops: none of its later rows publish until someone deals with that row. A key whose earliest row is failed waits until the row's retry is due. When the row runs out of retries it becomes abandoned and the key stops. Other keys, rows without a key, and the other priority lane keep publishing.

The cleanup job no longer deletes abandoned rows that have a partition key, since deleting one would let the key's later rows publish out of order.

To restart a halted key, either retry the abandoned row or skip it. The outbox guide shows how to find abandoned rows. Retrying sends the row again with a fresh retry budget:

UPDATE outbox
SET status = 'pending', retry_count = 0, locked_by = NULL, locked_until = NULL
WHERE id = '<row id>';

Skipping the row marks it published without sending it. Its event is dropped for that broker, and the key's next row publishes on the next tick:

UPDATE outbox
SET status = 'published', published_at = CURRENT_TIMESTAMP,
    locked_by = NULL, locked_until = NULL
WHERE id = '<row id>';

Do not delete the row. Outbox reconciliation runs when the server starts and can rebuild a deleted row of a recent event as pending. The rebuilt row would then publish after the key's later rows.

SQL Server stores DateTime as DATETIME2

On SQL Server, a DateTime field used to map to a DATETIME column, which rounds to increments of about 3 milliseconds. Protean writes microsecond-precision timestamps, so two values written less than 3 ms apart could read back equal, and a value read back could differ from the one written.

A DateTime field now maps to DATETIME2, which keeps 100-nanosecond steps. Date fields still map to DATE. Other databases are unchanged.

  • New deployments: Nothing to do. Tables Protean creates on 0.19 get DATETIME2 columns.
  • Existing deployments: Protean does not change an existing table, so its columns stay DATETIME and keep rounding. Convert them.

The column that matters most is the outbox's created_at. The outbox claims rows in created_at order, so rounding it can publish two commits' events interleaved (see When can the claim order differ from commit order?).

Convert a column

ALTER TABLE <table> ALTER COLUMN <column> DATETIME2 NULL;

Write NULL for an optional field and NOT NULL for a required one. ALTER COLUMN sets the column's nullability to what the statement says, and makes it nullable when the statement says nothing.

SQL Server refuses to alter a column that an index or a constraint uses. Drop an index with DROP INDEX <index> ON <table> and create it again after the ALTER. A DateTime(unique=True) field has a UNIQUE constraint with a name SQL Server generated. Find the name in sys.key_constraints, drop it with ALTER TABLE <table> DROP CONSTRAINT <name>, and add it back with ALTER TABLE <table> ADD UNIQUE (<column>) after the ALTER.

Convert the outbox

All five outbox timestamps are nullable, and ix_outbox_active uses created_at. The statements below use the default table name outbox:

DROP INDEX IF EXISTS ix_outbox_active ON outbox;
ALTER TABLE outbox ALTER COLUMN created_at DATETIME2 NULL;
ALTER TABLE outbox ALTER COLUMN published_at DATETIME2 NULL;
ALTER TABLE outbox ALTER COLUMN last_processed_at DATETIME2 NULL;
ALTER TABLE outbox ALTER COLUMN next_retry_at DATETIME2 NULL;
ALTER TABLE outbox ALTER COLUMN locked_until DATETIME2 NULL;
CREATE INDEX ix_outbox_active
    ON outbox (status, priority DESC, created_at, sequence_number);

This also does the ix_outbox_active rebuild for SQL Server.

Rows already in the table keep the point in time DATETIME stored, but they can read back with different microseconds. DATETIME stores steps of 1/300 second, and DATETIME2 shows them in full, so .123 reads back as .123333. Code that compares against a timestamp it read before the conversion no longer finds a match.

The processed_message table, which records the messages each handler has handled, has a processed_at column that also stays DATETIME. Convert it the same way, with ALTER TABLE processed_message ALTER COLUMN processed_at DATETIME2 NULL. Until then, the cutoff for removing old records can be off by up to 3 ms.

eventstore verify exits 2 when it cannot connect

protean eventstore verify used to exit 1 with a logged traceback when it could not connect to the event store. Under --json, stdout stayed empty.

It now exits 2, the code for a usage or environment error:

  • Under --json, stdout holds one result envelope with status set to error and the connection error under data.error.
  • Without --json, the connection error is printed as one line on stderr, with no traceback.

A script that checked for exit 1 no longer gets it when the command cannot connect. Other errors during the scan still exit 1 with a traceback. If a CI job treated any non-zero exit as a failure, nothing changes for it.

Generated .proto files import each other

protean schema generate --format protobuf used to copy every value object and entity an element referenced into that element's file, as a sibling message. The referenced element also had its own file. When both were in the same package, protoc rejected the tree with "already defined" when it compiled the files together.

Each value object and entity is now defined only in its own file. A file that uses one adds an import of that file, by its path under schemas/:

syntax = "proto3";
package shop.ordering;
import "Order/entities/Package.v1.proto";

message Order {
  string id = 1;
  repeated .shop.ordering.Package packages = 2;
}

What this changes for you:

  • Compiling the files. Pass -I <output>/schemas to protoc, so it can find the imported files. This applies even when you compile one file at a time.
  • Type names. A referenced type is named by its full name with a leading dot (.shop.ordering.Package). A value object from another module keeps its own full name, such as shared.kernel.Money. It is no longer copied into the referencing element's package, so generated code that used the local copy (sales.ordering.Money) must use shared.kernel.Money.
  • Nested messages. A value object that belongs to no aggregate cluster, such as one only projections use, has no file. Entities that refer to each other through HasOne or HasMany would import each other in a cycle. In both cases the type is defined as a nested message inside the file that uses it, such as .shop.ordering.CustomerCard.ShippingAddress.
  • Timestamps. Dates are written as .google.protobuf.Timestamp, with a leading dot, so a package with a google part cannot shadow it.
  • Name clashes. protean schema generate now fails with an error when two elements would be written to the same file, in any format. Before, the second file silently replaced the first. Rename one of the two classes.

Field names, field numbers and labels are unchanged, so the bytes on the wire are the same. Tools that compare message names, such as a schema registry's compatibility check, Any type URLs and generated class paths, see the new names.

generate_proto_schema() called without the new import_path argument, and generate_proto_schemas(), still return one self-contained file per element with sibling messages.

Deprecation versions must be strings

deprecated= used to accept a version written as a number and turn it into a string. A number cannot keep a trailing zero, so 0.20 became "0.2". That sorts before 0.18, so diff_ir(..., current_version="0.18") classed a field due for removal in 0.20 as safe to remove. protean ir diff and the protean-check-compat hook pass no current version, so they class every deprecated removal as premature and were not affected. A declared since: 0.20 showed as "deprecated since v0.2" in diagnostics.

deprecated= on an element, a field or a FieldSpec now raises ConfigurationError when since or removal is not a string. removal: None still means there is no removal version.

# fragment
# Accepted before and read as "0.2". Now raises ConfigurationError.
name = String(deprecated={"since": "0.18", "removal": 0.20})

Quote both versions:

from protean.fields import String

name = String(deprecated={"since": "0.18", "removal": "0.20"})

diff_ir() and classify_changes() now check every deprecated block in both IR files before comparing them, on elements and on fields, whether or not anything changed. They raise ConfigurationError naming the element or field when:

  • since is not a string, or removal is neither a string nor null.
  • since is missing.
  • The block is not an object, such as "deprecated": "0.15".

A current_version that is not a string raises too. Before, the diff read a block only on a removed published event or a removed field. It read a numeric version as text, and could crash with an AttributeError or KeyError on a block that is not an object or has no since.

protean ir diff prints the error and exits 1. The protean-check-compat hook prints it and blocks the commit, under strictness = "warn" too. With strictness = "off", both skip the comparison.

A baseline that Protean generated never holds a number. The old code wrote 0.20 into it as the string "0.2", and the diff cannot tell that the value was meant as 0.20.

To upgrade:

  1. Quote every since and removal version in deprecated=.
  2. Regenerate the IR baseline, so that a version that lost its zero, such as "0.2", is written again as "0.20".

The IR's $schema URL moved

The IR used to name its schema https://protean.dev/ir/v{X}/schema.json, which does not resolve. New IR files name https://docs.proteanhq.com/assets/ir/v{X}/schema.json, where the docs site serves every schema version. Each schema's $id moved to the same URL.

No action is needed. protean ir check and protean ir diff ignore $schema, so the old URL alone does not make a baseline stale or show up as a change. It picks up the new URL the next time you regenerate it. If a tool of yours matches the $schema value, accept both URLs.

New in ir diff: value type changes on Dict and List fields

ir diff now compares the value type of a Dict or List field that keeps its type. Moving a field from Dict() to Dict(value_type=ValueObject(TierPrice)) used to show up only in the text diff: the compatibility report listed nothing for it and the command exited 2. Rows stored while the field was untyped can fail to load once it is typed.

Adding, changing or removing a value type is now a breaking field_type_changed, and a published event lists it as contract_field_type_changed. Under strictness = "strict", a diff that exited 2 now exits 1, so a CI job that runs protean ir diff or the protean-check-compat hook fails on a value type change. Under strictness = "warn", it exits 0 and the hook passes. See Dict and List value types.

Email port removed

The email subsystem was deprecated in 0.17, and 0.18 warned that it would be removed in 0.19.0. It is now removed. Sending email is an application concern, so Protean no longer ships a port for it.

These names are gone, and using them raises AttributeError or ImportError:

  • The @domain.email decorator and the DomainObjects.EMAIL element type.
  • Domain.send_email(), Domain.get_email_provider() and the Domain.email_providers attribute.
  • domain.registry.emails.
  • BaseEmail and BaseEmailProvider (protean.core.email).
  • EmailProviders, DummyEmailProvider and SendgridEmailProvider (protean.adapters.email, and the two names re-exported from protean.adapters).

The sendgrid extra is gone too, so pip install "protean[sendgrid]" no longer installs the SendGrid client. If your code calls SendGrid directly, install the sendgrid package yourself.

An [email_providers] block in domain.toml is now ignored. Creating the Domain no longer warns about it, and protean check no longer reports it. You can delete the block. An [email_providers] block inside an environment section (for example [production.email_providers]) is still loaded when that environment is active, so a ${VAR} placeholder in it must still resolve. Delete those blocks too.

protean check no longer has the DEPRECATED_EMAIL and DEPRECATED_CONFIG codes. A suppress_checks list or a [lint].suppressions table that still names them keeps working, because nothing emits those codes now. You can drop those entries.

To keep sending email, do it from an event handler or a subscriber that calls a notification service in your application:

# fragment
@domain.event_handler(part_of=User)
class WelcomeNotifier:
    @handle(UserRegistered)
    def send_welcome(self, event: UserRegistered) -> None:
        notifications.send_welcome_email(event.email)

Here notifications is your own module. It can wrap SendGrid, SMTP or any other service.

State-based aggregates save their events only through add()

A Unit of Work used to save the events of every state-based aggregate it had read, whether or not add() was called on it. The aggregate's field changes were saved only through add(). A handler that read an aggregate, called a method that raised an event and never called add() stored the event, but the database kept the old state:

# fragment
@handle(ConfirmOrder)
def confirm(self, command: ConfirmOrder) -> None:
    repo = current_domain.repository_for(Order)
    order = repo.get(command.order_id)
    order.confirm()  # sets status and raises OrderConfirmed
    # No repo.add(order): before 0.19, OrderConfirmed was saved and
    # published, but status stayed unchanged in the database.

An aggregate's events are now saved only when add() has saved the aggregate in the same Unit of Work. This is the rule event-sourced aggregates already followed. Events raised on an aggregate that was only read are not saved at commit. Protean logs nothing and raises nothing. An aggregate removed with _dao.delete() does not count as saved either, so events raised on it before the delete are not saved.

To keep the event, pass the aggregate to add(). That saves its field changes too:

# fragment
@handle(ConfirmOrder)
def confirm(self, command: ConfirmOrder) -> None:
    repo = current_domain.repository_for(Order)
    order = repo.get(command.order_id)
    order.confirm()
    repo.add(order)

Call add() after the last change. An add() on an aggregate with no change since it was read and no pending events writes nothing, so it does not keep an event raised after it. If the aggregate changes again after add(), call add() again.

Look for this pattern in command handlers, event handlers, application services and process managers: an aggregate loaded with get() or a query, a method called on it that raises an event, and no add() before the Unit of Work ends. Tests that expect such an event in the event store or the outbox fail after the upgrade, which is one way to find the places to fix.