Migrating to 0.16

From 0.15 to 0.16

Protean 0.16 is a stability and reliability release. Most upgrades are drop-in. The changes that may require operator action are the outbox column bounds, the Elasticsearch client upgrade, and a few operational-default and behavioral changes described below. Run protean upgrade-check --domain=<your-domain> to have these surfaced for your project, with the generated outbox ALTER SQL where applicable.

On this page:

Outbox column bounds: A one-time ALTER TABLE for existing outbox tables
Outbox unique index: Drop and recreate the recommended index as composite over (message_id, target_broker)
Outbox target_broker NOT NULL: Backfill NULLs, then add the constraint
Elasticsearch 8.x client: Installs now default to the 8.x client, which requires an Elasticsearch 8.x server
Connection pool defaults: Raised SQLAlchemy pool sizes for PostgreSQL and MSSQL
Health-check port: protean server now binds a probe server on :8080
Observatory binds localhost: The dashboard is loopback-only by default
Deprecations: --debug in favor of --log-level DEBUG

Outbox column bounds

Tier-3 structural change

The string fields on the Outbox aggregate now declare max_length. SQL providers therefore emit VARCHAR(N) columns instead of TEXT / VARCHAR(MAX). This unblocks index creation on the outbox table: SQL Server refuses to index VARCHAR(MAX), MySQL requires a blind prefix length on indexed TEXT, and bounded columns are cheaper to store and scan everywhere.

The affected fields and their bounds:

Field	`max_length`
`status`	32
`target_broker`	128
`stream_name`	255
`type`	255
`locked_by`	128
`correlation_id`	255
`causation_id`	255
`message_id`	255

message_id, causation_id, and correlation_id are bounded at 255 rather than a UUID-sized value because they hold composite Protean message ids (for example testdomain::order-<aggregate-id>-3) or caller-supplied tracing strings, not bare UUIDs.

The data and metadata_ columns remain unbounded JSON blobs.

Who is affected

New deployments: Nothing to do. Tables created on 0.16 get the bounded VARCHAR(N) columns automatically.
Existing deployments with a populated outbox table: Protean never alters a populated table. The table keeps working as-is, but to gain the storage and indexing benefits (and to match the schema Protean now emits) run the one-time ALTER TABLE for your backend below.

Pre-flight verification

No value Protean writes to these fields exceeds the new bounds. Before applying the migration, confirm your existing data also fits:

SELECT
  MAX(LENGTH(status))         AS max_status,
  MAX(LENGTH(target_broker))  AS max_target_broker,
  MAX(LENGTH(stream_name))    AS max_stream_name,
  MAX(LENGTH(type))           AS max_type,
  MAX(LENGTH(locked_by))      AS max_locked_by,
  MAX(LENGTH(correlation_id)) AS max_correlation_id,
  MAX(LENGTH(causation_id))   AS max_causation_id,
  MAX(LENGTH(message_id))     AS max_message_id
FROM outbox;

On SQL Server, substitute LEN(...) for LENGTH(...) (SQL Server has no LENGTH function).

If any column exceeds its bound, investigate before applying the migration.

Migration recipes

=== "PostgreSQL"

```sql
ALTER TABLE outbox
  ALTER COLUMN status TYPE varchar(32),
  ALTER COLUMN target_broker TYPE varchar(128),
  ALTER COLUMN stream_name TYPE varchar(255),
  ALTER COLUMN type TYPE varchar(255),
  ALTER COLUMN locked_by TYPE varchar(128),
  ALTER COLUMN correlation_id TYPE varchar(255),
  ALTER COLUMN causation_id TYPE varchar(255),
  ALTER COLUMN message_id TYPE varchar(255);
```

=== "MySQL"

```sql
ALTER TABLE outbox
  MODIFY status varchar(32) NOT NULL,
  MODIFY target_broker varchar(128) NULL,
  MODIFY stream_name varchar(255) NOT NULL,
  MODIFY type varchar(255) NOT NULL,
  MODIFY locked_by varchar(128) NULL,
  MODIFY correlation_id varchar(255) NULL,
  MODIFY causation_id varchar(255) NULL,
  MODIFY message_id varchar(255) NOT NULL;
```

=== "SQL Server"

```sql
ALTER TABLE outbox ALTER COLUMN status varchar(32) NOT NULL;
ALTER TABLE outbox ALTER COLUMN target_broker varchar(128) NULL;
ALTER TABLE outbox ALTER COLUMN stream_name varchar(255) NOT NULL;
ALTER TABLE outbox ALTER COLUMN type varchar(255) NOT NULL;
ALTER TABLE outbox ALTER COLUMN locked_by varchar(128) NULL;
ALTER TABLE outbox ALTER COLUMN correlation_id varchar(255) NULL;
ALTER TABLE outbox ALTER COLUMN causation_id varchar(255) NULL;
ALTER TABLE outbox ALTER COLUMN message_id varchar(255) NOT NULL;
```

=== "SQLite"

SQLite does not enforce declared column lengths, so the migration is a
no-op for storage. New databases get the `VARCHAR(N)` declaration;
existing databases continue to function unchanged.

Outbox unique index becomes composite

Tier-3 structural change

The recommended unique index on the outbox table changed from message_id alone to a composite over (message_id, target_broker). A single published event is dual-written to the outbox once per target broker (the internal broker plus every external broker), all rows sharing one message_id, so a unique index on message_id alone rejected the framework's own dual-write with a UniqueViolation. The framework now also always writes a non-NULL target_broker (the configured internal broker name, default "default"), because PostgreSQL and SQLite treat NULLs as distinct in a UNIQUE index, which would otherwise let duplicate message_id rows through in single-broker mode.

Who is affected

New deployments: Nothing to do. Tables created on this release get the composite uq_outbox_message_id_target_broker index automatically.
Existing deployments that created the uq_outbox_message_id index on 0.16.0: drop it, backfill any rows where target_broker IS NULL to the internal broker name, then create the composite index. Replace 'default' below with your configured [outbox] broker value if you set one.

Migration recipes

=== "PostgreSQL"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
DROP INDEX IF EXISTS uq_outbox_message_id;
CREATE UNIQUE INDEX uq_outbox_message_id_target_broker
  ON outbox (message_id, target_broker);
```

=== "MySQL"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
DROP INDEX uq_outbox_message_id ON outbox;
CREATE UNIQUE INDEX uq_outbox_message_id_target_broker
  ON outbox (message_id, target_broker);
```

=== "SQL Server"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
DROP INDEX uq_outbox_message_id ON outbox;
CREATE UNIQUE INDEX uq_outbox_message_id_target_broker
  ON outbox (message_id, target_broker);
```

=== "SQLite"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
DROP INDEX IF EXISTS uq_outbox_message_id;
CREATE UNIQUE INDEX uq_outbox_message_id_target_broker
  ON outbox (message_id, target_broker);
```

Outbox `target_broker` becomes NOT NULL

Tier-3 structural change

Outbox.target_broker is now declared NOT NULL (0.16.2). It defaults to the configured internal broker name ([outbox] broker, default "default"), and the framework always sets it on write. The constraint closes a gap in the idempotency index above: because a UNIQUE index treats NULLs as distinct on PostgreSQL and SQLite, a row with a NULL target_broker slips past the (message_id, target_broker) uniqueness and reopens the duplicate-publish window. Legacy rows with a NULL target_broker are coerced to the default broker name on read, so existing data keeps working.

Who is affected

New deployments: Nothing to do. Tables created on 0.16.2 get the NOT NULL target_broker column automatically.
Existing deployments: Protean never alters a populated table. Backfill any NULL rows and add the constraint to match the schema Protean now emits. Replace 'default' below with your configured [outbox] broker value if you set one.

Migration recipes

=== "PostgreSQL"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
ALTER TABLE outbox ALTER COLUMN target_broker SET NOT NULL;
```

=== "MySQL"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
ALTER TABLE outbox MODIFY target_broker varchar(128) NOT NULL;
```

=== "SQL Server"

```sql
UPDATE outbox SET target_broker = 'default' WHERE target_broker IS NULL;
ALTER TABLE outbox ALTER COLUMN target_broker varchar(128) NOT NULL;
```

=== "SQLite"

SQLite cannot add a `NOT NULL` constraint to an existing column in place; it
requires a table rebuild (create a new table with the constraint, copy the
rows, drop the old table, rename). New databases get the constraint at
`CREATE TABLE` time. If you run the outbox on SQLite and need the constraint
on an existing database, recreate the table on 0.16.2 or perform the rebuild
manually.

protean upgrade-check --domain=<your-domain> surfaces this migration (and the composite-index one above) with the generated SQL for your database, on the enforcing backends.

Elasticsearch 8.x client

Tier-2 behavioral change

The elasticsearch extra now allows the version 8 client, and a fresh install resolves to it by default (elasticsearch>=7.17.9,<9.0.0). This unblocks urllib3 2.x and resolves the related security advisory.

The version 8 client only connects to an Elasticsearch 8.x server. It rejects older servers during its product-compatibility check.

Who is affected

Running an Elasticsearch 8.x server: Nothing to do. The 8.x client is the default and connects normally. Bare host entries (for example localhost:9200) are normalized to scheme://host:port automatically, with USE_SSL selecting the scheme, so existing configuration keeps working.
Running an Elasticsearch 7.x server: Pin the 7.17 client, which connects to both 7.x and 8.x servers:
```
pip install "protean[elasticsearch]" "elasticsearch<8" "elasticsearch-dsl<8"
```
The 7.17 client also benefits from urllib3 2.x, so the security advisory is resolved on this path too. Plan an Elasticsearch 8.x server upgrade to move onto the default 8.x client in a future release.

Connection pool defaults

Tier 2 (operational-defaults). The SQLAlchemy pool defaults were raised to pool_size=5 and max_overflow=10 for PostgreSQL and MSSQL (previously 2 and 5). With pool settings unset, each worker may open up to workers × 15 connections per domain (was workers × 7).

Who is affected

Deployments on PostgreSQL or MSSQL that relied on the previous default pool sizes, especially with several workers against a database with a low max_connections ceiling (PostgreSQL defaults to 100).

What to do

Verify the database's max_connections has headroom for workers × 15. To keep the previous behavior, set them explicitly per database:

[databases.default]
pool_size = 2
max_overflow = 5

A LOW_POOL_SIZE warning is emitted when pool_size is below 5; silence it with PROTEAN_ENV=development or PROTEAN_ENV=testing. See Harden the server for sizing guidance.

Health-check port

Tier 2 (operational-defaults). protean server now starts a health-check HTTP server on :8080 by default (/healthz, /livez, /readyz). If the port is already in use the engine logs a warning and continues without probes, so the change is non-fatal.

What to do

Nothing, unless :8080 collides with another service. Move it or disable it:

[server.health]
port = 8081        # or
enabled = false

Observatory binds localhost

Tier 2 (behavioral, secure by default). protean observatory now binds 127.0.0.1 by default instead of 0.0.0.0. The Observatory is unauthenticated and exposes domain internals and Dead Letter Queue management (retry/delete) endpoints, so it is loopback-only unless you deliberately expose it.

What to do

If you reached the Observatory from another host (for example a Docker port mapping), bind a non-loopback address explicitly:

protean observatory --domain my_app --host 0.0.0.0

Binding a non-loopback address logs a warning. Expose it only on a trusted network behind an authenticating reverse proxy; never on the public internet.

Deprecations

Tier 1. The --debug flag on protean server and protean observatory is deprecated in favor of --log-level DEBUG. It still works but emits a DeprecationWarning and will be removed in v0.17.0.

protean server --log-level DEBUG        # was: protean server --debug

Migrating to 0.16

Outbox column bounds

Who is affected

Pre-flight verification

Migration recipes

Outbox unique index becomes composite

Who is affected

Migration recipes

Outbox target_broker becomes NOT NULL

Who is affected

Migration recipes

Elasticsearch 8.x client

Who is affected

Connection pool defaults

Who is affected

What to do

Health-check port

What to do

Observatory binds localhost

What to do

Deprecations

Outbox `target_broker` becomes NOT NULL