Skip to content

ADR-0007: Domain-Scoped OpenTelemetry Providers

Field Value
Status Accepted
Date 2026-03-16
Author Subhash Bhushan
Epic 6.1 OpenTelemetry Integration (#742)

Context

Protean needed production-grade observability (distributed tracing and metrics) to support ops teams shipping telemetry to backends like Grafana, Datadog, and Jaeger. OpenTelemetry (OTEL) was the natural choice: it is vendor-agnostic, widely adopted, and provides the W3C TraceContext propagation that Protean's TraceParent value object already uses.

OTEL's recommended usage pattern is to register a single global TracerProvider and MeterProvider at process startup via trace.set_tracer_provider() and metrics.set_meter_provider(), then retrieve them anywhere with trace.get_tracer(). This works well for conventional applications that have exactly one telemetry configuration per process.

Protean's Domain class, however, is designed to be instantiated multiple times in the same process, most commonly in tests, where each test case may create its own Domain(name="Test") with different configuration. The OTEL global registration model creates three problems in this context:

  1. Single-assignment semantics: OTEL globals are write-once: the first call to set_tracer_provider() sets the provider, and OTEL silently ignores later calls (or logs a warning). A second domain in the same process cannot override the provider with its own configuration.

  2. Shutdown leaves broken globals: Calling shutdown() on the global provider puts it into a terminal state. Any spans created afterward are silently dropped. Tests create, activate, and shut down domains repeatedly, so only the first test gets real spans.

  3. No per-domain configuration: Different domains may need different service names, exporters, or resource attributes. A single global provider cannot express this.

Protean also already had an existing observability system (the Observatory) which provides zero-config, real-time tracing for developers running protean server locally. Observatory uses Redis Streams and SSE, producing flat MessageTrace events. OTEL needed to coexist with Observatory without replacing it: Observatory serves developers; OTEL serves ops teams. The two systems share instrumentation callsites but have independent emission paths, connected only at the /metrics endpoint where OTEL's PrometheusMetricReader takes over when available.

Decision

We store TracerProvider and MeterProvider as attributes on the Domain instance, never setting OTEL globals. All access goes through domain-scoped helpers:

# In application code -- always safe, always returns something usable
tracer = domain.tracer  # lazy property, triggers init on first access
meter = domain.meter

# Under the hood (src/protean/utils/telemetry.py)
def get_tracer(domain, name="protean"):
    provider = getattr(domain, "_otel_tracer_provider", None)
    if provider is None:
        return _NoOpTracer()
    return provider.get_tracer(name)

Providers are created during init_telemetry(domain), which is called lazily on first access to domain.tracer or domain.meter. A sentinel flag (_otel_init_attempted) prevents repeated initialization attempts. shutdown_telemetry(domain) flushes and destroys the providers, then resets the sentinel so a new initialization cycle is possible, critical for test isolation.

The Domain class exposes two lazy properties:

  • domain.tracer: Returns a configured OTEL Tracer or a _NoOpTracer
  • domain.meter: Returns a configured OTEL Meter or a _NoOpMeter

Both use deferred imports so the opentelemetry package is never loaded unless telemetry is actually accessed.

Telemetry is disabled by default (telemetry.enabled: False in domain config). All OTEL packages are optional extras (pip install protean[telemetry]). When telemetry is disabled or the packages are absent, all public functions return lightweight no-op objects. These objects implement the minimal OTEL interface, so instrumentation code never needs conditional guards.

Consequences

Positive:

  • Each domain instance owns its telemetry lifecycle. Tests create isolated domains with InMemorySpanExporter and assert on spans without cross-test contamination.
  • Multiple domains in the same process (e.g., a multi-bounded-context deployment) can have independent service names, exporters, and resource attributes.
  • Clean shutdown/re-init cycles work correctly. Protean sets no global provider, so no shut-down global can silently drop spans.
  • Observatory and OTEL coexist cleanly at instrumentation callsites. The /metrics endpoint serves OTEL Prometheus exposition when available, falling back to the hand-rolled implementation when not.

Negative:

  • Diverges from OTEL's documented best practice. Contributors familiar with OTEL will first call trace.get_tracer() (the global) and need to learn the domain-scoped pattern.
  • Every instrumented callsite must have access to the Domain instance (typically via current_domain context or direct reference) to obtain a tracer or meter.
  • Some third-party OTEL auto-instrumentation libraries expect global providers (e.g., opentelemetry-instrumentation-requests). They will not automatically participate in Protean's domain-scoped traces. An application has to bridge them to the domain's providers manually.

Alternatives Considered

Global providers (OTEL default). Rejected because of the single-assignment, broken-shutdown, and no-per-domain-config problems described in Context. These are fundamental to OTEL's design and unlikely to change upstream.

Per-thread providers. Would solve test isolation for single-threaded test runners but not for async or multi-domain production scenarios. Also adds complexity and does not solve the configuration problem.

Monkey-patching OTEL globals per test. Fragile, requires careful teardown ordering, and breaks if any library caches a reference to the old provider.

References