Observability & Alerting Guide

This guide covers wiring Prometheus metrics, Kubernetes health probes, runtime worker gauges, structured logs, and OpenTelemetry tracing in production.

1. One-Line Production Setup (http.productionProfile)

Call zmodu.http.productionProfile before registering any routes (router.mountAll or server.addRoute), because addRoute snapshots the active global middleware chain at registration time:

var server = zmodu.http.Server.init(io, allocator, 8080);
defer server.deinit();

var profile = zmodu.http.ProductionProfileState.init(allocator);
defer profile.deinit(allocator);

try zmodu.http.productionProfile(&server, .{
    .max_connections = 4096,
    .header_timeout_ms = 10_000,
}, &profile);

Endpoints registered by productionProfile:

Override default paths via metrics_path, liveness_path, and readiness_path.

2. HTTP Golden Signals

productionProfile exports the following series by default:

Bounded Route Labels

Golden signals carry a route label populated from ctx.route_template (the matched pattern such as /orders/{id}, never the raw URL path). Unmatched routes map to route="__unmatched__". Label cardinality is capped by ProductionConfig.max_route_series (default 64); additional patterns map to route="__other__".

Never place high-cardinality identifiers (tenant_id, user_id, order_id) into Prometheus labels. Use createCounterFamily / createHistogramFamily for small bounded dimensions, and register all custom metrics at startup before the first GET /metrics scrape freezes the registry.

3. Runtime Execution Plane Metrics (37 Gauges via MetricsBridge)

HTTP metrics do not observe worker mailboxes, worker pools, or timer lag. When using app.runtime(), wire Runtime.MetricsBridge:

var bridge = try Runtime.MetricsBridge(PrometheusMetrics).init(&rt, &metrics);
metrics.setScrapeHook(bridge.publish);

3.1 CPU Worker Pool & Blocking Pool (9 Gauges Each)

3.2 Mailboxes, Timers & Supervision (13 Gauges)

4. Outbox & Database Pool Metrics

// 1. Transactional Outbox consumer metrics and background polling
var consumer = zmodu.outbox.OutboxConsumer.init(allocator, &backend, .{}, ctx, handler);
try consumer.setMetrics(profile.prometheus.?);
try consumer.startPolling(io, 1000);
defer consumer.stopPolling();

// 2. Database connection pool gauges sampled on scrape
const pool_active = try metrics.createGauge("db_pool_active", "Active pooled DB connections");
const pool_waiters = try metrics.createGauge("db_pool_waiters", "Requests waiting for a DB connection");
metrics.setScrapeHook(struct {
    fn sample(ud: ?*anyopaque) void {
        const db: *data.Client = @ptrCast(@alignCast(ud.?));
        if (db.poolMetrics()) |pm| {
            pool_active.set(@floatFromInt(pm.current_active));
            pool_waiters.set(@floatFromInt(pm.current_waiters));
        }
    }
}.sample, &db);

5. PromQL Queries & Alert Thresholds

# Request rate (QPS, 5m window)
sum(rate(http_requests_total[5m]))

# 5xx error ratio
sum(rate(http_responses_5xx_total[5m])) / clamp_min(sum(rate(http_requests_total[5m])), 0.001)

# P95 / P99 latency
histogram_quantile(0.95, sum(rate(http_request_duration_milliseconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_milliseconds_bucket[5m])) by (le))

6. Logs, Tracing & Panic Attribution

• Request trace correlation: tracingMiddleware (enabled by productionProfile) reads or generates X-Trace-Id (ignoring untrusted headers longer than 128 bytes or containing non-printable characters) and calls ctx.setTraceId(). Inside handlers, call const log = ctx.logScope("orders"); so every log line includes trace_id=....
• OTLP Export: Use zmodu.observability.OtlpExporter (exportSpans) to push spans over http:// or https://.
• Panic attribution: Export pub const panic = zmodu.panicHook; in your root main.zig to print the active METHOD /path to stderr when a panic occurs.