Health and observability

Observe LeapView at the process, dependency, delivery, data, and query layers. A single “up” signal cannot distinguish a healthy process with a broken active project from a slow query caused by an invalid model grain.

Liveness and readiness

/healthz is the lightweight liveness endpoint. /readyz represents readiness for serving. The generated command checks readiness:

leapview healthcheck \
  --url https://leapview.example.com/readyz \
  --timeout 5s

Use liveness for process restart decisions and readiness for traffic admission. Keep both inexpensive; they should not execute a full dashboard query. Use separate synthetic probes for end-to-end analytical behavior.

The unauthenticated readiness response contains only stable check names and fixed states. Reviewed delivery-startup diagnostics may expose documented non-secret codes; platform, analytics, runtime, lease, and arbitrary custom-check errors collapse to failed, and active project identifiers are not returned. Use restricted application logs, metrics, and authorized operator interfaces for internal error details rather than widening the readiness payload.

After a provider restore, production may set LEAPVIEW_RECOVERY_SET_ID to one canonical recovery-set UUID for exact pre-traffic validation. When unset, readiness performs the ordinary PostgreSQL startup checks. When set, startup reads only that ID, requires the set to be published with one exact passed immutable validation attempt/result, and fail-closes unless its target pointer/revision, generation, publication, native snapshot seal and catalog identity, admitted compatibility tuple, and serving-artifact identity match the active projections. It also recomputes the canonical validation-evidence digest and binds the exact attempt/frontier, both database recovery identities, object versions and provider frontiers, and relation manifest/closure to that selected set. /readyz performs no recovery writes: it does not create validation attempts or record validation results. The check does not replace provider object probing; follow the PostgreSQL operations guide and Backup and restore guide for that recovery evidence.

Metrics

LeapView exposes Prometheus metrics behind LEAPVIEW_METRICS_BEARER_TOKEN. Production validation requires a strong token. Restrict the endpoint by network policy as well, inject the token into the scraper securely, and avoid logging it.

Monitor at least process resource use, request rate and latency, error status, read/write executor saturation, queue depth and timeouts, refresh activity, storage capacity, and managed upload failures. Alert on sustained conditions and user-visible symptoms rather than every transient supersession.

Baseline health alerts

The repository-owned rule file at deploy/observability/prometheus/leapview-alerts.yaml provides portable alerts for an unavailable scrape target, sustained application 5xx responses, fatal process-owned DuckDB health, named PostgreSQL pool saturation/acquisition pressure, and dashboard refresh error-budget burn. The rules assume that Prometheus assigns LeapView targets the stable scrape label job="leapview"; instance must identify the bounded scrape target. Alert identity is limited to those scrape labels plus the static service and severity labels, with the PostgreSQL pool alerts retaining the bounded pool name. Operation, publication state, occurrence, schedule, artifact, target, request or trace identities, principals, projects, and resources are aggregated away and never become alert labels.

Validate the syntax, PromQL behavior, firing delay, labels, and annotations with the pinned Prometheus toolchain:

task observability:alerts:check

The task downloads the official Prometheus archive for the supported Linux or macOS architecture, verifies its pinned SHA-256 digest, caches promtool under .tmp/tools, then runs promtool check rules and the healthy and firing rule fixtures.

Copy or mount the rule files into the Prometheus deployment and reference them from the Prometheus configuration:

rule_files:
  - /etc/prometheus/rules/leapview-alerts.yaml
  - /etc/prometheus/rules/leapview-recording-rules.yaml

Reload Prometheus only after validation succeeds. The target-unavailable rule relies on Prometheus's standard up metric and therefore requires the target to remain present in Prometheus service discovery; a target omitted from the scrape configuration cannot alert. The 5xx rule uses leapview_http_requests_total, aggregates method, route, and status dimensions down to job and instance, and requires a positive five-minute error rate continuously for ten minutes. leapview_duckdb_fatal_health is present only when LeapView owns a process-local DuckDB environment.

Audit writes use the caller-owned PostgreSQL transaction and append directly to the immutable audit table. Monitor append failures through the application error rate and PostgreSQL health signals; Prometheus does not expose payload, actor, or delivery-queue dimensions.

Every alert carries a runbook_url for its canonical response procedure. The baseline alerts link directly to target-unavailable response, sustained HTTP 5xx response, DuckDB fatal-health response, and the PostgreSQL pool saturation, headroom, acquisition, and churn procedures. Outcome-reliability burn and latency burn alerts likewise deep-link to their alert-specific procedures. Use the correlated incident investigation workflow to move from their bounded alert identity into metrics, request logs, optional upstream trace identity, domain evidence, and an objective recovery record. These links define evidence collection, safe mitigation, and recovery checks; they do not perform remediation. Configure routing and receivers in the operator-owned Alertmanager deployment.

Dashboard refresh reliability SLI and SLO

The repository-owned recording rules at deploy/observability/prometheus/leapview-recording-rules.yaml derive dashboard refresh reliability only from leapview_dashboard_refresh_duration_seconds_count. A finished refresh is eligible unless its outcome is canceled. complete is good; partial, error, and the bounded fallback other are bad. Cancellations are excluded because refresh supersession and stream closure are expected lifecycle behavior rather than evidence of failed dashboard work.

leapview:dashboard_refresh_reliability:ratio_5m records the five-minute good-to-eligible ratio. leapview:dashboard_refresh_reliability:ratio_30d records the rolling 30-day ratio. Both aggregate command and outcome dimensions to the stable job and instance scrape labels. Validate the rules, counter-reset behavior, empty-traffic handling, aggregation, and label contract with the existing pinned toolchain:

task observability:sli:check

Ten companion recordings make the ratio's underlying traffic volume explicit. The eligible_events recordings cover 5-minute, 30-minute, 1-hour, 6-hour, and 30-day windows and include every outcome except canceled; their corresponding bad_events recordings include partial, error, and other. They use increase over the source histogram counter, aggregate to the same job and instance labels, and handle counter resets. For a nonzero eligible volume, the reliability ratio is equivalent to one minus bad events divided by eligible events over the same window.

Prometheus must retain at least 30 days of source samples for a complete rolling window. Until that history has accumulated after initial deployment or retention loss, the 30-day ratio and event volumes reflect only the available portion of the window. increase may return fractional estimates at range boundaries, so event volumes are operational context rather than accounting totals. Canceled-only traffic records zero eligible and bad events while still emitting no reliability ratio. Completely missing source metrics emit neither volume nor ratio; missing or idle traffic is never reported as perfect reliability. Sparse traffic can make the five-minute ratio volatile and the 30-day ratio statistically weak, so operators should inspect the eligible event recording before interpreting either series.

The repository-owned dashboard refresh reliability objective is 99% over a rolling 30-day window. The LeapView engineering/operations owner owns this initial target. It is a realistic baseline for detecting meaningful degradation while production behavior is still being established, without requiring an immature system to meet a stricter objective before sufficient operational history exists. Revisit the target after production reliability data establishes normal dashboard refresh behavior.

leapview:dashboard_refresh_reliability:objective_ratio_30d records the 0.99 objective wherever the 30-day reliability ratio is evaluable. leapview:dashboard_refresh_reliability:error_budget_consumption_30d divides the observed bad-event fraction by the approved 1% bad-event allowance. A value below 1 is within budget, 1 is exhausted, and a value above 1 is overdrawn. leapview:dashboard_refresh_reliability:error_budget_remaining_30d records one minus consumption, so a negative value remains visible after exhaustion rather than being clamped. All three recordings retain only job and instance.

The objective and budget recordings are absent when eligible traffic is zero or missing, including canceled-only traffic; absence must not be interpreted as compliance. Low traffic remains mathematically valid but statistically weak, so evaluate budget state together with eligible_events_30d. Prometheus cannot distinguish a complete 30-day history from a partial window after deployment or retention loss, and the budget recordings naturally reflect only the samples available. They inherit the SLI's increase estimation behavior and are not accounting totals.

The 5-minute, 30-minute, 1-hour, and 6-hour burn_rate recordings divide each window's bad-event fraction by the approved 1% error-budget allowance. A value of 1 spends budget at exactly the sustainable rate; 6 and 14.4 spend it six and 14.4 times faster respectively. A burn-rate recording is absent when its eligible volume is zero or missing, including canceled-only traffic.

The critical LeapViewDashboardRefreshFastBurn alert fires only when its five-minute and one-hour burn-rate recordings are both at least 14.4 for the same job and instance and the one-hour window contains at least 10 eligible refreshes. The dual windows reject a short spike that is not also significant over one hour; the volume floor prevents very sparse one-hour traffic from paging. There is intentionally no separate five-minute floor or additional hold duration. A firing alert means the current failure rate is rapidly consuming the existing 99% SLO budget; follow the linked dashboard refresh response. Alertmanager routing remains operator-owned.

The warning LeapViewDashboardRefreshSlowBurn alert fires only when the 30-minute and 6-hour burn rates are both at least 6 for the same job and instance and the six-hour window contains at least 60 eligible refreshes. It detects sustained moderate degradation that can exhaust the 30-day budget without meeting the critical fast-burn threshold. The dual windows provide persistence filtering, so there is no additional hold duration. Missing, canceled-only, or lower-volume traffic cannot fire the alert. Follow the linked slow-burn response.

The warning and critical alerts are independent and can both fire during severe degradation; operator-owned Alertmanager routing may inhibit the warning when the critical alert is active. This SLO measures only completed dashboard refresh work. It does not measure refreshes still in flight, HTTP or SSE transport continuity, authentication, static assets, direct API queries, or an end-to-end synthetic journey. The repository defines no additional burn tier, long-window budget alert, paging receiver, or Alertmanager routing. Services below either alert's volume floor require direct inspection of the 30-day budget recordings. Because Prometheus evaluates the recording and alert groups independently, firing and recovery may trail source conditions by one rule-evaluation interval.

Dashboard refresh latency SLI

The dashboard refresh latency SLI measures only refreshes whose normalized outcome is complete. A completed refresh is timely when its recorded end-to-end duration is less than or equal to five seconds, using the cumulative le="5" bucket from leapview_dashboard_refresh_duration_seconds. The inclusive five-second boundary matches an existing histogram bucket and the current dashboard qualification expectations. Partial, error, other, canceled, and in-flight refreshes are excluded rather than classified as slow; the outcome reliability SLI above remains the canonical measure for those terminal failures.

leapview:dashboard_refresh_latency:timely_completed_ratio_5m records the five-minute timely-to-completed ratio. leapview:dashboard_refresh_latency:timely_completed_ratio_30d records the rolling 30-day ratio. Both use Prometheus counter-reset-aware rates and aggregate the histogram's command, outcome, bucket, and any other source dimensions to job and instance. They emit no series when completed-refresh traffic is zero or missing, so absence must not be interpreted as timely behavior.

Ten companion recordings make the ratio's population explicit. The completed_events recordings increase the completed-refresh histogram count over five-minute, 30-minute, one-hour, six-hour, and 30-day windows. Their corresponding slow_completed_events recordings subtract the cumulative le="5" bucket increase from the completed count, so they represent completed refreshes above the five-second boundary. For a nonzero completed volume, the timely ratio is equivalent to one minus slow completed events divided by completed events over the same window. The full recording names use the leapview:dashboard_refresh_latency: prefix, retain only job and instance, and exclude every non-complete outcome. Missing or canceled-only source traffic emits no latency volume rather than a synthetic zero.

These volumes are operational evidence, not accounting totals: Prometheus increase() can return fractional estimates at range boundaries. Use them to distinguish a low-sample ratio from broad latency degradation, but do not treat the estimated event count as an exact ledger.

The repository-owned dashboard refresh latency objective requires at least 99% of completed server-side refreshes to finish within five seconds over a rolling 30-day window. The Production Reliability / Engineering Operations owner responsible for LeapView operational reliability targets owns this policy. The initial target follows the established dashboard reliability objective pattern and protects against meaningful user-visible latency degradation without imposing a stricter objective before representative production latency history exists. Revisit it after sufficient representative history is available. This completed-only latency SLO is distinct from the outcome reliability SLO: failed, partial, and canceled refreshes remain outside the latency population and are measured by the outcome reliability model where applicable.

leapview:dashboard_refresh_latency:objective_ratio_30d records the 0.99 objective wherever the 30-day timely-completed ratio is evaluable. leapview:dashboard_refresh_latency:error_budget_consumption_30d divides the rolling slow-completed fraction by the approved 1% allowance. A value below 1 is within budget, 1 is exhausted, and a value above 1 is overdrawn. leapview:dashboard_refresh_latency:error_budget_remaining_30d records one minus consumption and remains negative after exhaustion rather than being clamped. All three recordings reuse the existing latency ratio and event-volume series and retain only job and instance.

The objective and budget recordings are absent when completed traffic is zero or missing, including canceled-only traffic; absence must not be interpreted as compliance. Low traffic remains mathematically valid but statistically weak, so evaluate budget state together with completed_events_30d. Prometheus must retain 30 days of source histogram samples for a complete rolling window; after deployment or retention loss, the ratio, volumes, and budgets naturally reflect only available history and inherit the event recordings' estimation behavior.

The five-minute, 30-minute, one-hour, and six-hour latency burn_rate recordings divide each window's slow-completed fraction by the 99% SLO's 1% allowance. A value of 1 spends latency budget at the sustainable rate; 6 and 14.4 spend it six and 14.4 times faster respectively. Burn-rate recordings are absent when completed volume is zero or missing.

The critical LeapViewDashboardRefreshLatencyFastBurn alert fires only when the five-minute and one-hour burn rates are at least 14.4 for the same job and instance and the one-hour window contains at least 10 completed refreshes. The warning LeapViewDashboardRefreshLatencySlowBurn alert fires only when the 30-minute and six-hour burn rates are at least 6 and the six-hour window contains at least 60 completed refreshes. The dual windows reject degradation present in only one window; each completed-event floor limits sparse-traffic noise. Missing or canceled-only traffic cannot fire either alert. Neither alert has an additional hold duration, so recording-rule and alert evaluation can trail source conditions or recovery by one interval. Follow the linked latency fast-burn response or latency slow-burn response.

The warning and critical latency alerts are independent and can both fire during severe degradation; operator-owned Alertmanager routing may inhibit the warning while the critical alert is active. Latency and outcome-reliability alerts measure different populations and can also fire together: latency alerts detect slow completed refreshes, while outcome alerts detect partial, error, or other terminal outcomes among eligible refreshes. Investigate both rather than treating either as duplicate evidence. This latency SLO defines no percentile recording, additional burn tier, routing policy, or automated remediation. It covers completed server-side refresh coordination through terminal target completion, not browser rendering, transport delay, or an end-to-end synthetic journey, and bucket classification cannot expose degradations that remain on the same side of the five-second boundary.

Query-result cache cutover rollout gates

Serving-generation cutover reuses a stable partition result cache while keeping execution flights generation-owned. Qualify a cache rollout with deterministic correctness and race gates before interpreting production metrics:

task test:cache:rollout

The command requires 100 consecutive passes of the cutover correctness and metrics checks and 20 consecutive passes of the focused race checks. The selected qualification checks have no conditional skip paths. Any failure, race report, stale-store acceptance, sentinel leak, unexpected physical execution, or Arrow ownership imbalance fails the rollout. The qualification remains opt-in and focused; it does not expand the ordinary pull-request test matrix.

The hard correctness and memory gates are:

Capture latency evidence on the same otherwise-idle host, Go toolchain, CPU count, and power profile for the merge-base baseline and rollout candidate:

task bench:cache:cutover:full

The full task records ten 500 ms samples at one logical CPU for the public resultcache lifecycle and governed materialize runtime paths. The quick task is only a smoke check:

task bench:cache:cutover:quick

The first Phase 2 full run establishes and archives the bootstrap baseline, because the benchmark harness does not exist on the pre-Phase 2 merge base. After that baseline is established, every future cache rollout compares its candidate output with the archived baseline or a fresh full run from its merge base.

Compare the ordinary Go benchmark output with benchstat. A candidate fails when a statistically significant latency increase exceeds the threshold below. An increase above the threshold without significance must be rerun on the same host; it is not treated as a pass until the comparison is conclusive or an explicit rollout exception records the evidence and owner.

During a controlled canary cutover, retain at least 30 minutes of samples and verify all of the following:

• leapview_query_result_cache_entries remains equal to leapview_query_result_cache_arrow_holds.
• leapview_query_result_cache_bytes never exceeds the configured node byte limit, and retained entries never exceed the configured node entry limit.
• leapview_query_result_cache_scopes{state="dormant"} becomes nonzero during the zero-reference interval, then leapview_query_result_cache_scope_transitions_total{transition="reactivated"} increases when the compatible generation opens.
• leapview_dashboard_query_cache_hits_total{source="cutover_retained"} increases after the warmed canary query is served by the reactivated generation.
• leapview_cache_invalidations_total{cache="stable_result"} does not increase during cutover unless the operator intentionally invalidates the partition.
• The absolute 30-minute stable-result eviction delta is zero: sum(increase(leapview_cache_evicted_entries_total{cache="stable_result"}[30m])) == 0. Any positive delta indicates memory pressure or unexpected churn and fails the canary; investigate before rollout rather than diluting the signal with generation-byte store volume.

These are rollout gates, not new cache semantics or alerting rules. Persistent production alerts and any threshold adjustment require separately reviewed operational evidence.

Frequency-aware cache admission evaluation

FAI-535 evaluates whether a frequency-aware admission guard should protect frequently reused dashboard results from bursts of one-off, high-cardinality queries. The evaluation does not change the production cache. LeapView keeps its exact synchronous stores, LRU eviction, Arrow ownership, per-runtime and node budgets, cutover lifecycle, and existing store outcomes.

The repository-owned workload calibrates its LRU model against the production resultcache.Pool, then compares that model with an admission-only candidate using a bounded, aging four-row frequency sketch. Both lanes retain the same LRU victim order so the workload isolates admission rather than also claiming sampled-LFU behavior. The fixed trace warms two eight-visual dashboard bundles in separate runtime scopes, interleaves 5,120 unique requests of four different sizes, and reloads both dashboards twice after every burst. A cache hit has a fixed 0.2 ms modeled cost; KPI and chart misses have fixed 35 ms and 90 ms costs. The p95 result is therefore a deterministic workload model, not an end-to-end wall-clock measurement.

Run the bounded smoke check with:

task bench:cache:admission:quick

Capture ten 500 ms samples at one logical CPU with:

task bench:cache:admission:full

The initial comparison was captured on Linux amd64 with Go 1.25.14 and one logical AMD EPYC-Rome CPU. Both lanes executed the same 7,936-request trace.

The decision is not to adopt the policy in production from this evidence. The trace demonstrates that frequency-aware admission can protect hot bundles under the targeted adversarial workload, and a separate cold-only trace checks that equal-frequency requests preserve LRU retention. It does not measure real dashboard latency, process CPU time, process RSS, Arrow allocation, or the production pool's concurrent overhead. It also does not qualify production frequency accounting for coalesced waiters and second-chance lookups, stable Arrow versus generation-byte entries, cross-scope node pressure, invalidation, or serving-generation cutover. A production proposal must resolve those semantics and repeat the comparison through the real concurrent pool before it can add a rejection outcome or change retention behavior. Until then, this benchmark is evidence for the hypothesis only, not a rollout gate.

Structured logs

Collect structured application logs from the service output. Preserve timestamp, severity, operation, route, status, duration, principal where safe, project, environment, request/correlation ID, deployment ID, revision digest, and refresh generation when available.

Secrets, bearer tokens, passwords, raw OAuth payloads, and sensitive query data must not appear in logs. Restrict log access according to the most sensitive metadata retained.

LeapView establishes X-Request-ID and X-Correlation-ID before process-wide middleware and route handling. A non-empty client request ID is preserved for idempotency compatibility; otherwise LeapView generates one. A missing correlation ID defaults to the request ID. Both canonical values are returned in response headers, including responses rejected by early security middleware, so proxy and application logs can agree on public request time, status, and correlation identity.

When an upstream service supplies valid W3C traceparent and tracestate headers, LeapView validates and carries that remote trace context through the request. Request and panic logs add the parsed trace_id and upstream_span_id; they never log the raw trace headers, baggage, authorization, cookies, query values, or request bodies. Missing or malformed trace context is ignored without rejecting the request or generating a trace ID, and the request ID remains LeapView's local correlation identity. This contract does not create spans, sample traces, or export telemetry.

The correlated incident investigation workflow defines the deployment-neutral pivots between alert labels, metrics, request logs, optional inbound trace identity, dashboard refresh records, generation and serving-state evidence, and the incident record. It also marks joins that are inferential because request identity is not propagated through every refresh or background-work boundary.

Delivery signals

Track project deployment IDs, environment, acting principal, candidate validation results, managed revision pins, activation outcome, and active deployment. Uploading an artifact or staging a data revision is not the same as successful activation.

Alert when production has no active deployment, a rollout repeatedly fails, or the active deployment differs from the intended promotion record.

Data and refresh signals

Track refresh generation, target project asset, queued/running duration, terminal status, cancellation or supersession, and active serving state. Monitor source and output row counts, unexpected schema changes, data-file growth, and available disk space.

An expected superseded refresh is not necessarily an incident. Repeated failures of the latest generation, growing queue delay, or inability to activate a valid candidate are actionable.

Query events and audit

Query events help identify slow or failing workloads by operation, project, duration, and diagnostic metadata. Audit events answer who changed security or administrative state. They serve different purposes and have separate retention controls.

Use leapview admin maintenance in dry-run mode to review bounded retention before applying deletion. Preserve relevant events externally when organizational policy requires longer history.

Synthetic verification

After deployment or upgrade, run a small authenticated sequence:

1. Check readiness.
2. Request the current principal.
3. List an expected project resource.
4. Describe a known semantic model.
5. Execute one bounded semantic or dashboard query.
6. Confirm the active managed revision for production.

Keep the synthetic principal read-only and scoped to the test project. This verifies routing, auth, active project state, and analytical execution without granting deployment privilege.

Recovery and retention boundaries

LeapView exposes readiness, delivery, refresh, query, and audit evidence for diagnosis. It does not schedule or execute production backup, restore, image upgrade, host rollback, or recovery-qualification drills. PostgreSQL backup and point-in-time recovery, DuckLake catalog protection, and object-store versioning, replication, and restore are external provider operations. Keep their evidence and encryption-key procedures with the PostgreSQL operations guide and Backup and restore guide; do not infer recovery from application metrics or object listings.

For an incident, preserve readiness responses, authenticated metrics, credential-scrubbed logs, deployment identifiers, and the provider's recovery point evidence. Keep traffic stopped until the provider-native recovery is complete and the active deployment, authorization, managed-data revisions, and representative governed queries have been verified.

See Operational troubleshooting, Audit events, and the environment reference.