Dashboard native Arrow response contract

This document defines the native-v1 correctness oracle for a future native Arrow response from dashboard table data queries. It does not activate that response, change the current dashboard endpoint, or authorize a production migration. FAI-543 and later experiments must satisfy this contract before their performance results can be considered.

The existing unversioned dashboard Arrow response is a compatibility boundary: it projects every cell to UTF-8, represents a source null as an empty string, and returns its next cursor as an initial response header. A native-v1 response instead preserves governed Arrow values and uses a completion trailer. Those behaviors are not wire-compatible. The current stream must not silently change; a native response requires explicit client opt-in and the response must identify native-v1 in both its header and schema metadata.

Scope

The first eligible implementation is an ordinary detail/table query. It must use the same authorization, governance, admission, audit, result-budget, sorting, and pagination inputs as the current direct dashboard query. Matrix, pivot, calculations, multi-block shaping, Datastar SSE, retained-result cache, and warm-cache delivery are outside this contract's first implementation.

Schema contract

The IPC stream schema is the post-governance projected schema, not the source table schema. In particular:

• Fields appear in governed projection order. Field names are the requested aliases where aliases exist; physical source names must not replace aliases.
• Arrow physical types are preserved. Boolean and signed/unsigned integer widths, floating-point widths, dates, timestamps, UTF-8 strings, binary values, decimals, and dictionary encodings must not be projected to strings.
• The field's nullable declaration is the final governed effective nullability described below. Each array's physical validity bitmap is preserved. A null must remain null; it must not become zero, false, an empty string, or an empty byte slice.
• A decimal preserves its precision, scale, sign, and exact unscaled value.
• A timestamp preserves its Arrow unit and timezone. A timestamp with timezone UTC represents the same UTC instant; a timezone-neutral timestamp remains timezone-neutral rather than acquiring a server-local zone.
• Binary values preserve their exact bytes, including zero bytes and empty non-null values.
• A dictionary field preserves its index type, value type, ordering flag, dictionary values, indices, and validity bitmap. Expanding a dictionary to strings is a contract change.
• Only response-safe governed metadata from the closed allowlists below is emitted. It must not expose generated SQL, physical connection or source identifiers, policy expressions, credentials, principals, or other internal state.

Nullability authority

native-v1 uses governed effective nullability with conservative fallback. The server-owned output schema descriptor for the final post-governance projection is authoritative; neither a source-column declaration nor the DuckDB Arrow field flag is independently authoritative.

An emitted Arrow field has Nullable=false only when the final projected expression is proven incapable of producing SQL NULL after every governed transformation. A nullable, unknown, incomplete, or conflicting derivation is emitted as Nullable=true. In particular:

• a source NOT NULL declaration does not by itself prove that the final projection is non-null;
• the nullable side of a left relationship traversal is nullable;
• a column mask that can produce NULL makes the output nullable;
• a calculation or derived expression is nullable unless its complete operator and input semantics prove otherwise;
• metric nullability includes its governed empty-set policy; and
• filters or the values observed in one page, including a page with zero nulls, never narrow the declared nullability.

Unknown always maps to nullable. The rule is stable for empty results: an empty stream uses the same derived output schema descriptor as a non-empty execution of the same governed query. It must not infer nullability from the absence of record batches.

Runtime validity bitmaps remain the authoritative physical statement about which values are null. The stream forwards them without null-to-value conversion. If a record contains a null for a field declared non-null, the execution has violated the governed output contract; Arrow encoder acceptance does not make that stream valid.

Implementation boundary

The schema foundation represents each projected field with an internal query.OutputFieldDescriptor containing:

• the final alias;
• the public governed logical type;
• effective nullability represented internally as proven non-null, nullable, or unknown; and
• internal derivation provenance sufficient to explain the decision without exposing source, policy, SQL, or connection details on the wire.

query.Planner.DescribeOutputSchema derives the descriptor before a response schema could be emitted from validated final PlanIR, including relationship traversal, column masks, metric empty-set policy, calculations, and derived expressions. Unknown derivations remain nullable. The descriptor is matched to the governed alias and projection order; it is not reconstructed from observed batches or client-authored metadata.

arrowquery.OutputSchemaSink is the un-routed enforcement boundary. It reconciles only the Arrow field nullability declaration required by the descriptor, preserves physical Arrow types and validity buffers, respects borrowed-batch callback lifetimes, and validates every emitted record batch. Its internal provenance is opaque and is not serialized as response metadata. No production route currently constructs this sink; this foundation does not authorize a handler change or native streaming migration.

An empty result is a valid Arrow IPC stream containing the full governed schema and zero record batches. It is not a schema-less stream and it is not a JSON empty array.

Reserved LeapView metadata

The leapview.* namespace is server-owned. Upstream/source metadata cannot set or override it. The allowlists are closed: unknown keys are rejected whether they use the leapview.* namespace, another namespace, or no namespace. Adding a response metadata key requires a contract revision and corresponding oracle coverage.

All schema keys and leapview.logical_type are authoritative server values; producer values cannot override them. display.label is the only producer metadata currently approved for forwarding. SQL, engine, connection, physical source, and other producer metadata are rejected. In particular, a cursor is never placed in schema metadata: it is disclosed only through the response trailer after successful completion.

Response protocol

A successful native response is an Arrow IPC stream and has these required headers before the body is committed:

Version negotiation

The legacy and native contracts share the dashboard visual query route but are separate, explicitly negotiated representations. A native request supplies both:

Accept: application/vnd.apache.arrow.stream
X-LeapView-Arrow-Contract: native-v1

Legacy behavior remains unchanged: an unversioned Arrow request receives the all-string schema, the existing initial X-Next-Cursor header semantics, and the existing exact-or-capped AvailableRows behavior.

A successful native response echoes X-LeapView-Arrow-Contract: native-v1. An Arrow request without the contract marker remains on the legacy all-string representation. A marker without the Arrow media type, or any unknown contract version, fails closed with 406 Not Acceptable; the server must never silently downgrade it. This foundation does not activate the native representation in the production dashboard handler. Activation remains a separate adoption change.

Native-v1 page model

The default page limit is 100 rows. The accepted range is 1 through 1,000 rows, inclusive. Native interactive pagination emits at most 10,000 cumulative rows for one cursor chain. The server executes a limit + 1 probe under the same governed query while there is capacity below that cumulative cap, emits at most limit rows, and uses only the probe row to decide whether continuation exists. It does not calculate or promise an exact total and does not expose AvailableRows. Clients that require exact or capped totals remain on the legacy contract.

The first request has offset zero. A middle page resumes from the offset in a validated native cursor. If the probe row exists and the IPC writer closes successfully, the server writes an opaque X-Next-Cursor trailer. The final page, an empty result, and a page that reaches the 10,000-row cap have no trailer value. Exactly limit physical rows is a final page; limit + 1 physical rows is the first condition that permits a cursor. At offsets where a full requested page would cross the cumulative cap, the emitted/query limit is reduced to the remaining allowance and no continuation probe is executed beyond the cap.

The trailer name is declared before commitment on every successful native response so clients can consume final and continuing pages uniformly. Initial headers and Arrow schema metadata never contain the native cursor. Probe rows count toward physical execution and result budgets even though they are not emitted. The empty response remains a complete native schema with zero rows and no cursor.

Dashboard-native cursor domain

Native-v1 uses the signed d3 cursor domain. Legacy dashboard d1 cursors and semantic-query q1 cursors are not interchangeable with d3; every decoder rejects cursors from the other domains. A native cursor expires after 15 minutes and binds:

• the native-v1 contract version;
• dashboard, page, and visual identity;
• server-computed canonical digests of normalized filters, selections, and effective sorting;
• the requested page limit;
• the final governed effective-policy identity;
• the serving snapshot;
• the next offset, cumulative rows consumed, and 10,000-row cap; and
• its expiry.

The signed payload contains a digest of the request/governance scope rather than raw filters, selections, sort definitions, policies, principals, SQL, source identifiers, or physical connection details. Governance and authorization are recomputed for each page before the cursor binding is accepted. A changed contract, identity, normalized request state, limit, or policy identity is an invalid cursor (400). A different serving snapshot is a cursor conflict (409). Clients treat the entire cursor as opaque.

Error and commit boundary

Before the Arrow response is committed, failures use the API's JSON problem shape with Content-Type: application/problem+json and no Arrow contract headers. Existing resource-concealment rules remain authoritative.

Request cancellation stops query work and releases borrowed batches and other leases. If the connection remains writable and no bytes have been committed, the failure follows the normal problem mapping; a disconnected client may observe only connection termination.

A runtime null in a field declared non-null is a native contract failure. If detected before response commitment, the server returns the normal structured JSON problem response and does not start an Arrow stream. If detected after commitment, the server terminates the Arrow stream, does not append a JSON fallback, and does not publish a successful X-Next-Cursor trailer value.

After the Arrow response is committed, the server cannot switch formats. A query, IPC, cancellation, or transport failure terminates the stream. There is no JSON suffix or fallback, no successful completion signal, and no X-Next-Cursor trailer value. Consumers must treat an unreadable or incomplete IPC stream as failed even if the HTTP status was already 200.

Cursor publication is ordered after successful IPC close. Cancellation, timeout, row/byte budget failure, admission failure, partial write, IPC write failure, or IPC close failure therefore cannot publish a successful cursor.

Security and governance invariants

The native transport changes representation only. Before any schema or batch is written, the request must pass the same boundaries as the current governed direct query:

1. Resolve the authenticated principal and active serving snapshot.
2. Authorize the dashboard/model dependencies and preserve any resource concealment behavior.
3. Apply row policies and column masks and compute the effective policy fingerprint. The emitted schema is the masked projection; it cannot reveal denied source columns or pre-mask physical types/metadata.
4. Acquire workload admission using the same class, identity, operation, and memory estimate as the control query.
5. Apply row and byte budgets to the schema, emitted rows, and pagination probe. A transport encoder cannot bypass budget accounting.
6. Record the same audit actor, credential/effective subject, operation, resource target, start, success, and failure outcomes as the control query.

FAI-543's borrowed DuckDB batches may be observed only synchronously inside the sink callback. The sink must finish reading or encoding a batch before the callback returns, must not retain borrowed buffers, and must not populate or read the retained dashboard result cache.

Correctness qualification

A candidate is rejected before performance comparison if it differs from the control in column order, aliases, physical types, values, null positions, metadata, sorting, offset/limit behavior, cursor behavior, cancellation, partial-write handling, authorization, row policies, masks, admission, budgets, or audit identity. The executable fixtures in internal/dashboard/http/arrow_contract_test.go lock the native value and wire semantics. Existing authorization, admission, budget, and audit tests lock the shared governed execution boundaries.

Only after those gates pass may an experiment compare the candidate with the current api_direct dashboard path using the same query and physical query behavior. Warm-cache measurements are a guardrail, not a comparable lane.

Nullability qualification must cover, at minimum:

• base NOT NULL and nullable fields;
• aliases and projection ordering;
• empty results;
• fields on both sides of a left relationship traversal;
• masks that preserve, replace, or introduce nulls;
• metrics with null and zero empty-set policies;
• calculations and derived expressions with proven and unknown semantics;
• multiple record batches; and
• non-null declaration mismatches detected both before and after response commitment.

Client compatibility and activation

No first-party browser component currently decodes this API Arrow stream, but API and generated-client consumers may rely on the current string/null projection. Therefore:

• native delivery requires the two-header native-v1 opt-in above; merely sending the Arrow media type continues to select the existing legacy stream;
• the response must return X-LeapView-Arrow-Contract: native-v1, and clients must reject unknown contract versions;
• clients must support native Arrow types, validity bitmaps, dictionaries, and HTTP trailers before opting in;
• generated clients continue to prefer JSON unless native Arrow support is explicitly selected.

The request marker must be added to TypeSpec/OpenAPI when production activation is proposed. This contract foundation intentionally remains unrouted so the generated and live endpoint cannot claim support before native serving, resource policy, and client qualification are complete.