Prepared state engine

Atomic byte-record batches, branch heads, scoped receipts and allocation-free publication.

Status: implemented in src/engine/state.zig, tested with Zig 0.16.0. This is a pure, single-owner in-memory state layer over the persistent map. It does not open files, write a journal, serve requests, authenticate a caller, enforce document schemas or execute graph queries. A call to this module's commit is an in-memory publication, not a durability acknowledgment.

The engine provides immutable retained snapshots, branch-local byte records, atomic mutation batches, explicit branch-head comparisons, deterministic receipts, branch-local idempotency and current version checks. These are foundations for the database integration, not complete UNIVERSE persistence conformance.

Prepare, journal, publish#

The durable owner uses this sequence:

1. Hold the external writer gate. Authenticate and authorize the actual operation, bound its input, and keep its typed command bytes immutable through journal encoding.

2. Under the publication mutex, call Engine.prepare(request). Release the publication mutex after preparation. No visible records, heads, sequence, versions or receipts change at this stage.

3. If Prepared.isReplay() is true, do not append a journal record. Return the stored receipt through the owner's synchronized replay path after current authorization checks.

4. Otherwise, encode the exact request, append it at Prepared.receipt().sequence, and complete the journal's durability operation. Readers may acquire and use the old committed snapshots while I/O runs.

5. Reacquire the publication mutex and call Engine.commit(&prepared). This transfers already prepared roots and metadata without allocation. Release the writer gate only after publication and outcome handling.

All state validation and allocation occurs before journal I/O. Fork preparation may reserve hash-table capacity, but cannot make the new branch visible. Preparation failure leaves visible state unchanged. Before any append, Prepared.deinit() safely cancels a preparation. Calling it after a successful commit only releases an emptied handle.

commit has only three errors: WrongEngine, StalePreparation, and PreparedConsumed. A stale or wrong-owner failure after syncing a record is an integration failure with potentially durable state. Stop writes and reopen/recover; do not report an ordinary rejected transaction and continue. The writer gate must span preparation, append/sync and publication so another mutation cannot invalidate the prepared base sequence.

An uncertain append/sync failure also requires the journal owner's recovery protocol. The state layer cannot decide whether a failed system call left a durable record. Releasing a preparation does not undo disk writes.

The publication mutex protects access to the engine's mutable fields during prepare, snapshot, status inspection and commit. It can be released around I/O because the separate writer gate excludes other mutations. A returned snapshot owns its immutable map and can be read outside both locks with a suitable allocator. The engine itself supplies neither mutex.

Public contract#

Command has four alternatives:

• batch: branch_id, expected_head, and ordered mutations.

• fork: source_branch_id, target_branch_id, and expected_source_head.

• drop: branch_id and expected_head.

• advance_versions: the next explicit authorization_epoch and catalog_version.

Batch mutations are put { key, value }, create { key, value }, and delete key. Keys and values are arbitrary byte strings. Put inserts or replaces; create requires absence; delete requires presence. Later mutations see earlier changes in the same batch. If any operation fails, none of the batch is published. Empty batches are rejected. A successful unchanged put still creates an audited command/receipt and advances the branch head.

Prepared state owns copied ordinary record bytes. Receipts and intake results are not inserted until commit. It does not retain the caller's typed Request for later journal serialization. Mutating those original command slices after preparation could cause the owner to log a different intent. Keep the exact request immutable or use a separately owned canonical buffer throughout the durable operation.

An engine and a preparation are owning values; do not copy them as independent owners or move/reinitialize an engine while a preparation refers to it. A preparation must be released. Snapshots use clone() for another owner and deinit() to release it. The allocator and backing state must outlive all retained snapshots and preparations, even if the engine has been destroyed.

Branch and sequence semantics#

Every new accepted command advances one global sequence, including forks, drops and version changes. A data batch changes only its target branch's head. A fork gets its creation sequence as its initial head and captures its source's record root at the checked source head. Version-control commands advance global sequence without changing any data branch's head. Matching retries advance neither.

A snapshot records branch_id, head, observed_sequence, observed version fields and its retained data root. Global sequence records when it was acquired; it does not expose another branch's mutations. Historical version fields are observations, not permission grants.

A child and grandchild share an immutable source root initially. Later mutations copy changed paths and remain isolated. There is no recursive read-through-parent lookup for ordinary records. Data-root retention itself is constant work, independent of record volume. A complete fork also inserts a receipt and catalog metadata. Receipt insertion appends into the preallocated receipt table when this engine is the only writer. The branch hash table may resize. The full operation is not guaranteed constant time.

Intake results do not share a parent map root. A fork copies the accounting hint and freezes result visibility at the parent's retained slot count at that moment. Later parent results stay on the parent. The child still sees the results that were visible at the fork, including results the parent had inherited, up to 16 levels.

Deleting a branch releases its current root and retains its identity/lineage tombstone. Other branches and retained snapshots continue owning their roots. The branch ID cannot be reused, even after deletion. A repeated historical fork or batch may return its old receipt after deletion; that receipt reports the old operation and does not resurrect the branch or assert that it is currently live.

base_head is provenance metadata. This module has no arbitrary historical-root lookup, merge-base registry, three-way merge or persistent retention catalog. To keep a historical readable root, retain a Snapshot; later layers must give retained roots durable identities and an explicit retention policy.

Idempotency and current authority#

Receipts live in a preallocated append-only table sized to max_receipts, keyed by branch scope and request ID. Batch/drop requests use their branch; fork requests use the target branch; version changes use catalog scope 0. Forking shares ordinary records without copying parent replay identities. Reusing a request ID on a different branch is an independent operation, subject to that branch's own existing receipts. There is no eviction. A full table rejects a new receipt with ReceiptLimit. A matching replay still succeeds at the cap. Dropped branches leave their receipt slots occupied. For a durable store the cap is chosen at create (default 100,000, range 1,024–1,000,000), persisted as table.limits, and reused on every open; see ADR-002.

The engine validates current authorization and catalog context before returning a historical receipt. A matching request ID and hash returns exactly the original receipt even when the old expected branch head is no longer current. A valid different intent under the same scoped ID fails with IdempotencyConflict. A supplied hash that does not match the typed command fails with InvalidRequestHash.

Authorization context is separate from the intent hash. After an authorization or catalog-version change, the owner must reauthorize and supply current context; this can return an unchanged historical receipt without executing the command again. These counters only detect stale context. They do not establish who the caller is, what that caller may access, or whether that caller may advance the counters. The trusted owner must enforce those decisions, including authorization on reads and delivery of historical results.

advance_versions cannot decrease either value and must increase at least one. Explicit values are recorded in the command so recovery does not consult wall-clock time, generate IDs or infer a new policy version.

Deterministic intent and recovery#

The command hash uses SHA-256 over domain bytes rural.state.command.v1 followed by a zero byte, then a one-byte command tag. Integers and lengths are unsigned 64-bit little-endian. Variable bytes are length-prefixed. Mutation order is significant.

Mutation tags are put 1, create 2, delete 3. Put/create encode key bytes followed by value bytes, each length-prefixed. Delete encodes only its length-prefixed key. Request identity and current authorization context are outside this hash. The independent golden fixture for a main-branch put of a -> bc at expected head 0 is ac3eb62114595548b63226dd1cd6fcadcc8d52424a46ff7db033d5b71aae8527.

Journal encoding is the durable owner's responsibility. It must preserve all typed fields and bytes and use separately enforced parse/frame limits. This hash definition does not prescribe JSON serialization or establish a journal format. Arbitrary byte values require a lossless representation if a text transport is used.

replayDurable requires the exact next global sequence. It revalidates typed intent and reconstructs catalog/version changes and receipts deterministically. Duplicate durable request records are rejected: an ordinary idempotent retry should never have produced another journal entry. Invalid sequence/hash/semantics or allocation failure leaves that record unapplied; the owner must treat recovery as incomplete rather than skip it and begin serving writes. State reconstructed this way is only as trustworthy and durable as the journal owner that supplied it.

Admission bounds and remaining resource work#

Defaults cap 1,000 mutations, 4,096 key bytes, 8 MiB per value, 16 MiB canonical structured request bytes, 100,000 records per branch, 1,024 branch catalog slots and 100,000 receipts. Limits are fixed for the engine instance. Durable stores persist the receipt/result table cap at create (1,024–1,000,000; default 100,000) and ignore a smaller or larger caller option on reopen. The request-byte limit counts identity, hash, context and the exact command-hash preimage; it does not replace a network/JSON wire-byte cap.

The record bound applies to each intermediate candidate state in an ordered batch. For a branch it counts ordinary map records, visible intake results, and one slot when the accounting hint is present. Tombstones consume branch slots. Receipts survive branch deletion and consume receipt slots. Neither is silently evicted; reaching a limit rejects new applicable operations. Matching receipt replay still works at capacity.

Both retained tables are allocated and touched at Engine.init, so later unique-owner commits of receipts, intake results, and accounting updates do not allocate. A second engine (cloneForStaging) shares those tables and stages into a private 128-slot delta, folding only after it is again the only writer. Snapshots retain the tables for their lifetime and do not become writers: the unique engine keeps appending, and a snapshot's captured floor hides slots appended after it was taken. Snapshots read and release the tables without the publication lock, so their reference counts are atomic and each index cell is published with release/acquire ordering after its slot is written. A dropped branch keeps its inherit link, so a live descendant still sees results it inherited through that branch. Intake-result values are capped at 384 bytes and the accounting hint at 512 bytes; a larger value is ValueTooLarge. Ordinary record values keep max_value_bytes. ADR-002.

Service status reports main_records as the main branch's ordinary records plus its visible intake results and accounting hint, so the soak's record time-to-cap follows result growth. Snapshot.count and BranchInfo.record_count stay map-only. BranchInfo.side_records is the extra count.

These are admission limits, not an exact resident-memory ceiling. The default tables are about 60 MB and stay within an 80 MB budget at 100,000 receipts. Retained snapshots, temporary preparation paths, allocator overhead, input parsing buffers and concurrent reader work need separate owner-level budgets. Ordinary publication frees replaced references and can release many no-longer-shared nodes. Unique-owner receipt and intake-result publication does not allocate. Large reclamation and fair scheduling remain integration concerns.

Verification#

Durable intent payloads now write RURALTX3; strict legacy RURALTX2 reading remains available. An explicit presence byte distinguishes an absent logical request hash from a present all-zero hash in both the durable payload and derived receipt cache. Mixed-version journals replay without rewriting historical frames. Legacy zero encodings remain absent because their lost distinction cannot be recovered. Old binaries cannot read newly written TX3 records. Exact layout and compatibility.

Target the module through the shared root so its relative map import has the same module identity:

zig test src/root.zig --test-filter state

Tests cover invisible/cancelled preparation, non-allocating data and fork publication, ordered atomic batches, expected-head conflicts, recursive root sharing, stable snapshots, parent deletion, branch-local idempotency, current context checks before replay, wrong-owner/stale preparation rejection, deterministic recovery, an independent hash fixture, admission limits, retained tombstones/receipts, sequence overflow, snapshots outliving the engine, and the preallocated receipt/result cap (unique-owner commits allocate no further bytes; the next receipt at the cap is ReceiptLimit; a branch record cap counts intake results). Retained-result tests also cover visibility frozen at fork with a dropped intermediate branch and a checkpoint round trip, snapshots across in-place appends and a staged fork before and after fold, full receipt and result tables leaving state unchanged while replay still works, and concurrent snapshot reads and releases while the unique writer replaces the same key.

Exhaustive allocation-failure tests cover a mixed branch/mutation/version/deletion workload. Additional failure injection verifies that each failed preparation preserves the visible root, head, sequence, receipt count and current versions, with temporary allocations released. They establish the pure state behavior; disk sync/recovery fault injection, locking and full application acceptance belong to the durable integration layer.