Storage, execution and distribution for a visible world

Status: Proposed architecture and research, 20 September 2026. This appendix specifies a possible RURAL design; it is not an implementation, compatibility statement or performance result. Database development remains paused. The current source contains an in-memory persistent map, prepared branch state, journal and framing foundations. It does not deliver the SQL, graph, warehouse, distributed or complete application system described here. See the recorded project status, fractal architecture (not published) and analytics contract (not published).

The proposal is to make a world independently versionable without making every version an independent physical copy. Storage, query execution and distribution must preserve that distinction through failures.

1. One committed world, several representations

A proposed world stamp identifies the authority domain, branch, committed position, schema root, object root, relationship root and physical-index manifests. Separate references bind a code observation, declared governance revision and semantic-definition versions. Current execution authority is checked independently; retaining an old stamp never retains a revoked permission.

Documents, typed rows, relationships, events, blobs, empty tables and schema definitions receive durable identities. An empty schema remains a catalog object. A relationship has its own identity and provenance. A pipe records its source, destination, transformation version, admission policy and progress. Rendered geometry references these objects; it is not the database's authoritative representation.

SQL, the document API and graph operations compile into one bounded, typed execution plan. Initial SQL should explicitly enumerate supported selection, filtering, projection, grouping, joins and mutations; unsupported syntax fails before execution. This is a RURAL dialect, not a promise of PostgreSQL wire, SQL-standard or SurrealQL compatibility. The same mutation through different APIs must have identical constraints, authorization and receipt semantics.

The value specification must precede an optimizer:

• Preserve absent properties separately from explicit null. SQL projections may map absence to NULL only through a documented conversion, with a presence predicate available. PostgreSQL's JSON documentation illustrates why JSON null, SQL NULL and missing paths require separate rules. PostgreSQL 18 JSON functions.

• Use checked integers and explicit fixed-scale decimals; overflow is an error. Money additionally carries a currency/unit. No implicit floating-point accounting.

• Specify string ordering, timestamp precision, timezone conversions and identifier normalization. Persist their versions before indexing depends on them.

• Give graph edges endpoint and deletion rules: restrict, explicitly cascade, or deliberately retain a dangling reference. Bounded traversal cannot silently become exhaustive traversal.

An index is a versioned physical view, not a second truth. Exact indexes join the transaction or advertise a usable position with a correctness-preserving tail. Full-text analyzers and vector indexes additionally name tokenizer/model versions. Approximate retrieval reports its approximation; authorization filtering and version checks remain exact. A deleted or unauthorized candidate must never become evidence merely because an older index returned it.

2. Publication is a protocol, not a pointer assignment

The proposed local commit sequence is:

1. Validate current authority, canonical request identity, bounds and expected branch head; reserve preparation and journal capacity.

2. Prepare the candidate roots and constraints under one ordered owner. Initially, reject an outdated expected head rather than attempt incomplete predicate-conflict detection.

3. Persist the exact replayable operation, receipt and required metadata; cross the declared local synchronization boundary.

4. Publish the prepared roots and sequence through a short ownership gate; then acknowledge.

Cancellation before commitment releases preparation. After durable commitment, publication/recovery must finish even if the caller disappears. A lost reply means unknown outcome: retry the same scoped identity and intent. Conflicting reuse fails. Receipt retention needs durable producer epochs and explicit replay horizons; unlimited historical deduplication is not free.

General SQL must not be replayed while independently generating time, randomness, external reads or model output. Either the ordered owner executes a tightly specified deterministic command with captured inputs, or it validates a transaction against the serialization point and logs the resolved write set plus necessary catalog effects. An optimistic read from an old snapshot is insufficient proof that its later writes are valid.

Current code provides useful pieces: root retention shares immutable data; state preparation separates allocation from publication; recursive forks initially share a root. However, Engine.commit also releases previous branch and receipt roots. Allocation-free publication is therefore not a constant-time guarantee. Proposed remediation is a bounded retirement queue: reserve its capacity before commitment, exchange ownership briefly, and reclaim outside the publication gate. When retirement cannot keep up, admission slows; history is not silently discarded. These are source observations, not new measurements. Persistent-map foundation, state-engine foundation.

3. A crystal has several histories

Represent a workspace using independent identities:

A Git commit identifies a tree state; one checkout does not acquire separate code and governance HEADs merely because the folders differ. Git commit-tree manual.

The proposed long-term default is a RURAL governance ledger with versioned file projections and explicit import proposals. Code remains Git-managed. File edits enter as proposals against an expected governance revision/content hash; adoption creates a new revision, and a recoverable projection operation updates files. Existing Git-authoritative repositories need an explicit migration choice, not silent replacement. Deployment records bind the complete tuple. Historical governance is explanatory evidence; it does not grant runtime authority.

A fork pins its source root and catalog, records a durable intent, prepares a sibling worktree if requested, verifies checkout identity, and publishes a ready workspace. Failures leave resumable or compensatable intent states. Git, filesystem and database work cannot honestly be described as one atomic transaction. Git worktrees share repository data while retaining per-worktree HEAD/index state, so repository-wide ref operations also need coordination. Git worktree manual.

Recursive data branches point directly to retained roots; reads do not walk an ever-growing parent-overlay chain. Later parent writes do not flow into children. Three-way merge compares a retained common base, source and target, rechecks the target head, and validates schema/uniqueness/edge constraints. A first version should reject conflicting whole-object changes. Merging data never replays payments, deployments or other external effects.

Thousands of operations against one crystal means bounded queues, independent sessions and conflict handling—not thousands of simultaneous writers to the same checkout or invariant.

4. Cleanup is a governed storage operation

Age, filename and a merged Git branch are insufficient deletion authority. An artifact record needs immutable identity, owner, generation, exact location, producing inputs, rebuild recipe, active runtime dependencies and retention reasons. Classify authoritative data, reproducible output, evidence, credentials and caches separately.

Proposed cleanup proceeds through inventory, eligibility proof, durable quarantine intent, fenced recheck, reversible quarantine where supported, grace period, exact deletion and receipt. Check current sessions, ordinary application startup dependencies, backup/restore jobs and open workspace intents. Unknown ownership, changed generation, dirty worktree, offline device or shared resource stops automatic deletion. An expired lease alone does not prove a process is dead; uncooperative external writers require exclusive ownership or manual resolution.

Operate through registered handles and verified directory identities, reject symlink escapes, and revalidate immediately before destructive filesystem actions. Git's normal worktree removal requires cleanliness; portable/offline worktrees can be locked against pruning. Do not bypass those guards with force. Git worktree manual.

Docker cleanup belongs in an optional privileged adapter with exact owned resource identities. Container inactivity does not establish that its images or volumes are disposable. Docker's broad pruning commands deliberately span multiple object classes; a crystal coordinator must not translate a local request into host-wide pruning. Docker pruning documentation.

After cleanup, verify the ordinary consumer path, not only a synthetic smoke test. Cleanup receipts and reclaimed-byte totals become visible operational objects, with bounded diagnostic detail.

5. Parallel work needs a resource ledger

Partition work into interactive reads, required mutations/intake, analytical computation and maintenance. Reserve service capacity for each; allow borrowing only while the protected class remains healthy. Use bounded workers, byte-accounted queues, maximum batch age and per-domain admission. MVCC removes some logical interference, not CPU, cache, memory-bandwidth or disk contention.

Optimize authority checks, point reads, root acquisition, deduplication and durable receipt latency first. Analytical scans need throughput and predictable interference. Semantic enrichment, geometry refinement and optional index rebuilding may lag with visible progress. Recovery, deletion and compaction still need reserved capacity: postponing them indefinitely eventually prevents useful foreground work.

Every task carries limits for CPU/work units, resident memory, spill bytes, scanned bytes, graph frontier, output bytes, deadline and retained-root lifetime. Enforce the aggregate budget as well as each task's budget. A small output limit does not bound the scan or intermediate join that produces it.

The planning invariant is:

M_control + M_writer + M_cache + Σ M_queries + M_indexes + M_retired + M_transport ≤ M_budget < M_host

Account for allocator overhead, worker stacks and native I/O buffers; reserve operating-system headroom. Fixed tasks that cannot spill must reserve their worst permitted allocation or reject. Do not market a buffer-cache cap as a whole-process memory guarantee.

For admitted rate λ, processing rate μ and burst length T, pending work grows by max(0, λ − μ) × T. With subsequent arrivals b, drain time is backlog / (μ − b) only if μ exceeds b. At an illustrative 100,000 events/s and 1,000 payload bytes, ten minutes contain 60 GB before receipts, indexes or replication. These are capacity arithmetic, not achieved rates. RURAL analytics capacity contract (not published).

Disk admission additionally reserves recovery, compaction, migration overlap, retained versions and quarantine space. A preflight free-space check cannot prevent another process consuming the disk; failed synchronization must never yield a successful durable receipt.

6. Outgrow memory without abandoning snapshots

The proposed disk engine combines immutable paged object/adjacency trees, immutable typed column segments and separately chunked blobs. Durable block identifiers replace process pointers. A pinned root retains its manifests and reachable blocks, not every block in RAM. Branch sharing is accounting over the union of reachable blocks; branch count multiplied by database size is the wrong physical-space model.

Clean cached pages can be evicted and reread. Unpersisted changes cannot. Temporary query spill is disposable after its owning query ends, whereas database pages remain reachable through roots. DuckDB's engineering work separates streaming, intermediate spill and persistent-page caching; high-cardinality aggregation demonstrates why streaming input alone cannot bound memory. These are established design lessons, not dependencies. DuckDB memory management, external aggregation research, ICDE 2024.

Implement partitioned external aggregation, external sort and spillable joins incrementally. Bound indivisible values, partitions, recursion and temporary files. Unsupported spill paths fail with a resource explanation. Column pruning, zone maps and sequential batches precede elaborate compression. Zig offers explicit allocation/layout control; it does not prove this executor faster than mature alternatives.

Garbage collection traces live branches, active readers, retained history, backup/restore pins, index builds and unfinished workspace operations. Snapshot leases have explicit retention budgets. A query exceeding them fails or renews through admission; the collector cannot invalidate a reader's live pointer. Privacy deletion requires coordinated purge/tombstone handling across derived views, retained copies and restore paths; logical deletion is not proof of physical erasure.

7. Analytics has several clocks

Keep accepted, query-visible and warehouse-complete positions distinct. A warehouse position belongs to a projection and definition version; across partitions it is a vector. Event time, recorded time and source coverage are additional dimensions, not replacements for processing positions.

At query position S, a projection complete through H may be combined with changes in (H, S] only while pinning both under compatible identity/version rules. Replacements retract old contributions, deletions remove them, and a corrected aggregate report replaces its prior report instead of becoming another event. Unsupported reconciliation waits or returns an explicit older view; it never performs a naive union and claims freshness.

Projection transactions atomically commit contributions and consumed positions. A crash must advance neither independently. Ingestion receipts certify admitted storage, not business truth: browser claims, authenticated purchases and imported reports remain different populations. Unknown coverage is unknown, not zero.

Expose useful operations and progress as cataloged objects. Durable mutations, external effects and accounting decisions need receipts; recording every memory access as another durable, billable event would create recursive overhead without improving the user's evidence.

8. Offload, replication and backup solve different problems

Cold placement is an explicit operation: pin a manifest; copy immutable chunks; verify lengths, authenticated integrity and retrieval; record the remote location and failure contract; atomically commit a placement generation; then evict eligible local copies. Failed or interrupted moves retain the old placement. Background scrubbing and restoration drills remain necessary after the initial verification.

Remote adapters must advertise actual conditional-write, versioning, listing, durability and deletion semantics. Amazon S3 documents conditional writes, but that does not establish identical behavior in every compatible service. The database manifest owner remains the serialization authority unless an alternative is proven. S3 conditional requests.

Unavailable cold data produces an unavailable/incomplete result with a coverage statement, never a fabricated zero. Shared-data caching has separate local quotas, provenance, eviction and consent policies; caching a peer's object does not transfer ownership or guarantee its continued availability. Confidentiality-domain boundaries also constrain deduplication.

Replication maintains eligible current copies for availability. Backup preserves separately retained recovery points against mistakes, corruption and compromised operational credentials. Neither a replica nor an uploaded chunk automatically constitutes a verified backup.

A proposed backup manifest names the root set, consistent partition cut, required journal suffix, schema/codec versions, receipt/source-epoch state, deletion fences and protected key references. Restoration uses a fresh isolated target: verify all reachable objects, replay without gaps, check logical invariants, reconcile accepted receipts and apply required current revocation/purge policy before enabling access. PostgreSQL explicitly distinguishes backup verification from restoring and checking a functioning database. PostgreSQL 18 pg_verifybackup.

Budget recovery time by the critical path through fetch, verification and replay; stages may pipeline, but the bottleneck still bounds completion. Specify recovery-point loss against the actual local, replicated or backup boundary. Migration and rebuilding temporarily require old and new copies. Key recovery and independent deletion authority are part of the restore design, not administrative footnotes.

9. Distribution evolves through ownership domains

First prove one durable owner. Next introduce independent partition owners, then replicated groups inside a declared trust boundary. A baseline candidate is three voting replicas per partition: a durable majority and healthy connectivity allow one replica failure. This is not a universal performance choice, global transaction order or protection against malicious replicas. Raft's original protocol requires additional leadership checks for linearizable reads and client duplicate suppression; naming a consensus algorithm supplies neither automatically. After the read barrier, the serving replica must apply through the required position before pinning its root. Raft extended paper.

Untrusted providers require a separately specified Byzantine threat model, execution verification and admission/membership protocol; signatures and encrypted disks alone do not supply them. No design here promises strong shared writes on every side of a partition. Removing a global operator kill switch must preserve owners' authority to stop their own instances, revoke access and request deletion under the declared retention contract. Resilience requires independent discovery, placement and recovery paths as well as running replicas; it cannot guarantee operation when all authorized providers, keys or viable communication paths disappear.

Partition migration must fence ownership. A proposed sequence is: retain/copy the old state and receipt history; catch up its journal; durably close the old generation at a cut; verify the new owner through that cut; compare-and-swap the routing generation; activate the new owner. Recovery resumes from recorded stages. The old owner rejects later writes; ambiguous replies retry their identities. Prefer temporary unavailability over two active owners. Cross-partition atomic mutations remain unsupported until a recoverable coordinator protocol and failure tests exist.

Peer federation is different from replicas of one authority. Domains exchange explicitly shared projections through durable outbox/inbox records carrying identity, content commitment, schema, provenance and permission scope. Receiving an object authenticates an assertion, not its truth. Peers reconcile cursors and duplicate identities without importing executable governance. Offline peers may author their own branches; shared-document merging needs a defined conflict/CRDT protocol, and globally conserved balances or unrestricted unique-name allocation need coordination or preallocated rights. Invariant-confluence research formalizes why coordination avoidance depends on the operations and invariants, not the word “decentralized.” Bailis et al., extended paper.

AO's independent processes and message passing are useful inspiration; its Arweave-backed history and runtime are separate architectural commitments. RURAL should first define a portable domain-message contract, then investigate an optional AO adapter. No AO service becomes a dependency of the local database, and permanent publication cannot be the default home for revocable private data. AO concepts.

10. Grow while upgrading

Version the wire protocol, durable format, typed plan, indexes and semantic definitions independently. A proposed rollout reads old/new formats while initially writing the old common format. Replicas advertise capabilities; new features activate only after the eligible set and restore tooling support them. etcd's documented mixed-version operation provides a concrete example of common-version gating and constrained rollback. etcd 3.5 to 3.6 upgrade.

Build replacement indexes/segments beside the active ones, retain a starting root, catch up changes, verify equivalence, then publish a new manifest generation. Old readers keep old manifests. Budget both representations until their pins expire. A schema migration needs explicit dual-read/write or a bounded write pause; a format flag alone cannot make incompatible values safe.

Declare the last rollback-safe point before activating irreversible writes. On one machine, binary replacement may require restart/recovery; online data migration does not imply zero-downtime executable replacement. Distributed rolling upgrades require membership health and failure-domain checks. Never load arbitrary new native code into the durable writer merely to claim hot upgrades.

11. Falsifiable delivery gates

Measure useful durable operations and correct answers under mixed workloads, including tail latency, retained bytes, spill, synchronization, amplification and recovery. The proposed advantage is coordinated operational context with controlled resource cost. It remains a hypothesis until those gates demonstrate it.

Source register

All external sources accessed 2026-09-20. References inform engineering constraints; no cited system is a required runtime dependency.

1. PostgreSQL 18 JSON functions: null, missing-path and conversion semantics.

2. Git commit-tree, documentation 2.55.0: commit/tree identity.

3. Git worktree, documentation updated 2.54.0: shared state, locks and removal safeguards.

4. Docker pruning: resource scope and deletion behavior.

5. DuckDB memory management, 2024: streaming, spill and cache lifecycles.

6. Kuiper, Boncz and Mühleisen, Robust External Hash Aggregation in the Solid State Age, ICDE 2024: out-of-core aggregation.

7. Amazon S3 conditional requests: explicit storage preconditions.

8. PostgreSQL 18 pg_verifybackup: verification boundaries and restore testing.

9. Ongaro and Ousterhout, extended Raft paper, 2014: consensus, reads and client retries.

10. Bailis et al., Coordination Avoidance in Database Systems, extended version: operation-specific invariant confluence.

11. AO concepts: actor processes, messages and persistent history.

12. etcd 3.5 to 3.6 upgrade guide: mixed-version operation and rollback boundary.