Architecture overview

The proposed branching-world architecture, concurrent readers, intake and explicit security and performance limits.

Architecture direction, 19 September 2026. The Operator approved starting the local alpha and expanded the requirements to fractal engineering, concurrent analysis/intake, server workloads, semantic networks and strong attack resistance. This document revises the earlier serialized-query foundation. It is a design, not a claim of implemented features, audited security or measured performance.

1. Philosophy: every experiment deserves a coherent world#

A useful database should let us ask what exists, what happened, what we believed at a particular time, what changes in an alternative branch, and why an answer is justified. It should do that while new events continue arriving.

RURAL's proposed unit of work is a world: a named, versioned view of data, code context, schemas and declared governance. A world can produce child worlds. Unchanged storage is shared; changes remain isolated until an explicit, validated promotion. The Oracle sees an accountable view of the world, not an unbounded pile of text.

Six principles:

1. Change creates a version. Users still create, update and delete normally; the engine preserves the versions required by active snapshots, branches and retention policy.

2. A branch is a reference before it is a copy. Forking unchanged data should primarily publish metadata, while divergent edits pay their actual cost.

3. Readers keep their world. A query pins a coherent snapshot; later writes publish another state rather than rewriting what it is reading.

4. Meaning has a source. Relations, metrics, assertions and inferences carry definitions and evidence. Similarity is not identity; a signed claim is not necessarily true.

5. Authority does not spread by copying. A branch or imported graph cannot grant itself credentials, permissions or the right to trigger external actions.

6. Computation approaches the data. Filter, aggregate, compare and resolve changes before sending compact evidence to AI.

This combines established ideas into a UNIVERSE-specific system. It does not claim that MVCC, persistent trees, bitemporality or semantic graphs are new inventions. Datomic demonstrates immutable database values; XTDB distinguishes the time data was recorded from the period it describes. We borrow useful concepts while preserving RURAL's ordinary mutable CRUD interface and chosen dependency boundary. Datomic transaction model, XTDB temporal concepts.

2. The places and their dimensions#

Repository islands remain the source foundation. The main floor, on that foundation, contains operations and governance. The sky brings website/product analytics, forms and business events. Oracle/MCP provides bounded, authorized insight across them.

The analytics backbone design makes sky intake and the warehouse central responsibilities, including viral bursts, durable receipts, producer retries, protected query resources and later replicated/global delivery. The local world model must support this path without claiming unmeasured capacity.

A record can participate in several explicit dimensions:

“Multidimensional” means these coordinates are first-class and composable. It does not mean building an index for every possible combination. Physical indexes follow observed queries; unused dimensions do not automatically multiply storage and write cost. Metadata can use interned IDs and sparse fields rather than repeating large envelopes in every value.

Distinguish the policy observed in an old world from permission to read that world now. Revocation applies to historical reads, cached answers, snapshots and evidence expansion.

3. Fractal engineering: logical recursion, controlled disk layout#

A workspace can branch from a branch indefinitely in the logical model, subject to configured resource/lineage limits. Its identity and ancestry are separate from its filesystem path.

Four different objects#

• Git commit/branch: code history and a selected starting state.

• Git worktree: a physical checkout with its own HEAD/index, sharing the repository's object database and many references.

• RURAL branch/root: a coherent data view and subsequent data changes.

• Workspace manifest: the durable association between those things, source observations, declared governance and execution restrictions.

Git linked worktrees are not isolated independent repositories. A “grandchild” should normally become another managed sibling checkout created from an explicit commit, while the manifest records its ancestry. Deeply nesting checkout directories is unnecessary and risks accidental recursive scanning, cleanup mistakes and path growth. Git remains an external development integration through UNIVERSE, not a dependency of the RURAL database runtime. Git worktree documentation.

Proposed workspace manifest:

workspace_id, parent_workspace_id, fork_operation_id
repository_id, code_commit, working_copy_manifest_if_dirty
database_branch_id, base_commit, immutable_root_id
schema_version, declared_governance_version
checkout_location, lifecycle_state, retained_storage_budget

A dirty working copy requires an explicit bounded content manifest or an explicit exclusion. A commit hash cannot describe uncommitted contents. Credentials, active jobs, sockets, payment actions and outbound messages are never duplicated as live capabilities by a fork.

Cheap database forks#

A fork pins a concrete committed root and catalog version, then durably publishes a new branch descriptor. Child and parent initially share immutable nodes/segments. Later parent writes cannot leak into the child. A child of a child starts from its resolved root; reads must not search an ever-longer chain of ancestor overlays.

The target is fork work proportional to metadata and changed paths, not the entire dataset. This excludes Git checkout materialization, remote transfers, index rebuilds and fsync latency: those costs remain real. New pages/versions are allocated for divergence. A branch retaining old data consumes storage even when no current query uses it.

The alpha can use a persistent in-memory map with immutable roots and durable replay/checkpoints, provided fork/lookup cost does not grow with ancestor depth. A paged persistent tree/segment catalog is the later disk-scale representation behind the same snapshot API. Do not implement a full database copy per fork and call it efficient branching.

Git plus database is a recoverable workflow#

There is no atomic transaction spanning Git, arbitrary filesystem changes and the database. Use an idempotent coordinator:

1. Persist a fork intent with exact source identifiers and an operation ID.

2. Pin/create the RURAL branch durably.

3. Create the owned checkout in a bounded staging location using explicit Git arguments.

4. Verify commit/content identity, ownership and data-root accessibility.

5. Publish the workspace manifest as ready only after all required components are verified.

After a crash, resume the recorded operation or report a recoverable incomplete workspace. Never guess ownership, delete a user's dirty checkout, or prune shared repository state to make recovery appear successful. This feature does not authorize creating worktrees in real user repositories without an explicit operation request; tests use disposable fixtures.

Merging worlds#

Use a common-base, source, target three-way comparison. Start with whole-record, edge and schema-object comparisons: independent object changes can be proposed for merge; different changes to the same object conflict. Delete-versus-modify, relation cardinality and uniqueness require validation even across distinct objects. Automatic field-level merge is later work. Recheck target/root versions at commit to avoid applying a stale merge plan.

Rebuild derived indexes/aggregates where appropriate; do not blindly merge physical index pages. Raw source events use their stable identities so merging branches cannot double-count intake. A data merge never replays emails, payments or deployments. Live external effects require a separate authorized execution record.

For the first alpha: explicit fork, inspect, diff, conflict detection, bounded merge/promotion and branch deletion with retention checks. Automatic semantic conflict resolution and transparent cross-server branch migration are later work.

4. Disk correctness and lifecycle#

A proposed immutable root is not visible merely because its bytes were written. The durable publication protocol must cover its data, schema/index state, deduplication, source cursor and branch metadata coherently.

Required ordering:

• Validate/admit the complete bounded transaction against current authoritative state.

• Write checksummed, versioned transaction/root information and establish the configured OS durability boundary.

• Publish the committed root to readers with correct memory synchronization; only then acknowledge the transaction.

• Build checkpoints/segments off to the side, verify them, publish their manifests atomically and durably, and retain prior recovery material until the new state is known durable.

Actual page/log ordering is specified with the chosen format before implementation. POSIX rename, fsync and platform-specific durable-flush assumptions need direct failure tests; none independently proves survival of every device failure. Corruption in committed history is not permission to truncate acknowledged data silently.

Reachability roots include live branches, open readers, retained history, backups, restore operations and in-progress workspace intents. Garbage collection cannot reclaim anything still reachable. Use bounded retention, snapshot leases and per-branch retained-byte accounting. Readers exceeding their resource contract fail explicitly; they do not pin unlimited storage forever. Conservative leaked space is preferable to premature reclamation, but must be visible and repairable.

Sharing also has a confidentiality boundary. Do not enable cross-security-domain content deduplication casually: equality leakage and key management can defeat isolation. Hashes are integrity identifiers, not encryption. Secure physical erasure of SSDs/backups is a separate requirement, not implied by DELETE.

5. Reads continue while writes commit#

The old plan's serialized query executor is superseded. Use MVCC/immutable roots with concurrent query workers from the foundation.

Readers acquire a stable root in a short publication step, then scan outside the mutation critical section. The initial alpha can have one ordered mutation owner and many snapshot readers. Later, independent partitions acquire their own mutation owners so unrelated writes can use multiple cores.

Snapshot isolation alone is not serializability. The first write path should validate bounded transactions at the owner, including the expected state used by host authorization and application reads. An arbitrary old query result cannot become a write without conflict validation. This distinction is established in MVCC research. Fast serializable MVCC paper.

What concurrency does and does not mean:

• Writes do not require stopping all snapshot readers.

• Readers do not hold mutation locks throughout a scan.

• Conflicting writes, uniqueness checks and cross-partition invariants still require ordering or coordination.

• Readers, writers, compaction and inference still share finite CPU, memory and I/O. “Nonblocking” does not mean zero interference or constant latency at any offered load.

• Multi-partition “latest” queries need a coherent published root set; taking an unrelated latest snapshot from each owner is not a global transaction snapshot.

Use explicit resource classes for interactive operations, intake, large analysis and maintenance. Bound queue bytes, scan work, graph frontiers, intermediate rows, query memory, snapshot age and retained bytes. Admit, queue within a bound, or reject; never accept unlimited work then rely on the OS to recover from exhaustion.

6. One logical history, specialized physical views#

Hot mutable documents, forms and graph adjacency prefer row-oriented access. Large historical analytics scans benefit from typed column segments and selected aggregates. Both derive from committed data with one definition of record identity/version/deletion.

The server milestone introduces immutable column segments containing row/version identifiers, typed columns, null/missing validity, offsets, checksums and min/max statistics. Add dictionary/integer/run-length encoding when measurements justify it. No required Arrow/Parquet/Kafka/Redis service or third-party compression library is implied.

Let a projection cover changes through commit H and a reader pin snapshot S. Support explicit modes:

• Answer consistently through H and report that freshness.

• Merge a version-aware committed tail (H, S] for an answer at S.

• Wait with a deadline for a projection covering S.

The merge resolves replacements and tombstones by identity. Naive union double-counts corrections. A 100->103 provider revision replaces 100; a changed dimension moves contribution between buckets. Pin the projection manifest and required tail together before reclamation. Analytical MVCC designs show why compressed segments and update versions need separate treatment. DuckDB analytical MVCC.

Queries exceeding memory need bounded external algorithms/spill or explicit rejection. A high-cardinality group-by is not bounded merely because its final output is small. Old snapshots retain storage and can force growth; branch retention has the same cost. DuckDB memory management, LMDB design paper.

7. Semantic networks and decentralized worlds#

A semantic cluster should be a versioned query/projection over typed relationships, not a magic vector field. Preserve asserted, inferred and hypothetical relations separately, each with source snapshot, definition/model version and provenance. Similarity scores do not justify merging identities or authorizing access.

Useful first mechanisms are typed edges, named contexts, explicit relation definitions and bounded traversals. Later clustering jobs run on pinned snapshots and publish rebuildable results. Embeddings can be supplied by an optional external UNIVERSE capability; they are not a required model runtime inside RURAL. RDF's named-graph work illustrates both contextual graphs and the need to specify their meaning rather than assume every claim is globally true. W3C RDF concepts, dataset semantics.

For large social networks, partitioning by ownership/locality can make common neighborhoods cheap, while supernodes and cross-partition relationships remain expensive. Model deleted/revoked observations, provenance, fan-out limits, duplicate messages and moderation decisions explicitly. A public-key identity alone does not solve Sybil attacks or make claims trustworthy.

AO offers a relevant direction: autonomous processes communicating through messages, with separate computation/storage responsibilities. Its official implementation context includes Arweave and HyperBEAM/Erlang, so adopting that stack would violate the current core dependency boundary. RURAL can later expose a protocol adapter, durable per-owner mailboxes and explicit causal references without making AO a prerequisite. AO introduction, HyperBEAM introduction.

Federation needs contracts for identity, replication, conflict resolution, deletion, trust, quotas, offline operation and causal ordering. Strong local transactions and explicit inter-domain messages are a plausible starting point. Instant globally consistent writes, unlimited partition tolerance, zero coordination cost and lowest latency everywhere are not simultaneously deliverable properties. No AO compatibility or decentralized security guarantee is claimed by this design.

8. Oracle: ask once, refresh cheaply#

Extend the compact MCP contract with explicit branch/root/snapshot coordinates. A query receipt binds its plan, branch, data/schema/metric versions, authorized scope and evidence. Context expansion must retrieve the same version or report its unavailability.

Prepare a question once. Reuse its typed plan, perform work near the data and refresh only when relevant changes occur. A dashboard can update deterministic computed values without calling the language model for every event. For unsupported incremental aggregates, return a small invalidation and rerun the prepared query instead of pretending arbitrary incremental view maintenance exists.

Result/context caches require current authorization checks and version-aware keys. Policy changes can invalidate a response even when data did not change. Give the Oracle compact answers, contradictions, freshness and evidence handles; keep bulky receipts and raw data accessible outside its prompt. Measure full interactions including discovery, retries and expansion, not an artificially tiny isolated response.

9. Attack resistance is a design, not a slogan#

Use one first-party binary with separable roles where useful; no privileged public database console is implied.

For public intake, a restricted process should communicate over local IPC to the owner and have no direct data-file handles. Per-source/tenant limits constrain accepted bytes, records, rate, durable disk use and processing work. Parsing rejects excessive nesting/lengths and unsupported encodings. Deadlines and bounded buffers cover slow clients; decompression is disabled initially or separately budgeted. CORS is not authentication, and a token embedded in browser code is not a secret.

Typed query plans and bound values avoid query-string interpolation. Still test parser, optimizer, traversal, aggregation and index amplification attacks. An input LIMIT does not bound a graph expansion or a huge intermediate join. Protect catalog discovery, errors, cache keys, timing-sensitive metadata and backup paths, not only returned rows.

A submission receipt must distinguish durably accepted from business processing completed. Idempotency is scoped and persisted; callbacks/webhooks are not executed inside commits. Signed webhook verification, replay windows and provider identity belong in the concrete adapter. Schema-valid anonymous form data remains untrusted content.

Transport and dependency constraint#

No outside libraries does not automatically produce secure software. Zig code still needs integer/lifetime/concurrency checks, fuzzing and review. Build/runtime safety settings must be documented; fast builds cannot quietly disable checks needed for the threat model.

Public TLS on Linux is a real unresolved gate under the strict dependency rule. Kernel TLS does not supply the complete user-space handshake. Native platform TLS APIs may be used where permitted and sufficient; a required third-party TLS library/proxy must not be smuggled into the design. Do not ship newly handwritten cryptography and call it audited. Resolve the transport through a permitted, reviewed implementation or an explicit future dependency-policy decision before public exposure. Local/private operation can proceed meanwhile. Linux TLS documentation.

The single-server release requires protocol/parser fuzzing, hostile-load tests, authorization tests, replay/tamper tests, dependency/build review, crash/restore tests and an independent security review of the exposed boundary. Test that one scope cannot read another via aggregates, branch sharing, histories, handles, errors or caches. Include restore-time permissions, malicious source text and stale grant races.

No database can promise immunity to all attacks or prevent a volumetric attack from saturating its upstream network. Network-level filtering/capacity remains a deployment concern, not a property we can prove with an engine unit test.

10. Performance, cost and the test that could falsify the idea#

Optimize cost per useful durable operation and correct answer, including storage, CPU, memory, I/O, retention, replication when enabled, and AI tokens. Compare equivalent semantics and durability; do not win a benchmark by weakening acknowledgement or omitting indexes/retention.

Calibrate the admitted-rate/SLO envelope on named hardware before making product claims. Report attempted, admitted, completed, rejected and retried work; do not hide rejection when reporting throughput. Proposed mixed-load experiments can investigate a transactional p99 no worse than 2× isolated baseline while analysis retains at least half its isolated throughput, but those are hypotheses to validate, not achieved guarantees.

Retention dominates cost: 10,000 observations/second at 1,000 bytes each is 864 GB/day before indexes, history or replicas. Fast code cannot erase that arithmetic. Policies for aggregation, raw retention, compression and cold storage must be explicit and preserve the intended evidence.

11. Build sequence#

Local alpha, now authorized: normal database/UNIVERSE/MCP capability plus immutable read roots, concurrent snapshot readers, branch-of-branch support, isolated workspace fixtures, conflict-aware diff/merge and correct root retention. Start with one mutation owner. This changes the foundation now; it is not postponed until the engine becomes impossible to adapt.

Single-server milestone, required next: Linux validation, disk-scale event segments, bounded parallel partition writers where useful, forms/intake isolation, mixed-load cost evidence, and completed transport/security gates. It remains a separate acceptance milestone so a local alpha is not falsely advertised as a safe internet service.

Federated/decentralized milestone: explicit domain/message and trust protocols, partitioned semantic graphs, replication/recovery, cross-domain queries and optional AO integration. Research/development work stays visible; none is implied complete by the alpha.

The Project Elaboration maps these commitments to tasks. The UNIVERSE model still defines the main-floor/sky meaning and compact MCP API; this document supersedes its original serialized-query implementation suggestion.

12. Required communication and typed specialization#

30 September 2026 clarification: fractal composition also covers recursively specialized stores and secure decentralized communication. The secure fractal cells design records Matrix/Signal research, numeric/boolean/decimal/time-series/vector profiles and the shared semantic, authority, key-lifecycle and recovery boundaries. AO/Quilibrium-style execution is a required architecture target; selecting particular providers and proving runtime compatibility remain distinct decisions. These requirements guide core contracts now while implementation and performance remain to be qualified.