Backup and portability

Define what a complete database backup must preserve and what remains outside it.

For non-compacted stores, P12 backs up one validated, complete durable journal capture. A capture is a versioned RURALBK1 file containing the journal length, store identity, final sequence, SHA-256 checksum, the source receipt-table cap (max_receipts, format version 2), and the full journal image. Format version 1 artifacts are refused with UnsupportedBackupVersion; no receipt-cap default is substituted. The capture is taken frozen under the database writer gate and streamed without that gate, written to an exclusive staging file, synced, then published with a no-replace rename. It contains no owner token, runtime descriptor, socket, queued job, or external-effect activation state.

Restore accepts only a private, single-link regular file opened without following symlinks. Its bounded framing, checksum, store identity, sequence, and journal frames must all validate. It creates an exclusive private staging directory, writes and syncs journal.log and table.limits (from the current capture), then opens the database to replay and rebuild in-memory projections. Successful replay precedes exclusive final destination publication. An existing destination is rejected. A truncated, corrupt, version-mismatched, or oversized backup fails before destination creation. Validated restores use private staging and replay before final destination publication. Restored runtime credentials must be created by the service bootstrap; pending external intentions require explicit local reconciliation and are never dispatched by storage recovery.

max_backup_bytes bounds the complete capture, including its 80-byte v2 header (256 bytes plus checkpoint sections for v3), on both creation and restore. An insufficient creation budget fails before a staging file is created. The writer gate then freezes the clean journal prefix (length, store identity, and sequence). That hold is the admission cost: the capture streams the frozen prefix afterward, without the gate, through a fixed 64 KiB buffer, checking each frame and the SHA-256 image checksum without allocating the image. Appends during the copy extend the live log past the prefix, so the artifact stays a consistent earlier snapshot. Committed journal bytes are never rewritten or compacted, and a failed append only truncates back to the last durable boundary, which is at or past the frozen length. If the unlocked copy finds a bad frame or checksum, it poisons the source under the gate, so later transactions and backups return RecoveryRequired. The backup header, plain fsync, and no-replace rename stay after the copy. For the zero-base v2 streaming path, the on-disk format is unchanged and a streamed capture restores like earlier v2 artifacts. Checkpoint-dependent v3 captures use the separately versioned format below.

On an unpublished write, sync or rename failure, creation closes and attempts to remove only the staging file it exclusively created, permitting a same-name retry when cleanup succeeds. Pre-existing staging and final files are preserved. A failure syncing the directory after publication leaves the final capture in place and returns BackupPublicationUnknown. inspectAndSync validates the original artifact against the expected store, then syncs its file and directory before returning its original manifest. It never captures newer source writes under the same identity. Resync failure also reports publication uncertainty. Cleanup I/O failure or process termination can leave staging material; inspect that material explicitly rather than overwriting it automatically.

restore.intoNewDirectoryFrom accepts separate source and destination directory handles; intoNewDirectory preserves the original same-directory API. Both return an owned open database that the caller must close. Neither starts a service or restores runtime credentials. Shared validation checks metadata before and after the bounded read and verifies the complete journal checksum. This does not promise protection from arbitrary hostile concurrent filesystem mutation or hardware failure.

Focused storage tests exercise exact budgets, preallocation refusal, corruption, exclusive staging and destination behavior, partial writes, post-rename sync uncertainty, same-artifact retry after later source writes, foreign-store refusal, public permissions, symbolic and hard links, separate directories, and allocation rollback. Live service admission, artifact quotas, authenticated commands, and CLI runtime evidence belong to their composition layer.

Offline compaction and table caps#

rural compact / compact.intoNewDirectory copies CURRENT, the latest checkpoint generation files, and the required table.limits sidecar, then writes a version-3 journal suffix. Reopening the compacted store under default options keeps the source receipt-table cap. Compact still never modifies source bytes.

A compacted journal has base_sequence != 0 and requires its published CURRENT, checkpoint manifest and image. Checkpoint-dependent captures use RURALBK1 version 3, with a 256-byte header followed by those three sections and the complete frozen compacted journal. Non-compacted creation stays at v2; restore accepts v2 and v3 artifacts. Version 1 is refused before destination creation. Older readers refuse v3.

The v3 header preserves v2 identity, final sequence, journal length/checksum and receipt-cap offsets. At bytes 80/84 are the exact CURRENT/manifest lengths (104/160); byte 88 holds the image length. Bytes 96/128/160 hold section SHA-256 digests; bytes 192–255 must be zero. Section lengths, digests, checkpoint codecs, store/generation identities and compacted journal base sequence/logical offset must agree. The caller's complete-artifact byte cap covers every section. Images and validated artifacts are allocated under this finite cap; journal creation still streams through the fixed 64 KiB buffer. Checkpoint decoding uses its existing separate owned-image bound. This does not promise image-size independent peak memory.

Creation holds the checkpoint mutex through selected-generation copying and journal capture, pinning generation files against publication/pruning. Ordinary writes continue after the journal prefix freeze. It does not publish a new checkpoint. Service backup also skips its optional post-backup checkpoint publication for a compacted store: replacing CURRENT without changing the compacted journal base would break source reopening. Its checkpoint_publish field is null when no attempt occurred, including this case and artifact retry.

Restore validates the complete artifact before creating an exclusive private .rural-restore-staging-<destination> directory. It writes and syncs checkpoint sections, journal and receipt limits, then replays using the existing database open path and checks final identity/sequence. Only a successful replay permits no-replace publication to the requested new destination. On macOS, a bounded Darwin renameatx_np(RENAME_EXCL) helper publishes directories because the pinned Zig Io fallback hard-links files. Other platforms retain the Io primitive. Before publication, failure attempts to remove only this invocation's staging directory. After publication, sync/reopen failure preserves the destination and returns RestorePublicationUnknown from the storage API and CLI error exit; inspect the complete destination explicitly rather than overwrite it. Cleanup failure or interruption may leave staging material. The clear WAL restore.intoNewDirectory / intoNewDirectoryFrom profile preserves captured logical state, revisions, branch heads, sequence and receipt capacity, then publishes a fresh store identity with an empty retry ledger and receipt sequence floor captured sequence + 1. Requests bound to the original store refuse with WrongStore; historical retries remain valid only on that original store. A new restored-store intent uses the new identity, current epoch/context and current branch head, commits once and then replays its own receipt.

The separately qualified native C02 restore profile preserves captured store identity, incarnation and receipts; it requires trusted fencing of the original writer. Neither profile restores runtime credentials or grants authority.

The original W04 refusal remains applicable to a checkpoint-dependent journal embedded in an old journal-only artifact. Missing/corrupt checkpoint sections or mismatched base identities fail, rather than silently recover from a different source. Full D24 corpus/native qualification remains a separate acceptance gate.

The offline compactor retains its existing 1,024-branch checkpoint decode limit. V3 framing validation bounds counts by encoded byte/owned-image budgets rather than adding that branch cap. Restore still applies the caller's configured branch/key/value limits through database replay; only receipt capacity is embedded in the backup header. A finite independent compacted fixture with 1,025 branches verifies a configured 2,048 branch/receipt capacity through capture and restore. This does not expand the offline compactor's supported input or qualify other checkpoint publication entry points.

The disposable public recovery drill compares schema, two documents, a real relation/index read, complete original probe receipts and retained fork root contents/snapshot identity across compacted backup/restore. Default restored service startup keeps the source receipt cap. This historical witness covers retained branch roots and lineage; general compacted history.snapshotAt or public compare reconstruction remains unqualified because that owner does not seed replay with the checkpoint. No pre-compaction journal history promise is made. Full corpus/native recovery qualification remains separate.

Canonical durable admission#

New stores persist their chosen receipt capacity in table.limits. Existing stores and offline compaction require that file: absence returns MissingStoreLimits, without adopting caller defaults or regenerating the file. The current LIM1 file must be exactly 16 bytes, regular and opened without following symlinks. Invalid length or content is refused; filesystem refusals for symlinks or directories remain typed I/O errors. Metadata is checked around the read under the store owner lock. This is not protection against an arbitrary concurrent filesystem attacker.

Admission refusal preserves journal, limits and backup bytes and does not publish a restore or compaction destination. Opening the store may still create or sync its owner-lock file; refusal does not promise zero filesystem effects. Existing older stores and artifacts remain untouched and require explicit handling outside these APIs. There is no automatic migration.