Durability and recovery

The single-owner local journal, append and sync semantics, and bounded tail recovery.

Status: implemented A02 journal foundation in journal.zig, using the frame format. This is a single-owner local journal, not a database, transaction engine, server, backup system, or replication implementation.

API#

• Journal.open(allocator, io: std.Io, parent: std.Io.Dir, directory_name, options) !Journal opens or initializes one child directory. The child name must be one component, excluding empty names, ., .., separators and NUL. The caller owns parent; the journal owns its child directory and files.

• Options requires max_payload and max_log_bytes. The latter includes the file header and bounds whole-log allocation on open/replay. create_if_missing defaults to true. The explicit synchronization mode is os_sync.

• storeId() !StoreId returns the immutable 128-bit store identity for a clean journal. StoreId is [16]u8; the bytes are opaque, without UUID version/variant encoding. lastSequence() returns the last validated sequence. nextSequence() requires a writable journal and returns the next sequence, rejecting exhaustion.

• appendExpected(expected_sequence, kind, payload) !u64 rejects an unexpected sequence before allocation/I/O. append(kind, payload) is the convenience form. Both return only after the frame has been written and file synchronization has succeeded.

• replay() !Replay copies and validates the complete log and captures store_id and last_sequence. Replay.next() !?Frame skips the initial file header and returns ordered payloads borrowed from the replay buffer. They remain valid after the journal closes, until Replay.deinit() frees the buffer. Acquiring this snapshot requires the engine's writer gate; writing a backup from the owned copy afterward does not.

• validateSnapshot(bytes, max_payload) !Metadata validates a complete journal image without allocating. It returns the store identity and final sequence; invalid images return InvalidJournal. Callers must separately admit total byte length.

• close() releases resources and the OS lock; it does not remove the lock file. A journal is a move-only owner and must not be copied or called concurrently. The engine serializes access.

recovery is either clean or required: RecoveryIssue. An issue includes its reason, first problematic offset, validated prefix length/last sequence, file length when known, and underlying frame/I/O error when applicable. Recovery-required handles reject replay and appends. A valid prefix is diagnostic information, not permission to apply a partial database history.

The engine should allocate and validate its proposed state before appendExpected, then publish that prepared state without allocating after success. This module does not perform that state publication or validate application payloads.

Files, ownership and initialization#

The dedicated directory contains owner.lock and journal.log. New directories request mode 0700 and new files mode 0600, subject to the OS umask. Only regular, single-link files are accepted. Existing final symlinks are not followed. The caller supplies a trusted parent directory; these checks do not defend against another principal allowed to rename or replace the directory or files while the journal is running.

The persistent lock inode is opened with a nonblocking exclusive OS advisory lock. A second cooperating owner receives AlreadyOwned. Kernel lock release on process exit replaces stale-PID recovery; normal close never unlinks the lock file. Deleting/replacing a live lock inode can invalidate exclusivity and is outside the supported ownership contract. Do not share this directory over a network filesystem or let other processes write its files without this protocol.

An initial frame has catalog kind, sequence zero. Version 4 uses a 32-byte payload: RURALJNL (8 bytes), little-endian journal version 4 (4 bytes), four zero reserved bytes, then the immutable 16-byte store identity. Version 5 (compacted journals and later segments) uses a 48-byte payload with the same prefix shape, version 5, the store identity, then little-endian base_sequence and base_offset binding the retained suffix to a published checkpoint or the preceding segment. The entire payload is protected by the normal frame checksum. New empty stores still create version 4. Version 1 remains unsupported_file_version. Data frames after a version-4 header start at sequence 1; after a version-5 header they continue from base_sequence + 1.

Offline compaction (rural compact, ADR-006) copies CURRENT and the latest checkpoint into a new directory and writes a version-5 journal containing only the suffix after the checkpoint offset. The source store is never modified.

Initialization writes and syncs the header, syncs the persistent lock file, then syncs the containing directory and supplied parent directory before returning. Reopen also syncs the lock file and directory entries, covering interrupted creation before directory sync. The caller must already have made ancestors of the supplied parent durable; this module does not create arbitrary ancestor paths. std.Io must be an actual OS-backed implementation for the durability claim.

Interrupted initialization leaves its files in place. Existing zero-length/partial files are never reinitialized or silently replaced. Initialization write/sync failure returns InitializationOutcomeUnknown; inspection on the next open determines whether the header is complete. Directory-sync errors are propagated and initialization is not reported successful.

Segments (GROVE phase 1, RG-03)#

Options.segment_bytes: ?u64 = null makes rolling opt-in. Null preserves single-file behavior, including LogLimitExceeded at max_log_bytes. With a value, an append that would push the active segment past that size rolls to the next segment first.

Segment 1 is always journal.log; existing stores need no migration. Segments 2 through 999999 are journal-NNNNNN.log with six digits, starting at journal-000002.log. journal-000001.log, gaps and invalid segment names are refused.

Segment k >= 2 starts with a version-5 header carrying the same store identity, base_sequence equal to the last sequence of segment k-1, and base_offset equal to the logical offset where its first data frame starts. The logical image concatenates all of segment 1 and the data frames of later segments, excluding their headers; it is a valid single-file version-4 journal.

Open validates every earlier segment as clean and contiguous, then opens the last as active. replay, replayBounded and copyPrefix use the concatenated image with their existing bounds. Journal.file and size refer to the active segment. Rolling requires a version-4 first segment; compacted stores do not roll in phase 1.

Append and acknowledgement#

The writer checks sequence, payload size, log capacity and current file length, allocates the frame, writes it at the known end, and synchronizes the file before publishing its new in-memory sequence or returning success. This implementation synchronizes each append; callers may batch multiple logical records inside one opaque transaction payload. There is no unsynced success mode.

A write or sync error may occur after some or all bytes reached storage. The journal returns AppendOutcomeUnknown, preserves the original pre-attempt boundary, records the underlying error, and refuses further writes/replay. Allocation/admission errors happen before writing and do not poison the handle. The engine must not acknowledge or publish a transaction whose journal append returned an error.

On reopen, complete valid contiguous frames are replayable and are synced before writable open succeeds. Their suffix may contain operations whose success reply was lost, including a crash after synchronization and before the application received success. Valid frames are not evidence that the caller received acknowledgement. Retried logical requests require persistent engine-level idempotency.

Incomplete suffixes and deliberate recovery#

Options.torn_tail = .refuse preserves every incomplete or corrupt suffix and returns recovery required. Opt-in .repair_unacknowledged truncates only a single trailing incomplete or invalid frame at valid_bytes: the file must end inside that frame or exactly at its declared end for an invalid frame. A complete frame header must parse and carry sequence last_sequence + 1; a partial header is eligible only as an incomplete frame. Header damage, sequence mismatches and earlier corruption remain recovery required. Repair truncates, syncs and re-inspects before permitting appends, reporting torn_tail_repaired with the truncated byte count and sequence. Under the interrupted-append assumption this is safe because acknowledgement follows sync: a trailing partial frame numbered last+1 was never acknowledged. External truncation or corruption of acknowledged bytes cannot be ruled out from framing alone; the caller explicitly opts into that assumption.

discardFailedAppend() is the narrow deliberate repair operation. It is available only on the same still-open handle that received AppendOutcomeUnknown. The handle knows its last prior successful boundary and that this single subsequent attempt never returned success. It validates that the prefix through that boundary is still a complete contiguous journal, truncates only the failed attempt, and syncs the file before restoring writable state. The caller must explicitly choose to abandon that unacknowledged attempt. Replaying it instead after reopening may also be valid; choose one outcome through the engine's idempotency policy.

A truncate/sync failure during this deliberate repair returns RepairOutcomeUnknown and leaves the handle blocked. Retrying the same repair on that handle is permitted after the I/O condition is resolved. A process crash during repair loses the in-memory proof: the next open inspects normally and applies only its explicitly selected torn-tail policy. discardFailedAppend() on a reopened handle returns CannotProveUnacknowledged. Operator-led backup recovery or a separately designed, externally justified repair protocol is required for a malformed reopened journal.

This protocol assumes cooperating owners and correct positional-write behavior. It detects accidental framing corruption; it does not authenticate the writer, guarantee protection against malicious file replacement, or repair earlier damaged records.

Durability and capacity limits#

os_sync uses std.Io.File.sync. Zig 0.16.0's standard POSIX threaded backend calls fsync. On macOS this implementation does not call F_FULLFSYNC. The current evidence supports OS-sync-before-success and tested reopen/I/O-fault behavior; it is not a certification of power-loss survival across every filesystem/controller/device. Linux/macOS are the explicit platform scope; only platforms actually tested should be reported as validated.

The journal is local to one storage device. Losing the host, device, directory or every valid copy can lose the data. Replication, off-host backup, encryption, power-failure testing and disaster recovery remain separate requirements. Opening or mutating a live journal through a shared network filesystem (NFS, SMB, and similar) is unsupported: one owner process holds the files; remote clients use the server API (see ADR-008).

Open and replay read the log into bounded memory; encoding allocates a bounded additional frame. Existing files exceeding max_log_bytes fail with LogLimitExceeded before log-buffer allocation. There is no checkpoint rotation, compaction, retention or larger-than-memory replay in this module. When the limit is reached, new appends fail explicitly. Long-term ingestion needs the next checkpoint/retention milestone.

Tests#

Inline tests use real temporary directories for append/reopen, owned replay lifetime, exclusive ownership and stable lock inode, bounds, sequence expectations, allocation failure, incomplete initialization, future file versions, torn suffixes, checksum corruption, sequence gaps, and a valid unknown-acknowledgement suffix. An injected std.Io vtable writes a real partial frame or fails synchronization on the journal/directory descriptor. These tests check that no success/new sequence is published before sync, failed directory sync prevents successful open, the handle remains blocked after unknown outcomes, and deliberate same-handle repair refuses a damaged prior prefix and synchronizes before allowing more writes.

The injected I/O tests are fault-path evidence, not physical power-cut tests. Database-state replay, engine idempotency, checkpoint/backup restore, server intake and entire-host loss are not implemented by this module.