Skip to main content

WAL — Snapshots & reader postures

An op-log grows forever. A snapshot is a materialized state a new reader adopts so it replays only the tail instead of the whole history. Snapshots live in a sibling LWW document, <documentKey>__snapshot, and are client-generated by a trusted role — the server holds no keys and cannot fold.

interface WalSnapshotDoc {
  state: Record<string, unknown> // sealed CrdtState (full state incl. tombstones)
  uptoTs: number                 // the contiguous ts-prefix this snapshot folded (also the resume point)
  writerSeq: Record<string, number> // per-author highest sequence covered
  producedBy: string             // snapshot-role identity (Ed25519 pubkey hex)
  authorPubkey: string           // == producedBy
  authorSignature: string        // signDoc over { state, uptoTs, writerSeq, producedBy }
}

Wire a WalSnapshotStore (read/write that document) into the WalDocument options to enable snapshots; omit it to always cold-start from ts = 0.

Why snapshots matter​

  • Cold-start cost — a new reader loads the snapshot and replays only ts > uptoTs instead of the whole log: O(tail) not O(history). This matters most because the dominant replay cost is the per-element Ed25519 author verification (~1–2 ms/element in pure JS), not the fold — so bounding the number of elements replayed is the lever, and a snapshot is how you pull it.
  • A correctness requirement under delegated — replaying from ts = 0 needs a key for every epoch the log spans. A recipient added after an epoch rotation only holds the current epoch, so for them a current-epoch snapshot is the only way to read history — not just an optimization. The trusted writer re-seals the materialized state under the current epoch, making a snapshot a natural re-keying boundary.

Producing a snapshot​

const snap = await doc.snapshot() // requires a snapshotStore

snapshot() pulls the full log, author-verifies and folds every element, exports the full CrdtState, seals it with the encryptor, signs the snapshot with signer.signDoc on the __snapshot key, and writes it to the store.

Two obligations on whoever may call it:

  • Snapshot writeRole. The snapshot must be written through the role-gated push path so the server enforces who may write <name>__snapshot. Do not write it from an in-process afterWrite plugin with ambient authority — that bypasses the writeRole gate the whole trust model depends on.
  • Full-history key custody (under delegated). To materialize, the producer must decrypt the entire log — i.e. hold a CEK for every epoch it spans. A late-joining trusted writer (current epoch only) cannot produce a complete snapshot until it is re-wrapped the old CEKs.

Adopting a snapshot on read​

open() reads the snapshot and accepts it only if:

  1. its author signature verifies (verifyDocAuthor over the signed content);
  2. producedBy === authorPubkey; and
  3. it passes the isSnapshotAuthor gate (if you provided one) — i.e. the signer actually holds the snapshot role.

An accepted snapshot's state is decrypted and adopted; the cursor resumes at uptoTs; the snapshot's writerSeq seeds the per-writer baseline (so ops pruned below it are not flagged as gaps). An unauthorized snapshot (failing the role gate) is ignored and the reader cold-starts. A forged snapshot (bad signature or producedBy mismatch) fails per onAuthorError and is recorded as rejected.

The resume invariant​

uptoTs names a contiguous, gap-free ts-prefix: the server assigns strictly-increasing tail timestamps, so the snapshot folded exactly ts ≤ uptoTs and the cursor folds exactly its complement ts > uptoTs — no drops, no double-counts at the boundary. The one duplication that can occur (a client retry re-appending the same op at a higher ts) is harmless because ops are idempotent.

Reader postures​

Set posture in the options. All three ship; pick per your threat model.

trust​

Adopt a role-signed snapshot's state directly and replay only the tail. Lowest cost; relies on the snapshot-role identity's integrity. snapshotVerified stays null (no re-derivation was performed).

trust-retain-tail (default)​

Like trust, but always keep the most recent retainTailN (default 64) author-verified elements in memory (retainedTail()), so recent history stays independently re-verifiable against the signed op-log even though older history was adopted from the snapshot.

re-derive​

Treat the snapshot as a hint: re-fold the signed prefix ts ≤ uptoTs from the log and compare the full canonical state against the snapshot's claimed state. The result is exposed as doc.snapshotVerified:

const doc = new WalDocument({ /* … */, posture: "re-derive", snapshotStore })
await doc.open()
doc.snapshotVerified // true = matches | false = diverges/forged | null = no snapshot adopted

The comparison is over full state (registers, list nodes, clocks, tombstones, ids) — not the materialized projection — so a validly-signed snapshot that materializes identically but, say, carries an inflated register clock that would suppress a future legitimate write is still flagged false. The materialized value is always correct here because the log itself was fully replayed and per-element verified.

snapshotVerified values: undefined before open(); null when no authorized snapshot was adopted; true/false from a re-derive comparison; false when a present snapshot was rejected as forged.

Compaction & retention (not yet implemented)​

Once a snapshot durably covers uptoTs, a trusted writer could prune log elements at or below it (trading history for storage), with knobs like keepFullHistory vs compactBelowSnapshot + a retainTailN of re-verifiable ops. This is not implemented yet — today snapshots are a pure cold-start optimization and the log is never pruned by the library. Tombstone garbage collection is entangled with this (tombstones can only be dropped inside a snapshot's state, and only if every reader resumes from it), so it is deferred together.

Note the interaction for when it lands: prune only below the durably-acknowledged winning snapshot, and merge concurrent snapshots by max(uptoTs) (a larger contiguous prefix strictly dominates), so a compaction never strands a snapshot whose tail was pruned beneath it.