skaidb v0.320.9

Anti-entropy, repair and space reclamation

Part of Running a skaidb cluster.

How replicas converge without an operator — read-repair, hinted handoff, digest-gated passes, incremental content digests — what REPAIR CLUSTER and RECLAIM do, how a wedged engine lock is kept from stalling peers, and how to audit a member's indexes.

Anti-entropy: repair & space reclamation

Replicas drift apart whenever a member misses a write — it was down, it shed load, a coordinator died mid-statement — and converge again without an operator through three mechanisms layered by cost: read-repair on the reads that happen anyway, hinted handoff for the writes a member is known to have missed, and periodic anti-entropy passes that compare digests and move only what differs. REPAIR CLUSTER runs a pass on demand, RECLAIM frees the space former owners no longer own, and CHECK INDEX audits one member's indexes against its rows. This page describes each, what it costs the engine lock and the disk, and how to read what they log.

Read-repair and hinted handoff

Replicas converge automatically through read-repair (a quorum read writes the winning version back to stale replicas — those holding an older version, none, or the losing version of an exact stamp tie — pushed in the background, so the reader's answer never waits on a stale peer; an unreachable one gets a hint) and hinted handoff (a write to a down replica is buffered and replayed when it returns). Hints are held in memory up to a per-replica cap and spill to a per-replica on-disk log beyond it — so a replica that stays down or keeps shedding for a long time loses no writes (bounded memory, durable across restarts) rather than dropping the overflow. A coordinator killed while holding hints loses none of them: every hint kept in memory is also appended to that replica's crash journal (hints/<id>.hintjournal) before the statement returns — one write(2) into the page cache, no fsync on the write path, so it survives a process kill (kill -9, OOM) at a cost of ~1-2 µs per hinted row, and only on writes that missed a replica. The asynchronous replication tail (the replicas beyond the write's consistency level) and schema (DDL) hints are journaled the same way before the statement returns, so a coordinator killed between the acknowledgement and the send leaves nothing to anti-entropy. A dedicated journal thread fdatasyncs new appends every 100 ms (so a power loss can take at most that much; anti-entropy covers it) — cluster.hint_journal_sync = "write" (live-mutable; default "tick") instead syncs the journal before each hinting statement returns, one fsync per hinting write — truncates a journal once its bucket is delivered, and compacts one that is mostly dead records by copying its live records to a new file. None of that I/O runs under the lock every coordinated write takes, nor on the background worker that carries the replication tail. A hint withdrawn for clock skew gets a withdrawal record (one append, accepted even at the cap), so it is never replayed. A journal is capped at cluster.hint_max_disk_mb (past it the hint stays in memory only); every append is written at the journal's own length, and a failed one is cut back to the last whole record. On open the node moves any journal left by the previous process onto the replica's disk log (whole records, withdrawn hints dropped, within the same cap) and the ordinary drain delivers it. A bucket re-buffered after a failed flush stays at its cap, the excess going to the disk log. (The asynchronous replication tail — the replicas beyond the write's consistency level — is queued in memory and is not journaled until a send to it fails: a coordinator killed between the ack and that send leaves those replicas to anti-entropy, as before. Time-series and schema hints are still memory-only.) A drain that aborts (peer busy, restart mid-pass) is retried by a 60-second ticker while any backlog exists, with delivered work logged — a large backlog deferred at an unreachable peer logs its size instead of waiting silently. The logs also look after themselves: hints older than cluster.hint_max_age_secs (default a day — many times the anti-entropy ceiling, so repair has long since covered them) are expired instead of replayed; a log for a replica that left the ring is deleted; a drain claim orphaned by a crash re-adopts onto the live log (a claim a drain in flight still holds is left alone); a log on disk always counts as pending, so it drains even after a restart; and each replica's log is capped at cluster.hint_max_disk_mb (both knobs live-mutable via SET CONFIG; default 1024 — at the cap new hints fall through to anti-entropy, counted and reported, exactly the pre-spill behavior but loud). Every such drop lands on skaidb_cluster_hints_expired_total. A hint whose table was dropped meanwhile is discarded at drain time instead of being refused by the peer until it ages out (skaidb_cluster_hints_discarded_total); until then 200 rows of a table dropped while a member was away held that member's hints_pending at 200 for a day. DDL gets the same treatment. A statement is broadcast to every member and needs a quorum to succeed; a member that does not acknowledge it (busy, down, or its circuit open) is retried once right away, and if it still has not applied it the statement is queued as a schema hint for that member — replayed by the same drain, in order and before the row hints, so a table created while a member was unreachable reaches it within a minute of it answering rather than at the next schema sync, and the writes hinted for that table in the meantime land right behind it. Schema hints count in hints_pending, expire on the same hint_max_age_secs, and a member that answers and refuses a statement has it dropped (on the expired counter) rather than blocking the ones behind it. Hints queued by DIFFERENT coordinators are not ordered against each other, so every replicated statement carries its coordinator's stamp and applies under last-writer-wins: a DROP that reaches a member before the CREATE it followed records the drop there anyway, and the late, older CREATE is then a no-op — a table created and dropped within seconds never comes back on one member alone. Tables created WITH (memory = true) are excluded from hinted handoff's durability story and from repair and reshard data motion entirely: they are ephemeral by contract (empty on restart, repopulated by their writers). For a full sweep — e.g. after a node was down a long time — run an active repair (Node::repair/repair_cluster), which reconciles every co-replica pair in both directions, including the catalog: databases, tables, and indexes are synced both ways, so a node that missed a DDL broadcast while it was down gets the missing schema too. Schema reconciles by last-writer-wins with tombstones — every DDL is HLC-stamped and a DROP leaves a versioned tombstone — so a drop that happened while a node was down propagates to it on rejoin, and a lagging node holding the now-dropped object does not resurrect it (the tombstone's newer stamp wins). A genuinely newer re-CREATE still wins over an older drop.

Engine lock holds

A member whose engine lock is wedged steps aside. When one hold of a member's engine lock lasts more than 10 s with other work queued behind it (a stalled disk read, a runaway operation), the member answers every request from its peers "busy" at once — health probes included — instead of letting them queue. Peers then read from and write to the other replicas (queuing hints for it) rather than waiting out a timeout per request, so one wedged member no longer slows statements on the healthy ones. It serves again as soon as the lock frees. Its log names the holder — the 10 s engine lock wedged line carries the oldest holder's thread, call site and kernel state, and a hold past a minute dumps every holder and waiter (ENGINE LOCK HELD … holders and waiters:) — and skaidb_engine_lock_longest_hold_ms / skaidb_cluster_wedged_refusals_total show it.

A pass shares the node with its queries: the compare takes the engine lock one page at a time, on the coordinator and on every peer serving the scan, and pauses between pages, so the longest a statement waits behind the repair is one page's decode. A single page that holds the lock longer than a second is logged as repair page hold N ms on <table> — that line, not a flood of slow-query entries, is the signal that a pass has started stalling statements again. Time-series repair — summaries, window reads and merge frames — runs on the table's store handle with the engine lock released; only a table with rollups keeps the engine lock for the merge (the rollup backfill needs it). The same holds for every time-series READ: aggregates (partials), raw gathers, label metadata and the peer handlers serving them take the engine lock only to look the table up, and the store's own lock only to snapshot its block list — block I/O and decoding run with neither held, so a whole-table aggregate on a slow disk no longer queues a writer and then every statement on the member behind it (the 2026-10-02 wedge). A member serving a time-series read for a coordinator stops when the coordinator drops the request (KILL QUERY, its deadline or its client gone — the requester probe runs every 100 ms), and serves its leg under the requesting statement's deadline and scan_byte_budget (fold state or raw samples), carried on the request; a request carrying none (repair, an unmetered PromQL read, a coordinator older than this) is bounded only by the probe. A cancelled statement gets no second-chance re-send, and a member that stops its leg at the statement's limits fails the statement with that reason — whether or not the other members would make quorum — rather than having its series silently left out. Standalone nodes (witnesses, the BI node) still run a SQL SELECT on a time-series table under their database read lock. A divergent series is reconciled per day-window: both sides fold a (count, checksum) per window first and only the windows that differ are pulled and pushed, so a single missed sample costs one window, not the series' whole history.

Continuous passes

Continuous anti-entropy. Beyond read-repair (on reads) and hinted handoff (for writes to a briefly-down replica), each node runs a full repair pass on a timer — cluster.anti_entropy_interval_secs (default 60s, 0 disables) — so a node that missed a broadcast while it was up (e.g. a DDL that committed at quorum while this node was momentarily behind) converges on its own, with no operator action. Passes are staggered per-node so the cluster doesn't repair in lockstep, and a pass is skipped while the node is shedding — repair is the largest transient allocator in the process and is pure background work, so one deferred interval costs nothing next to pushing a node already under memory pressure into an OOM. The skip is logged.

Adaptive cadence (cluster.anti_entropy_adaptive = true, off by default). The fixed timer has a structural problem: a pass's cost scales with data, its value scales with how much has actually diverged, and divergence is bursty — caused by restarts, missed writes and membership changes, not by the passage of time. With adaptivity on, the interval becomes a bounded variable that follows what passes actually find:

  • a pass that reconciled rows snaps the interval back to anti_entropy_interval_secs (the floor);
  • a pass that converged (found nothing) relaxes it ×1.5, up to anti_entropy_max_interval_secs (0 = auto, 4 × the floor);
  • a deferred or failed pass changes nothing — a skip is not evidence of convergence.

The ceiling doubles as the guaranteed full-sweep bound: adaptive rest is bounded exposure, never unbounded trust. Divergence events pull the next pass in rather than waiting out a relaxed interval — a hint stored (a replica just missed a write), a hint replay draining, a member joining — debounced so events move the next pass earlier but never schedule more passes than the floor already permits. All three knobs are live: SET CONFIG cluster.anti_entropy_adaptive = 'true' per node, no restart.

Watch it through skaidb_ae_interval_seconds (the live interval), skaidb_ae_passes_total{outcome=…} and skaidb_ae_rows_reconciled_total (see METRICS.md); alert on time() - skaidb_ae_last_pass_at_seconds exceeding the ceiling, which is what distinguishes a healthy backed-off loop from a stuck one.

Digest-gated passes. A repair pass first exchanges a compact XOR digest (4096 buckets over key ‖ hlc ‖ op, ~32 KB) per (table, peer) pair; equal digests prove the pair converged and skip the full paged compare, so a steady-state pass ships digests instead of tables. The digest sees stamps, not bodies: HLC stamps are unique per member, so two coordinators can stamp two writes of one row identically (same millisecond, same logical counter), and two replicas holding the two versions digest alike. Such a tie resolves by the deterministic tie rule wherever both versions meet — a replica that receives the second one keeps the winner, whichever arrives first, by any path (replication, hinted handoff, a repair push); a QUORUM/ALL read repairs a replica holding the loser; a row compare (the full one, or a bucket that differs for any other reason) pulls or pushes the winner. A tie whose winner never reached one replica (its hints expired) stays on that replica until one of those happens. Digest computation itself is cheap: it scans value-free stamps — each SSTable carries a <file>.stamps sidecar holding just (key, hlc, op) per entry, so no row value is even decompressed. Encrypted tables have a sidecar too (sealed like the data blocks); tables written before the sidecar existed — or before encrypted ones got theirs — fall back to data blocks until compaction rewrites them. On full-copy clusters (replication factor ≥ member count) each node also caches its digest per table, keyed on a (schema stamp, write-sequence) version, so an idle table's digest is served from memory — a fully converged pass touches no table data at all. The cache can never mask divergence: any write bumps the version on the node that has it, forcing a fresh digest — and therefore a mismatch — on at least one side of the pair.

Stamp-equal rows are settled by body fingerprint. Two versions of a row can carry one stamp (two writers, one clock tick); the digest cannot see that, so a repair pass walks the differing buckets as (key, stamp, put-or-delete, body fingerprint) pages from both sides and reads a body only where the versions may differ — a stamp-equal pair whose fingerprints are both known and equal costs nothing. The fingerprint comes from the stamps sidecar (format 3, written when storage.stamps_body_fingerprints is on); a member or a file without one is compared by body exactly as before, so an unknown fingerprint never hides a difference, and a peer on an older release that does not know the fingerprint page is walked with the row page. The witness pull applies the same rule to a pulled put at the stamp of the local version.

Incremental content digests. On multi-node deployments, every full-copy table also carries an engine-maintained digest: the same 4096-bucket XOR fold a repair scan computes, kept current by the write path itself (each write folds the key's previous winner out and the new one in, at the single choke point every write variant funnels through), built once by a paged background scan, and invalidated whenever the merged view changes outside the write path (a compaction purging tombstones or TTL-expired rows, RECLAIM's row drops) — the invalidated table simply falls back to scans until the background rebuild finishes. A ready digest is persisted (digest.bin beside the table, written at every flush install and at the clean-shutdown flush) and loaded at the next open — but only when the WAL replays nothing, since a replayed record is a write the file may not hold. That is why a clean shutdown flushes every memtable, however small (the memory-pressure flush keeps a 64 KB floor; a table whose trickle of writes stayed in its WAL would otherwise replay them and lose its digest); an unclean stop simply rebuilds as before. A clean stop also leaves a clean_shutdown_ms mark in the data directory, and the next start-up's catch-up pass compares digests AS OF that mark (minus the usual as-of lag) instead of now − lag: rows written while the node was away are stamped past it and omitted from both sides, so a table that was converged at shutdown compares equal and is not walked at all — those rows arrive by hinted handoff, and the next pass verifies them at a fresh cutoff. A key updated while the node was away still differs and is walked (only the digest buckets such keys fall in); a stop longer than ~4 minutes (the AS-OF rings' depth) takes the ordinary cutoff now − lag, so the rows written while the node was away are walked in by their buckets — and it still defers the tables only a scan could verify (below), provided every peer taking part in the catch-up confirms that its hints for the node are complete since the stop: it has kept hints since before the stop (a member wiped and resynced meanwhile has not), it is not resyncing, the gap is within cluster.hint_max_age_secs (a day by default, less a ten-minute margin for the replay), and it dropped no row hint at its hint-log cap, expired none and lost none to an unclean stop of its own at or after the mark — for any member, with the clock-skew allowance to spare. A peer that cannot confirm it, or one from a release that predates the question, turns the deferral off and the pass scans those tables, which is then the repair. BACKUP and RESTORE never carry the mark (nor the node's hint-history and hint-loss records beside it); a member's directory put back by hand from a cold filesystem copy must not carry it either — delete clean_shutdown_ms from the copy before the start, or the node presents a stop it did not make and defers verification of the rows written since the copy was taken. The peers' digests at the mark are captured for every table in the first seconds of the catch-up, while their rings still cover it, and this node's side is computed when the pass reaches the table — so a catch-up that takes longer to get there (search catch-up, hundreds of tables, a loaded peer, a slow local disk) still compares at the mark. The clean-shutdown flush holds the engine lock only for the short steps (the one write hold before it saves the vector indexes, settles the applier watermarks and drops the caches — it flushes nothing; a hold of a second or more is logged with its parts): every memtable with anything in it is frozen once (tables and index engines alike, whatever their size — at most four seals per write hold, the next hold taking the next engines; what a peer replicates after an engine's seal replays at the next open), the SSTables are built in parallel (up to eight builders) with no lock held and installed one per short hold (each install persists its table's digest; the untouched tables' digests are written first, one short read hold each; an engine whose build fails is left alone for the rest of the stop), the time-series heads are flushed on their own handles (biggest first, rollups fed, a rollup source's open bucket left for its window flush) and the archive copies still owed run off the lock — within a 240 s budget (the packaged unit allows 420 s to stop), so a node shutting down under heavy traffic on a slow disk keeps the digests of every table it got through, the next open replays no WAL for anything the pass reached, and the peers' requests to the stopping member are not wedged for the length of it (held across every fsync, the lock made a 1,841-memtable stop a 233 s wedge). The open itself logs where it spent its time: one skaidb: open of table|index|ts table <name> took N ms (…) line for any single engine or store over 1 s (WAL replay, SSTable load, digest load; block opens and head replay for a time-series store), then one summary — skaidb: engine open took N ms (catalog, tables [wal replay, sstables, digests], indexes, ts [blocks, heads], vectors, search, recovery) — and skaidb_engine_open_seconds carries the total. While the open runs it says where it is every 10 s (skaidb: engine open +30s — tables 119/415 (opening <table> for 4 s)), and after the listeners bind a member still catching up logs every 30 s what its maintenance jobs are doing (skaidb: startup +90s (not strict-ready) — anti-entropy pass: tables 37/415 — verifying: <table>@<peer> (since +12 s), …; search catch-up <index>: index 1/3 …: 20480 rows read, 20480 queued (9s)), the same text SHOW MAINTENANCE shows in progress, then catch-up phase over. An open that outlasts the ~4-minute AS-OF window loses the digest compare at the mark, so this line is the first thing to read when a member's catch-up walks whole tables. A table dropped while a pass is working through it is skipped rather than failing the pass. Repair pages adapt their row count to how long each page holds the engine lock (target 200 ms), so a table of large rows is read in small pages instead of stalling writes for seconds. A TTL table carries a time-bucketed digest: its stamps fold into 64 windows spread over the TTL (never narrower than 5 minutes), an AS-OF answer XORs only the windows more than one whole window above cutoff − ttl, and the scan fold on the other side omits the same rows — so the wall-clock reclaim of expired rows, which only ever removes rows below that floor, never invalidates it. (It used to, which kept every TTL table on the full stamp scan: 587 s per pass for one production spans table, an I/O burst every fourth pass that put seconds onto the writes racing it.) The cost is about 2 MB of RAM per TTL table and a row's last window before expiry going unverified; a TTL under 30 minutes carries no digest and is stamp-scanned as before. Any table whose digest cannot answer at the mark — a TTL too short for one, a digest rebuilding after a compaction purged rows it could not follow (delete-heavy tables: a TOAST companion, a tag table under a sweep), none persisted after an unclean stop, or no peer able to answer at the mark because its recent-write ring overflowed while the node was away — could only be verified by a full stamp scan, which no mark bounds, so the startup pass after a clean stop defers it to the routine pass (logged as startup catch-up — deferred N table(s) whose digest cannot answer at the start-up cutoff, naming them; a node resyncing from empty defers nothing, since no hint carries rows it never held): the rows written while the node was away reach it by hinted handoff — its peers retry every minute, and hints_pending on them reaches 0 when done — and the routine pass verifies those tables exactly as it always has. This is what keeps a restarted member's first anti-entropy pass at seconds instead of a scan of every table (15-17 minutes on a production cluster before it), and so what a gated rolling restart (/ready?strict=1) actually waits for; an orchestrator that wants the outage rows landed before taking the next member down waits for the peers' hints_pending to reach 0 as well. A peer that does not answer a 3-second probe at the start of a catch-up attempt is left out of that pass (logged by name). Completeness is then judged over the peers that answered, so a peer that accepts connections but never replies cannot hold the catch-up, or strict readiness, indefinitely. Nothing about the silent peer's tables is recorded as verified, so routine anti-entropy reconciles it once it answers again, and its peers' pending hints still stop a gated roll. The digest gate answers in O(1) where a full verify would re-scan millions of stamps per pass, and a digest that differs narrows the work instead of abandoning it: the compare is restricted to the buckets whose values differ (both sides still walk their shard, but only those buckets' rows cross the wire and enter the merge-join — a few divergent keys cost a few buckets of about table/4096 rows each, not the whole table). And the exchange happens as of a cutoff (~30 s in the past): each side omits keys whose current winner is stamped past it, so writes still in flight are excluded on BOTH sides and two converged replicas match even mid-ingest — without the cutoff, any write landing between the two computations forced the full paged compare every pass. Divergence younger than the cutoff is invisible for exactly one pass. Because the digest is trusted to skip work, a missed write site would read as permanent convergence — so every pass re-verifies ONE digest-backed table (rotating) by full scan and compares; a mismatch on a quiet table is counted and logged as a bug, and either way the digest is dropped and rebuilt. TTL tables carry the time-bucketed form described above (digest.bin holds it in its own layout); standalone nodes never build one and their write path never pays the digest's prior-stamp lookup.

Hot-table backoff. A table under continuous ingest defeats the versioned digest cache: every write voids it, and because the two sides scan at different instants, in-flight writes make even converged replicas' plain digests mismatch — forcing the full paged compare on every pass (measured: one 4.4M-row live-ingest table cost ~400 s of every hourly pass, per node). The as-of digests above remove most of that cost; the backoff remains as belt-and-suspenders and for tables without a digest. A table that changed since its last clean verify (a pass where every peer agreed and nothing needed repair) is deferred, and re-verified every 4th pass — bounded divergence-detection lag (4 × the anti-entropy interval, worst case) in exchange for passes that stop re-verifying data the quorum write path and hints already delivered. The backoff never applies where it could mask divergence someone is waiting on: an explicit REPAIR CLUSTER sweeps everything (on every member), a resyncing node verifies everything, a table in a placement transition is verified every pass (the finalize protocol counts on full passes), a verify that repaired a real volume of rows re-verifies every pass until it comes back clean (a handful of "repairs" per pass is tolerated as in-flight write noise — the merge-join's two sides scan at different instants, so a live-ingest table never verifies at exactly zero), and a deferral is honored only while cluster membership matches the peer set the clean verify covered (a join/decommission re-verifies everything). A table whose own verify keeps failing (its peer busy/unreachable) never earns a deferral and is re-verified every pass — but one table's busy peer does not disable the backoff for the rest (under continuous ingest most passes have some transiently busy pair, and those are exactly the passes worth keeping cheap). A pass that deferred anything logs how many tables it skipped. An idle table whose peer diverged is never deferred — deferral requires the local copy to have changed — so one-sided divergence is still caught on the next pass.

What REPAIR CLUSTER answers. The statement runs a full pass on the member that takes it, then one on every other member (at the same time as each other; a member already mid-pass — a restarted one in its startup catch-up — finishes that pass first and then runs the sweep's), and answers three things: repaired — the rows every pass that reported copied, summed; complete — true only when every member's pass reconciled every table with every one of its peers; and, when it is not, one incomplete.<member> line per member saying what that member's pass left uncovered:

incomplete.<member> meaning
N table(s) not reconciled with <peer> (unreachable or busy) that member's pass gave up on the pair after its in-pass retries — <peer> is down, restarting (its circuit still open), or write-locked
<peer> left out (unresponsive when this member's startup catch-up began) the member is still inside its own catch-up, which excluded <peer>
its pass stopped early (the member is overloaded) background work yielded to foreground traffic part-way
no answer within 120 s — its pass may still be running; … the member's pass outlasted the sweep's wait; the pass itself is not cut short
did not answer the repair request (…) the member is unreachable
its pass failed: … the pass ended in an error, quoted
ran a pass, but its version does not report what the pass covered a member on an older version during a rolling upgrade

repaired 0 with complete true means the replicas agree as of the pass. repaired 0 with complete false means nothing was learned about the named members — a sweep issued while a member is restarting typically names that member; run it again once the member is back (its own catch-up pass and hinted handoff are converging it meanwhile).

The sweep compares replicas as of now and defers nothing. The routine pass compares them as of 30 s ago — a restarted member's catch-up, as of its clean-shutdown mark — so that writes still in flight do not read as divergence, which leaves divergence younger than that to the next pass; an operator asking for a repair right after an incident is asking about exactly those rows, so the sweep covers what was written seconds before it and what was written while a member was away. Writes in flight during the sweep cost it a few bucket compares and may be counted in repaired.

Automatic catch-up on (re)join. When a node starts and finds peers, it runs a catch-up pass in the background as soon as a peer is reachable — the same repair (schema + data) — so a node that was down converges on its own without an operator running anything. This covers schema that quorum-DDL couldn't reach while the node was offline, plus any row writes beyond what hinted handoff replayed. (A brand-new node added with cluster add-node is bootstrapped explicitly with schema + its share of the data.) While that startup pass runs the node is not strictly ready, so it is not paced like the hourly background pass: it verifies four tables at once with no pauses between pages or tables, backing off only when a peer answers busy or the node's own load permit says so — a restart's catch-up takes minutes, not the quarter hour a paced walk of hundreds of tables took. (Four lanes of back-to-back short write holds behind the writer-preferring lock can make a restarted member's first statements wait 5–127 s; skaidb_engine_lock_wait_ms_bucket shows such a convoy.)

Automatic join (self-announce). A node that starts with seeds pointing at an existing cluster but that the cluster doesn't yet know about will announce itself to a reachable seed, which runs add_member and broadcasts the new membership to every node — so you no longer have to run cluster add-node by hand, and you avoid the half-join trap (a node that pulls data via catch-up but was never admitted to the ring). The announce is a no-op when the seed already lists the node (symmetric seeds), and is rejected if the joiner's replication factor doesn't match the cluster's — fix the RF and restart rather than form a cluster whose coordinators disagree on each key's replica set. That is the only final rejection: a join the seed rolled back (the joiner's engine stayed busy, another transition was open, a migration batch went unacked past its retries) is re-announced — every 2 s for the first 30 s, then once a minute — until the node sits in a finalized ring. Being listed by a seed is not the end of it: the begin broadcast lists the joiner while its join is still running, and that join can roll back, after which the node is out again and announces again; only the settled ring stops the announcer. The announce RPC itself waits as long as the join takes (up to an hour) rather than timing out on the ordinary ten-second internode deadline, so a long migration never leaves the joiner believing it was refused. Joins are still serialized; do one at a time. \cluster cross-checks each peer's membership view and flags membership_disagreement when a peer you route to doesn't list you.

Members need not share one client port. The ring carries internode addresses; each member's SQL port is learned from its host stats and /status endpoints advertises every member on its own port (a peer that has not reported one yet — or an older binary — is shown on the answering node's port until it does). A member in a container beside another skaidb on the same host can therefore use a different quic_port.

After resharding, reclaim (Node::reclaim/reclaim_cluster) physically frees space for keys a former owner no longer holds. See RESHARDING.md.

/status, /metrics and /ready never block on the engine. The unauthenticated /status serves cluster identity (name, aliases, witness rows) from a cache a background thread refreshes every few seconds — it runs no SQL inline, so a health probe answers in milliseconds regardless of what holds the engine lock. Every probe reads the engine through a bounded try: while the lock is write-held (a RESTORE, a long DDL) /metrics answers from the last snapshot of the storage statistics (skaidb_stats_age_seconds is its age) with skaidb_engine_lock_longest_hold_ms growing, /status reports engine_lock_busy_ms, and /ready answers after a 1 s wait at most. What a held lock means for readiness depends on the form: /ready?strict=1 answers 503 engine lock held (an exclusive operation is running) as soon as that 1 s try fails — the gate for rolling operations — while plain /ready (and ready in /status) stays ready until the lock has been out of reach for a 30 s grace, then turns 503 / false with the same reason. A statement that waits a few seconds behind a flush or a DDL is delayed, not failed, so a load balancer has nothing to route around; a lock held past the grace is a node that serves nothing. Staleness-age fields (seen_age_secs, sync_age_secs) are computed from row timestamps at render time, so they stay accurate even from cached rows. Point external monitors at it freely; a slow /status now means the process itself is unhealthy, not that a repair pass is running.

Repair rides out a slow peer. A repair RPC that draws a busy reply, an I/O timeout, or the local circuit breaker's refusal is retried with backoff rather than failing the pass — a receiver applying a merge chunk holds its engine read guard, so being briefly late is normal, and one late chunk used to abort an entire time-series leg. Retrying is safe because repair requests are idempotent: a merge is keyed by timestamp, so re-sending a chunk whose acknowledgement was merely lost changes nothing.

Time-series repair reconciles in BOTH directions. A series' primary compares per-series (count, checksum) summaries with each replica — over the samples stamped at or before the pass's cutoff (now minus the as-of lag; a recent clean-shutdown mark during a start-up catch-up), so a series both sides had settled by then compares equal however many samples arrived since. A sealed block carries a checksum per chunk (summaries.dat), so a replica's summaries compose from its block indexes without decoding a chunk; a block written before that file existed is decoded once and carries the checksums on from its next compaction. The peers' summaries are asked for while the local fold runs, not after it. Every 24th routine pass and an operator sweep compare whole series, which is also what reaches a sample stamped past the cutoff that one replica alone holds (a future-stamped sample, a table stamped in µs/ns); when they differ it pulls the replica's samples for a window, merges them locally, and pushes the union back. Only the windows whose own summaries differ are exchanged, and during a start-up catch-up after a clean stop only the windows from two days below the mark onward — what the node missed while away is stamped there; older history is the routine pass's. Pushing alone cannot converge — a replica holding samples the primary lacks would stay different after every push, and because the summaries still disagreed the primary would re-push the series' entire history on every later pass, indefinitely. Samples are immutable facts keyed by timestamp, so the union is the authoritative view and merging is idempotent on both sides.

Auditing a member's indexes (CHECK INDEX)

Anti-entropy, read-repair and hints converge rows; nothing compares a member's local index entries with its rows. A member whose copy of an index lost entries (or holds entries no row produces) keeps serving: its index-only counts answer short — at every consistency level when it coordinates the count — ordered index reads at ONE miss rows, a MATCH or NEAREST misses the row whose document or vector it lost, while every row reads back fine. CHECK INDEX is the audit, for every kind of index a member keeps: secondary (UNIQUE and FK-supporting ones included), geo, search (full-text), vector, and GLOBAL indexes of full-copy tables.

How to audit a member. The statement runs on the node it is sent to and never scatters, so connect to each member in turn (a driver or skaidbsh pointed at that member's address), then per database:

USE app;
CHECK INDEX ALL;                       -- every index of `app`
CHECK INDEX ALL ON mail;               -- ... of one table (one walk of it)
CHECK INDEX i_mail_tag;                -- one index
CHECK INDEX ALL RESUME 'ci1.…';        -- continue a `partial` run

One row per index. status is ok, drift (missing, stale or duplicates above zero), partial (the statement timeout stopped it: rerun with RESUME '<resume>' until a row has no token; counts are per statement — add them up; the indexes of one table cut together carry the same token), skipped: building (a backfill is running; its index is not used for reads until it finishes) or skipped: sharded (a GLOBAL index of a table this member does not hold whole: its entries live on other members; the repair pass's sharded verify covers it). samples names up to five primary keys per kind (missing [42] -> ["s3"]: row 42 produces the entry s3, which the index lacks; a geo entry shows its point, a search index document, a vector index vector, or vector 2 for a MULTI row's third vector). in_flight counts rows and entries whose index maintenance was still on its way while the check passed — queued, a search write the index has not committed yet (it commits about once a second), a text waiting to be embedded, or a GLOBAL entry whose row or entry was written in the last 30 seconds (the entries are replicated writes of their own and can trail the row) — judged again at the end: a busy table always has some; they are not drift, and a rerun when the table is quiet judges them. duplicates (UNIQUE indexes only) means two stored rows hold one value: that is a data problem, not an index problem — find the rows from the sample and fix one; no index operation repairs it.

What it costs. It reads the table once for all of its indexes and each index twice, in key order and in pages (a vector index in node-id order, from memory; a search index's documents are read with the table, a page of keys from its term dictionary at a time): every page is one short read hold of the engine lock (about 8 ms, cut at 25 ms), with a pause after it, so queued writers go first — a member keeps serving, and memory stays at one page plus a hash ledger per index of at most 6 MB (12 MB for one table's indexes together). The ledger matches the index against what the rows produced, so only entries that do not match have their row read back: a clean index costs no random read. Measured on a 2 GB table (2.1 M rows of about 1 KB, release build, NVMe, a writer inserting throughout): a clean check took 23–30 s, its longest hold was 16 ms, and the writer never waited more than 13 ms (p99 5–9 ms). On an at-rest-encrypted copy of a member with 2.47 M rows and two indexes it took 86 s for both, held the lock at most 25 ms, and found exactly the entries an independent forensic count had established as lost. CHECK INDEX ALL checks a table's indexes together: one walk of the table serves all of them, then each index is walked on its own against its ledger (with every kind on one table, the longest hold measured was 15 ms). The ledgers live for one statement, so a check the timeout stops part-way is finished by its RESUME entry by entry — a random read per remaining entry (56–68 s for the same 2 GB index, against 23–30 s). CHECK INDEX ALL therefore does not start a table when less of the statement is left than the longest table so far took, and a table cut after another one had the statement first is walked again from its first row by the RESUME (its partial rows then report no rows: the next statement counts them); only a table whose check does not fit one whole statement is finished entry by entry. On a large table whose indexes together do not fit one statement, check one index per statement (CHECK INDEX <name>), so each has a whole statement to itself; an index that still does not fit needs a longer storage.statement_timeout_secs on that member (set at start) to be checked with its ledger. Audit at quiet times on a member whose disk is shared. The heavy-scan cap (iwm.max_concurrent_scans) applies: a refusal (too many concurrent scans) is retryable. KILL QUERY stops it; SHOW QUERIES shows its progress.

What to do with the result.

  • ok everywhere: nothing to do.
  • drift on a secondary, geo, GLOBAL or search index on a member: repair that member's copy with CHECK INDEX <name> REPAIR (or CHECK INDEX ALL REPAIR) on that member. It walks again and fixes exactly what it finds: each finding is re-checked against the stored row — its last-writer-wins version across every storage source, so a losing write an older release left in front of its winner is never taken for the row — and the index under a short write hold (a row whose maintenance is queued again is left to it), the missing entry is put back or the stale one removed, the index WAL is synced after the hold, and a line per index goes to the log (skaidb: CHECK INDEX <name> REPAIR: … entries repaired). The index keeps serving throughout. On the 2 GB table above, repairing 1,000 lost entries took 27–35 s (the walk) with a longest hold of 13 ms; on the encrypted copy above, the repair put back all 828 lost entries of each index and the next check was ok. Rerun CHECK INDEX to confirm ok. A GLOBAL index's entries are rows of a replicated table: the repair writes this member's copy and anti-entropy carries it to the other replicas, as the repair pass's own GLOBAL verify (which heals and collects entries on every pass) does. A search index is repaired key by key the same way: a row missing its document is indexed again from its stored version (stamped with the row's HLC, as the maintenance path stamps it), a document no stored version stands behind is removed, and the index is committed after each write hold (outside it), so the repair is durable when REPAIR returns. This is the fix for a member whose search index lost documents — not REBUILD SEARCH INDEX, which is replicated DDL: EVERY member clears its documents at once and the index answers "rebuilding" everywhere until each member's paged backfill finishes (the backfill runs in the maintenance worker — 1024-row pages under the engine WRITE lock, indexing and the final commit with no lock held — so members keep serving everything else; it used to stream the whole table under the write lock and froze every member).
  • drift on a vector index: REPAIR reports it and changes nothing (its log line says so); REBUILD INDEX <name> re-derives the index from the rows on every member. Check the other members first: if only one member drifted, rebuild when the index can be out of service for the length of its backfill.
  • The alternative, REBUILD INDEX <name>, is cluster-wide DDL: EVERY member wipes its copy and backfills it from its rows, and the index serves no reads anywhere until each member's backfill finishes (queries fall back to their pre-index plans — scans). Its backfill pages hold the engine WRITE lock: on the 2 GB table the rebuild took 16–19 s per member (the index out of service for all of it), the wipe held the lock about 30 ms, each 2048-row page 12–15 ms (p99 19–24 ms), and a writer waited up to 130–170 ms when a page met a memtable flush. Prefer REPAIR for drift on one member; REBUILD INDEX is for an index whose every copy is suspect, or a vector index that drifted (which REPAIR does not change).

Around a roll, and after an unclean stop. Where the running release has CHECK INDEX, audit every member before a roll (one at a time is enough; the check changes nothing), keep the result rows and repair what drifted, so drift after the roll is new; audit again after it. Rolling onto CHECK INDEX from a release without it, keep cluster.index_count_check = "log" (the default) through the roll, then audit and repair every member, and switch to enforce only once every member reports ok — under enforce, every QUORUM/ALL count on a quiet table whose index drifted is a row read until it is repaired. A member stopped uncleanly (killed, OOM-killed, its host reset) replays the index maintenance it still had queued when it starts again; audit it once it has caught up.

What that replay covers. Each table's applier watermark on disk promises that every write stamped at or below it has its index entries applied and durable, and the replay re-applies every row version above it. So the watermark stays below a queued task, a failed one waiting for its retry, a full-text write no commit has made durable, a vector write applied since its index's last snapshot and stamped below that snapshot (until the next vector checkpoint, every 10 minutes), and an inline write whose index entries are not synced yet. A write that arrives stamped at or below the watermark on disk — a hint, a repair push, a write that waited out a long engine hold, a table's first queued write — is stored only after the lowered watermark is written, in an engine hold of its own: a row is durable the moment it is appended as far as a killed process is concerned, and before its index entries once another writer's group commit syncs the table's WAL. A merge of SSTables waits (at most two minutes, or until level 0 reaches three times its compaction trigger) while one of them holds a version whose index maintenance may still be queued, so that the replay can still name the entries of the version that write replaced (a full-text or vector write waiting for its index commit or snapshot holds the WAL, not merges). Before a merge exceeds that bound, a durable marker is set; an equal-stamp overwrite that erases an older body sets the same marker. The marker is cleared once the queued maintenance it covered has run. A member that stopped before then reconciles the table's secondary and geo entries against its rows after it starts, in the background in short engine-lock holds — it serves at once, and its index-only counts of that table read rows until the reconcile is through. Only marked tables pay that extra scan. A member also covers the writes it coordinates itself (indexed inline, in the hold that stores the row): a table's first such write puts the table's watermark on disk before its row is stored, so a table whose clients all write through one member is replayed like any other. A standalone server and a witness use the same cover: every table with something to maintain carries an applier watermark (created for existing tables at startup and by a new table's first write, on disk before its row is), a statement holds it below its first stamp per table until its row and index WALs are synced, a witness stores each pulled page as a member stores a write (the watermark lowered below the page's stamps on disk first, index WALs synced after the lock), and the maintenance tick raises the watermarks at most once a second, only when one can move — in a write hold it takes anyway for flushes, else only if the engine write lock is free (it never queues behind a reader), and once with a blocking wait after 30 s of finding it busy — and writes the file at most every 10 s with the lock released, retrying a failed write at the next tick (settled at a clean stop). So a power loss between a row's fsync and its index entries' is repaired by the open-time replay on every node kind. No fsync per write is added. An embedded database (the library, no server) keeps no watermark unless the embedding calls enable_embedded_index_watermarks(); with it on, its own statements raise and persist the watermark (at most once a second) and the same open-time replay repairs that window there — off, only CHECK INDEX … REPAIR repairs it (and no flushed WAL segment is ever held).