Operating a cluster day to day

Part of Running a skaidb cluster.

Rolling restarts, backups and point-in-time recovery, what the node does under damage, memory pressure, a full disk and a stop signal, and the operational limits worth knowing before they find you.

Rolling restart: planned maintenance on a member

There is no maintenance mode or "drain clients" command, and none is needed for a restart: a graceful stop is the drain. systemctl restart/stop (and the package's try-restart on upgrade) send SIGTERM, and the node then (1) answers /ready with 503 not ready: shutting down and sets /status shutting_down: true, (2) closes its REST, binary and MQTT listeners — a new connection is refused, which a driver or skaidbsh treats as "not this node" and fails over without ambiguity, (3) shuts down idle keep-alive and pooled connections so clients reconnect elsewhere on their next statement, (4) lets the statements already running answer, for up to server.shutdown_drain_secs (default 15; SET CONFIG it higher for a workload of long statements — up to 60, the most a stop spends draining so that the flush that follows fits the unit's TimeoutStopSec=420), and only then (5) flushes memtables, commits search writers and exits (see Graceful shutdown under Resilience). A client that talks to ONE node only, with no failover list, sees refused connections for the length of the restart; give such clients the members' endpoints (drivers discover them from /status endpoints).

The workflow for one member, in order:

Check the ring is healthy first: every member /ready?strict=1 → 200, hints_pending 0 everywhere (skaidbsh cluster status), no resharding window open (/admin/resharding active: false). Taking a member out while another is already behind is how a QUORUM write finds only one replica.
systemctl restart skaidb (or stop, do the host work, start). With RF ≥ 2 the other replicas serve reads and writes meanwhile; writes the stopped member misses are hinted to it and drained when it returns.
Wait for /ready?strict=1 → 200 on the returned member (skaidbsh cluster wait-ready --strict), then for its hints_pending to reach 0 on its peers, before touching the next member. One member at a time.

A member that will be gone for long (hours — a disk replacement, a host move) is better decommissioned (REMOVE NODE, which drains its keys to their new owners first) and re-added afterwards, so the ring keeps full replication meanwhile instead of accumulating hints; a member whose data directory is lost comes back through the wipe-resync path in Replacing a member (wipe-resync) and the resync state. server.read_only is a write-freeze for a witness or a maintenance window on the DATA, not a drain: it answers mutations with an error rather than routing them away.

Backups on a cluster

BACKUP TO '/path' backs up the answering node's shard (a crash-consistent copy of its data directory — each node backs up its own). RESTORE FROM is refused on a live cluster: swapping one node's data underneath quorum reads would silently diverge replicas. To restore a node: stop it, restore its data directory offline, start it, and let repair converge it.

A backup is complete only when its marker says so. Every backup — BACKUP TO, each node's directory under BACKUP CLUSTER TO, full or incremental — writes BACKUP_IN_PROGRESS into the target before its first copy and BACKUP_COMPLETE (the file list with sizes, a checksum of catalog.json, the completion time) atomically as its LAST step. A target holding BACKUP_IN_PROGRESS without a valid BACKUP_COMPLETE is what an interrupted backup looks like — a node killed mid-copy, a full disk — and ls shows it. RESTORE refuses it (backup at '<path>' is incomplete: the backup that wrote it was interrupted …), refuses a backup whose files disagree with its marker (… is damaged: <file> …). The next BACKUP into an interrupted target takes it over — files that match their source are kept, the rest are copied again, and it ends with a valid completion marker — so a scheduled backup recovers by itself after a killed run; only a target another backup of the same node is still writing is refused. The engine's exclusive lock is held for the copy alone: the WAL cut and the completion marker (a walk of the whole target) are written after it is released (locked_ms vs elapsed_ms in the result). A backup with neither marker was written by a skaidb without them; it restores with a warning column saying its completeness was not verified, and the next BACKUP into it adds the markers. There is no override that restores an interrupted backup. The live node's process lock (.lock) and disk reserve (.skaidb-reserve) are never part of a backup.

BACKUP CLUSTER TO '<path>' has every member back up to path on its own filesystem. Members that share one filesystem — a backup mount — must not share one directory (two members syncing the same directory delete each other's files, and the completion marker then refuses the set): write {node} in the path and each member substitutes its own id in filename-safe form, BACKUP CLUSTER TO '/mnt/backups/{node}' → /mnt/backups/192_168_7_174_7100, …/192_168_7_175_7100, …, each with its own CLUSTER_CUT. A path without the placeholder is used as given.

BACKUP CLUSTER TO is refused while a membership change is in flight (cluster backup refused: a join of <node> is migrating keys (epoch N, dual-ring window open); poll /admin/resharding until active=false, then retry): a set taken mid-change straddles two rings and holds a half-migrated shard with nothing in it to say so. The coordinator checks its own view, and every member checks its own before copying, so a coordinator whose view lags is caught by the member that still holds the window.

Restoring a backup onto other hosts. A node's backup carries its data and its identity: the persisted topology file (the ring and epoch it was a member of) and its hints/ logs (writes it still owed that ring's members). Restored onto a node whose own address is not in that ring — a copy of the cluster on new hosts, a different cluster size, a staging replica — the ring is foreign, and the node treats it as such at start-up: it logs persisted topology (epoch N, members …) does not list this node (…) — … ignoring it, keeps the file aside as topology.foreign, starts from its configured seeds at epoch 0, and discards the hint logs addressed to nodes outside those seeds. It never routes to, hints for, or proposes membership to the cluster the backup came from. The procedure is therefore: (1) on each new host, restore the backup with the standalone config (seeds = [], as for any RESTORE), (2) switch to the new cluster's config — seeds listing the new hosts only, the same replication_factor on each — and start them; they form the new ring from seeds and anti-entropy converges whatever differs between the copies (BACKUP CLUSTER plus RESTORE … TO TIMESTAMP <cut> on every node gives them an identical starting point). Never point a restored node's seeds at the source cluster unless it is meant to join it. Restoring a node's backup onto the same node keeps its ring and epoch, which is what a same-node restore needs.

Point-in-time recovery on a cluster

With storage.wal_archive_dir set, each node keeps every sealed WAL segment and RESTORE FROM '<path>' TO TIMESTAMP '<when>' replays them on top of a backup — undoing a bad write without losing the work done since the last backup. See QUERY_SYNTAX.md for the statement.

On a ring this is an operator procedure, not a statement:

Stop every member. A node restored under a live ring has its recovered rows immediately overwritten by peers that still hold the bad write.
Run the same RESTORE … TO TIMESTAMP on each node's data directory offline, with an identical target. Replicas converge only if they replay to the same instant.
Start the ring. Anti-entropy closes whatever a node that was down during the window missed.

No coordination protocol is involved, and that is deliberate: HLC stamps are cluster-wide and last-writer-wins is deterministic, so independent replay of the same window on each node reaches the same state. What it needs is a stopped ring, not a distributed algorithm.

In practice, per node (RESTORE is a statement, so the node must be up — just not in the ring):

Swap in a standalone config: same data_dir, same keys, same archive, but seeds = [] (standalone mode is what accepts RESTORE) and statement_timeout_secs = 0 (replay can exceed any interactive timeout).
Under systemd, clear the unit's filesystem sandbox for the window. ProtectSystem=full plus StateDirectory/ReadWritePaths bind-mount the data directory into the service's namespace, and RESTORE renames the live directory aside — which fails with EBUSY against a bind mount and EACCES against a root-owned parent. A runtime drop-in (removed afterwards) is enough: printf '[Service]\nProtectSystem=off\nStateDirectory=\nReadWritePaths=\n' > /run/systemd/system/skaidb.service.d/restore.conf, systemctl daemon-reload, and make the data directory's PARENT writable by the service user for the window.
Start the service, wait for the LISTENER (a large data directory opens long after systemd reports active), run RESTORE FROM '<backup>' TO TIMESTAMP '<target>', stop, undo steps 1–2.

How long it takes. A restore replays the recovery window, not the archive: each backup records where it sits in its node's WAL archive (WAL_CUT, written by BACKUP TO / BACKUP CLUSTER), and the restore skips every archived segment up to that point — the backup holds those records already. What is left is the segments sealed since the backup, up to the target: their records apply in batches with one WAL sync per segment, and they are not written to the archive a second time (the archive is the same size after a restore as before it). The result row says how much of each happened (segments, records, segments_skipped), and so does one log line (restore: replayed …). So recovery time scales with the write volume between the backup and the target — take backups often enough that this is small — and not with storage.wal_archive_retention_secs. A backup that carries no WAL_CUT (taken by an older skaidb, or streamed to a client) and a backup restored against another node's archive are replayed from the archive's first segment: correct, only slower. Each node must therefore restore from ITS OWN backup and archive. While the statement runs the node's /metrics, /status and /ready keep answering (skaidb_engine_lock_longest_hold_ms shows the restore holding the engine).

Coordinated snapshots make the target exact. BACKUP CLUSTER TO '<path>' backs up every member and mints ONE cut instant after all backups complete (recorded in each backup's CLUSTER_CUT); restore every node TO TIMESTAMP that instant and the ring lands on a single consistent cut instead of "roughly the same moment".

A table's archive directory is named by the table's encoded name (the same %-escaping as its data directory, so u/a and x/a never share one). Archives written by an older release under the last raw path component are never moved: while such a legacy directory (row or time-series) still holds segments older than the archive root's .names-encoded stamp, a point-in-time restore leaves that table at backup time and lists it in the result's tables_skipped column; the legacy directories are swept under storage.wal_archive_retention_secs and disappear once empty.

If a restore fails after the swap — the restored data does not open, or the archive replay fails (a window crossing a schema change, say) — the previous data is put back crash-safely: the failed directory is renamed to <data_dir>.failed-restore-<n>, the set-aside copy renamed into place and the parent synced, then the failed copy deleted; the process lock file moves with the data. After a crash during that, the previous data is either back at <data_dir> or still at <data_dir>.pre-restore[-<n>] (move it back with the node stopped); a leftover <data_dir>.failed-restore-<n> is only the failed attempt — delete it. The suffixes are appended to the directory's full name (skai.1 → skai.1.pre-restore).

While the backup is copied in, the data directory holds a RESTORE_IN_PROGRESS file naming where the previous data was set aside; it is removed once the restore (copy, open and any archive replay) has finished. A server refuses to start on a data directory holding it — a crash or kill cut that restore short, so the directory is a partial copy — and on a MISSING data directory with a .pre-restore* or .failed-restore-* directory next to it, instead of creating it empty; the error says which directory to move back. (So a kept <data_dir>.pre-restore* also blocks a deliberate wipe-and-resync start on an emptied data directory: delete the old copy first.) If putting the previous data back fails too, the node refuses every statement until it is restarted (it would otherwise serve, and accept writes into, an empty stand-in database), and the error names where the previous data is.

The previous data directory is left beside the restored one as <data_dir>.pre-restore. It is a full copy — DELETE IT (and the backup) once the restore is verified, or a tight node runs out of disk and the inflow watermark starts refusing writes. The restored data directory keeps the mode of the one it replaced (the package's 0750), and so does the copy kept aside; when you put a saved skaidb.toml back by hand, use cp -p so it keeps its root:skaidb ownership and mode — a config the service user cannot read stops the node at start with Permission denied.

Each node archives its own shard's segments, so the archive is per-node like the backup. A witness mirrors data but is not a recovery point: it applies the bad write too.

Resolution is uniform across table kinds. Row tables seal on the wal_archive_max_lag_secs timer (5 minutes by default), and TIME-SERIES tables ROLL their WAL segment on the same timer — a roll, not a flush, so the head, blocks and compaction cadence are untouched; sealed segments are copied to the archive and stay in place for crash replay until their checkpoint retires them.

Cost. wal_archive_max_lag_secs rotates the WAL on a timer; it does NOT flush, so the LSM's flush cadence and SSTable shape are unchanged. Tables that come due together are sealed one per maintenance tick (100 ms) rather than all in one engine-lock hold, so a fleet of active tables reaching the timer at once costs a run of short holds, not one long one. The seal (one WAL rotation) is the only archive step under the engine lock: the copy of the sealed segment into the archive, with its fsyncs, and the retention sweeps run with no engine lock held and report back in a short hold, so a slow archive volume slows the archiver, not the node. A segment stays on disk until its copy is confirmed — a flush's install never drops a segment the archive still owes, and a refused copy is tried again after 10 s — and skaidb_wal_archive_backlog_segments counts what is owed and not being copied right now (alert on a sustained value). An open lists the sealed segments it finds that the archive does not hold as owed again (a crash between a seal and its copy used to leave them listed nowhere, and the next install dropped them); a segment sealed while archiving was suppressed (a restore's replay) carries a .nocopy marker next to it and is never copied. A resharding cleanup (RECLAIM's Engine::retain) archives the table's sealed segments before it reads or rewrites anything, and a copy the archive refuses defers the cleanup — the table stays as it was and RECLAIM reports the error for the operator to run again — so the rewrite never leaves a gap in the archive. A wait for a copy in flight that times out marks the copier stuck process-wide: no other engine waits for one until a copy reports back. A time-series table's roll takes no engine lock on a cluster member (it runs on the table's own store lock, the copies into the archive on none); elsewhere it is one roll per tick, and a table whose store is busy waits for the next tick instead of being waited for. The archive holds roughly what the node writes: measured on a production node taking ~46 MB/h of writes, a 6-hour window is about 0.3 GB. Only tables archive: a restore replays a table's segments, and that apply writes its index entries, so secondary and geo indexes keep no archive (an index engine archiving under its own name would share the directory of a table named like it, and a restore would replay the index into that table). An index directory an earlier release left in the archive ages out under the same wal_archive_retention_secs — one directory per maintenance pass — and is removed once empty. Put the archive on the same filesystem as data_dir (segments are then hard-linked rather than copied) but OUTSIDE it, or BACKUP will copy the archive into every backup.

Resilience

Damaged table files quarantine the table, not the node. When a table's files are damaged the node starts, every other table serves, and the damaged table is quarantined: every statement against it — reads and writes alike — errors table 'db.t' is quarantined (files damaged at startup: corrupt <file> [at offset N]: <cause>) — restore its directory from a backup or wipe-resync this node, or DROP TABLE to discard it (never "does not exist"), its catalog entry is kept so nothing can silently re-create over the damage, nothing on disk is deleted or rewritten, the startup log names the file, and skaidb_storage_quarantined_tables counts it — alert on non-zero. Remediation: restore the table's directory from a backup, wipe-resync the node from its peers (RF≥2), or DROP TABLE to discard it. This covers encrypted files too: each file's kek_id proves whether the right KEK is configured, so an authentication failure under the correct key reads as the disk rot it is and quarantines — while a wrong or mis-rotated keyring still fails the open loud (a key mistake must never masquerade as one damaged table). Environment errors (permissions, I/O) also still fail the open loud.
What counts as damage. WAL: a record that fails its checksum is classified by where it is and what follows it. Each active log has a durable frontier — the last offset a completed fsync covered, kept in wal.log.frontier next to it and written after every fsync (a page-cache write; the only fsync it adds is one per flush or seal, when the frontier is reset ahead of the log). A bad record at or beyond the frontier is the unacknowledged tail of appends a crash interrupted — a torn tail, whatever follows it: after a host crash a later page of a multi-page group commit can be on disk while an earlier one still holds zeros, and that must not quarantine a table that lost nothing. The tail is cut off, zeroed and fsynced before the open returns, and logged ERROR wal <path>: torn tail at offset N — dropped B bytes of unfinished log (R records replayed before it; the log was durable through offset F); the tail was zeroed. A write is acknowledged only after its fsync returned, and the frontier is recorded only after that fsync, so no acknowledged record is in a dropped tail. Below the frontier, in a sealed segment, or in a log with no frontier file (written by an older binary — the open creates one): nothing but zeros or end-of-file after the bad record (and no later segment holding records) is still a torn tail; records or any non-zero data after it, a zeroed hole with records beyond it, or a bad record at the end of a sealed segment that later files follow, is damage in the middle of the log: the table quarantines with the file and offset named; replay never silently stops at the damage and treats the prefix as the whole log. The frontier on disk can trail the last fsync by the kernel's writeback delay (after a process crash or a clean stop it is exact): an acknowledged record in that gap is protected by its fsync, and only a device that loses bytes it reported durable, together with a host crash, can turn its loss into a logged torn tail instead of a quarantine. A restart in the same boot does not read a log's reserved tail (the page cache holds exactly what was written); after a reboot each tail is checked once. SSTable: a bad magic, a footer/index/Bloom-filter checksum mismatch, or a block index whose blocks are not contiguous, all at open. Manifest: a MANIFEST that is missing or empty while .sst files are present, not UTF-8, or with a malformed line (the error names the line) — the table files are never swept as "orphans" on the strength of a missing or unreadable manifest; only an intact manifest's unreferenced files are build debris. One state without a manifest is not damage: the debris of a first flush that an older binary did not finish (it wrote the manifest only when the first file was installed). When the log still holds every row version of every table file present, the files are removed, the open logs WARNING <dir> — no manifest, but N data file(s) … left by an interrupted first flush and the table opens from the log; a single file that does not open as a table (the build was cut short) is set aside as <file>.sst.unfinished rather than deleted, and only while the log holds records. A file with one row the log does not hold, an empty log, or several unreadable files quarantine as a lost manifest, files untouched. Deletes are rename-first: every table/index directory a DROP, a partition expiry or a REBUILD deletes is first renamed into <data_dir>/.trash/<pid>-<n> (a fixed short name in a directory no user object can be named like; parents fsynced), then removed, so a crash leaves the whole directory or none of it; the next open empties .trash. A new table or index never adopts a directory with its name that no catalog object owns (it is set aside and removed), and REBUILD INDEX marks the index building in the catalog before it deletes the old files (a crash reopens it rebuilding). Missing file: a data file the manifest names that is gone (the error names it), or a WAL file that held records no SSTable has: each wal.log.frontier slot also carries the log's file inventory — the sealed segments (wal.log.seg.NNNNNN) still waiting for a flush, and where records start in the active log. A segment enters it when it is sealed and leaves it, durably, when the flush that covers it has built its SSTable (one small fsync of the frontier, outside the engine lock), before it may be deleted. An open that finds such a segment missing, or no wal.log although durable records were confirmed in it, refuses the table (it used to recreate an empty log with a warning, and never noticed a missing segment): corrupt <file>: WAL file is missing — … held records no SSTable has …. Recovery, in order of preference: put the file back (a backup, or — with WAL archiving on — the archive's copy <wal_archive_dir>/<encoded db.table>/NNNNNN.wal, which is the same bytes for a segment already archived), restore the table directory (backup or point-in-time restore), or wipe-resync the node from its peers. To accept the loss and open with what remains, create an empty ACCEPT_WAL_LOSS file in the table's directory (tables/<db.table>/) and restart: the open logs WARNING wal … opened WITHOUT … and removes the marker. On a member, anti-entropy then re-fetches the lost rows from the other replicas. A directory written by a binary that kept no inventory refuses nothing (the first open records one). The time-series WAL keeps its own inventory (timeseries/<db.table>/wal/ inventory, rewritten at every seal and before a checkpoint deletes the segments it retired), so a missing time-series segment quarantines its table the same way; the marker goes in timeseries/<db.table>/ (next to wal/), and a segment an older binary's checkpoint retired after the inventory last saw one is retired, not lost. Time-series: a damaged WAL segment or block.
SSTable blocks are checksummed (format 2). Every file written by this version carries a CRC32 trailer on each data block, on the block index and on the Bloom filter, and a CRC over the footer; the index, Bloom filter and footer are verified at open (a mismatch quarantines the table), each data block when it is read. A damaged block is a statement error storage error: corrupt <file>.sst at offset N: data block checksum mismatch naming the file and block, on every read path (point read, scan, range, reverse, the stamps sidecar) — no row from that block is ever returned, a streaming SELECT ends with that error instead of a shorter result, and the table is quarantined by that read (later statements say files damaged while serving: …; the gauge counts it). A flipped byte inside a compressed block used to decode to a plausible wrong value, and a flipped index key made point reads miss keys a scan still found. Encrypted files already authenticate every block with AES-GCM; format 2 adds the footer CRC there. Format-1 files (every file written before this version) have the same field positions and no checksums: they open and read exactly as before and are rewritten as format 2 by their next compaction. The manifest gains a #skaidb-manifest-v1 first line; a header-less manifest still loads. Older binaries cannot open format-2 files (they quarantine the table), so rolling back below this version needs a restore from backup.
Indexes rebuild; the catalog is guarded. A secondary or geo index whose files are missing or damaged at open is derived data: its files are removed, the index is marked building in the persisted catalog, and it is rebuilt from its table (a server pages the backfill in the background; the planner leaves it alone meanwhile) — the node starts. A data directory whose catalog.json is missing while tables/ or timeseries/ hold data refuses to start with catalog <path> is missing but the data directory holds N table director(ies) with data (db.t, …) — refusing to start as an empty node. Restore catalog.json from a backup (the table files are intact and reattach to it), or move <tables> and <timeseries> aside to start fresh, instead of starting empty and later reattaching the old files to re-created tables. A corrupt catalog.json refuses with catalog <path> is corrupt (<parse error>): the schema cannot be trusted … restore catalog.json from a backup.
A slow or unresponsive replica cannot hang a write. Every internode connection has a bounded read/write timeout, so a peer that is up but not answering (thrashing under memory pressure, a kernel that accepted the socket while the process is stalled) is failed fast — the coordinator meets the write quorum from the responsive replicas and hints the slow one for handoff, rather than blocking on it. (A refused connection already failed fast on connect; this covers the connected-but-silent case.)
No fsync under the engine lock — rows, secondary indexes, search indexes and UNIQUE reservations alike. A statement that claims or releases a UNIQUE value hands its syncs up with the rows': claims are synced first, then the rows, then (one short write hold to append them) the releases — so a power loss can leave a reservation without its row change (a stale reservation, which CHECK INDEX reports and REPAIR clears after the 10 s grace) or, in the window before a statement returns, an unacknowledged row without its reservation (reported as missing, repaired the same way), never a release kept while the row change it belonged to was lost. A write holds the engine lock for its memtable + WAL-buffer append only; the WAL fsync runs after the lock is released (group commit: one writer at a time syncs a WAL through everything appended so far, and every commit that fsync covers returns when it completes — one fsync per group of concurrent writers). A coordinator's own copy of a single-row write appends its secondary-index entries under the lock too and syncs the index WALs, then the table WAL, after releasing it — before the copy counts toward the consistency level, so an acknowledged write is on disk on every replica that counted. A coordinator's own copy of a time-series append takes the engine lock for the store lookup only (the store has its own lock and fsyncs its own WAL); a table with rollups appends under the engine read lock. The same holds for a full-text index: the write path queues the index operation and never touches the Tantivy writer, the refresher thread (or, for a read-your-writes search, the coordinator just before it) commits with no engine lock held, and a memory-pressure release freezes memtables one per short hold and commits dirty index writers outside. So a slow disk makes that member's fsyncs slow, not its engine lock: other statements on the member — and the quorums its peers need — keep flowing.
A quorum read is never failed by a suspended-but-healthy peer. The per-peer circuit breaker makes a flapping peer cheap by failing calls to it fast (after its cooldown a background prober re-tests it while every caller keeps failing fast — with a probe the peer answers only once it has written and synced a small file in its data directory, so a member whose disk cannot sync stays suspended though it still answers pings; an older member answers it as a ping — so a peer that never replies costs no client statement anything — skaidb_cluster_breaker_probes_total counts the re-tests — and every internode dial is bounded by a 3 s connect timeout, so a black-holed peer trips the breaker within seconds instead of holding each statement for the OS's SYN retries), but a peer that has just restarted — or that briefly declined reads as busy — stays suspended on every coordinator for the cooldown after it is serving again. Counting those peers as non-responders would fail QUORUM reads on a cluster where every member is up, which is exactly the shape a rolling upgrade produces. So when, and only when, a read is about to fall short of its quorum, the coordinator gives each unanswered peer one breaker-bypassing probe: a bounded (2 s) connect/read/write attempt that ignores the breaker. A success satisfies the read; the peer's circuit closes for every other caller only when an answer proves the peer can complete a durable write — the prober's probe or an acknowledged write — so a member that answers from memory while its disk cannot sync is not sent writes again, and a recovered member is still picked up within the prober's next round. This covers point reads, paged scans (LIMIT pushdown, candidate resolution, streaming aggregation) and time-series gathers. Healthy reads never pay for it, and a genuinely dead peer costs the bounded timeout at most once per statement — the alternative being an error. When the probe also fails, the "read quorum not met" error names each peer and why it failed.

Memory

Memory-pressure release and load shedding. A node watches its memory against its limit (cgroup when set, else system RAM), measuring non-reclaimable usage — the cgroup charge minus reclaimable file-backed page cache (mmap'd SSTable/WAL/search segments the kernel evicts before OOM), so a cache-filled node isn't falsely shed while it still has real headroom. Two tiers:
Past 75% it actively releases: freezes table/index memtables (≥4 MB) for the background flusher and commits every dirty search-index writer (Tantivy holds indexed-but-uncommitted documents in heap buffers; a node that stops taking writes otherwise never commits them and rides its limit until the fault storm or the OOM killer gets it — observed in production). The pass is staged so reads keep flowing through it: the engines to release are listed under the read lock, each is frozen in its own short write-lock hold (one WAL seal), the SSTables are built by the flusher thread and the search commits run with no engine lock held. Release actions are paced (at most every 10 s).
Past 85% it also sheds writes — rejecting new writes (client and inbound-replica) with a retryable "memory pressure" error — and the release pass turns aggressive (flushes memtables down to 64 KB). The release is driven by the memory sampler, not a client write — otherwise a shedding node would deadlock (it rejects the very writes that would trigger a flush) — and covers every memtable, since pressure spread thin across many tables leaves each below the per-engine flush threshold while the sum pins the node. The flag clears at 70% (hysteresis).

Shedding is loud: entering it logs the anon/file split plus jemalloc's allocated/resident/retained numbers, a distress line repeats every 60 s while it persists ("releases are not freeing enough; OOM risk"), and recovery logs the episode's duration. Anti-entropy passes log their duration (and allocator stats) whenever they reconcile rows or take ≥60 s. The guard is the node's memory protection: the packaged unit sets no kernel memory.high (MemoryHigh=infinity), since that limit counts page cache and would hold a node whose data exceeds its RAM under constant kernel reclaim. Usage and stall pressure are read from the container's cgroup, or from the service's (or its slice's) own cgroup when that carries a memory.max below the container's. Reads and DDL are never shed; a coordinator that gets a shed rejection from a replica hints it and proceeds at quorum. Watch skaidb_memory_shedding_writes / skaidb_memory_used_bytes (METRICS.md); a node stuck shedding is undersized for its workload.

Disk

A node that runs out of space blocks its own writes, not the cluster's. Below iwm.disk_low_watermark the node enters IWM state 3: /ready answers 503, /status shows iwm.disk_write_block: true, and INSERT/UPDATE/remote-write get the retryable error disk full: node is blocking writes, retry — the drivers back off or fail over to another member, and the writes it misses are hinted to it like any other outage. Reads, DELETE, DDL, retention and compaction keep working: the node gives back the reserve file it keeps for exactly this (iwm.disk_reserve, 64 MB by default, .skaidb-reserve in the data directory) so tombstones and compaction output have somewhere to go. A write that had already committed is never lost to a full disk — it is in the WAL and the memtable, and the flush it triggered is retried once space exists. Restarting on a full disk is safe. skaidb_iwm_disk_rejections_total, skaidb_iwm_disk_enospc_trips_total and skaidb_iwm_disk_reserve_present (METRICS.md) show the guard at work; the recovery steps are in INSTALL.md.

Graceful shutdown

Graceful shutdown. SIGTERM/SIGINT (what systemd's stop/restart send) first drain the node: /ready answers 503 not ready: shutting down and /status shows shutting_down: true; the REST, binary and MQTT listeners close, so a new connection is REFUSED (nothing was sent — a driver fails over to another member without ambiguity); idle keep-alive and pooled connections are shut down, so a client's next checkout sees EOF and reconnects elsewhere; and the statements already running get up to server.shutdown_drain_secs (default 15, live-mutable) to answer — a REST response written in that window says Connection: close. The server log records drained N in-flight statement(s) in M ms (K idle connections closed), or a WARNING with how many the budget cut. Only then does the node flush memtables and commit search-index writers before exit, so the next start replays almost nothing — an unclean kill costs a search-index catch-up from the last committed watermark. The stop also records each table's newest stamp, so the next open serves every vector snapshot and search commit that nothing is stamped after at once, with no catch-up walk (after an unclean kill the bound is unknown and the catch-ups page the tables, under the read lock). The flush waits at most ~10 s for the engine lock, so a wedged node still exits inside systemd's kill window. Every phase of the stop is bounded — statement drain (shutdown_drain_secs, at most 60 s), MQTT disconnect + broker-state flush (15 s), drivers-registry flush (30 s), index-maintenance drain (20 s), memtable flush + content digests (240 s) — 365 s at worst, 320 s at the default drain, under the packaged unit's TimeoutStopSec=420. The last log line before clean shutdown gives each phase's duration: shutdown phases — drain 0.004 s, mqtt 0.000 s, registry 0.002 s, index maintenance 0.000 s, flush 1.310 s; total 1.316 s (worst case 320 s), with (budget elapsed) after a phase that ran out of its budget. The dedicated metrics/health listener keeps serving through the drain, so monitors watch the node leave rather than lose it.
Full-copy counts are local. When replication_factor >= members (every node holds every row), unfiltered COUNT(*) is answered from the local engine's key statistics — no cluster gather (which would materialize the whole merged table on the coordinator). The local answer is O(keys) with no value decode, at the same freshness trade the search paths already make. A member in the resync state (rebuilding from an empty or part-filled store) does not take this shortcut at any consistency level — it gathers from the complete peers until its catch-up completes.
Compaction commits before it deletes. Retired SSTables are removed only after the manifest durably points at their replacement, so a kill can never leave the manifest naming deleted files (which would make the engine refuse to open). A manifest entry that fails to open is logged with the exact file and manifest path.
Bulk index builds stream the table. Building or rebuilding a search index (CREATE SEARCH INDEX, startup catch-up, or an automatic rebuild) reads the source table one row at a time rather than gathering the whole shard into memory first, so indexing a large table stays within a bounded footprint (the writer heap, sized from memory_target, plus one row) and does not OOM a small node. DB workload must never OOM a node.

Operational notes & limitations

Same RF on every node, and seeds that name real peers. A node's own identity (bind_addr:internode_port, or advertise_addr:internode_port) must be reachable by peers at exactly that string — it is what the announce sends and what peers dial — and every seeds entry must parse as host:port (an entry that does not, such as "not a host:::", refuses the start). A node need not list itself: the announce admits it (see Automatic join); a symmetric list on every node is simply the easiest way to keep configured and the ring in step. bind_addr may be 0.0.0.0 only together with advertise_addr.
Distinct data_dir per node (and distinct ports when co-located). Every server process holds an exclusive lock on <data_dir>/.lock for its lifetime, taken before the engine opens, so a second process on the same directory refuses to start (data_dir … is in use by another skaidb process (pid N …)) instead of replaying the WAL beside the live node and flushing over its acknowledged writes. The lock, not the file, is the signal: a file left behind by kill -9 is taken over at the next start.
No membership gossip/consensus. Static membership is via seeds; runtime changes are best-effort broadcasts ordered by an epoch and persisted, so a restart reloads the live ring. A fresh node auto-announces to a seed to get admitted (CLUSTER_MEMBERSHIP.md), and a member that missed a broadcast catches up by a membership pull: every member's host stats carry its epoch, whether its window is open, and its member list, and a node that sees a peer on a newer settled epoch adopts that ring on its next probe (the routine host-stats refresh, or the heal tick while it holds a window) — a rollback or finalize a busy member never received reaches it within seconds instead of waiting for the next change. There is still no consensus: two concurrent topology changes aren't linearizable. Do one membership change at a time.
Transactions are single-node. BEGIN/COMMIT/ROLLBACK work on the embedded engine and over driver connections to a STANDALONE server (per-connection transaction state, crash-atomic commits via a redo journal); the cluster coordinator autocommits per statement and refuses transaction control — there is no distributed transaction coordinator.
Key-listed searches during a roll. A search whose WHERE pins the whole primary key sends the key set to every member inside the search query (docs/SEARCH.md, key-list restriction). A member on a version without it cannot decode that query; the coordinator re-asks it with the keys taken out — all its hits, as keys and scores (statistics too, under global BM25) — and keeps only the listed keys. The answer is unchanged and no shard counts as missing; that member just pays the unrestricted search's index cost until it is rolled. The sharded top-k and aggregate fast paths decline on such a member and fall back to that gather.
Joins gather to the coordinator. A single-table WHERE is pushed to the shards, but a SQL JOIN pulls the tables to the coordinating node.
Bulk-apply QoS. Inbound bulk batch appliers (drain, rebalance, hint replay, repair) run under admission control — at most half the cores (clamped 1–4) apply batches concurrently, so a migration flood can't monopolize the CPU and starve foreground queries (measured during a decommission: cpu PSI ~80% with io ~0 — pure CPU saturation from FTS-indexing inbound rows). The admission wait is bounded (2 s): a saturated node answers "busy" fast instead of parking excess connection threads on the gate — an unbounded queue would let a rejoining node facing every peer's catch-up flood accumulate thousands of abandoned threads (senders time out at 10 s and retry on fresh connections) while making no progress. Rejected/timed-out senders degrade to hints, which repair backstops. "Busy"/shedding replies count toward the sender's circuit breaker like transport errors, so a saturated peer gets its full cooldown (10 s) instead of a full-rate retry hammer: the circuit closes on the prober's ping or on a real answer to a second-chance probe, never on a busy/shedding reply and never on the periodic clock probe — without this the rejecting node would burn cores decoding a flood it keeps refusing. Inbound repair scans (ScanPage/LocalScan) take the engine read lock with a bounded wait (1.5 s) and answer "busy" rather than parking behind a catch-up write flood; the repairing peer retries a busy answer with backoff (500 ms; a peer that stays busy trips the breaker within a few attempts), then treats it as unreachable for that pass and retries next interval. TS repair merges and summaries (TsMerge/TsSummary) instead queue for the lock: they arrive serially (one in flight per pass), so parked threads can't pile up, and a bounded poll starved under continuous ingest — a writer is almost always waiting, and waiting writers block new read attempts — failing passes that should have converged. A merge frame holds the lock only to find its table: the merge, and the rollup backfill of a table with rollups, run on the table's own store lock. The drain also pauses migration_pause_ms (floor 10 ms) between chunks. node_stats carries cpu_pressure_pct (PSI) so saturation is visible per node.
Broad filters resolve in one merged scan. A pushed-down WHERE (or an index scan) gathers candidate keys per member and re-reads them for the authoritative LWW version. Few candidates re-read as quorum point reads; past ~256 the coordinator switches to a single paged, LWW-merged pass over the table intersected with the candidate set — same read-quorum guarantee, one scan instead of one RPC fan-out per key (a count(*) over a 100k-row match would otherwise issue 100k sequential quorum reads).
A flapping peer is circuit-broken. A zombie node (TCP up, application unresponsive) would otherwise make every replicated write burn the full internode I/O timeout — worse than a cleanly-down peer. After 3 consecutive failures a peer's circuit opens: calls to it fail fast (the coordinator hints immediately) for a 10 s cooldown, then one call re-tests. Liveness probes bypass the breaker and close it on success. A frozen peer (SIGSTOP, a host swapping it out: its kernel accepts connections, nothing answers) is caught sooner: a peer that leaves two clock-probe rounds in a row (every 2 s, 1.5 s each) unanswered while another peer really answers has its circuit opened, so writes hint it ~4-6 s in instead of waiting out three 10 s I/O timeouts (~18 s). A round where no peer answered — a refused connection or a mixed-version reply is no answer — counts for nothing (that is this node's own stall, or a roll's restarting member), so a coordinator never trips a healthy peer on its own hiccups; once the peer answers a clock probe within 800 ms again, a circuit opened this way is re-tested at once with the breaker's durable probe and closes on its Pong (a member whose disk cannot sync stays open).
Disk-spilled hints drain in bounded pages, only to live peers. The hinted-handoff overflow log replays a page (1024 records) at a time and is left untouched while its peer is down, so a large log never balloons a restarting node's memory. The drain also runs after a restart for logs inherited from the previous process.
Unfiltered scans page, merge, and honour LIMIT. SELECT … FROM t with no WHERE gathers every shard last-writer-wins, paged so the coordinator holds a few pages at a time rather than whole shards. Every page — a member's answer to the gather (ScanPage) and the coordinator's own — ends at 2,000 (peer) / 20,000 (local) rows, 4 / 8 MB, or about 20 ms of its engine-lock hold (the clock is read every 16 rows), whichever comes first, and says whether more rows follow; a member from before this bound pages by rows alone and its short page is its last (mixed versions interoperate during a roll). A plain LIMIT n (no ORDER BY) is pushed into the gather: sources are paged in lockstep and a row is emitted once every still-active replica has scanned past it, so the scan stops after the first n rows are sealed instead of materialising the whole table.
A PK pinned by =/IN is a point-read set. When every primary-key column is pinned by an equality or a literal IN list (bound array parameters included), the coordinator resolves the exact candidate keys (≤ 1000; composite keys cross-multiply) and reads them as batched, pipelined quorum point reads to their replica sets, however many — the "fetch these N ids" shape never scatters a filter, and never pages the key span between the smallest and largest id. That holds for a streamed result too: each keyset page (… ORDER BY <pk> LIMIT n AFTER (…)) is the next n listed keys past the cursor, read as the same batched point reads at ONE, QUORUM and ALL.
Subqueries and joins on a whole primary key probe it. A correlated [NOT] EXISTS / [NOT] IN / scalar subquery, a LEFT JOIN … WHERE <key> IS NULL anti-join or a plain [LEFT] JOIN whose equalities pin the inner table's entire primary key reads that table as batches of primary-key IN reads (≤ 256 keys each, so every batch is multi-key point reads per replica set, never the merged pass), at the statement's read consistency — the inner table is never gathered to the coordinator (see QUERY_SYNTAX.md → Correlated subqueries).
ORDER BY <indexed> LIMIT k at QUORUM on a table every member replicates first tries an agreed window (k ≤ 2048): the coordinator walks its own top k rows (decoding only the columns the statement reads) and takes each row's version stamp, while quorum − 1 peers — taken in rotation, peers with an open circuit last — each answer one SortedWindowStamps: their own top k keys (no overfetch) with the version stamp of each, never a row body. When every consulted window holds the same keys at the same versions, the coordinator's window is the answer, with no candidate re-read: any write acknowledged by a quorum is on a consulted replica, so a row outside the window sorts, by its newest values, behind k rows whose agreed values are the newest. That needs each consulted replica's index to place every row by its current value, and index entries land after the row, on the maintenance applier: a replica (coordinator or peer) declines while an entry of the index range it walks still trails its row — and only then: queued maintenance that leaves that range's entries alone (writes to other columns, full-text work) does not make the window decline, so it keeps serving on a table under writes. Any difference (a lagging or newer replica, a delete one side has not applied), pending maintenance, an unreachable peer or one that predates SortedWindowStamps, or an AS OF read falls back to the distributed sorted top-k below, which re-reads and repairs. A multi-key order (ORDER BY d DESC, id DESC, ORDER BY d, x DESC — any trailing columns, any directions) is an agreed window too when a secondary index walks its leading key: every consulted window is the top k rows on the leading key plus its boundary tie run — every further row whose leading value equals the k-th row's — asked of the peers as SortedWindowTies. Such a window holds every row that can rank in the top k under the full order (a row strictly behind on the leading key has k rows ahead of it, whatever the later keys say), the replicas must agree on all of it (keys and versions, tie run included), and the executor sorts it by the full order and cuts. A tie run longer than 256 rows past k (a coarse leading key) declines (metric window_tie_overflows), as does a leading key that is the primary key's first column or has no index, and a peer that predates SortedWindowTies (it answers "unknown request op"); each falls back like any disagreement (metrics agreed_windows_multikey / window_tie_rows). Any agreed window, single-key or multi-key, also declines when the index walk would rank rows unlike the sort: the walked range holds a NULL or missing leading value (the index places those lowest, the sort places NULLs last ascending and first descending) unless the filter bounds the leading column, or holds numbers under more than one encoding (Int next to Float: the index keeps them apart, the sort compares values). A bounded probe of the index (one seek per region, no row read) decides it. Each consulted replica stamps a window row from its walk's own read of it — that read's stamp and a fingerprint of the bytes it decoded (one read and one fingerprint pass per row, not a second versioned read; metrics window_rows_stamped / window_row_rereads) — so the stamps add no rows examined. A keyset probe walk makes one such read per page, so a QUORUM anti-join walk under a single-column ORDER BY examines what it examines at ONE (a 44k-row mailbox of 15 KB rows, 1 untagged in 40, NOT EXISTS … LIMIT 250: 1.25 s and 92k rows examined → 0.5 s and 21k in-process; LIMIT 500 used to fail the row budget). The same holds for a tie-broken order such as ORDER BY internal_date DESC, id DESC: its pages are multi-key agreed windows (12k × 15 KB rows, LIMIT 250: 92k rows examined, 0.91 s wall and 1.8 s CPU in-process → 21k, 0.33 s and 0.47 s — what the single-column order costs).
ORDER BY <indexed> LIMIT k at QUORUM is a distributed sorted top-k (k ≤ 10000): every member returns its local index-ordered top candidates (4× overfetched to absorb per-shard staleness in the sort column and boundary ties), the bounded union is re-read at quorum, and the executor re-sorts — ~members × 4k row reads instead of gathering the whole match set. The members' candidate walks run at the same time (the peers' walks are on the wire while the coordinator walks its own index; the coordinator's own plan first decides whether the order can be served at all — from the plan, reading no rows — so a declining plan never asks the peers). The re-read is batched: one multi-key read per consulted replica for every 128 candidates (up to 8192 candidates), keys whose reads consult the same peers sharing chunks, and every chunk on the wire before any reply is awaited (up to 16 at once) — the re-read costs about one round trip whatever the primary key's order, and the peers' row reads overlap the coordinator's. A chunk a consulted peer answers busy, short or not at all is asked of the replicas not yet consulted as one batch; only keys still short after that fall back to per-key quorum reads. Each member's candidate walk decodes only the sort column and the filter's columns and holds (and byte-charges) only the candidate keys. A WHERE pinning every primary-key column to literals (account = ? AND id IN (…), up to 1,000,000 keys, read 128 keys per round trip) is not a top-k at all: it is that bounded set of quorum point reads, re-sorted — walking the order index with the pin as a residual read every row of the range when fewer than 4k rows matched. When the coordinator is itself a replica (always, on a full-copy table) the re-read is lean: it reads its own copy first and sends each consulted peer its per-key version stamps; a peer ships a row body only for a key where its version wins last-writer-wins against that copy (a newer stamp, a winning tombstone, or the same stamp with different bytes) and answers the rest with its stamp alone. QUORUM semantics are unchanged — the newest version on any consulted replica wins, deletes are honoured, stale replicas (the coordinator's own included) are read-repaired — but agreeing replicas move stamps, not rows. A peer that predates the lean read answers each chunk already sent "unknown request op" and is re-asked for full rows (all re-asks in one wave), and chunks sent after its first refusal ask it the full read directly. On a full-copy table (RF ≥ members) the candidate union needs only a read quorum of members — a member that is restarting or busy is an excluded source, not a reason to abandon the walk for a whole-match-set gather (which the byte budget rejects for a large table); a sharded table needs every member. A member whose copy of the index is still building holds every row, so a range or filtered candidate union asks it for the filter's exact keys instead of counting it out — the coordinator keeps its own index-bounded leg even at two members, where a quorum is both and nothing could be excluded. Fewer matches than k is served bounded too when every member's walk ran dry below its window (a sparse filter). When no index-ordered walk serves the order at all, the exact gather behind a single-column ORDER BY col LIMIT k retains only the best k rows as they finalize — O(k) memory however many rows match (multi-key orders keep the full gather). A multi-key ORDER BY <indexed>, <more…> the agreed window above did not answer rides the same leading-column windows under a coordinator-side completeness proof: the top-k is served bounded when at least k rows sort strictly before every truncated window's weakest candidate on the leading column (ties there are decided by the later keys, which the full re-sort applies); a leading column with too few distinct values fails the proof and the exact gather answers. Conservative by construction: if any member is unreachable or cannot walk in order, or fewer matches than k resolve, the exact full gather answers instead; every returned row is quorum-fresh. (At consistency ONE on a full-copy cluster the ordered read serves entirely from the local replica's index walk.)
Schema repair converges index definitions, not just names. Every repair pass exchanges the full catalog as stamped idempotent DDL; a replayed CREATE … IF NOT EXISTS whose schema stamp advances replaces a differing search/vector index definition and rebuilds it — a node that missed the DDL that widened an index while down does not keep the stale, narrower definition forever. SHOW INDEXES' local column (ok/building/missing) exposes each node's live index state for cross-ring comparison.

See RESHARDING.md for the deeper design and the current edges.