skaidb v0.320.9

Replication, consistency and ordering

Part of Running a skaidb cluster.

How many copies a row has, what an acknowledgement means at each consistency level, how clustered UPDATEs stay linearizable, what every write error means for a retry, and how clocks bound it all.

Replication factor & consistency

  • replication_factor (RF) — how many nodes hold each key. RF 3 tolerates one node down at QUORUM. RF must be ≤ the number of nodes (it's capped at the node count otherwise). Use the same RF on every node.
  • Consistency (ONE / QUORUM / ALL), set as the cluster defaults:
  • ONE — ack after one replica (fast, weak).
  • QUORUM — majority of replicas (floor(RF/2)+1).
  • ALL — every replica.
  • Strong consistency when read CL + write CL > RF (e.g. QUORUM+QUORUM with RF 3 → R2 + W2 > 3). With weaker levels, the remaining replicas are updated in the background and converge via anti-entropy.
  • Per session: SET CONSISTENCY ONE|QUORUM|ALL on a binary-protocol connection (or \consistency in skaidbsh) overrides the defaults for that session's statements. REST is stateless and rejects it.
  • A slow replica does not hold the cluster's writes up. A write asks the replicas its level needs, the prompt ones first; one that has not answered within 250 ms while another replica can take its place — or once the write's level is met — is not waited for. It still gets the write (and a hint, in case it never applies it), and for the next 10 s it is asked last. QUORUM point reads, the consensus rounds of linearizable updates and time-series inserts (sent to every replica at once) work the same way. skaidb_cluster_slow_replica_skips_total counts it; a rate that follows one member points at that member's disk.

Secondary indexes and acknowledged writes

A write is acknowledged once its row is durable in the table on enough replicas; each replica maintains its secondary indexes afterwards, from a per-node queue (milliseconds behind, tens of them behind a multi-hundred-row batch). A statement that reads through an index still answers for the table:

  • Lookups and mutations never trail. Each member knows which of its rows are stored but not yet indexed (the table's un-indexed tail) and completes every unordered index read from them: the candidate keys a coordinator gathers for WHERE <indexed col> …, the local index gather at ONE, the index-narrowed count. UPDATE / DELETE … WHERE <indexed col> issued the moment an INSERT is acknowledged — through any coordinator — affects every matching row; there is no wait and no dependence on the queue's depth. A tail row is judged by its current version, so a row that moved out of the predicate is not returned and one that moved into it is. Completing a read costs it nothing per queued row: the index entries a queued write replaces and produces are recorded with it when the row is stored, so placing the tail against a read's range is a comparison of entry keys, and only a row the index actually trails inside that range is read.
  • Index-only counts never trail and never wait either. A count a covering index (or an intersection of indexes) answers stays an index range count on a table under writes: the entry count is corrected by the tail — plus every entry a stored row produces that the index does not hold yet, minus every entry it still holds for a value the row has left — which is the table's own answer, at every consistency level the local count serves. It declines to the streamed count only behind a bulk backlog: a member that is resyncing, or more than 2048 maintenance tasks queued for the table (a migration, a hint replay).
  • Index-only counts hold the engine lock a page at a time. A member counts an index range in passes of about a millisecond each, every pass under its own read hold of the engine lock, released between passes so the writes a pass held up go first — as coordinator, as a replica asked for its count, and on a standalone server for SELECT count(*) … WHERE …. A writer waits for one pass, never for the range (one hold over a 600k-entry range would keep every write on the member waiting about 180 ms). The storage engine follows every write to an entry a count has already passed, so the answer is exactly the count one hold would give in the hold of the last pass: entries written, deleted or moved within the range meanwhile are each counted once or not at all, as they stand then, and the tail correction is taken in that same hold — a row stored and acknowledged before it is always counted. Intersection counts (several indexes) page the same way, and so do the counts that narrow through a numeric index range beside a filter no index serves: their walk reads each candidate row, and it follows the table's writes row by row — a row written while the count runs, or whose index maintenance is still queued, is counted as it stands in the last pass, never twice and never missed — so they too give the count one hold would give then. On a table with a TTL, whose rows expire with no write to follow, they are taken in one hold, as before.
  • Table walks hold the engine lock a page at a time too. A count no index serves (the row walk behind count(*) … WHERE …), SELECT DISTINCT <column> at consistency ONE on a full copy, and the unfiltered count(*) on a full copy walk the table in passes, each under its own read hold, released between passes — on a member and on a standalone server alike. The unfiltered count is exact through the same write-follow as the index-only count; a filtered count or a distinct folds each page as it stands when it is read, so a row written behind the walk's cursor is not revisited. One hold for a whole walk, on a disk that stalls, was every statement on the node waiting behind it.
  • Multiple NULL-safe exclusion counts require membership agreement. Conjunctions with two or more (col != value OR col IS NULL) predicates can use paged primary/index key-set operations (see INDEXING.md). ONE counts the local set. QUORUM/ALL on fully replicated tables compare both count and SHA-256 of the ordered eligible primary keys from every replica the level requires. Equal cardinalities of different sets are a disagreement. Every statement checks afresh, regardless of cluster.index_count_check; a mismatch, running out of replicas that can prove the count, and local planner declines use the normal LWW row merge. A replica that cannot supply a proof (unreachable, an older binary, past the reply wait) is replaced by the next replica, as for any index-only count. A replica stops counting for a coordinator that has given up. Range watches keep each local proof consistent across page releases; writes inside an input range allow two restarts, then use rows. Other account ranges do not invalidate a count. Type/coverage gates and bounded cursors prevent either row-body gathers or candidate-sized key sets on the admitted path; the row fallback retains its ordinary scan and byte budgets. EXPLAIN's cluster.access reports this set-count path and its conditional row fallback; cluster.fan_out distinguishes local ONE from peer proof requests at QUORUM/ALL, with the full-copy placement requirements. Fully pinned counts report count_point_keys and point-read routing, including qualified =/IN pins. A declined local plan reports its reason and row-count routing. EXPLAIN does not request proofs or establish that replicas currently agree.
  • Other index-only counts at QUORUM and ALL check the replicas the level needs. At ONE the coordinator's own index count answers. At QUORUM / ALL on a table every member holds whole, the coordinator also asks as many other replicas as the level needs for their own index-only count — one small request each, in parallel and while the coordinator counts, answered from the replica's index with no row read, so an agreeing count costs one round trip more than at ONE; a replica that cannot answer (unreachable, busy, resyncing) is replaced by the next. A replica that has not answered shortly after the coordinator's own count (its count's time plus 50 ms, between 100 and 300 ms) is left out for 10 seconds — a frozen or swapping member costs one count that wait, not every count the transport's timeout — and when no replica is left to ask, the coordinator's own count answers. Once checked, the same count (table, filter and level) is served from the coordinator's index for 5 seconds without asking again — under log (the default), any count of the table at that level, whatever its filter, so counts whose literal changes every time (a parameterized WHERE user = ?, a sliding time window) ask once per table every 5 seconds too: each replica's answer is a count on that member, and asking on every count of a hot filter, or of every new literal, cost every member that count's work. When every answer is the same, that is the count. When they differ they are asked again — unless the coordinator's table took a write while they counted: then the range is being written (writes in flight between replicas) and the coordinator's own count answers at once, as a count under writes always has. Asked again, answers that now agree are the count; answers that MOVED mean the same as a write seen — the coordinator's count, no pause, no row read; and only a difference that holds still for three rounds (the last 5 ms after the second) is a disagreement: counted in skaidb_cluster_index_count_disagreements_total and logged (at most once per table a minute) with the index and each member's count — a member whose index has entries out of step with its rows; CHECK INDEX <name> REPAIR on that member fixes it (see Auditing a member's indexes). What the count then answers is cluster.index_count_check (live-mutable):
  • log (the default): the coordinator's own count, as before the check — the drift is reported, the answer and its cost unchanged;
  • enforce: the rows' count (the row-merging quorum read, which unions every replica's index keys; on a 150k-row range about 80× an index count), logged next to the index counts, and served again with no replica asked and no row read while the table takes no write (at most a minute);
  • off: the coordinator's own count, no replica asked.

So with enforce, one member's index drifting from its rows changes the answers that member coordinates only at ONE, and at QUORUM / ALL only while the table is being written. Turn it on once CHECK INDEX ALL reports ok on every member. A pinned table (or one whose placement is changing) has no whole copy on every member and counts from the rows at these levels. During a rolling upgrade, a member on a release without the request answers "unknown request op"; it is not asked again for a minute, and the count answers without it. - Index-ordered reads (ORDER BY <indexed col> LIMIT n, the sorted top-k, agreed windows) wait, on every member that walks for them, until the queue has written the index entries of what that member had acknowledged when the read began — read-your-writes through the index on the coordinator that took the write, and at QUORUM through any coordinator. There is no wait when nothing queued for the table changes an index entry (writes to columns no index covers, full-text work). The wait is bounded (250 ms) and is skipped when the queue is more than 2048 tasks deep, as during a bulk load: the walk then runs on the index as it is and its order can trail the table by the queue's lag. skaidb_cluster_index_settle_waits_total counts the waits, skaidb_cluster_index_settle_timeouts_total the reads that ran without the queue having caught up.

skaidb_cluster_index_tail_keys is the number of rows in the un-indexed tail on a member (every table). It follows the maintenance queue and is 0 whenever the queue is idle.

A member indexes the writes it coordinates in place and the copies it receives as a replica from its queue, so the same row written through two coordinators at once can have a newer version stored — and indexed — on a member before that member's queued task for the older version runs. The queued task then works from the row as it is stored: it removes only the entries the stored row no longer produces and adds entries only for the stored version, so every member's index ends up holding exactly its rows' entries whatever order the writes arrive in. The queue survives an unclean stop (a kill, an OOM kill, a host reset): a member replays, at its next start, every write whose queued task had not run — the point it replays from trails the wall clock by about 10 s and stays below the oldest task still queued and the oldest full-text write no search-index commit has made durable yet, and a copy that arrives stamped older than that point (a hint, a repair push) moves the point below itself, on disk, before the copy is acknowledged — as does every other copy acknowledged for that table while the move is still being written. A task whose index write fails (a full disk) stays queued and is retried — after a second, then backing off to once a minute, applied as the queue is, a few dozen per engine-lock hold — so the member's index catches up once the disk accepts writes again; one still failing after 30 minutes is given up, counted in skaidb_cluster_index_maintenance_abandoned_total and logged with its table, for CHECK INDEX … REPAIR. Until then the replay point, and the flushed WAL segments of that table, stay behind it (skaidb_cluster_wal_segments_held). The tasks waiting for a retry count with the queue against its 256 MB bound — writes wait while the two together are past it — and hold at most 256 MB themselves (skaidb_cluster_index_maintenance_retry_bytes): a task failing past that is given up at once, counted and logged the same way. skaidb_index_superseded_entries_kept_total counts the entries a queued task left in place because the stored row still produces them and nothing else queued for the row would have put them back — entries that member's index would otherwise have lost.

Full-text and vector indexes keep their near-real-time visibility (refresh_ms, the HNSW applier).

ONE on a full-copy cluster is a local pass — use it for analytics

When RF ≥ member count every node holds every row, so a read at ONE is served by a single pass over the local replica instead of gathering the same rows from several replicas and last-writer-wins-merging them on the coordinator. This applies to every read shape: point reads, COUNT(*), filtered counts, DISTINCT, ordered reads, and plain and grouped row gathers (GROUP BY, time_bucket).

The difference is large on aggregate reads. Measured on a 3.69M-row table over a 24-hour window (~40k matching rows), RF=3 over 3 members:

QUORUM ONE
count(*) 1612 ms 116 ms
GROUP BY service 2050 ms 115 ms
GROUP BY service, name + ORDER BY avg() — 164 ms
time_bucket hourly buckets — 109 ms

Dashboards and reporting queries are the intended users: they read data every node holds in full and do not need read-your-writes. The trade is that ONE returns one replica's view, which may lag an in-flight write by a beat — so do not use it for a read that immediately follows a write. A resyncing or partially-backfilled node never serves these paths (see Resync state), so ONE cannot read an incomplete copy.

The local pass shares the node with everything else on it. Where it walks a table — no key, key range or index serves the filter — it holds the engine's read lock a page at a time, released between pages, so a write waits for a page rather than the whole scan; a parallel GROUP BY's workers page their own chunks. Each page reads the table as it is at that moment: a key comes back at most once, and one that existed throughout the scan is never missed — but a row moved to another key mid-scan can come back under both keys or neither, and a multi-row transaction committing mid-scan can be seen half-applied. AS OF reads page the same way (a row written after the instant is hidden on every page). A walk in INDEX order (ORDER BY <indexed column> LIMIT k with a filter that keeps it walking, an indexed range, a grouped fold over a range) pages too: it follows the writes into its index range and never returns a row twice — a row whose indexed value changed mid-walk and moved behind the walk is not returned — and a range written faster than it can follow (or an index rebuilt under it) ends the statement with a re-run error. The streaming GROUP BY on a cluster_by prefix pages as well. A standalone server (a witness, the BI node) runs every SELECT this way.

Foldable grouped aggregations (COUNT/SUM/AVG/MIN/MAX, plain args) stream at any consistency: rows fold into per-group state as the gather produces them — O(groups) memory, exempt from scan_row_budget — so a GROUP BY over a window bigger than the row budget answers whether the coordinator takes the ONE local pass or the QUORUM merge. ONE is still the faster choice here (it skips the cross-replica merge entirely — see the 2050 ms vs 115 ms row above, which is coordination cost, not a memory difference), but it is not the only consistency that can answer a large grouped window. Non-foldable aggregates (PERCENTILE, COUNT(DISTINCT …)) still materialize at every consistency.

Aggregations with a search predicate (WHERE MATCH(…)) are the exception: they push into the search index (see docs/SEARCH.md, "Aggregations"). Every member reads its index under the engine READ lock; the commit a read-your-writes search needs runs before it with the lock released, and the index's own fsync (its NRT commit) always runs after a write's lock is released — the same rule as the WAL fsync of the write itself (see CLUSTER_OPERATIONS.md "Resilience"). On a full-copy cluster any member's index answers locally; on a sharded cluster (RF < members) every member aggregates only the key-space it primarily owns and the coordinator merges the partials — plain metrics as per-shard finals, AVG as SUM+COUNT, distinct counts as exact term sets: ungrouped as one set, keyword-grouped as one exact term set per group unioned by bucket key, so a value held on several shards under the same group counts once and no group's set reaches another's. The merge is exact-or-decline (every member, one membership epoch, a 65,536 bound on groups and on terms per group); anything it cannot serve exactly falls back to the deduped row gather.

Grouping on a cluster_by prefix: the streaming merge

A table declared WITH (cluster_by = (c1[, c2…])) stores and replicates its rows in (c1, c2…, primary key) order on every member, so every source of a gather — each replica, each shard, each partition child — walks presorted by that key. A foldable GROUP BY whose columns are (any order of) a leading prefix of cluster_by uses that: the coordinator merges the sources in key order and finalizes each group the moment every source has moved past its key, holding one open group instead of a group map. Result cardinality never touches the byte budget; ORDER BY a leading run of those columns (ascending) is the walk's own order and the merge stops at OFFSET + LIMIT; an ORDER BY on an aggregate re-sorts the streamed groups (a bounded top-k under a LIMIT).

Exactness across replicas is the paged gather's: at any consistency other than ONE-on-a-full-copy the rows come through the LWW-merged scan, which delivers each key once (the newest version wins) in key order across every replica and shard, so a row three replicas hold counts once. At ONE on a full copy the local engine walks its own copy (and k-way merges its partition children itself).

What keeps the ordinary hash fold: a filter served by a secondary index, a point read or a geo index (the merge needs a key-order walk — EXPLAIN's group aspect says hash fold then), an open transaction, a cluster_by adoption in flight, and — on the merged gather only — a partitioned table whose partition column is not the clustering head (its children interleave on the clustering key; ONE on a full copy still merges them locally). Putting the partition column at the head of cluster_by, the recommended shape, makes the children concatenate presorted. EXPLAIN on a coordinator adds a cluster.group line saying which route the gather takes; the skaidb_cluster_ordered_group_merges_total counter (METRICS.md) counts the statements a coordinator served by the merge.

Note consistency is a property of the session/connection, so a client that pools connections generally needs a separate pool for its analytics reads rather than a per-query flag.

Adopting cluster_by on a running cluster

ALTER TABLE t SET (cluster_by = (c1[, c2…])) re-keys an existing table online when it is a full copy on every member (its replication factor is at least the member count and it has no pins): every member holds every row before and after, so the re-key never moves a row between members. The DDL broadcasts; each member renames its copy into a legacy child and opens a new-layout companion. Until the cutover, reads serve from the legacy child (complete: every write lands there too), and every write lands in both layouts: the new one keyed by (cluster_by…, pk), the legacy one by the old key. The member that owns the table's name pages the merged legacy view at read quorum and replicates each row into the new layout with its original stamp, so a copy of an older version always loses to a write made during the adoption. When that pass completes it issues a replicated DROP of the legacy child, and each member swaps the companion into place as it applies it (a write a peer still addressed to the companion lands on the table). A range-partition adoption on a cluster runs the same way, copying rows into their range children. A sharded table refuses the ALTER — its new key would hash to other members — so declare cluster_by at CREATE TABLE there. The same limits as the standalone adoption apply (no indexes, streams, FOREIGN KEYs, partition_by, distribute_by, TOAST or memory on the table while it adopts; recreate indexes after), and a transaction (BEGIN ATOMIC PARTITION) or keyset paging (AFTER) on the table is refused until the drain finishes.

Linearizable updates (per-key consensus)

Plain writes resolve by last-writer-wins, with a deterministic tie rule: an exact cross-node HLC stamp collision resolves the same way on every replica, never by arrival order. Clustered UPDATEs and INSERT … ON CONFLICT upserts additionally serialize per row through a per-key consensus round (single-decree Paxos among the row's replicas, a majority of them must answer), so concurrent read-modify-writes of one row never lose updates, at every consistency level. A round that gives up under contention says whether a retry is safe: CAS contention … (not applied; safe to retry …) only when every replica of the row has confirmed the statement's update can never commit, and CAS outcome unknown … (may have been applied; do not retry blindly) otherwise. A statement recognizes its own landed update by a bounded history of recent commit origins carried in the row (the last 1,024 commits, none older than ten minutes — about 30 KB on a key written that hot; every commit and consensus message of the row carries it, which is why it is bounded). A statement stalled behind more than that many rival commits cannot tell a pruned commit of its own from none and answers CAS outcome unknown … (the row's commit history no longer covers this statement's proposals). While a replica of the row is down that proof is out of reach, so a statement that spends its proposal budget there resolves for up to ~8 s (64 backoff rounds, holding one of the node's CAS slots) and then answers "outcome unknown". The statement deadline, KILL and a cancelled client (iwm.disconnect_cancellation) are checked between rounds and bound the whole statement, backoff sleeps included: a statement that has not yet proposed anything (still queued for its turn on the key, still waiting for a CAS slot, or between its first prepares) answers resource limit: statement timeout … (not applied; safe to retry …); one with a proposal in flight resolves it first and answers CAS outcome unknown … (resolution interrupted: …). At consistency ALL a round fails before proposing — not applied; safe to retry; last: promises k/n replicas (consistency ALL needs every replica) — when a replica does not promise (three prepare rounds, then the verdict); a commit that reached a quorum but not every replica answers CAS applied; replicated to k/n replicas (ALL not met) — the update IS committed and reaches the rest by hinted handoff; do not retry it. The batched rounds of a multi-row statement give each row the same two verdicts (a row short of a promise proposes nothing; a commit batch one replica did not take is CAS applied; replicated to k/n … for its rows). A multi-row statement answers "not applied" only when none of its rows committed; if some did, the error is CAS partial outcome: N of M rows … (partially applied; do not retry blindly).

Hot keys

Every concurrent read-modify-write of one row is its own consensus round, so the writers of a single row serialize behind each other and their throughput is bounded by round-trip time, not CPU: hundreds of committed increments per second on a loopback cluster, tens per second across a container network (an uncontended round is three to four durable exchanges, so on a disk with a 100 ms fsync it is ~1 s). Contenders that arrive through the same coordinator take turns on a per-key FIFO queue there (one round of one statement per key per coordinator at a time, waiting statements hold no CAS slot), so a key written by a hundred clients is contended three ways on a three-member cluster, not a hundred ways; a queue already 64 statements deep refuses the next one at once with CAS contention: N statements already queued for this key on this node (not applied; safe to retry …), and a statement whose deadline passes while queued answers resource limit: statement timeout … (not applied; safe to retry …) without ever proposing. Between rounds a round backs off exponentially with full jitter (up to ~1 s), never past the statement deadline. Every extra contender still adds queueing and retries (the skaidb_cluster_cas_*_total counters show the rounds and retries per commit). Design the hot spot out rather than tune it: spread a counter over N rows (id = 'visits:' + <shard 0..N-1 chosen by the client>) and SUM them on read; batch on the client (SET n = n + 5 after five local increments); or append one event row per increment — an INSERT of a fresh key is a plain last-writer-wins write with no round — and aggregate, which CREATE ROLLUP … AGGREGATE keeps current for you. The retry contract above is unchanged: "not applied; safe to retry" is exact, and "outcome unknown" is reconciled by a re-read.

Batched rounds

An UPDATE or upsert touching many rows runs the rounds of rows that share a replica set batched — up to 128 rows per batch, one prepare, one propose and one commit exchange per step, each costing every replica one WAL sync for the whole batch rather than one per row (on a cluster where every node holds every row, that is every row of the statement). A batch moves each row's current version from one replica only (the others report its version stamp), and a replica keeps a proposed row value in its consensus state only until its own copy of the row carries it. With cluster.cas_paxos_compress = true (live, default off) it keeps that value LZ4-compressed; a version without the compressed layout forgets such state, so turn it on only once no member will be rolled back to one, and turn it off (then let UPDATE traffic settle) before any rollback. Each row keeps its own consensus state: a row whose round meets contention continues alone without holding up the rest, rows too large to batch (a batch carries about 2 MB of row data) take the per-row rounds, and while any replica runs a version without the batched exchange (a rolling upgrade) every row takes the per-row rounds. At most 16 rounds — a whole batch counts as one — run at once per coordinator across all its multi-row statements; single-row statements have 64 slots of their own, so they never wait behind the multi-row ones (skaidb_cluster_cas_running{kind}, skaidb_cluster_cas_permit_waits_total). A row that fails (a CHECK on the new value, say) fails the statement and no further batch starts, but the statement is not atomic: other rows of the failing row's own batch may still commit. So batch many rows into one statement (WHERE id IN (…) with every primary-key column pinned is a point-write set, and so is a multi-row INSERT … ON CONFLICT whose rows each name a different key — a statement repeating a key runs its rows one after another, so each sees the one before) rather than issuing them one by one; the skaidb_cluster_cas_batch_* and skaidb_cluster_cas_paxos_* metrics show the batching at work (METRICS.md) — see the UPDATE semantics under Statements in QUERY_SYNTAX.md. There is no special "primary" to configure.

Write errors and what a retry means

A write error names what happened to the data, because the two families behave differently:

  • read quorum not met: R/N members responded (<addr>: <cause>; …) — a read (or the read leg of a DELETE … WHERE) could not gather its quorum. Nothing was written; retrying is always safe.
  • write quorum not met: k/n acks — applied on k of N replicas and not rolled back; the rest receive it by hinted handoff; retry only an idempotent statement (…) [<addr>: <cause>; …] — a blind write (INSERT, upsert, DELETE, a multi-row VALUES batch, a time-series append) reached fewer replicas than its consistency level requires. The write is applied on the k replicas that acknowledged it (the coordinator's own copy included) and is not rolled back: the coordinator hints it to the replicas it missed, and hinted handoff and anti-entropy deliver it when they are reachable. The replicas named in brackets are the ones that answered with a refusal (a shed, a disk guard, a clock-skew rejection); the others did not answer. A client that retries must therefore treat the first attempt as possibly applied (the info outcome of a Jepsen-style history). With no ack at all the text says so instead of "applied on 0": write quorum not met: 0/n acks — no replica confirmed it, but it is hinted to the replicas that missed it and may still be applied (outcome unknown …) — a replica may have applied it without its ack arriving, and the hints deliver it later. There is no idempotency key, so:
statement retry after write quorum not met
INSERT … VALUES (k, …) (a plain insert is an upsert) safe — the same row again
INSERT … ON CONFLICT DO UPDATE SET c = v (absolute value) safe
DELETE FROM t WHERE pk = k safe
UPDATE t SET c = v WHERE pk = k (absolute value) safe — but note it is a consensus update, see below
UPDATE t SET n = n + 1 …, … ON CONFLICT DO UPDATE SET n = n + 1 not safe — applies twice if the first attempt landed
time-series INSERT of samples safe — a sample is identified by series and timestamp; the later copy wins

A write quorum not met on a minority side of a partition is the case to think about: the write is on the isolated coordinator, it replays onto the majority when the partition heals, and under last-writer-wins it then outranks an acknowledged majority-side write with an older stamp. Clients that need a write to be either fully acknowledged or absent use the consensus path (UPDATE / INSERT … ON CONFLICT) at the consistency level they need. - Consensus updates (UPDATE, INSERT … ON CONFLICT): every error carries its verdict — (not applied; safe to retry …), CAS outcome unknown … (may have been applied; do not retry blindly), CAS partial outcome … (partially applied; do not retry blindly) or CAS applied; replicated to k/n replicas (ALL not met) … do not retry — including a statement ended by its deadline or KILL (see the contract at the top of this guide). The resource limit: statement timeout text keeps its prefix and gains the verdict. - clock skew: … (not applied; safe to retry) — this coordinator's clock is beyond the cluster's skew bound and it refuses to coordinate writes; nothing was applied. See Clock synchronisation. - write superseded: the row's current version is stamped N ms in the future … — a blind write reached its replicas, and one of them holds a version of the row stamped further ahead than the clock-skew bound allows (written while some member's clock was wrong). The write is stored at its own stamp and not rolled back, but that version outranks it on every read, and a retry changes nothing until the replica holding the version is resynced or wall time passes its stamp (see "A member whose clock was wrong" below). A multi-row statement says how many of its rows are affected; the others are written normally.

Write ordering across coordinators

Blind writes — INSERT (an upsert), DELETE, multi-row VALUES batches — are last-writer-wins by stamp, and a statement's stamp comes from the clock of the member that coordinates it. A member's clock is ahead of every write it has applied, not of every write the cluster has acknowledged: a QUORUM write taken by one member reaches the last replica a moment later, and until then that replica's clock can sit below the write's stamp (two clocks inside one millisecond order by their logical counters; a millisecond of offset between hosts does the same).

What a client may rely on. A blind write issued after another write to the same key was acknowledged wins over it, whichever members coordinated the two, when both ran at QUORUM (or at any two levels whose acknowledging replica sets must overlap, W₁ + W₂ > RF). INSERT, DELETE, INSERT sent back to back through three different members leaves the row of the last INSERT. A driver that spreads statements over its seed list, or fails over between two statements, needs no key affinity for its own writes to stay in order.

How. A replica that takes a client write synchronously answers with the stamp of the version it already holds whenever the incoming write does not supersede it — a newer stamp, row or tombstone, or an equal stamp that wins the deterministic tie — and the coordinator checks its own replica the same way. When any of them reports, the coordinator stamps the write again above the newest reported stamp and replicates it once more; the acknowledgement the client gets is that second round's. Two write quorums share a replica, so one of the replicas that answer holds the earlier write. The first version stays in the replicas' logs below the version it lost to, as history. A multi-row statement is treated per row: only the rows a replica reported are stamped again, in statement order. A row's companion writes — its value-TOAST blobs, its global-index entries, the legacy leg of a cluster_by or partition adoption — follow the same rule, so an index entry never stays behind its row. skaidb_cluster_write_restamps_total counts the re-stamps; each costs that write one extra replication round and nothing else.

What it does not cover.

  • Writes at ONE. The acknowledgement vouches for one replica; a write through another member right after it can be stamped below it and lose. The window is the first write's replication tail (milliseconds), and read-your-writes at ONE holds on the member that took the write.
  • Concurrent writes — neither acknowledged before the other was issued. They land in stamp order, and either order is a correct outcome. The coordinator stamps a write again once; a version that then appears above the second stamp is a concurrent writer's.
  • Rows written by a transaction's COMMIT: they carry the transaction's commit stamp. Consensus updates (UPDATE, INSERT … ON CONFLICT) never needed it — they are linearizable per row.
  • Hinted handoff, read-repair, anti-entropy and migration deliver each version at the stamp it was written with.
  • A rolling upgrade: a replica on a version without the report answers a plain acknowledgement, and the write keeps its first stamp. The guarantee holds once every member runs a version that has it.
  • A reported version stamped further in the future than the clock-skew bound allows is not outranked — the coordinator's clock must not follow it. The statement fails with write superseded: … (above).

Clock synchronisation

Blind writes are last-writer-wins on the coordinator's clock: each node mints a hybrid logical clock stamp from its wall clock (advanced by every stamp it applies or reads, never moved backwards), and the newest stamp wins on every replica. Two concurrent writes to one key through two coordinators therefore order by the coordinators' clocks, and so do two sequential ones at ONE when they are closer together in time than the clocks are apart — there a member whose clock is 5 s fast silently wins against writes made 4 s after its own, and one whose clock is 5 s slow silently loses. (Sequential writes at QUORUM are kept in order by the replicas' newer-stamp report — Write ordering across coordinators — but only up to the bound below: a version stamped further ahead than that is never outranked.) Worse, every member that applies or reads a stamp from a far-future clock would carry it forward in its own stamps until wall time caught up. Run NTP or chrony on every member (containers inherit the host clock; keep the hosts synchronised) and leave the bound below in place.

The bound. cluster.max_clock_skew_ms (default 500, live-mutable, 0 disables every check) is the largest wall-clock difference tolerated between any two members. Every member measures every peer's offset NTP-style every 2 s with a dedicated clock request: the peer answers with its wall clock (the reading is exact to half the round trip) from its connection thread alone — no engine lock, no file, no subprocess — on a pooled connection, so the probe costs a member nothing however busy it is, and it leaves the peer's circuit breaker alone. A peer on a release without the clock request is asked its host statistics instead, which carry the same clock. Each member reports the offsets as skaidb_cluster_peer_clock_offset_ms{peer} and in /status peers[].clock_offset_ms, and keeps a verdict in /status clock_skew (offset_ms — this node against the median of its measured peers, max_abs_offset_ms, measured_peers, peers_beyond, skewed, writes_refused, hlc_ahead_ms, hlc_stored_ahead_ms, hlc_rewinds, limit_ms, action). A member is skewed when it is beyond the bound against more than half of the peers it has a reading for, on two consecutive probes (one clean probe clears it); skaidb_cluster_clock_skewed is 1 meanwhile, an error line repeats once a minute (clock skew: this node's clock is N ms ahead of/behind the cluster …), and /ready?strict=1 answers 503 clock skew: …. A member with no measured peer — standalone, or every peer down, wedged or running a binary that reports no clock — has no verdict and refuses nothing. With two members (or one measured peer) both sides are beyond the bound against "a majority" and both act; a third member resolves the tie.

Beyond it. With cluster.clock_skew_action = "refuse_writes" (the default) a skewed member refuses to coordinate client writes — blind writes, time-series appends, consensus updates and DDL — with clock skew: this node's clock is N ms ahead of/behind the cluster; writes refused until it is within cluster.max_clock_skew_ms (NTP/chrony) (not applied; safe to retry) (skaidb_cluster_clock_skew_write_refusals_total); plain /ready answers 503 as it does for a node shedding writes, so a load balancer routes around it. It keeps serving reads and applying replication from its peers. With "warn" it only logs, counts and flags strict readiness. Independently of the action, every member rejects a stamp that arrives more than the bound (+100 ms) ahead of its own clock — a replicated or hinted write, a repair row, a DDL version, a consensus ballot or commit stamp — with clock skew: … stamped N ms ahead of this node's clock rejected (skaidb_cluster_clock_skew_rejections_total, one log line a minute), so a skewed coordinator's write fails its quorum loudly (write quorum not met … [<peer>: clock skew: …]) instead of the cluster adopting its clock; a repair page or hint batch carrying one such row still delivers the rest. A member's node clock likewise never merges a stamp beyond the bound from a read winner or a consensus floor (skaidb_cluster_clock_future_stamps_ignored_total). The two-member case and a member whose clock is slow deserve a note: a slow member rejects its peers' honest stamps as "future", so it stops taking replication (hints accumulate for it on the coordinators, within cluster.hint_max_disk_mb) and its own writes are refused — it is a read-only, stale replica until its clock is fixed, after which hint replay and anti-entropy catch it up; QUORUM reads through it stay correct because they gather from the healthy replicas.

A member whose clock was wrong. Fix the clock first (NTP); the verdict clears on the next probe and writes resume. Two things may remain. (1) Its node clock sits where the skew left it — it minted stamps at the wrong time, and an HLC does not run backwards by itself — so for as long as that lead exceeds the bound every client write through the member is refused with clock skew: this node's clock (HLC) is N ms ahead of its wall clock … (not applied; safe to retry). The state is reported like the skew itself: /ready answers 503 in both modes (clock skew: writes refused (this node's clock (HLC) is ahead of its wall clock; /status clock_skew.hlc_ahead_ms)), /status clock_skew shows writes_refused: true with hlc_ahead_ms (the lead) and hlc_stored_ahead_ms, skaidb_cluster_clock_writes_refused is 1 and skaidb_cluster_clock_hlc_ahead_ms carries the lead, and an error line once a minute says for how long writes stay refused. What happens next depends on what the stamps were used for. If nothing the member stored carries one — it minted them for reads or for consensus ballots its peers rejected, or its writes were refused before they were applied — the lead exists in memory only (hlc_stored_ahead_ms 0): on the second clock probe in a row that finds the member's wall clock within the bound (a few seconds) it lowers its node clock to the wall clock, or to just past the newest stamp it did store, logs … lowered to the wall clock; client writes are accepted again (skaidb_cluster_clock_hlc_rewinds_total), and writes resume without a restart. If it stored rows at those stamps (hlc_stored_ahead_ms > 0; see (2)), lowering the clock would mint stamps below rows it holds, so it keeps refusing until wall time passes them — the size of the jump. A restart with a synchronised clock ends the refusal at once in either case (the node clock lives in memory and is seeded from the wall clock at start) and leaves the rows of (2) where they are. (2) Rows it coordinated while skewed — before the verdict took effect (at most two probe intervals) or with the action set to warn — sit on its own replica stamped in the future (its peers refused them; the clients were told write quorum not met … applied on 1 of N). Such a row wins every read of its key, is invisible to AS OF now() and to anti-entropy (which compares as of a recent instant) until wall time passes its stamp, and will then overwrite any honest write to that key made in between. To discard them, wipe and resync the member (stop it, empty its data directory, start it; it rebuilds from its peers — the same procedure as replacing a node) before wall time reaches the stamps; a consensus update of such a key answers CAS refused: the row's current version is stamped N ms in the future … (not applied; safe to retry) until then, and a blind write whose synchronous replicas include the one holding the row answers write superseded: … (the write is stored, behind that version). A key whose consensus state on some replica holds a promise to a far-future ballot refuses honest ballots from that replica until wall time passes it; a quorum of the other replicas still serves the key.

What the bound does not do. Within the bound, last-writer-wins still orders concurrent writes by clock: two writes to one key through different coordinators, neither acknowledged before the other was issued, can land in either order. Writes that are sequential as a client sees them (the second issued after the first was acknowledged) order correctly at QUORUM whichever coordinators take them — a replica holding the earlier write reports its stamp and the coordinator stamps above it (Write ordering across coordinators); at ONE the window is the replication tail of the first write (milliseconds). For keys written by several clients at once use the consensus path — UPDATE and INSERT … ON CONFLICT are linearizable per row and do not depend on the clock for correctness — or route each key's writer to one coordinator; read-your-writes holds on the coordinator that took the write and, at QUORUM/QUORUM, on every member.