Adding, removing and replacing members

Part of Running a skaidb cluster.

Joining and decommissioning nodes while the cluster serves traffic, what configured versus actual membership means, how a failed join rolls back, forced removal, and the wipe-resync path for a member whose data is gone.

Add or remove nodes at runtime (online resharding)

The ring can change while serving traffic — a node can join (and receive its share of the keyspace) or be gracefully decommissioned (drain its keys first). The full mechanics — pending-ranges dual-write during a join, single-sender migration, epoch'd membership, throttling/resume — are in RESHARDING.md.

Drive them with skaidbsh, the unified shell/admin client (shipped alongside skaidb). It talks to any node's REST endpoint over an authenticated POST /admin/* control plane (RBAC: the role needs Admin on the whole cluster; membership changes are serialized server-side, one at a time):

# Point it at any node (REST port defaults to 7080, override with --rest-port);
# add --user/--password if the server requires auth.
skaidbsh --host 10.0.0.1 cluster status            # show the ring, epoch, members, RF

skaidbsh --host 10.0.0.1 cluster add-node 10.0.0.4:7100    # join: migrates its share in
skaidbsh --host 10.0.0.1 cluster remove-node 10.0.0.3:7100 # decommission: drains, then leaves
skaidbsh --host 10.0.0.1 cluster remove-node 10.0.0.3:7100 --force  # a node that is down or
                                                           # wedged: drop it WITHOUT draining
skaidbsh --host 10.0.0.1 cluster repair            # anti-entropy: converge all replicas
skaidbsh --host 10.0.0.1 cluster reclaim           # free space former owners no longer own

status prints JSON like:

{ "clustered": true, "node_id": "10.0.0.1:7100", "epoch": 3,
  "replication_factor": 3,
  "configured": ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.3:7100"],
  "self_in_ring": true,
  "members": ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.4:7100"],
  "peers": [
    { "id": "10.0.0.2:7100", "in_config": true, "in_ring": true,
      "reachable": true, "hints_pending": 0, "lag_ms": 4 },
    { "id": "10.0.0.4:7100", "in_config": false, "in_ring": true,
      "reachable": true, "hints_pending": 0, "lag_ms": 7 }
  ],
  "discrepancies": {
    "configured_not_in_ring": ["10.0.0.3:7100"],
    "ring_not_configured": ["10.0.0.4:7100"]
  } }

Configured vs. actual. configured is what seeds says membership should be; members is the live ring the coordinator actually routes and replicates to. They diverge in normal operation: cluster add-node admits a node that was never in anyone's seeds (it shows up under ring_not_configured), and a seed that has not (yet) been admitted to the ring shows up under configured_not_in_ring. The latter is exactly the trap to watch for: a node started with seeds pointing at the cluster will pull data via background catch-up (it sees peers, so anti-entropy runs) without ever joining the ring — it serves stale-but-converging reads while no one routes writes to it. Such a node appears in configured_not_in_ring until it is admitted — which the node's own announce normally does on its own (see Automatic join below); the trap is left only when that announce is finally rejected (a replication-factor mismatch). A joining node's own seeds may omit itself (list only peers): it announces the identity it is reachable at, not its seed list. The tell on the node itself is self_in_ring: false, meaning it is coordinating/catching-up but owns no ring tokens. The unauthenticated GET /status carries the same configured / self_in_ring / configured_not_in_ring / ring_not_configured fields (without the per-peer liveness probe); \cluster adds reachable, hints_pending, and lag_ms per peer (see METRICS.md for the matching Prometheus gauges).

Under the hood these call the coordinator: add-node broadcasts the new ring, bootstraps the joiner's schema, and streams it the keys it now owns (dual-writing during the move so concurrent writes stay correct); remove-node opens the same dual-ring window (the smaller ring with the old one unioned in, so a write that arrives mid-drain reaches the key's new owner too), has the leaving node drain its keys to their new owners, then finalizes the smaller ring and tells the leaving node it is out. The same operations are also available as raw HTTP (POST /admin/status, /admin/add-node with {"addr":"…"}, /admin/remove-node with {"id":"…"} or {"id":"…","force":true}, /admin/repair, /admin/reclaim), as plain SQL from any client — SHOW CLUSTER, ALTER CLUSTER ADD NODE 'host:7100' / REMOVE NODE 'id' [FORCE], REPAIR CLUSTER, RECLAIM, plus SHOW CONFIG [LIKE] / SET CONFIG and SHOW SLOW QUERIES (identical RBAC and audit as the HTTP endpoints) — and as skaidb_cluster::Node library methods.

A long migration keeps the skaidbsh request open until it finishes. The push rate is tuned per node with SET CONFIG cluster.bootstrap_duty_pct = '<1..90>' (live, on the members that serve the push — the ceiling on the share of their capacity a bootstrap may take; default 50). The batch size and pause behind it (set_migration_throttle in RESHARDING.md) are the engine's internals, not an operator setting. Run one membership change at a time.

A join that fails is rolled back. If the joiner cannot be reached, its schema bootstrap fails, or a member's push fails, every member goes back to the previous ring at a new epoch and the joiner is told it is not a member — nothing stays half-open, and the next add-node (or the joiner's own re-announce, which retries on its own) starts clean. Nothing is lost by this: migration copies rows and never removes them from their old owners, and writes made during the window went to both rings. A "busy" bootstrap answer (the joiner's engine was write-locked — typically by its own catch-up) is retried with backoff for about 16 s before the join counts as failed. A migration batch the joiner does not ack is re-sent with backoff (1, 2, 4, 8, 15, 30 s, then once a minute — about five minutes in all) before the join fails: batches are idempotent and checkpointed, and a joiner on a slow disk misses an ack now and then without being broken. When the budget is spent the rollback says so (rebalance to <joiner>: batch of N rows not acked after K attempts over S s — join rolled back (epoch E)). The coordinator waits for every member's push to finish — for as long as the data takes — before it finalizes, so the joiner is a settled replica only once it holds its rows. The one window that can still be left open is a coordinator that crashes mid-join; a surviving member finishes or rolls it back after ten idle minutes, re-adding the same node finalizes its own stuck join at once (a re-add while a different membership change is open is refused with HTTP 400 "another membership change is in progress" — retry later), and remove-node --force closes it by dropping the node.

A node that cannot be drained — down, or wedged so that it accepts connections and never answers — refuses a graceful remove-node ("leaving node unreachable"); the drain is what keeps every key at full replication. remove-node --force (REMOVE NODE 'id' FORCE, {"force":true}) drops it from the ring without draining: the survivors keep the copies they already hold (each key had replication_factor owners, so at most one copy is lost) and anti-entropy re-replicates the keys to their new owners — run cluster repair to do that immediately. A forced removal also goes through while a graceful removal of the same node is still parked on its drain: that one is told to give up (it returns "aborted by a forced removal") and the forced one proceeds. Tables that pin the node still refuse; re-pin first.

Every ring change is decided by a majority of the members it starts from. A begin, finalize, rollback, heal or forced removal is a proposal on the current epoch, accepted by a majority of the current ring (the node being added or removed does not vote; a FORCE removal is decided by the survivors); each member persists what it promised and accepted in topology.vote before answering, and installs the new ring only once it was chosen. So there is at most one ring per epoch cluster-wide: a committed join and a reported FORCE removal are never undone by a later heal, and a member isolated through a change cannot roll it back — it adopts the committed ring when it reconnects. The liveness rule follows: a change needs a majority of the members reachable (both of them in a two-member cluster); a window whose peer is gone stays open until the peer returns or the operator forces the node out. Two FORCEs issued on opposite sides of a partition fork the cluster — re-add one side. A member that rejected a ring as stale says so (MembershipStale) rather than acknowledging it, and a redelivered ring it already holds is acknowledged once more. During a rolling upgrade, members on an older release that do not know the vote count as stateless votes only when the members that do know form a majority on their own, so a roll cannot split the ring; avoid membership changes mid-roll regardless. Leftover topology.tmp.* files from a crash mid-write are swept at start.

A graceful removal is a two-phase change, like a join. Before the drain starts, every member — the leaving node included — adopts the smaller ring with the old ring unioned in, so a write that arrives while the drain runs is written to the key's new owner as well as its old ones and a read consults both; once the drain completes, the smaller ring is finalized. /admin/resharding therefore reports active:true for the length of a removal too, and ONE reads on the survivors see every write acked before remove-node returned, with no anti-entropy pass in between. A drain that fails rolls the removal back (everyone returns to the old ring at a new epoch; the error is the drain's), and a removal whose coordinator died mid-drain is rolled back by a surviving member after ten idle minutes — the leaving node still holds every key it owned, so nothing is lost; run remove-node again. Removing an id that is not a member is accepted and says so: the response carries removed: false and not_a_member: <id> instead of removed: <id>.

Replacing a member (wipe-resync) and the resync state

To rebuild a member on a fresh disk, or to move it onto an encrypted or compressed engine, stop it, wipe its data directory, and start it again with the same configuration; it backfills everything from its peers. What makes this safe against live reads is the resync state: a node whose store is empty when it starts on a multi-member ring is flagged resyncing before it serves its first statement — from its own evidence, with no peer asked yet — and stays flagged until its startup catch-up has completed a full anti-entropy pass and the deferred index maintenance that pass queued has drained (rows a bulk apply lands reach the table before its indexes; the flag waits for both, up to ten minutes, and a covering-index count declines to the row walk on a resyncing member). While flagged it never answers a count, scan, aggregate, distinct, ordered read or local search from its own (incomplete) copy at any consistency level, ONE included: those paths gather from the complete peers instead, so a client connected to the rebuilding member gets whole answers throughout. The flag shows in GET /status (resyncing, resync_progress, resync_source), in every member's resyncing_endpoints (drivers and skaidbsh route around it), in the UI as resync, and /ready?strict=1 answers 503 for it (so a rolling operation waits it out); the log says resync detected — empty data directory on a N-member ring at start, resync confirmed — backfilling from … once a peer's host stats show it holds what this node lacks (real data and more than twice this node's footprint, or more than twice its live keys plus a small margin), and resync complete — serving normally when the catch-up lands. The presumption is settled by a probe round at start and then every second: a node that starts empty beside peers that are also empty (a new cluster, possibly with a member down) has its flag withdrawn as soon as at least half its peers have answered and none holds more than it — within a second or two of starting. A peer whose engine is busy gives no reading and decides nothing; a round with no reading leaves the flag standing. A node that restarted part-way through a backfill (its store is not empty but holds less than half a peer's data) is detected by the periodic footprint probe the same way; an unreadable local reading there counts as unknown, never as zero.