Small clusters: one, two and three members

Part of Running a skaidb cluster.

What each member count buys you, what it costs when a member is down, and the settings that make a two-member cluster behave the way its operator expects. Three members is the smallest configuration in which no statement fails while one member restarts; this page is about what happens below that.

The arithmetic

Everything on this page follows from three rules.

• Copies. Each key lives on replication_factor members, capped at the member count. Two members with replication_factor = 3 hold two copies of every row; every member is a full copy.
• Majorities. QUORUM means floor(n / 2) + 1 of n. For a statement's reads and writes at QUORUM/ALL, n is the key's replica count. For the per-row consensus behind UPDATE, INSERT … ON CONFLICT, UNIQUE claims and BEGIN ATOMIC commits, n is the key's replica count as well. For DDL, n is the member count. With two members every one of those majorities is 2 of 2: both members must answer.
• Who counts. Only ring members count. A witness mirrors data but holds no vote and is never a replica; clients hold no vote either.

One member

A node with an empty seeds list runs standalone: no ring, no replication, no quorum, and every statement including transactions has the single-node semantics described in QUERY_SYNTAX.md. It is the right shape for development, for an embedded deployment, and for a witness. Its availability is the machine's.

Two members

What you gain over one

• A live second copy of every row, on another machine, with a continuously converging mirror that needs no operator: hinted handoff for the writes a member misses, read-repair, and anti-entropy passes for everything else.
• Reads and plain writes ride through a member outage. At consistency ONE a SELECT, INSERT or DELETE is answered by the member a client can reach; the other member catches up from hints when it returns. The drivers fail over between the hosts in their list on their own.
• Half the hardware of three, and every member is a full copy, so the local fast paths apply: unfiltered counts and ONE reads of a full-copy table are answered from the local engine without a cluster gather (CLUSTER_CONSISTENCY.md).

What a member outage costs

While one of the two members is down, unreachable, or restarting:

• QUORUM and ALL reads and writes fail. Both replicas are the quorum. Run the defaults at ONE (below) or every statement stops with read quorum not met / write quorum not met.
• Every consensus statement fails, at any consistency level. UPDATE and INSERT … ON CONFLICT on a replicated row answer CAS contention … not applied; safe to retry; last: promises 1/2 of 2 replicas after their prepare rounds; an INSERT or UPDATE into a table with a UNIQUE index fails its claim the same way; BEGIN ATOMIC commits fail; DDL answers DDL quorum not met: 1/2 members applied. These are the statements that need a second durable vote to be safe, and a two-member cluster has no one to give it. The exceptions are UPDATEs that already take the plain read-merge-put path (tables with TOAST, a global index, a foreign key or partitions); they keep working at ONE with last-writer-wins semantics for the duration.
• Every rolling restart is a short outage for those statements. With three members the restart of one is invisible; with two it is a window in which updates, unique inserts, transactions and DDL return retryable errors. Applications must treat not applied; safe to retry as exactly that, and schedule restarts accordingly.
• No second failure is absorbed. Losing the other member leaves no replica at all.
• Membership changes need both. Every ring change is decided by a majority of the current members, which with two is both; a graceful REMOVE NODE also drains through the leaving member. A dead member can only be dropped with FORCE, decided by the survivor alone (CLUSTER_MEMBERSHIP.md); the survivor then holds the only copy until a new member joins and resyncs. Never issue a FORCE on both sides of a partition: that forks the cluster.

Settings for two members

Setting	Value	Why
cluster.default_read_consistency / cluster.default_write_consistency	"ONE"	The shipped default is QUORUM, which with two members means both. At ONE reads and plain writes survive a member outage; UPDATEs stay linearizable regardless.
cluster.hint_max_disk_mb, cluster.hint_max_age_secs	sized to the write rate times the longest outage you want to ride through	Hints the survivor cannot keep fall through to anti-entropy, which converges the member later but slower (defaults 1024 MB and one day; both live-mutable).
mqtt.state_consistency	"one"	The broker's session and subscription state otherwise needs both members for every change (MQTT.md).
replication_factor	3 on both	It is capped to the member count, so two members hold two copies and a third member joining later restores three without a config change.

A rolling restart of two members: restart one, wait for /ready?strict=1 and for its peer's hints_pending to reach 0, then the other, exactly as for three (CLUSTER_OPERATIONS.md), accepting that the consensus statements error during each half.

Three members

A majority is 2 of 3, so one member down costs nothing: every statement keeps working at QUORUM, including UPDATEs, unique claims, transactions and DDL; the returning member catches up from hints, and a rolling restart or upgrade is invisible to clients. replication_factor = 3 makes every member a full copy, so the local fast paths still apply. Reads at QUORUM plus writes at QUORUM are strongly consistent (R + W > N). Losing a second member drops to the two-member behaviour above until it returns. This is the smallest cluster to run anything that must not see maintenance windows, and the shape most of this guide assumes.

More than three

Beyond three members the replication factor stays at the configured value, so members stop being full copies: a key lives on replication_factor of them and reads gather from a subset. Member outages are absorbed as long as each key keeps a majority of its replicas, placement is per table (CLUSTER_PLACEMENT.md), and value-sharded indexes (GLOBAL_INDEXES.md) replace the local index scans a full copy allowed.

Choosing

Members	A member restart	Needs an operator for	Hardware
1	downtime	everything	one machine
2	reads and plain writes continue at ONE; updates, unique inserts, transactions and DDL error until it is back	dropping a dead member (FORCE) and re-adding one	two machines
3	invisible	nothing routine	three machines

If the choice is between two members and three small ones, choose three. A third member needs no more than a few cores and the memory for its share of the data (INSTALL.md), and it is the difference between maintenance being a window and maintenance being nothing.