Setting up a cluster

Part of Running a skaidb cluster.

What turns a node into a cluster member, which ports it needs, how to form a three-node cluster on three machines or one, and how to verify the ring once it is up.

How a node decides it's clustered

seeds is empty (the default) → the node runs standalone: a single local engine, no internode networking.
seeds is non-empty → the node runs as a cluster member. The seed list is the full membership: every entry is a member's internode address host:internode_port, and the list must include this node itself.

A node's identity on the ring is bind_addr:internode_port — or advertise_addr:internode_port when cluster.advertise_addr is set. For the node to be part of the ring, that exact string must appear in seeds, and it is the address other nodes use to reach this one: a routable IP or a name every peer resolves. Set advertise_addr to that name and bind_addr to 0.0.0.0 when the node should listen on every interface, or when its interface address can change while its name cannot (a container re-attached to a network keeps its service name and loses its IP). Without advertise_addr, bind_addr must not be 0.0.0.0. All nodes should be given the same seed list.

Ports

Each node listens on three ports, all on bind_addr:

Purpose	Config	Default	Who connects
Client binary (fast path)	`server.quic_port`	`7000`	applications / drivers
Client REST	`server.rest_port`	`7080`	`curl`, HTTP clients
Internode RPC	`cluster.internode_port`	`7100`	other skaidb nodes
Prometheus metrics	`observability.prometheus_port`	`9090`	scrapers

The internode port must be reachable between every pair of nodes — open it in your firewall/security group across the cluster. Client ports only need to be reachable by clients.

Form a cluster (static seed list)

The supported way to stand up a cluster is to give every node the same seeds list at startup. Each node must also be told bind_addr (or advertise_addr) + internode_port so its own identity (bind_addr:internode_port, or advertise_addr:internode_port) matches its entry in seeds, and all nodes must share the same replication_factor.

Three nodes on three machines

Machines 10.0.0.1, 10.0.0.2, 10.0.0.3, internode port 7100 on each, RF 3:

# On 10.0.0.1
skaidb --data-dir /var/lib/skaidb \
  --bind-addr 10.0.0.1 --quic-port 7000 --rest-port 7080 \
  --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

# On 10.0.0.2 — identical, but --bind-addr 10.0.0.2
skaidb --data-dir /var/lib/skaidb --bind-addr 10.0.0.2 \
  --quic-port 7000 --rest-port 7080 --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

# On 10.0.0.3 — identical, but --bind-addr 10.0.0.3
skaidb --data-dir /var/lib/skaidb --bind-addr 10.0.0.3 \
  --quic-port 7000 --rest-port 7080 --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

Only --bind-addr differs between nodes; the --seeds list is identical everywhere. Start order doesn't matter — a node tolerates peers that aren't up yet (writes/reads just need their quorum).

Three nodes on one machine (local test)

Same idea with 127.0.0.1 and distinct ports + data dirs per node:

SEEDS=127.0.0.1:7100,127.0.0.1:7101,127.0.0.1:7102

skaidb --data-dir ./n1 --bind-addr 127.0.0.1 --quic-port 7000 --rest-port 7080 \
  --internode-port 7100 --seeds $SEEDS --replication-factor 3 &
skaidb --data-dir ./n2 --bind-addr 127.0.0.1 --quic-port 7001 --rest-port 7081 \
  --internode-port 7101 --seeds $SEEDS --replication-factor 3 &
skaidb --data-dir ./n3 --bind-addr 127.0.0.1 --quic-port 7002 --rest-port 7082 \
  --internode-port 7102 --seeds $SEEDS --replication-factor 3 &

Each node's 127.0.0.1:<internode_port> matches one entry in $SEEDS.

Config-file and env-var equivalents

Every flag is also a TOML key and an environment variable. A per-node config file (skaidb --config /etc/skaidb.toml):

[server]
bind_addr = "10.0.0.1"      # this node's reachable address
quic_port = 7000
rest_port = 7080
data_dir  = "/var/lib/skaidb"

[cluster]
seeds = ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.3:7100"]
internode_port = 7100
# advertise_addr = "db1.internal"   # ring identity when bind_addr is 0.0.0.0
replication_factor = 3
vnodes_per_node = 256
default_read_consistency = "QUORUM"
default_write_consistency = "QUORUM"
# max_clock_skew_ms = 500          # largest clock difference tolerated between members
# clock_skew_action = "refuse_writes"   # or "warn"; see CLUSTER_CONSISTENCY.md "Clock synchronisation"
# index_count_check = "log"        # or "enforce" / "off"; see CLUSTER_CONSISTENCY.md "Index-only counts"

Or env vars (handy for containers; CLI flags override these, which override the file):

export SKAIDB_BIND_ADDR=10.0.0.1   # or 0.0.0.0 together with SKAIDB_ADVERTISE_ADDR=db1.internal
export SKAIDB_INTERNODE_PORT=7100
export SKAIDB_SEEDS=10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100
export SKAIDB_REPLICATION_FACTOR=3
export SKAIDB_DEFAULT_READ_CONSISTENCY=QUORUM
export SKAIDB_DEFAULT_WRITE_CONSISTENCY=QUORUM
skaidb --data-dir /var/lib/skaidb

Confirm the resolved settings on any node with skaidb --print-config.

Verify the cluster

Write through one node and read it back through another — leaderless means any node accepts both:

# Create + insert via node 1 (REST on :7080)
curl -X POST 10.0.0.1:7080/query -d "CREATE TABLE users (PRIMARY KEY (id))"
curl -X POST 10.0.0.1:7080/query -d "INSERT INTO users (id, name) VALUES (1, 'ada')"

# Read via node 2 — sees the replicated row
curl -X POST 10.0.0.2:7080/query -d '{"sql":"SELECT * FROM users WHERE id = 1"}'

# Per-node metrics
curl 10.0.0.3:7080/metrics

Each node's startup log prints its endpoints (binary endpoint listening on …, REST endpoint listening on …).