Securing a cluster

Part of Running a skaidb cluster.

Internode authentication (shared token or mutual TLS) and how to rotate certificates without a flag day, client TLS, at-rest encryption, what it covers and what it does not, and the KEK rotation runbook.

Internode security

By default internode traffic is unauthenticated (internode_auth = "none") — fine on an isolated/trusted network, but anything that can reach a node's internode port can read data and change membership. Two modes lock it down; every node must use the same mode and material:

Token — a shared secret. Peers prove knowledge of it with a mutual HMAC-SHA256 challenge-response, so the secret never crosses the wire and each connection uses a fresh nonce (no replay). No encryption.

[auth]
internode_auth = "token"
internode_token = "a-long-random-shared-secret"      # or:
# internode_keyfile = "/etc/skaidb/cluster.token"     # file holding the secret

Cert — mutual TLS. Every node presents a certificate signed by a shared CA, and the channel is encrypted. Node certificates must carry the SAN DNS:skaidb (how peers verify each other without per-node hostnames) and extendedKeyUsage = serverAuth, clientAuth.

Generate the CA and per-node leaf certs with the built-in helper (no OpenSSL incantations, no locale footguns):

skaidbsh certs gen --out ./skaidb-certs --nodes 3
# writes ca.crt, ca.key, and node1..node3.{crt,key}, all 0600

Give each node its own leaf (node<i>.crt/node<i>.key) plus the shared ca.crt; keep ca.key off the nodes (it's the issuing root — store it offline for future cert minting). Then:

[auth]
internode_auth = "cert"
internode_tls_cert = "/etc/skaidb/node.crt"   # this node's leaf (SAN: skaidb)
internode_tls_key  = "/etc/skaidb/node.key"
internode_tls_ca   = "/etc/skaidb/ca.crt"     # CA that signs every node's cert

Both are also settable via flags/env (--internode-auth, SKAIDB_INTERNODE_*). A node that can't satisfy the configured mode is dropped at the handshake, before any RPC. Rollout: there's no mixed-mode window — a cert node and a none/token node cannot talk, so turn the mode on with the same material on every node and restart them together (a brief flag-day; clients fail over and retry). The effective mode is surfaced at GET /status as "internode_auth": "none" | "token" | "cert", so a monitoring check can catch a node that silently came up unauthenticated. Client auth is separate — SCRAM on the binary endpoint and HTTP Basic on REST, plus RBAC; see the README.

Encryption note: only cert mode encrypts the internode channel. none and token are plaintext on the wire (token authenticates but does not encrypt).

Rotating internode certificates (no flag day)

Unlike turning cert mode on, replacing certificates does have a mixed window, because internode_tls_ca may hold several CAs — every certificate in that file is added to the trust store, for both the peers a node dials and the peer certificates it accepts. Concatenating the old and new CA gives an overlap in which nodes on either certificate interoperate.

mTLS is mutual, and that dictates the order. A node handed the new CA alone rejects the old certificates its peers are still presenting — its own certificate being valid does not help. Deploying "new CA + new cert" to one node first therefore partitions that node. Bundles go everywhere before any certificate is swapped:

Bundle, node by node. Set internode_tls_ca to cat old-ca.crt new-ca.crt, leave each node's existing cert/key alone, restart. Safe in any order: a bundled node still accepts untouched peers.
Verify every node reports "internode_auth": "cert" on GET /status and the cluster is ready with the full member count.
Swap certificates, node by node. Replace internode_tls_cert / internode_tls_key with the new-CA-issued pair, restart. Mixed old/new nodes interoperate for as long as this takes — both sides still hold the bundle.
Drop the old CA once every node presents a new certificate: set internode_tls_ca back to the new CA alone and restart node by node. Old certificates stop being trusted, which is what completes the rotation.

Each step is a rolling restart, so quorum holds throughout at RF≥3. The four trust states above — including the two failure modes in step 3's warning — are pinned by cert_rotation_via_ca_bundle in skaidb-cluster, so the property this procedure depends on cannot regress silently.

Client TLS (driver ↔ cluster)

The binary (7000) and REST (7080) ports can be TLS-wrapped independently of internode mode. Set [encryption]:

[encryption]
client_tls = "opportunistic"   # off | opportunistic | required
tls_cert_file = "/etc/skaidb/server.crt"
tls_key_file  = "/etc/skaidb/server.key"

off (default) — plaintext only.
opportunistic — one port serves both: a TLS ClientHello is wrapped, a plaintext connection is served as before. Use this to migrate — point every client at TLS, confirm, then switch to required.
required — plaintext connections (including probes on the REST port) are refused. TLS only.

Clients authenticate with SCRAM/HTTP-Basic inside the TLS channel (client certs are not required — this is one-way server TLS). A misconfiguration (client_tls on but cert/key unset) fails the listener startup loud, never silently plaintext; the effective mode shows at GET /status as client_tls. The server cert can be any TLS cert; the cluster CA from skaidbsh certs gen works (its node certs carry SAN skaidb, and the CA and node certs carry Subject and Authority Key Identifiers, so clients that verify strictly per RFC 5280 — Python's default VERIFY_X509_STRICT, openssl verify -x509_strict — accept the chain). Connect with:

skaidbsh -H node --tls --tls-ca ca.crt          # verify against the cluster CA
skaidbsh -H node --tls                          # verify against the PUBLIC-CA roots
skaidbsh -H node --tls --tls-insecure           # self-signed / dev (INSECURE)
# --tls-server-name <name>  overrides the verified SAN (default: skaidb)

Bare --tls verifies against the public-CA roots (Mozilla's bundle, compiled in) — right for a server behind a public certificate, and a certificate error against a cluster-CA server (pass --tls-ca, or --tls-insecure for dev). The Rust driver's TlsVerify::System is the same policy.

The driver takes Client::connect_many_tls(endpoints, user, pw, Some(tls)).

At-rest encryption

Encrypt the row stores on disk with AES-256-GCM: every table's and index's WAL and SSTables, vector-index snapshots, the consensus log and the persisted content digests — the exact list, and what is not covered, is under What at-rest encryption covers. The scheme is envelope: a KEK from a keyfile wraps a per-file DEK that seals the data, so key rotation is cheap and the KEK never touches data.

skaidbsh keyfile gen --out /etc/skaidb/at-rest.key   # 32 bytes, 0600

[encryption]
at_rest_enabled = true
at_rest_kek_source = "keyfile"          # or "kms" — see below
at_rest_keyfile = "/etc/skaidb/at-rest.key"

KMS-held KEK. With at_rest_kek_source = "kms" no key rests on the node: at_rest_kek_command is run once at startup (through sh -c) and its stdout is the KEK — 32 raw bytes, 64 hex chars, or base64 of 32 bytes. It is the one-liner you would type against your KMS, so any service with a CLI works and the node's IAM/Vault identity, not a file, gates the key:

toml [encryption] at_rest_enabled = true at_rest_kek_source = "kms" at_rest_kek_command = "aws kms decrypt --ciphertext-blob fileb:///etc/skaidb/kek.enc --query Plaintext --output text" # rotation: the previous KEK's command, unwrap-only (same role as # at_rest_previous_keyfiles) at_rest_previous_kek_commands = ["vault kv get -field=kek secret/skaidb/kek-2025"]

A non-zero exit fails startup with the command's own stderr (a denied decrypt is spelled out, the key material never is); an empty or malformed output fails the same way. The command runs as the service user, once per process start — a KMS outage stops a node from starting, never a running one.

New files encrypt; existing plaintext files stay readable (mixed migration). What converts by itself once the node restarts with at-rest on: a table's or index's WAL starts sealing at its first flush (a log that still holds plaintext records is sealed off by the maintenance sweep and its plaintext segment is deleted when that flush lands); the consensus log is rewritten encrypted within seconds of the start (consensus log rewritten encrypted in the log); a content digest is resealed as it is loaded; a vector snapshot at its next save. Plaintext SSTables stay until a compaction rewrites them, which a cold table never does — to fully encrypt an existing node, do a rolling per-node resync: wipe the node's data dir and let it rebuild from peers onto the encrypted engine — one node at a time, RF keeps the cluster serving (the same shape as re-encrypting any replicated store).
Time-series tables seal per file as well: every new WAL segment and every new block gets its own DEK. A table that has plaintext files keeps reading them — its WAL is sealed from the next checkpoint on (every head flush, at the latest one 2 h block window; the plaintext segments are retired by that checkpoint, into the archive as they are when one is configured), and a plaintext block stays until compaction folds it into a sealed one. Damage to a sealed time-series file is reported like damage to a plaintext one — a WAL record that fails authentication with valid records after it, or a block region that fails it, is corrupt data: naming file and offset and quarantines that table; a record that fails at the very end of a segment is a torn write and is truncated away. A sealed time-series file on a node without its key fails startup (at-rest crypto:), never a quarantine. Versions without time-series at-rest support cannot read sealed time-series files: do not downgrade a node that has written them.
A missing or bad keyfile fails startup loud — the node never comes up silently unencrypted. at_rest_enabled is restart-scoped. The effective state shows at GET /status as at_rest.
Back up the keyfile off-box before enabling. Losing it makes all encrypted data unrecoverable — it is operator-critical.
The WAL, SSTables and the stamps sidecar are ciphertext on disk. The sidecar is sealed with the same per-file key in a separate nonce domain, so encrypted tables keep the value-free repair fast path (they used to have none, which made every digest scan decode full data blocks).

What at-rest encryption covers

Encrypted, under the data directory:

Path	Holds
`tables/<t>/wal.log`, `wal.log.seg.`, `sst/` (data + stamps sidecar)	row tables — rows, keys, versions
`indexes/<i>/…` (same files)	secondary and global indexes
`paxos/…` (same files)	the consensus log of linearizable (CAS) writes — an accepted proposal is a whole row body
`tables/<t>/digest.bin`	anti-entropy content digests — they list recently written primary keys
`vector/<index>.hnsw`	vector-index snapshots — each indexed row's key and embedding
`timeseries/<t>/wal/*.wal`	time-series tables' WAL — series labels and every unflushed sample
`timeseries/<t>/blocks/b-*/` (`chunks.dat`, `series.dat`, `meta.json`)	time-series blocks — samples, the series index (labels), block metadata
the WAL archive of time-series tables (`<wal_archive_dir>/timeseries/<t>/`)	sealed time-series WAL segments, archived byte-for-byte as they were written
the WAL archive (`storage.wal_archive_dir`) of row tables	sealed WAL segments, archived byte-for-byte as they were written

Memory tables (WITH (memory = true)) never reach the disk.

Not encrypted — plan for these when the threat is a stolen disk or a readable backup:

Path	Holds	Why it matters
`fts/<index>/`	full-text (search) indexes — term dictionaries, postings and positions of every indexed field, and each document's primary key	every word of the indexed columns is recoverable, and so are the keys
`hints/*.hintlog`	hinted-handoff spill: whole row writes queued for a replica that is unreachable, once the in-memory buffer is full	exists only while a replica is down; deleted when drained or expired
`hints/*.hintjournal`	hinted-handoff crash journal: a copy of the in-memory hints (whole row writes, and time-series sample batches) for a replica that is unreachable	truncated when delivered; adopted on the next open after a crash — row hints into the `.hintlog`, time-series batches into the in-memory TS buffer (and journaled again at once)
`catalog.json`	the schema: database, table, column and index names, view / procedure / trigger definitions	names and SQL text, no rows and no credentials (users live in an encrypted table)
membership / topology and `runtime/` bookkeeping	node ids, addresses, counters; a membership change in flight at the stop is saved WITH its previous members (`--prev`) and reopens as an open window, not a settled ring (a binary older than this format reads such a file as no ring and starts from its seeds — downgrade only with no membership change in flight)	no row data
`timeseries/<t>/checkpoint`	a WAL segment number and a flush boundary timestamp	no labels, no samples
log files (`[observability]`)	query, slow-query and audit logs	statement text as logged

For those, put the data directory (and the log directory) on an encrypted filesystem or block device (LUKS/dm-crypt, ZFS native encryption): it composes with at-rest encryption and closes the list above. A column that must never be on disk in the clear should not carry a search index.

grep -a -r <a value you inserted> <data_dir> after a flush is the check: it finds nothing in the encrypted paths.

Rotating the at-rest KEK (no data rewrite)

Rotation rewraps each file's small wrapped-DEK header under the new KEK — the sealed data, and the DEK itself, are never touched, so rotating a multi-GB node costs seconds of I/O, not a rewrite. Every encrypted file carries a kek_id (a fingerprint of the KEK that wrapped its DEK); a background sweep finds files tagged by a retired KEK and patches their headers in place, ~130 files per minute-tick per node, crash-safe (a journal sidecar replays an interrupted patch at the next open).

The keyring dictates the order, exactly like the certificate rotation above. A node given the new keyfile alone cannot open a single file wrapped by the old KEK — it fails startup loud. The old KEK must therefore be in the ring before the primary is swapped, and leave it only when no file references it:

Generate the new keyfile (skaidbsh keyfile gen --out /etc/skaidb/at-rest-2.key) and back it up off-box, like the first one.
Swap with the old key in previous, node by node:

toml [encryption] at_rest_keyfile = "/etc/skaidb/at-rest-2.key" # new primary at_rest_previous_keyfiles = ["/etc/skaidb/at-rest.key"] # old, unwrap-only

Restart. The node opens everything (old files unwrap via the previous key), writes every new file under the new KEK, and the sweep starts rewrapping old headers immediately. 3. Watch skaidb_storage_kek_stale_files to zero on each node. This gauge counts every file (SSTable, WAL, sealed WAL segment, persisted digest — of tables, indexes and the consensus log) still referencing a non-primary KEK. SSTables converge by in-place patch; a busy table's WAL converges at its next flush, a quiet (empty) WAL is re-keyed directly by the sweep; a digest is rewritten under a fresh key; the consensus log is rewrapped by its own maintenance pass (every 10 s). Time-series tables count in the same gauge — blocks, live WAL segments and archived WAL segments. A block is rewrapped in place (its meta.json, which carries the envelope, is rewritten; chunks and index are untouched); a table's live WAL converges by one checkpoint, which the sweep triggers (the head is re-recorded into a segment under the new KEK and the older segments are retired); archived segments are rewritten with a new header around the same sealed records, so point-in-time recovery needs only the new key afterwards. 4. Retire the old keyfile once the gauge reads zero: remove it from at_rest_previous_keyfiles, restart, then destroy the old keyfile — at zero, no byte on disk needs it (the node would now start without it).

Steps 2 and 4 are rolling restarts; quorum holds at RF≥3. Every state above — including that step 2's ordering is mandatory and that the gauge reaching zero really does mean the old KEK is unreferenced — is pinned by kek_rotation_rewrap_pins_every_rollover_state in skaidb-storage.