skaidb metrics & observability

This is the contract dashboards and monitoring agents build against. Every metric skaidb exposes is listed here with its type, labels, and meaning. The registry renders the Prometheus text exposition format with a correct # TYPE and # HELP per metric — counters use rate()/increase(), gauges are instantaneous, histograms expose _bucket{le=…} + _sum + _count.

Endpoints

skaidb serves these over HTTP/1.1. They are unauthenticated and read-only, so orchestrators, load balancers, and scrapers need no credentials and no admin rights:

Path	Method	Purpose
/metrics	GET	Prometheus scrape. Pull-model gauges are refreshed on each scrape. Never waits on the engine lock: while it is write-held (a RESTORE, a long DDL) the storage gauges are the last snapshot's and skaidb_stats_age_seconds is its age.
/health, /healthz	GET	Liveness — 200 ok whenever the process is up.
/ready, /readyz	GET	Readiness — 200 ready when the node will serve reads AND accept writes (engine open, not shedding, disk not blocked, not refusing writes for clock skew); 503 with the reasons in the body otherwise — including engine lock held (an exclusive operation is running) once the engine lock has stayed write-held for a 30 s grace (the probe itself never waits more than 1 s for the lock, and answers 200 inside the grace). ?strict=1 reports a held lock as soon as that 1 s wait fails, and also waits out resharding migration, resync backfill, the startup catch-up repair (/status catching_up), heavy search/index/vector rebuilds, and a clock beyond cluster.max_clock_skew_ms in warn mode (clock skew: …) — gate rolling operations on it.
/status	GET	Low-privilege topology read: ring/epoch/members and default consistency, no secrets. Answers while the engine lock is write-held: engine_lock_busy_ms says for how long it has been (0 = free), and ready turns false once that passes 30 s (the same grace plain /ready applies). Unauthenticated by default; set server.status_auth_required = true to require MONITOR on * (the probes below stay open regardless).
/admin/slow	POST	Sample of recent (masked) slow queries. Requires Admin.
/admin/status	POST	Full topology incl. member ids. Requires Admin.

The /metrics, /health, /ready, /status routes are served both on the REST port (server.rest_port, default 7080) and on a dedicated metrics listener (observability.prometheus_port, default 9090) when that port differs from the REST port. Point your scraper at prometheus_port to keep it off the data plane.

SHOW TABLES / SHOW INDEXES (see QUERY_SYNTAX.md) let a tool enumerate the catalog without /query data access.

The ops shortlist (page/panel on these first)

The ~15 metrics an external monitor should build against, before anything else on this page. "Page" = wake someone; "panel" = graph and alert lazily.

Metric	What it means	Guidance
skaidb_up	1 while the process serves.	Page on scrape failure / 0.
skaidb_iwm_state	Workload-manager state: 0 normal, 1 elevated, 2 shedding writes, 3 disk-blocked.	Page on ≥2 sustained >5 min; panel 1.
skaidb_cluster_clock_skewed	1 while this member's clock is beyond cluster.max_clock_skew_ms against a majority of its measured peers (it refuses client writes by default).	Page on 1; panel skaidb_cluster_peer_clock_offset_ms against the bound.
skaidb_cluster_clock_writes_refused	1 while this member refuses every client write for its clock — the measured skew above, or its node clock (HLC) left ahead of its wall clock by a corrected clock jump (skaidb_cluster_clock_hlc_ahead_ms).	Page on 1 for more than a minute (an in-memory lead clears by itself within seconds).
skaidb_memory_shedding_writes	1 while writes are rejected under memory pressure.	Page on sustained 1 (starts at 85% of limit, clears at 70%).
skaidb_memory_used_bytes / skaidb_memory_limit_bytes	Sampled usage vs the node's (cgroup) limit.	Panel the ratio; alert >90% for 10 min.
skaidb_iwm_disk_write_block (+ skaidb_iwm_disk_rejections_total)	1 while disk-growing writes are refused for lack of space.	Page on 1 — the node is read-only until space frees.
skaidb_storage_disk_bytes	Data-dir bytes actually on disk.	Panel vs volume size; alert >80%.
skaidb_cluster_hints_pending	Writes — and DDL statements (schema hints) — buffered for replicas that did not acknowledge them (also per-peer in /status peers[].hints_pending + lag_ms).	Alert on non-zero >15 min: a peer is down or unreachable. Must drain to ~0 before each step of a rolling upgrade.
skaidb_cluster_hints_expired_total	Hints dropped after cluster.hint_max_age_secs — those writes now reach the replica only via anti-entropy.	Page on increase: replicas were down longer than the hint window.
skaidb_ae_last_pass_at_seconds	Unix time of the last completed anti-entropy pass.	Alert when now - value exceeds the AE interval ceiling — repair has stalled.
skaidb_wal_archive_backlog_segments	WAL segments not yet archived (PITR).	Alert on non-zero sustained: the archive is stalled and the recovery window is eroding.
skaidb_cluster_search_shard_missing_total	Search queries answered with a shard missing (partial results).	Page on increase — silent wrong answers to users.
skaidb_storage_quarantined_tables	Tables quarantined for torn/corrupt files.	Page on non-zero.
skaidb_flusher_last_tick_age_seconds	Age of the last completed background maintenance tick.	Page >600 s: the flusher thread is wedged, wherever it stalled.
skaidb_storage_flushes_total / skaidb_storage_compactions_total	Background write-path health.	Panel the rates; a flatline under write load with rising skaidb_storage_memtable_max_bytes means the flusher is not keeping up.
skaidb_witness_oldest_sync_age_seconds	Staleness of the least-synced witness.	Alert when it exceeds a few pull intervals (witness recovery data going stale).
skaidb_maintenance_longest_job_seconds	Longest currently-running maintenance job.	Panel; alert on hours-long jobs (stuck rebuild).
skaidb_build_info	version/git_sha labels, always 1.	Panel for deploy tracking; no alert.

Conventions

• Counters end in _total (with a few standard exceptions like skaidb_*_seconds_count). Only ever increase; reset to 0 on restart.
• Gauges are absolute, can go up or down (e.g. skaidb_queries_in_flight).
• Histograms (skaidb_query_duration_seconds) bucket observations; query the _bucket/_sum/_count series with histogram_quantile().
• Cardinality is bounded. Per-table metrics are opt-in (observability.per_table_metrics = true) because the table count is unbounded. Error and consistency labels come from small fixed sets.
• Every series can be attributed to a node via skaidb_node_info (or by tagging the scrape target).

Build / runtime

Metric	Type	Labels	Meaning
skaidb_up	gauge	—	1 while the server is up.
skaidb_build_info	gauge	version, git_sha, rustc	Always 1; build metadata for tracking deploys.
skaidb_node_info	gauge	node_id, role	Always 1; node identity so federated scrapes are distinguishable at the source.
skaidb_start_time_seconds	gauge	—	Unix time the process started.
skaidb_uptime_seconds	gauge	—	Seconds since start.
skaidb_stats_age_seconds	gauge	—	Age of the storage statistics on the page (skaidb_storage_*, skaidb_wal_*, the cache and per-table gauges): 0 when this scrape read them from the engine. While the engine lock is write-held — a RESTORE, a long DDL — the scrape does not wait: it serves the last snapshot (a background probe refreshes it at least every 30 s while the lock can be had) and this is its age. Non-zero for more than a scrape or two means the engine lock is held — look at skaidb_engine_lock_longest_hold_ms; before the first successful read it counts from process start.
skaidb_engine_open_seconds	gauge	—	Wall-clock of the storage engine open at this start (catalog, table/index engines, time-series stores, vector/search index loads, recovery). The per-phase split is the skaidb: engine open took … log line; any single table or store over 1 s gets its own skaidb: open of … line.

git_sha/rustc come from the SKAIDB_GIT_SHA / SKAIDB_RUSTC build-time env vars; unset → "unknown".

Query path

Metric	Type	Labels	Meaning
skaidb_queries_total	counter	type	Statements executed (select/insert/update/delete/ddl/tx/other).
skaidb_query_duration_seconds	histogram	type	Execution latency. Use histogram_quantile(0.99, …).
skaidb_queries_in_flight	gauge	—	Statements currently executing.
skaidb_query_errors_total	counter	class	Failed statements by class: parse, constraint, storage, timeout, permission, other.
skaidb_query_stream_total	counter	path	Statements answered on the binary protocol's streaming lane: streamed page by page, or buffered when the plan needs the whole result first.
skaidb_promql_windowed_partials_total	counter	—	PromQL range queries answered by the windowed-partials fast path (pre-aggregated windows rather than a walk of the raw samples).
skaidb_rows_returned_total	counter	—	Rows returned to clients.
skaidb_rows_written_total	counter	—	Rows written (inserted/updated/deleted) — the write-throughput signal a bulk import shows up in (queries/s counts statements, so a multi-row batch is one query).
skaidb_rows_scanned_total	counter	—	Result cells examined (rows × width) — a proxy for result volume.
skaidb_slow_queries_total	counter	—	Statements slower than slow_query_ms.
skaidb_transactions_total	counter	kind	begin/commit/rollback (embedded engine).
skaidb_authz_denied_total	counter	—	Statements denied by RBAC.
skaidb_logins_total / skaidb_login_failures_total	counter	—	Auth outcomes.
skaidb_admin_total	counter	op	Admin control-plane ops (status/add_node/remove_node/repair/reclaim/slow).

Connections

Metric	Type	Labels	Meaning
skaidb_connections_active	gauge	endpoint	Open connections by endpoint (binary/rest/mqtt).
skaidb_connections_total	counter	endpoint	Connections accepted by endpoint.
skaidb_connections_refused_total	counter	endpoint, reason	Connections turned away at admission; reason="over_capacity" means server.max_connections (or the metrics listener's own ceiling — as many slots again as server.max_connections, never fewer than 64 — endpoint="metrics") was already reached; reason="reserved" means the connection landed in the superuser-reserved band above max_connections and did not present the superuser; reason="rate_limited" is the IWM connection rate gate.
skaidb_auth_refused_total	counter	endpoint, reason	Authentications refused before the verifier ran; reason="rate_limited" means the source IP was over auth.failed_login_rate_limit.
skaidb_metrics_listener_bound	gauge	—	1 when the dedicated metrics/health listener (observability.prometheus_port) is bound, 0 when its bind failed at startup — /metrics, /ready and /status are then served on the REST port only and /status.metrics_listener carries the error. Absent when no dedicated listener is configured. Alert on 0 (scraped via the REST port).

MQTT broker

Present when [mqtt] is enabled (see MQTT.md).

Metric	Type	Labels	Meaning
skaidb_mqtt_packets_total	counter	type, dir	Control packets by type (publish, puback, subscribe, …) and direction (in/out).
skaidb_mqtt_connect_total	counter	outcome	Connection admissions: accepted, rate_limited, over_capacity.
skaidb_mqtt_messages_dropped_total	counter	reason	Dropped messages: outbox_full, queue_full, retained_full, expired, too_large (a delivery over the client's Maximum Packet Size, or an inbound packet over mqtt.max_packet_bytes), acl, store_error.
skaidb_mqtt_sessions_expired_total	counter	—	Stored sessions discarded by the expiry sweep.
skaidb_mqtt_sink_rows_total	counter	—	Messages captured by [[mqtt.sink]] rules.
skaidb_mqtt_sink_dropped_total	counter	reason	Sink failures: error, ooo (out-of-order-window TS drops).
skaidb_mqtt_cluster_forward_failed_total	counter	—	Peer fan-out RPCs that failed (state self-heals via _mqtt.*).
skaidb_mqtt_cluster_forward_dropped_total	counter	—	Fan-out events dropped on queue overflow.
skaidb_mqtt_messages_published_total	counter	origin	Messages published through the broker by origin (sql = PUBLISH from SQL).
skaidb_mqtt_bridge_messages_total	counter	bridge, direction	Messages an MQTT bridge relayed between this broker and its remote (in/out).
skaidb_mqtt_bridge_reconnects_total	counter	bridge	Reconnects of an MQTT bridge to its remote broker.
skaidb_mqtt_parked_readers	gauge	—	Connections whose reader is parked on the epoll parker — an idle connection holds no thread.
skaidb_mqtt_parks_total / skaidb_mqtt_wakes_total	counter	side	Connection threads parked and woken, by side (reader/writer).

BI node

Present when [bi] is enabled (see BI.md). Only the node that serves /bi emits them — every other node redirects the route and has nothing to count.

Metric	Type	Labels	Meaning
skaidb_bi_queries_total	counter	outcome	BI app statements: ok, cached (answered from the result cache), error, refused (a write — a BI node reads a mirror), busy (the admission gate turned the request away at bi.max_concurrent_queries and the caller got 429). Alert on a sustained busy rate: dashboards are queueing behind each other, and the tiles that lose are simply not drawn. A steady refused rate is a client sending writes, not load.
skaidb_bi_query_seconds	histogram	—	BI statement latency. Every exit is observed, so _count matches the sum of skaidb_bi_queries_total — refusals included, at zero. Cache hits are in here too and are what pulls the low buckets up, so read histogram_quantile(0.99, …) rather than _sum / _count.
skaidb_bi_in_flight	gauge	—	Statements executing right now, against bi.max_concurrent_queries. Panel it against that ceiling; the time it spends pinned there is what the busy counter then charges for.
skaidb_bi_cache_entries	gauge	—	Results held in the BI result cache, against bi.cache_entries. Panel; no alert.
skaidb_bi_cache_lookups_total	counter	result	Result-cache lookups, hit/miss. The cache is invalidated by the mirror watermark rather than by a timer, so a table the pull cycle keeps moving counts every lookup as a miss. Panel the hit ratio: a ratio that collapses on a busy table is the cache working as designed, and the reason the same board costs more at some hours than others.
skaidb_bi_refreshes_total	counter	outcome	Scheduled dashboard refreshes: ok, error, skipped (a due board with nothing to run: no tiles, or only PromQL ones, which the scheduler does not warm). Alert on a sustained error rate — a board that fails to refresh keeps serving its last tiles, so it looks exactly like a board nobody is changing.
skaidb_bi_refresh_seconds	histogram	—	Time to refresh one dashboard's tiles. Alert when the p99 approaches the shortest board's refresh_secs: the scheduler runs bi.refresh_workers boards per tick, so a board slower than its own cadence is never current and every other board waits behind it.

PromQL tiles are the gap in the counters above: {"lang":"promql"} on /bi/api/query is evaluated outside the BI runner, so it reaches no counter here, takes no result-cache entry, and is not bounded by bi.max_concurrent_queries.

/bi requests are not their own REST class — they land in skaidb_rest_requests_total{path="other"}, so the counters above, not the REST table, are this node's request signal.

A BI node is a witness, and mirror staleness is reported by the members, not by the witness itself: alert on skaidb_witness_oldest_sync_age_seconds labelled witness="<its witness id>", scraped from a member. The BI node's own Data page shows the same ages, but no metric it exposes does.

Storage / LSM

Pulled from the engine snapshot at scrape time (aggregated across all table and index storage engines).

Metric	Type	Labels	Meaning
skaidb_storage_tables	gauge	—	Tables in the catalog.
skaidb_storage_indexes	gauge	—	Secondary + vector indexes.
skaidb_storage_memtable_bytes	gauge	—	Approx live memtable footprint, summed across all engines (memory tables included). The flush threshold is per engine and does not apply to this sum — compare memtable_max_bytes to flush_threshold_bytes instead.
skaidb_storage_memtable_versions	gauge	—	Row versions held in memtables across every engine. Read with memtable_bytes: many small versions (a hot key rewritten thousands of times, each version carrying its CAS history) and few large rows look the same in bytes alone. SHOW STORAGE reports the same figure.
skaidb_storage_memtable_max_bytes	gauge	—	Largest single-engine memtable footprint (active + frozen), excluding memory = true tables (their memtable is the table by design). Stays at or under skaidb_storage_flush_threshold_bytes when the background flusher is healthy: the maintenance sweep freezes any over-threshold engine even if no writes arrive. Alert on the ratio of the two sitting well above 1: the flusher is wedged.
skaidb_storage_sstables	gauge	—	On-disk SSTable count across levels.
skaidb_storage_disk_bytes	gauge	—	On-disk bytes across all SSTables.
skaidb_storage_flushes_total	counter	—	Memtable flushes installed as SSTables. With memory-plan-sized thresholds a flush can legitimately be a rare event — compare memtable_max_bytes to flush_threshold_bytes rather than expecting steady flush cadence.
skaidb_storage_build_failures_total	counter	—	SSTable builds (flush or compaction) that failed, counted on the engine that issued the job (a job of a since-dropped engine is not charged to its successor; a leftover file another engine's build left at a table's next file name is taken over, not counted). A committed write is never lost to one — the memtable keeps the rows and the build is retried after a backoff — but a rising rate means the node cannot write its memtables out (a full disk, typically) and memory is holding what should be on disk. Alert on this.
skaidb_storage_wal_poisoned_engines	gauge	—	Tables and indexes whose write-ahead log a failed fsync poisoned. After an fsync error the kernel may have dropped the pages it could not write, so a later fsync that succeeds proves nothing about them: the engine refuses writes (and is neither frozen nor sealed) until the node restarts, and the restart replays what really is on disk. Writes told "failed" may or may not be there. Alert on anything above zero: check the disk, then restart. Also wal_poisoned_engines in SHOW STATUS.
skaidb_storage_enospc_total	counter	—	Storage I/O errors that were No space left on device.
skaidb_storage_flush_threshold_bytes	gauge	—	The per-engine flush threshold in force on this node (32 MB default; the memory plan sizes it up to 1 GB under memory_target). The healthy ceiling for memtable_max_bytes.
skaidb_maintenance_longest_job_seconds	gauge	—	Age of the longest in-flight background job (SHOW MAINTENANCE). Digest builds and backfills finish in minutes; an hours-old job is stuck.
skaidb_flusher_last_tick_age_seconds	gauge	—	Seconds since the background flusher/maintenance thread completed a full tick — on every node: a standalone server runs the same 100 ms loop a member does (freeze/flush, compaction, TTL reclaim, archive upkeep, time-series head flushes), with the SSTable builds off the engine lock. The thread-liveness heartbeat: a growing age is a wedge, wherever the thread stalled.
skaidb_storage_compactions_total	counter	—	Compaction passes completed.
skaidb_storage_legacy_block_bytes	gauge	—	On-disk bytes in SSTables written under an older (smaller) block-size target. Compaction rewrites them at the current target, so this trends to zero; cold tables are finished by a background sweep (one small table per maintenance pass). Zero = block-size migration complete.
skaidb_storage_legacy_block_files	gauge	—	File count behind skaidb_storage_legacy_block_bytes.
skaidb_storage_quarantined_tables	gauge	—	Tables whose files were damaged (a WAL record failing its checksum with data after it, a bad SSTable footer/index/Bloom checksum, a missing/empty/unreadable MANIFEST next to table files, a data file the manifest names that is gone, a damaged time-series block or WAL segment) and were quarantined — at startup instead of refusing the whole open, or by the read that found a data block failing its checksum. Statements on them error with the cause and the file; restore the table's directory (backup or wipe-resync from peers) or DROP TABLE to discard. Non-zero should alert.
skaidb_storage_kek_stale_files	gauge	—	KEK rotation progress: files (SSTables, WALs, sealed WAL segments, persisted digests — of tables, indexes and the consensus log; time-series blocks, WAL segments and archived WAL segments) whose data key is still wrapped by a KEK other than the current primary. A background sweep rewraps them in place; zero means no file references a previous KEK and its keyfile can be retired (see CLUSTER_SECURITY.md, "Rotating the at-rest KEK").
skaidb_storage_compaction_bytes_total	counter	—	Bytes written by compaction.
skaidb_batch_spliced_rows_total	counter	—	Rows executed through the spliced executemany fast path (plain INSERT batches run as multi-row statements instead of once per row). Zero while drivers send batches = the per-row fallback is carrying them.
skaidb_wal_bytes	gauge	—	Live write-ahead log size.
skaidb_wal_archived_segments_total	counter	—	Sealed WAL segments copied into the point-in-time-recovery archive before deletion. Zero forever = storage.wal_archive_dir is unset.
skaidb_wal_archived_bytes_total	counter	—	Bytes of those segments.
skaidb_wal_archive_backlog_segments	gauge	—	Sealed WAL segments the archive still owes a copy of and that are not being copied right now — a refused copy (full or unmounted archive volume) stays counted until it lands, and the node does NOT delete the segment meanwhile, so the data directory grows. Deliberate: a gapped archive would still look restorable. A freshly sealed segment shows until the next archive pass starts its copy (within 100 ms on a standalone server; on a cluster member behind any flush or compaction batch the maintenance loop is in). Alert on a sustained value.
skaidb_wal_fsyncs_total	counter	—	WAL fsyncs issued since the process started, summed over the node's tables and indexes (compare to writes to see group-commit coalescing). Monotonic across WAL rotations and truncations; it drops only when a table is dropped or the process restarts.
skaidb_cache_hits_total / skaidb_cache_misses_total / skaidb_cache_evictions_total	counter	—	Read-cache effectiveness.
skaidb_cache_entries	gauge	—	Live read-cache entries.
skaidb_cluster_search_global_stats_rounds_total	counter	—	Global-BM25 statistics rounds: one per sharded full-text search while cluster.global_bm25 is on (the extra scatter that buys single-index ranking).
skaidb_cluster_search_shard_missing_total	counter	—	Ranked searches (vector ANN / full-text relevance) that ran with a shard missing because a peer could not answer. A vector search degrades rather than fails, so for it this counter is the only signal that results came from fewer shards. A full-text search tolerates up to RF−1 missing members, since every row still has an answering copy, and fails with a retryable search incomplete error past that. A rising rate means members are not answering searches.
skaidb_search_reindex_skipped_total	counter	—	Row writes whose full-text re-index was skipped because no field any search index covers changed and the index already held the previous version (a tag or counter update on a row with a large indexed body). Zero on write-once data; a high share on update-heavy tables is the saving working.
skaidb_search_indexes	gauge	—	Full-text search indexes open on this node.
skaidb_search_docs_total	gauge	—	Documents held across all full-text search indexes. A gauge despite the name (the current count, not a running total) — kept for dashboards.
skaidb_search_disk_bytes	gauge	—	Bytes of full-text search index segments on disk. Measured by a background walk at most every 5 minutes, so it can lag new segments by that much.
skaidb_search_rebuild_seconds	gauge	—	Wall-clock the last open spent reopening or replaying the search indexes.
skaidb_cluster_batches_refused_undecoded_total	counter	—	Inbound replicated batches (ApplyBatch) this node refused before decoding their rows, because it was shedding memory or out of disk. The sender hints the rows for later delivery. Rising together with the shed or disk-guard signals is expected; it shows the refusals cost no row decoding.
skaidb_jobs_runs_total	counter	—	Scheduled job runs executed by this node (any outcome). Jobs are singly-owned, so this is per-owner, not per-cluster.
skaidb_jobs_failures_total	counter	—	Job runs that ended in error. Also visible per job in SHOW JOBS (last_status, last_error, failures) and in _job_runs. Alert on a sustained rate: a job retries with exponential backoff, so a broken one goes quiet rather than loud.
skaidb_jobs_deferred_total	counter	—	Job ticks the workload manager deferred: the run did not start and was rescheduled. Not a failure, and invisible in the job's own state — a job that is being shed looks exactly like a healthy quiet one. A rising rate means scheduled work is losing to load.
skaidb_iwm_state	gauge	—	IWM node state: 0 normal, 1 elevated (memory in its release tier), 2 overloaded (shedding writes), 3 critical (disk writes blocked). Alert on dwell time at ≥1.
skaidb_iwm_disk_write_block	gauge	—	1 while disk-growing writes are blocked for lack of free space ([iwm] watermarks; hysteresis).
skaidb_iwm_disk_rejections_total	counter	—	Writes rejected by the low-disk guard.
skaidb_iwm_disk_enospc_trips_total	counter	—	Times a real ENOSPC write failure tripped the disk guard ahead of the free-space sampler. Any non-zero value means the volume actually hit zero free bytes — the watermark was not enough headroom for this node's compaction outputs; raise iwm.disk_low_watermark.
skaidb_iwm_disk_reserve_present	gauge	—	1 while the guard's reserve file (iwm.disk_reserve) is on disk; 0 while it has been released to give deletes/compaction room (writes are blocked), or when the reserve is off.
skaidb_iwm_sentinel_kills_total	counter	—	Statements killed by the Query Sentinel (iwm.sentinel = auto, fires only while shedding).
skaidb_queries_disconnect_kills_total	counter	—	Statements cancelled because their driver client closed the connection mid-statement.
skaidb_iwm_conn_rejections_total	counter	—	Connections refused by the IWM rate gate (iwm.conn_rate).
skaidb_iwm_ops_rejections_total	counter	—	Statements refused by the Overloaded ops ceiling (iwm.ops_rate_overloaded).
skaidb_query_stream_total{path}	counter	path	Streamed queries by server path: keyset paged on the primary key, ts paged in time slices (a time-series raw read) — the node holds one page either way — or buffered (node materialised, then chunked). Binary and REST streams both count.
skaidb_stream_events_lost_total	counter	—	Change-stream events that could not be appended to their log. The source write still succeeded, so any non-zero value means a stream's log is behind its table and a replay of that window has a hole in it. Should be 0.
skaidb_iwm_heavy_scans_running	gauge	—	Heavy read scans holding an admission slot right now (iwm.max_concurrent_scans).
skaidb_iwm_scan_admission_refusals_total	counter	—	Heavy scans refused because every admission slot was taken.
skaidb_block_cache_bytes	gauge	—	Decompressed bytes resident across all tables' block caches. The first place to look when node memory ramps during scan-heavy periods (anti-entropy sweeps, witness pulls).
skaidb_block_cache_hits_total / skaidb_block_cache_misses_total / skaidb_block_cache_evictions_total	counter	—	Block-cache effectiveness.
skaidb_bloom_negative_lookups_total	counter	—	Point reads resolved absent by the Bloom/SSTable layer.

Host system (per node)

Sampled from /proc (plus df for the data-directory filesystem) at each scrape; each node reports its own host. Memory is cgroup-aware — a container reports its own limit and usage, not the host's. CPU% and disk throughput are computed over the window since the previous sample.

Metric	Type	Labels	Meaning
skaidb_host_cpu_percent	gauge	—	Busy CPU as % of all cores over the last sampling window.
skaidb_host_cpus	gauge	—	Logical CPU count.
skaidb_host_mem_total_bytes	gauge	—	Total memory (cgroup limit when one applies).
skaidb_host_mem_used_bytes	gauge	—	Used memory (cgroup memory.current when limited, else MemTotal - MemAvailable).
skaidb_host_rss_bytes	gauge	—	The skaidb process's resident set size.
process_open_fds	gauge	—	File descriptors the skaidb process holds open (client and internode sockets, data files, logs). The standard process-collector name, so stock dashboards and alerts apply; Linux only.
process_max_fds	gauge	—	The process's open-file limit (soft RLIMIT_NOFILE, e.g. systemd LimitNOFILE); 0 when unlimited. Alert on process_open_fds / process_max_fds > 0.8.
process_threads	gauge	—	OS threads in the skaidb process (connection threads, workers, background loops) — the first place a thread leak shows. Linux only.
skaidb_host_disk_read_bytes_total / skaidb_host_disk_written_bytes_total	counter	—	Whole-host disk IO since boot (physical devices; partitions/loop/dm excluded).
skaidb_host_disk_total_bytes / skaidb_host_disk_available_bytes	gauge	—	The filesystem holding the data directory.

Per-table (opt-in)

Enabled with observability.per_table_metrics = true. Each carries db and table labels — only turn this on when the table set is small and known.

Metric	Type	Labels	Meaning
skaidb_table_live_keys	gauge	db, table	Live key versions: per-file counts cached at open plus the memtables' running counters — O(files) at scrape, never a row walk. A key rewritten across files counts once per file until compaction merges them.
skaidb_table_tombstones	gauge	db, table	Tombstone versions awaiting compaction (same accounting).
skaidb_table_disk_bytes	gauge	db, table	On-disk bytes for the table — every table kind, TIME-SERIES included. A time-series table's size comes from a background walk of its block directory at most every 5 minutes (never under a lock), so it can lag new blocks by that much.
skaidb_ts_table_series	gauge	db, table	Series count per time-series table.
skaidb_ts_table_samples_appended_total	counter	db, table	Samples appended per time-series table.
skaidb_ts_table_samples_rejected_total	counter	db, table	Samples rejected (OOO / series limit) per time-series table.
skaidb_ts_table_blocks	gauge	db, table	On-disk block count per time-series table — watch it FALL during a compaction-backlog drain.
skaidb_ts_table_maintenance_errors_total	counter	db, table	Best-effort retention/compaction failures (maintenance never fails an append).

Per-table live-key and tombstone figures are version counts kept per data file. A file's footer records its tombstone count, so the figures are exact from the moment a file is opened. A file written before the footer carried the count knows only its entry count: every entry of such a file counts as live and its tombstones as 0 until the background thread sst-counts (off every lock, read-only, at most 32 MiB/s of file) has counted it; every engine opened later (a restore, an adoption) is queued for it as well. Nothing reads a data file to answer statistics.

The per-table time-series figures never wait for a store: while a store's own lock is busy (a repair's summary fold, a head flush) they are the last values read under it, so a scrape, SHOW STATUS or a peer's host-stats probe cannot stall the node behind a time-series table.

Vector index

Metric	Type	Labels	Meaning
skaidb_vector_indexes	gauge	—	HNSW indexes.
skaidb_vector_indexed_total	gauge	—	Total vectors held in memory.
skaidb_vector_rebuild_seconds	gauge	—	Time to rebuild vector indexes on the last open.

Cluster / replication

Present only when the node is clustered (cluster.seeds set). Pulled from the coordinator at scrape time.

Metric	Type	Labels	Meaning
skaidb_membership_epoch	gauge	—	Membership epoch; alert on changes.
skaidb_cluster_members	gauge	—	Members visible from this node.
skaidb_cluster_resharding	gauge	—	1 while a join/decommission dual-write window is open.
skaidb_cluster_writes_total	counter	consistency	Coordinated writes by level (one/quorum/all).
skaidb_cluster_reads_total	counter	consistency	Coordinated reads by level.
skaidb_cluster_quorum_failures_total	counter	kind	Operations that failed to reach quorum (read/write).
skaidb_cluster_write_restamps_total	counter	—	Client blind writes (rows of INSERT/DELETE and their companion index/TOAST writes) this coordinator stamped again and re-sent because a replica already held a newer version of the key — a write acknowledged through another coordinator that this one had not applied when it minted its stamp (CLUSTER_CONSISTENCY.md "Write ordering across coordinators"). Each is one extra replication round, also counted in skaidb_cluster_writes_total. A steady rate means several coordinators write the same keys back to back (or blind-upsert one hot key concurrently); it is the cost of keeping a client's own sequence in order, not an error.
skaidb_cluster_slow_replica_skips_total	counter	—	Replica copies of a write (or a QUORUM point read, a CAS phase, a time-series append) this coordinator stopped waiting for: the replica had not answered within 250 ms while another replica could take its place, or once the operation's level was met. The replica still got the request (a write lands when it gets to it, and is hinted too); it counts as slow for 10 s — asked last, not waited for — unless it answers promptly first. A rate that tracks one member points at that member's disk or CPU; it is the cost of not making every write wait for the slowest replica, not an error.
skaidb_cluster_flush_backpressure_waits_total	counter	—	Write statements (and replicated applies) that waited, holding no engine lock, for this node's background flusher to get under four frozen memtables on some table or index — at most 2 s each. The write path never writes tables out itself; a steady rate means writes arrive faster than the disk takes them, and writers are slowed to its pace.
skaidb_cluster_read_repairs_total	counter	—	Read-repair writes pushed to lagging replicas.
skaidb_cluster_ordered_group_merges_total	counter	—	Grouped aggregations this coordinator served by the streaming merge on a cluster_by prefix (one open group resident, no group map — see CLUSTER_CONSISTENCY.md).
skaidb_alloc_purges_total	counter	—	jemalloc arena purges on IWM tier relax (iwm.jemalloc_purge_on_relax, off by default). Each purge logs the before/after allocator split — the evidence for whether the knob earns its keep.
skaidb_ae_passes_total	counter	outcome	Anti-entropy pass attempts: reconciled (fixed rows), converged (found nothing), deferred (pressure/peer mid-pass), failed.
skaidb_ae_rows_reconciled_total	counter	—	Rows fixed by anti-entropy — the divergence actually found, and the adaptive cadence's feedback signal.
skaidb_ae_interval_seconds	gauge	—	Current effective anti-entropy interval. Fixed unless cluster.anti_entropy_adaptive is on; then floor..ceiling, relaxing while passes converge, snapping to the floor when one reconciles.
skaidb_ae_last_pass_seconds	gauge	—	Duration of the last completed pass (deferred attempts excluded).
skaidb_ae_last_pass_at_seconds	gauge	—	Unix time the last completed pass finished. Alert on time() - this > ceiling — it is what distinguishes a healthy backed-off loop from a stuck one.
skaidb_cluster_hints_stored_total	counter	—	Hinted-handoff writes buffered for unreachable replicas.
skaidb_cluster_hints_replayed_total	counter	—	Hinted-handoff writes successfully replayed.
skaidb_cluster_hints_expired_total	counter	—	Hints dropped without delivery: older than cluster.hint_max_age_secs, over the per-replica cluster.hint_max_disk_mb cap, or logged for a replica that left the ring. Anti-entropy repair covers what they carried; a growing rate means a replica is persistently behind or gone.
skaidb_cluster_hints_pending	gauge	—	Hints currently buffered (all peers).
skaidb_cluster_hints_discarded_total	counter	—	Hinted writes discarded at drain time because their table was dropped since they were stored — nothing to replay, and the peer would refuse them. Not a loss: the table is gone everywhere.
skaidb_cluster_hints_pending_peer	gauge	peer	Hints buffered per peer — exact replication backlog for that node.
skaidb_cluster_ts_hints_dropped_total	counter	—	Time-series samples hinted handoff could not buffer for an unreachable replica (its 100k-sample hint buffer was full). Anti-entropy repairs them; growth while a replica is down is the loss that pass will carry.
skaidb_index_stale_entries_total	counter	—	Secondary-index entries a read found pointing at a row version that no longer exists — skipped (never served) and queued for the background sweep. A steady non-zero rate on an old index: rebuild it (DROP INDEX + CREATE INDEX).
skaidb_index_superseded_entries_kept_total	counter	—	Index entries queued index maintenance left in place because the row's stored version still produces them, where no other task for the row was queued to put them back. Each one is an entry this member's index would otherwise have lost until the row is next written: a superseded task — its row rewritten meanwhile by a write this member coordinated, by another queued task applied first, or by REBUILD INDEX — removing the stored row's own entry. A lower bound (a newer task for the row applied earlier in the same applier batch still counts as queued). Growth on a member taking writes to the same rows through several coordinators while its maintenance queue is backed up is expected and harmless: it counts the race being absorbed, not drift.
skaidb_cluster_cas_rounds_total	counter	—	Per-key consensus (CAS) rounds started by clustered UPDATEs and INSERT … ON CONFLICT upserts — each key of a batched round counts once per batched iteration.
skaidb_cluster_cas_retries_total	counter	—	CAS rounds retried on contention (ballot bumps / lost quorums).
skaidb_cluster_cas_contention_failures_total	counter	—	Clustered UPDATE and INSERT … ON CONFLICT statements that exhausted CAS retries; alert if growing — a pathologically hot key.
skaidb_cluster_cas_batch_rounds_total	counter	—	Batched CAS iterations: one prepare, one propose and one commit exchange covering up to 128 keys of a multi-row UPDATE or INSERT … ON CONFLICT upsert that share a replica set, each phase one WAL sync per replica for the whole batch.
skaidb_cluster_cas_batch_keys_total	counter	—	Keys carried by batched CAS iterations; rate(keys) / rate(rounds) is the mean batch size. An uncontended key takes two iterations (the commit, then the verifying prepare).
skaidb_cluster_cas_permit_waits_total	counter	—	CAS rounds that found every slot of their pool taken and waited for one: multi-key UPDATEs / upserts share 16 node-wide slots, single-key statements 64 of their own. Steady growth: statements queue behind each other on this coordinator.
skaidb_cluster_cas_running	gauge	kind	CAS slots held right now, by pool: kind="multi" (id-list rounds, of 16) and kind="single" (one-row statements, of 64). Pinned at the cap with cas_permit_waits_total growing = saturated.
skaidb_cluster_cas_batch_fallbacks_total	counter	—	Keys a batched CAS round handed to the per-key path: contended keys (promise or accept shortfall, a would-lose commit), keys still unresolved after six iterations, keys whose rows exceed a batch's 2 MB byte budget (one replica's prepare reply carrying the row bodies, or the proposed values), a key whose row could not be read from the replica holding its newest version or did not fit the batch's 2 MB fetch budget, and every key while a replica runs a version without the batched ops (a rolling upgrade). A steady rate on a settled fleet means hot keys.
skaidb_cluster_cas_paxos_syncs_total	counter	—	Durable Paxos-state WAL syncs this node served as a CAS replica: one per per-key prepare or propose, one per batched one. The replica-side fsync cost of linearizable UPDATEs and INSERT … ON CONFLICT upserts.
skaidb_cluster_cas_paxos_bytes_total	counter	—	Bytes of Paxos state this node wrote durably as a CAS replica, for every round it served (whichever node coordinated) — the Paxos engine's WAL volume. A propose stores the proposed row value in full (LZ4-compressed with cluster.cas_paxos_compress on); a later prepare on a replica whose row already carries that commit rewrites it as a marker of a few tens of bytes, so the bulk is about one stored row value per replica per updated row.
skaidb_cluster_cas_prepare_fetches_total	counter	—	Keys whose newest row version a batched CAS prepare saw only as a stamp from a replica that sends no row bodies, so the coordinator read the row from that replica (at most 2 MB of rows per batch; the rest run per-key). The replica that ships bodies — the coordinator's own, else the first peer — could not supply that version: it lags, or its prepare failed (a busy, write-locked engine, or unreachable), which fetches every key of the batch. Near zero on a healthy cluster; a steady rate means a lagging or busy body replica.
skaidb_cluster_fk_guard_waits_total	counter	—	Backoff rounds foreign-key writers spent waiting on the other side of a constraint (a child insert behind a parent delete's fence, or a parent delete draining child claims in flight). A steady rate means hot parent rows contended from both sides.
skaidb_cluster_fk_guard_stale_resets_total	counter	—	Foreign-key guard state (a claim or a fence) reset because its holder stopped heartbeating for the 10 s grace period — a coordinator died mid-statement. A steady rate on a healthy cluster is a bug; the constraint stays enforced, but the statement that died may have left an orphan VALIDATE CONSTRAINT finds.
skaidb_cluster_index_settle_waits_total	counter	—	Index-ordered reads (ORDER BY <indexed col> LIMIT, sorted top-k, agreed windows) that waited for the maintenance queue to index the rows this node had acknowledged before them — read-your-writes through an index's order. Lookups, mutations and index-only counts never wait: they are completed from the un-indexed rows. Each wait lasts the queue's lag (milliseconds; at most 250 ms). Follows the write rate on indexed tables; nothing to do.
skaidb_cluster_index_settle_timeouts_total	counter	—	Index-ordered reads that ran before the queue had caught up: it was more than 2048 tasks deep (no wait attempted) or did not drain within 250 ms. Their order can trail the table by the queue's lag (lookups and mutations never do — they are completed from the un-indexed rows). Expected during a bulk load; a steady rate otherwise means the maintenance applier is behind — check skaidb_cluster_maint_queue_bytes and the disk.
skaidb_cluster_index_tail_keys	gauge	—	Rows in the un-indexed tail on this node, over every table: stored (and acknowledged) while their index maintenance is still queued. Follows the maintenance queue (a few to a few hundred under writes; up to the queue bound during a bulk load) and returns to 0 when the queue is idle. A value that stays above 0 on an idle node is a leaked key — index reads on that table keep paying for it until a restart; report it.
skaidb_cluster_index_maintenance_retrying	gauge	—	Index maintenance tasks whose apply failed (an index or vector write refused by a full disk or an I/O error) and that wait for a retry — a second after the failure, then with a doubling backoff up to a minute. Each stays in its table's un-indexed tail (index reads and counts stay complete) and holds the table's applier watermark, and with it the flushed WAL segments the watermark gates, below its stamp until it applies. Normally 0; non-zero for more than a few minutes means the disk is still refusing index writes (see skaidb_cluster_wal_segments_held).
skaidb_cluster_index_maintenance_retry_bytes	gauge	—	Row bytes (old + new versions) the index maintenance tasks waiting for a retry hold. Bounded at 256 MB — a task failing past it is given up at once (skaidb_cluster_index_maintenance_abandoned_total) — and counted with skaidb_cluster_maint_queue_bytes against the queue's 256 MB bound, past which writes on this node wait. Normally 0; it climbs while index writes keep failing (a full disk).
skaidb_cluster_index_maintenance_abandoned_total	counter	—	Failed index maintenance tasks given up after 30 minutes of retries, so their table's WAL is not held without bound, or at once when the tasks waiting for a retry already hold 256 MB. Each is a row whose index entries may be missing or stale on this member: the log line (skaidb: index maintenance of a row of <table> given up …, or skaidb: <n> failed index maintenance task(s) given up (last on <table>) … at most once a minute for the bound) names the table; run CHECK INDEX ALL ON <table> REPAIR on that member. Alert on any increase.
skaidb_cluster_wal_segments_held	gauge	—	Flushed WAL segments of row tables kept on disk until their table's applier watermark passes them, over every table, refreshed about once a second on a cluster member and read at every scrape on a standalone server (which exports it too). A handful while writes flow on a member (the watermark trails the wall clock by about 10 s and stays below queued maintenance, uncommitted full-text writes, inline writes whose index entries are not synced yet, and vector writes applied below their index's last snapshot until the next vector checkpoint — up to 10 minutes after a burst of hints or repair pushes into a table with a vector index), 0 on a standalone server, whose index maintenance is inline; a count that keeps growing is a watermark held down — a failed maintenance task retrying (skaidb_cluster_index_maintenance_retrying) or a search index that cannot commit — and the disk it costs grows with the table's writes until it clears.
skaidb_cluster_index_union_incomplete_total	counter	—	Index-bounded reads that fell back to an unbounded scan because a peer could not contribute its candidate keys (unreachable, write-locked, breaker-open). Answers stay correct; a rising count means they are being served the slow way — look at peer health.
skaidb_cluster_index_count_disagreements_total	counter	—	Index-only count(*)s at QUORUM or ALL (this member coordinating) whose consulted replicas counted differently from their own indexes, by the same amounts over three rounds (the last 5 ms after the second) while none of the counts moved. With cluster.index_count_check = "log" (the default) the coordinator's own count was served; with enforce the rows' count (the row-merging quorum read), reused while the table takes no write. A count is checked at most once per filter and level every 5 s (under log, once per table and level whatever the filter), so this counts checks, not statements. Writes in flight between replicas are not counted: the coordinator's own count is served, as a count under writes always was, when its table took a write while the replicas counted or when any count moves from one round to the next (a range being written), and answers that agree on a later round are the count. So drift on a table under a constant write stream shows once the table has a quiet moment. A non-zero rate therefore means some member's secondary index is out of step with its rows (entries missing for rows it holds, or kept for values they left) — or, rarely, a replica that stayed one write behind for the length of the three rounds. The log line index-only count on <table> (index '<name>', …) disagrees between replicas (at most one per table a minute) names the index and each member's count (and, under enforce, the rows' count); CHECK INDEX <name> on each member finds the drifted copy and CHECK INDEX <name> REPAIR fixes it.
skaidb_cluster_point_read_stamp_confirms_total	counter	—	Keys of a batched quorum re-read (sorted top-k candidate unions, index point-read sets) a peer confirmed by version stamp alone — its copy was no newer than the coordinator's, so no row body crossed the wire.
skaidb_cluster_point_read_newer_bodies_total	counter	—	Keys of a batched quorum re-read where a peer held a newer (or same-stamp, different-bytes) version and shipped the body; the winner is read-repaired as usual. A high share against …_stamp_confirms_total means the coordinator's replica is lagging.
skaidb_cluster_window_rows_stamped_total	counter	—	Rows of an agreed-window quorum read (a QUORUM/ALL ordered LIMIT read on a fully replicated table, as coordinator or as a consulted peer) whose version this member took from the walk's own read of the row — one read and one fingerprint pass per row, no second read.
skaidb_cluster_window_row_rereads_total	counter	—	Rows of an agreed-window quorum read this member read a second time, versioned, to stamp — a walk branch that reads no stamp: a primary-key-ordered ascending walk, or a full-scan top-k (an order column with no index). Zero for index-ordered walks, primary-key descending walks and primary-key point reads.
skaidb_cluster_agreed_windows_multikey_total	counter	—	Multi-key ordered LIMIT reads (ORDER BY d DESC, id DESC LIMIT k and the like, QUORUM/ALL, fully replicated table) this member answered as coordinator from agreeing windows that carried the leading key's boundary tie run — no candidate re-read. Declines (disagreement, a tie run over 256 rows past k, a peer predating the tie window) take the overfetched sorted top-k and are not counted.
skaidb_cluster_window_tie_rows_total	counter	—	Rows those multi-key agreed windows held past their LIMIT through the boundary tie run (rows tied with the k-th on the leading key). Near zero when the leading key is fine-grained (a timestamp); a high ratio against …_agreed_windows_multikey_total means a coarse leading key whose tie runs each window walks and ships.
skaidb_cluster_window_tie_overflows_total	counter	—	Multi-key agreed windows the coordinator's own walk declined because the boundary tie run passed its cap (256 rows past the LIMIT); each such read walked that window and then took the overfetched sorted top-k. A steady rate means a leading key too coarse for the window (a status or day column): add a finer leading key, or expect sorted top-k cost on those reads.
skaidb_engine_lock_longest_hold_ms	gauge	—	The longest current hold of the engine lock, in milliseconds (0 when nobody holds it). Scrapeable while the lock is held — /metrics never waits on it. On a standalone server the value is the probes' own observation: milliseconds since the first probe (a scrape, /status, or the 5 s background probe) found the lock write-held. Normal holds are milliseconds; a shutdown flush or a large DDL can hold it a minute or two. Minutes with waiters piling up is a wedge: the log line ENGINE LOCK HELD … holders and waiters: (after 60 s, then every 5 min) names the holding thread, its call site and its kernel wait state.
skaidb_engine_lock_waiters	gauge	—	Threads waiting for the engine lock: blocked on it, or in a bounded wait (internode handlers and the probes re-try on a backoff and give up at their bound — a replica read answers busy after 1.5 s). A few under a write load is normal; a count that keeps growing together with skaidb_engine_lock_longest_hold_ms is a long hold.
skaidb_engine_lock_wait_ms_bucket	counter	kind, le	Blocked engine-lock acquisitions by how long they waited (ms, cumulative per le bucket 1/10/100/1000/5000/10000/+Inf; kind = read or write). This is the convoy the longest-hold gauge cannot show: a restarted member's first statements waited 5–127 s behind holds under 10 s each. rate of the high buckets during a restart = the catch-up starving foreground statements.
skaidb_engine_lock_wait_ms_sum / skaidb_engine_lock_wait_ms_count	counter	kind	Summed wait milliseconds and count of blocked acquisitions (mean wait = sum / count).
skaidb_cluster_wedged_refusals_total	counter	—	Internode requests this member answered "busy: engine lock wedged" at once — its engine lock was held more than 10 s with waiters queued — so peers used other replicas instead of waiting out a timeout on it. Rising = this member is out of service (see skaidb_engine_lock_longest_hold_ms).
skaidb_cluster_maint_queue_bytes	gauge	—	Row bytes (old + new versions) held by the deferred index-maintenance queue — secondary, vector and full-text index work applied after a write is acknowledged. Bounded at 256 MB together with skaidb_cluster_index_maintenance_retry_bytes: past it, writes on this node wait for the applier. Pinned near the bound means the applier is behind (full-text re-indexing of large rows, a slow disk) — slow the writers.
skaidb_cluster_maint_queue_tasks	gauge	—	Deferred index-maintenance tasks not yet applied, including the applier's batch and producers waiting to enqueue — the queue's depth in tasks, next to its bytes above. The channel holds at most 8,192 tasks; a sustained count near or above that (or the bytes near their bound) means the applier is behind and replicated writes wait for room. 0 on an idle member.
skaidb_cluster_maint_backpressure_waits_total	counter	—	Writes that waited because the maintenance queue, with the failed tasks waiting for a retry, was past its byte bound. Backpressure working as designed; a steady climb means the write rate exceeds what this node can index.
skaidb_memory_shedding_writes	gauge	—	1 while the node is shedding writes under memory pressure (rejecting new writes so it can drain instead of being OOM-killed). Alert on sustained 1.
skaidb_memory_used_bytes / skaidb_memory_limit_bytes	gauge	—	Sampled memory usage vs. the node's limit (cgroup when set, else system RAM); shedding starts at 85% and clears at 70%.
skaidb_memory_stall_pct	gauge	—	Share of the last 10 s in which every task of the node's cgroup was stalled on memory (PSI full avg10, rounded %). At 10% or more the node releases memtables and block caches even below its own thresholds, and logs memory pressure from outside this process once per 10 minutes: the host is reclaiming from a container whose limit exceeds the host's real headroom. Alert when it stays above 10%.
skaidb_memory_anon_bytes / skaidb_memory_file_bytes	gauge	—	Cgroup anon vs file split. The production memory-wedge signature is anon ratcheting up while file collapses toward zero — graph these together.
skaidb_alloc_allocated_bytes / skaidb_alloc_resident_bytes / skaidb_alloc_retained_bytes	gauge	—	jemalloc live heap / resident pages / OS-unreturned address space. resident − allocated ≈ fragmentation + unpurged dirty pages: distinguishes "something holds memory" from "the allocator won't give it back".
skaidb_cluster_replication_lag_ms	gauge	peer	Approx. ms between this node's HLC frontier and the latest write it has confirmed peer applied.
skaidb_cluster_peer_clock_offset_ms	gauge	peer	The peer's wall clock relative to this node's, in ms (positive = the peer is ahead), measured NTP-style on the periodic stats probe (every 5 s, exact to half the round trip). Absent for a peer that reports no clock (an older binary), is down or is wedged. Panel against cluster.max_clock_skew_ms; see CLUSTER_CONSISTENCY.md "Clock synchronisation".
skaidb_cluster_clock_offset_ms	gauge	—	This node's wall clock relative to the median of its measured peers (ms, negative = behind). Absent with no measured peer.
skaidb_cluster_clock_skewed	gauge	—	1 while this node is beyond cluster.max_clock_skew_ms against a majority of its measured peers (two consecutive probes). With cluster.clock_skew_action = refuse_writes (default) it refuses client writes meanwhile and plain /ready is 503; /ready?strict=1 is 503 in either mode. Page on 1.
skaidb_cluster_clock_writes_refused	gauge	—	1 while this node refuses every client write with a clock skew: error, for either reason: its measured clock is beyond the bound with refuse_writes, or its node clock (HLC) is ahead of its wall clock beyond the bound. /ready answers 503 in both modes meanwhile and /status clock_skew.writes_refused is true. Page on 1.
skaidb_cluster_clock_hlc_ahead_ms	gauge	—	How far this node's clock (HLC) is ahead of its wall clock (ms) when that exceeds the bound (+100 ms) — what a forward clock jump leaves once it is corrected; client writes are refused meanwhile. 0 otherwise. It drops to 0 within seconds when nothing the node stored carries a stamp from the jump (the clock is lowered, see …_hlc_rewinds_total); otherwise it counts down as wall time catches up (/status clock_skew.hlc_stored_ahead_ms > 0 tells the two apart).
skaidb_cluster_clock_hlc_rewinds_total	counter	—	Times this node lowered its clock (HLC) back to its wall clock after a corrected clock jump left it ahead with nothing stored at the stamps. Each is a clock jump on this host that NTP then corrected — look at why it jumped.
skaidb_cluster_clock_measured_peers	gauge	—	Peers with a clock-offset reading in the last minute. 0 = no skew verdict is possible (standalone, or every peer old, down or wedged).
skaidb_cluster_clock_skew_rejections_total	counter	—	Stamps from peers (replicated and hinted writes, repair rows, read-repair rows, DDL versions, consensus ballots and commit stamps) this node rejected because they were more than the bound (+100 ms) ahead of its own clock. Non-zero means some member's clock is off — the error log names the kind of stamp; the sender's write failed its quorum loudly.
skaidb_cluster_clock_skew_write_refusals_total	counter	—	Client writes this node refused with a clock skew: error: its own clock was beyond the bound (refuse_writes), or its node clock had been left in the future by an earlier skew.
skaidb_cluster_clock_future_stamps_ignored_total	counter	—	Remote stamps (read winners, consensus floors) the node clock declined to merge because they led the wall clock by more than the bound — the drag that would otherwise pin this node's own stamps into the future.
skaidb_cluster_peer_requests_total	counter	—	Internode RPCs issued by the coordinator.
skaidb_cluster_peer_errors_total	counter	—	Internode RPCs that errored or timed out.
skaidb_cluster_peer_errors_by_peer_total	counter	peer, kind	Transport-level internode call failures per peer: breaker (refused by this node's open circuit for that peer, i.e. fast-fails during an outage), timeout (connect/read/write), transport (refused, reset, name resolution, wire). Attributes peer_errors_total to a peer — the unlabelled total also counts reply-level failures (bad or busy replies).
skaidb_cluster_breaker_probes_total	counter	result	Circuit-breaker re-tests of suspended peers by this node's prober: ok closed the circuit, failed re-armed the cooldown. A steady failed rate is a peer still unreachable; client statements no longer pay for these trials.
skaidb_witness_last_seen_age_seconds	gauge	witness	Seconds since a registered witness last heartbeat its registry row.
skaidb_witness_oldest_sync_age_seconds	gauge	witness	Seconds since the LEAST caught-up mirrored table was synced. Alert on this one (e.g. > 3600).

Witness staleness: alert on sync age, not last-seen. A witness heartbeats its registry row every cycle, so last_seen_age stays small even while one table is stuck — a witness once failed every cycle for three days with a perfectly healthy heartbeat. oldest_sync_age is derived from the per-table watermarks and is the honest signal. Both are emitted by members (a witness is not a member and does not emit them about itself); the registry read behind them is throttled to once a minute. Absent until a witness registers, and oldest_sync_age is absent until its first cycle heartbeats a watermark. Set the threshold against the witness's duty_pct, not its interval_secs — at 25% a long cycle rests roughly three times as long as it ran, so gaps well past the interval are normal. The same two clocks are on GET /status as seen_age_secs / sync_age_secs per witness (see CLUSTER_WITNESSES.md).

These anti-entropy and quorum signals are correctness-critical — without them, read-repair, hinted handoff, and quorum failures are invisible.

Reading replication health per peer. skaidb_cluster_hints_pending_peer is the exact backlog — writes this node has buffered for a peer it couldn't reach. skaidb_cluster_replication_lag_ms is an estimate: it only advances a peer's baseline when a write is confirmed to it, so it climbs while a peer is unreachable and falls once hinted handoff/anti-entropy catch it up. A peer with no confirmed write yet (freshly added, or down since startup) is absent from replication_lag_ms — rely on hints_pending_peer and the reachable flag in \cluster for those. Both are emitted per current peer (ring ∪ configured seeds).

The same per-peer backlog and lag are surfaced without a scraper: GET /status carries a peers array (id, in_ring, in_config, hints_pending, lag_ms), and the UI members panel renders a backlog column (buffered writes owed to that node — nonzero is flagged) and a lag column (that node's replication lag), so you can see at a glance how far behind each node is.

The advisor

SHOW ADVICE turns the signals this page describes — the gauges in the ops shortlist, node_stats, the slow log and the maintenance lines in the server log — into findings with an audience (a database's own advice for its users; host and suspected-bug findings for administrators). See ADVISOR.md.

The node_stats table (replicated host statistics)

With observability.node_stats (default on), every node INSERTs its own host statistics — CPU, load, memory, disk I/O and space, uptime, restart count, and cgroup OOM kills — into the replicated node_stats table every node_stats_interval_secs (default 1 s, live-mutable): one row per node, keyed on the node id, stamped with the sample time (ts, epoch ms). The row replicates like any write, so any member serves the whole cluster's picture from a local read, and it is plain SQL:

SELECT node, ts, mem_used_bytes, restarts, oom_kills FROM node_stats;
-- memory-ramp composition (anon vs file vs allocator view):
SELECT node, mem_anon_bytes, mem_file_bytes,
       alloc_allocated_bytes, alloc_resident_bytes, alloc_retained_bytes
FROM node_stats;

oom_kills counts kernel OOM kills in the node's cgroup since the last RESET OOM COUNTER [ON '<node>'] (ADMIN) — or, absent one, since container boot. After an incident is investigated, reset it so the next nonzero reading is unambiguous fresh signal.

The UI's stats NODES table reads this (falling back to live probes for members without a fresh row, e.g. mid rolling-upgrade) and shows each row's age — a silently struggling node dims and its age climbs, instead of one missed probe flapping a live node to "unreachable". Node restarts log their start number, and when the cgroup's OOM-kill count advanced since the previous start, the log says the prior run likely died to the OOM killer. Probe-loss/recovery transitions and circuit-breaker events are logged on the coordinator as well.

Logs

Audit/query/login logs are written to stderr in human-readable text by default. Set observability.log_format = "json" to emit one JSON object per line (event, elapsed_ms, error, sql, …) so a log agent can parse them reliably. Query logs are masked (literals → ?) unless query_log_masked is disabled. A bounded, masked sample of recent slow queries is available at POST /admin/slow.

Log files

By default logs go to stderr. Set observability.log_file to a path to write all audit logs to a file instead (created if absent, appended otherwise):

[observability]
log_file = "/var/log/skaidb/audit.log"

Each log category can be split into its own file with a per-category override; an empty override falls back to log_file, and an empty log_file falls back to stderr:

Key	Stream
observability.query_log_file	executed-statement log
observability.slow_query_log_file	slow-query log
observability.error_log_file	error log
observability.login_log_file	login/auth log

[observability]
log_file = "/var/log/skaidb/audit.log"   # everything not overridden below
error_log_file = "/var/log/skaidb/error.log"
slow_query_log_file = "/var/log/skaidb/slow.log"

Categories pointed at the same path share one file handle, so their lines interleave safely. File sinks are write-behind: executed-statement ([query]) lines are buffered and flushed within 250 ms, while slow-query, error and login/auth lines are flushed as they are written; every buffer is also flushed on clean shutdown (SIGTERM/SIGINT) and before a config set swaps a log file. A tail -f of the query log therefore lags a statement by at most a quarter second, and a line is always written whole. All of these keys are runtime-mutable (config set, --* flags, and SKAIDB_*_LOG_FILE env vars), and a path that can't be opened falls back to stderr with a one-line warning rather than failing startup.

Every log line carries a timestamp: text-format lines (and every operational skaidb: line in the server log) are prefixed with an ISO-8601 UTC instant (2026-07-20T18:42:13.123Z …); JSON-format lines (observability.log_format = "json") carry it as a ts field instead, so each line stays independently parseable.

Query/slow-query/error lines carry the execution context: the authenticated user, the session database, and the access surface — via=driver (binary-protocol/driver connections), rest, es, ui, prom (PromQL evaluations, which log like queries), bi (the BI app on a BI node), or internal (background/system statements):

2026-07-20T18:42:13.123Z [query] 3ms user=agencik db=agencik via=driver SELECT … WHERE id = ?
2026-07-20T18:42:14.001Z [query] 1ms user=pi_air_quality db=pi_air_quality via=prom promql query_range avg(pm25{}[?m])

JSON format carries the same as user/db/via fields.

REST request activity

• skaidb_rest_requests_total{path="query"|"insert"|"es"|"prom"|"ui"|"ops"|"admin"|"other"} — REST requests served, per path class, timed end to end.
• skaidb_rest_request_duration_us_total{path=…} — total serving time in microseconds; divide by skaidb_rest_requests_total for the average response time (the overview tab's REST-activity card shows exactly this).