Grafana with skaidb

skaidb speaks enough of the Prometheus wire protocol that Grafana's built-in Prometheus datasource works against a node directly — no exporter, no sidecar. Ingestion comes in over remote_write, queries go out over the Prometheus HTTP query API, both on the REST port (default 7080).

1. Point a Prometheus datasource at skaidb (recommended)

In Grafana: Connections → Data sources → Add data source → Prometheus:

URL: http://<node>:7080
Auth: enable Basic auth and use a skaidb account when the node runs with authentication (the query API requires Select on the metrics table; a database-level grant satisfies it). Without auth enabled, leave it off.

Scoping a datasource to a database — or to any time-series table

The bare URL serves the metrics table in the default database. A path prefix on the datasource URL scopes the whole API (Grafana just appends /api/v1/… to the base URL, so this works with the stock Prometheus datasource):

http://<node>:7080/db/<database> — that database's metrics table (same remote_write semantics; remote_write to /db/<database>/api/v1/write ingests there too).
http://<node>:7080/db/<database>/table/<table> — any time-series table: the table's fields become the metric names. E.g. a table air_quality (SERIES KEY (sensor)) with fields pm25/co2 serves PromQL like pm25{sensor="pi1"} and rate(co2[5m]).

The permission check follows the scope: Select on that table (a grant on its database satisfies it) — so a database-scoped account works against its own data without any grant in the default database. A 403 from the API names the exact table and database it checked.

Grafana health-checks the datasource via /api/v1/status/buildinfo and /api/v1/metadata — both answered. Queries hit:

Endpoint	Purpose
`GET/POST /api/v1/query`	instant queries
`GET/POST /api/v1/query_range`	dashboard panels
`GET /api/v1/labels`, `/api/v1/label/<name>/values`	template variables
`GET /api/v1/series`	series lookup

All of them evaluate over the metrics time-series table that remote_write ingests into (the metric name is the name label).

Getting data in

Point any Prometheus-compatible shipper (Prometheus itself, the Grafana Agent/Alloy, vmagent, …) at the node:

# prometheus.yml
remote_write:
  - url: http://<node>:7080/api/v1/write
    basic_auth:            # when the node requires auth
      username: admin
      password: ...

Samples land in the auto-created metrics table (label columns intact) and are queryable from SQL too:

SELECT rate(value) FROM metrics WHERE name = 'http_requests_total' AND job = 'api';

Supported PromQL

Instant selectors with =/!=/=~/!~ label matchers (regex forms anchored, Prometheus-style), offset and the @ modifier (fixed unix time, start(), end()), rate / increase / delta / irate / idelta and avg/min/max/sum/count/last_over_time over range selectors ([5m] — the *_over_time family is what Grafana's Metrics Drilldown tiles use), the window-analytics family (present_over_time, absent_over_time, changes, resets, deriv, predict_linear(m[w], t), stddev/stdvar/mad_over_time, quantile_over_time(φ, m[w]) — the alert-rule staples), sum/avg/min/max/count/stddev/stdvar/group [by|without (...)], count_values("label", v), quantile(φ, v) (the drilldown's "Standard deviation" / "Percentiles" previews) and topk(k, v) / bottomk(k, v), vector arithmetic (+ - * / % ^ with PromQL precedence — ^ right-associative and tightest, unary minus between ^ and * — one-to-one matching by default, on(...)/ignoring(...) to control the match key, and group_left(...)/group_right(...) for many-to-one joins like the classic metric * on(instance) group_left(version) build_info), comparison operators (== != > < >= <=, filtering by default, 0/1-valued with bool), the set operators and / or / unless (with on/ignoring; the drilldown's extreme-values filter emits <expr> and <expr> > -Inf), subqueries [range:step] feeding any range function (max_over_time(rate(m[5m])[1h:1m]); an omitted step defaults to 60s, inner steps epoch-aligned like Prometheus), Inf/NaN literals, number-only expressions (the 1+1 datasource health check), histogram_quantile, label_replace / label_join, sort / sort_desc (instant queries), absent(v) (labels derived from the selector's = matchers — the standard missing-data alert), and timestamp(<selector>) / time() (the "last reading" / staleness stat patterns: max(timestamp(m)) * 1000, time() - max(timestamp(m))). Trailing commas in matcher blocks are accepted, Prometheus-style. The drilldown's full query catalog is pinned by the grafana_promql_compatibility test. Typical dashboard panels — sum by (job) (rate(http_requests_total[5m])), histogram_quantile(0.9, sum by (le) (rate(req_bucket[5m]))) — work as-is.

The per-sample math family (abs, ceil, floor, round, clamp/clamp_min/clamp_max, sqrt, exp, ln, log2, log10, sgn), the UTC calendar functions (minute, hour, day_of_week, day_of_month, day_of_year, days_in_month, month, year — the no-argument forms too), and vector()/scalar() are all supported; function names only bind when followed by (, so a metric that shares a name still selects. Scalar function arguments — quantile's φ, topk's k, clamp bounds, predict_linear's seconds — accept any constant arithmetic expression, so alert rules copied from the Prometheus docs (predict_linear(m[6h], 24 * 3600)) work verbatim.

rate/increase/delta use Prometheus's exact window-extrapolation algorithm (extrapolatedRate), and non-finite samples render as NaN/+Inf/-Inf API values like Prometheus — so panel values agree with a real Prometheus on the same data, not just approximately. Aggregation grouping accepts both positions (sum by (a) (x) and sum(x) by (a)), and a scrape-time literal name label (systemd units, cooling devices — stored as exported_name since the metric name owns name in storage) renders back as name, so {{name}} legends work.

Verified panel-by-panel: the full Node Exporter Full dashboard (Grafana #1860 rev 101, 284 queries) evaluated against a real Prometheus holding identical data (Prometheus scraped node_exporter and remote_wrote to skaidb): 253/253 panels with data matched exactly (≤0.1% relative, most bit-identical); the remaining 31 were empty on both sides (hardware metrics the test host lacks).

Out of scope: native histograms (no native-histogram storage), trigonometric functions, atan2. Panels using those need the fallback below.

Monitoring skaidb itself

The node's own operational metrics are a Prometheus scrape at GET /metrics (unauthenticated, gauge names under skaidb_*) — including per-node host system stats (skaidb_host_*: CPU%, cgroup-aware memory, process RSS, disk IO counters, data-dir disk space; see METRICS.md), so basic host dashboards need no separate node_exporter. Scrape it with your regular Prometheus and dashboard it like any other target — or remote_write that Prometheus back into skaidb and dashboard skaidb from skaidb. There is also a built-in web UI with live stats (incl. a per-node CPU/RAM/disk table) at http://<node>:7080/ui (UI.md).

Alerting

Grafana-managed alert rules work against the datasource like any other Prometheus source: the rule's query runs through /api/v1/query, and Grafana does the evaluation, state tracking and notification. The functions alert rules lean on — absent, predict_linear, quantile_over_time, histogram_quantile, the comparison operators — are all in the supported subset.

Rules worth starting from, using the node's own skaidb_* metrics (scrape /metrics into Prometheus, or remote_write it back into skaidb and alert on skaidb from skaidb):

# A node stopped reporting at all — the classic missing-data alert.
absent(skaidb_up)

# Errors are being returned to clients.
rate(skaidb_query_errors_total[5m]) > 0

# Tail latency. `type` separates ddl/query/mutation; drop the `by (type)`
# to alert on the whole node.
histogram_quantile(0.99,
  sum by (le, type) (rate(skaidb_query_duration_seconds_bucket[5m]))) > 2

# The data directory will fill within a day at the current rate.
predict_linear(skaidb_host_disk_available_bytes[6h], 24 * 3600) < 0

# The workload manager left its normal state — shedding or rejecting.
skaidb_iwm_state > 0

Set Pending period to at least two scrape intervals so a single missed scrape does not page anyone. absent() derives its labels from the selector's = matchers, so pin the node there (absent(skaidb_up{node="skai1"})) if you want one alert per node rather than one for the whole fleet.

A starter dashboard

Panels that answer "is it healthy" without any tuning. Every metric below is on /metrics; the full catalogue is in METRICS.md.

Panel	Query
Query rate	`sum by (type) (rate(skaidb_queries_total[5m]))`
Error rate	`sum(rate(skaidb_query_errors_total[5m]))`
p99 latency	`histogram_quantile(0.99, sum by (le) (rate(skaidb_query_duration_seconds_bucket[5m])))`
In-flight queries	`skaidb_queries_in_flight`
Connections	`skaidb_connections_active`
Rows read / written	`rate(skaidb_rows_returned_total[5m])`, `rate(skaidb_rows_written_total[5m])`
Block-cache hit ratio	`rate(skaidb_block_cache_hits_total[5m]) / (rate(skaidb_block_cache_hits_total[5m]) + rate(skaidb_block_cache_misses_total[5m]))`
Disk used by data	`skaidb_storage_disk_bytes`
Disk free on the volume	`skaidb_host_disk_available_bytes`
Memory (process RSS)	`skaidb_host_rss_bytes`
CPU	`skaidb_host_cpu_percent`
Compaction throughput	`rate(skaidb_storage_compaction_bytes_total[5m])`

skaidb_build_info and skaidb_node_info carry version and node identity as labels, so the usual metric * on(instance) group_left(version) skaidb_build_info join works for stamping panels with the running version.

2. Fallback: SQL over REST (Infinity / JSON API datasource)

For queries outside the PromQL subset — or any non-timeseries table — use a JSON-over-HTTP datasource (e.g. the Infinity plugin) against the SQL gateway:

Method: POST, URL: http://<node>:7080/query
Body: the SQL, either raw text or {"sql": "SELECT ...", "db": "mydb"}
Auth: HTTP Basic, same accounts as everything else
Response shape: {"columns": [...], "rows": [[...], ...]} — in Infinity set Format: table, Rows selector: rows, and map columns by index.

Time-series SQL (docs/TIMESERIES.md) gives you windowed aggregates the PromQL subset lacks:

SELECT time_bucket(1m, ts) AS t, avg(value)
FROM metrics
WHERE name = 'http_requests_total' AND ts >= now() - 1h
GROUP BY t ORDER BY t;

Notes

HTTP keep-alive is supported: a client that reuses its connection (Grafana's HTTP datasources do) pays the TCP/TLS handshake once, not per panel per refresh. Connection: close and HTTP/1.0 still behave classically; large row results stream chunked and close.
TLS: terminate at a proxy in front of the REST port; Basic auth wants TLS on untrusted networks.
Timestamps follow the Prometheus HTTP API conventions (float seconds; sample values as strings) — Grafana handles this natively.