Vector search (HNSW)

skaidb can store embeddings and run approximate nearest-neighbor (ANN) search over them with an in-memory HNSW index, including filtered search ("nearest neighbors where …"). This is the index family behind semantic search / RAG / recommendations.

Status: distributed (sharded scatter-gather), in-memory, persisted as a snapshot and reloaded on open (full rebuild only when no usable snapshot exists). Both index creation and the kNN query have SQL syntax (NEAREST, below). See limitations at the end.

Storing vectors

Vectors are ordinary document fields holding an array of numbers — SQL supports array literals:

CREATE TABLE docs (PRIMARY KEY (id));
INSERT INTO docs (id, cat, embedding) VALUES (1, 'news', [0.12, -0.04, 0.91, ...]);

Creating an index (SQL — works cluster-wide)

CREATE VECTOR INDEX docs_emb ON docs (embedding) DIM 768 USING cosine
DROP   VECTOR INDEX docs_emb

DIM (the vector dimension) is required; USING is cosine (default), l2, or dot. This is broadcast DDL: every node builds and maintains an HNSW over its own shard. The index is maintained automatically on INSERT/UPDATE/ DELETE (a replace soft-deletes the old vector and inserts the new one).

The DDL acks at schema-apply; each node backfills existing rows in paged background work (like secondary indexes). While a node is backfilling, SHOW INDEXES reports local = building there, and searches against the index answer "vector index is rebuilding — retry shortly" rather than silently serving a partial graph. On a single-node/embedded database the backfill completes before the DDL returns.

Quantization (QUANTIZED)

CREATE VECTOR INDEX docs_emb ON docs (embedding) DIM 768 USING cosine QUANTIZED

QUANTIZED cuts the graph's RAM: instead of exact f32 components, each in-RAM vector is stored int8 scalar-quantized (a per-vector scale, x_i ≈ scale·q_i) — 4× less vector memory and snapshot payload (measured: a 2000×64 index's snapshot shrank 824 → 456 KB; at 768 dims the vector payload dominates the graph, so the whole-index saving approaches 4×). The graph searches over the quantized vectors, then every query rescores: it over-fetches 4× the requested k, re-reads the candidates' exact f32 vectors from their rows, recomputes the true metric distance, and returns the best k — so _distance (and the ES _score derived from it) is always the exact distance, and ranking quality is largely recovered even where the quantized graph ordering is slightly off. The cluster path rescores at the coordinator on the rows it already re-reads at read consistency.

Constraints: a build-time choice — DROP + re-CREATE to change (the snapshot format differs: SKHNSW02 vs SKHNSW01; a mismatched snapshot falls back to a rebuild). Not combinable with EMBED — a managed index stores only the text in the row, so there is no exact vector to rescore against. Use it when vector RAM is the constraint; skip it for small indexes where the 4× oversampled candidate re-read isn't worth the memory saved.

Managed embeddings (EMBED, semantic_text)

A managed vector index embeds a TEXT column for you — the ES semantic_text workflow. Configure an embeddings endpoint in [inference], then:

CREATE VECTOR INDEX docs_sem ON docs (body) EMBED DIM 768;
SELECT id FROM docs NEAREST (body, 'natural language query', 10);  -- query auto-embedded

EMBED makes path a text column: on write skaidb embeds it via the provider (rather than reading a pre-computed vector array), and a string NEAREST query is auto-embedded. DIM must match the model; the index errors at create if [inference] is off or the dimension disagrees.

How the embedding model is used

skaidb never runs an ML model in-process — the single static binary stays. An "embedding model" is always an external HTTP model server that skaidb calls as a client. This keeps the database free of Python/CUDA/model weights and lets you point at whatever provider you run.

The wire contract (OpenAI embeddings API). For a batch of texts skaidb sends:

POST <inference.url>
Authorization: Bearer <inference.api_key>     # only when api_key is set
Content-Type: application/json

{ "model": "<inference.model>", "input": ["text one", "text two", ...] }

and expects the OpenAI-shaped response (order preserved, one entry per input):

{ "data": [ {"embedding": [0.01, -0.02, ...]}, {"embedding": [...]} ] }

Any server that speaks this shape works — OpenAI, Azure OpenAI, a local text-embeddings-inference (TEI) server, Ollama (/v1/embeddings), Cohere/others behind an OpenAI-compatible proxy, or your own endpoint. The returned vector length must equal the index DIM (skaidb validates and rejects a mismatch).

Configuration ([inference] block, see config/skaidb.toml): every key below can also be set (or overridden) via a SKAIDB_INFERENCE_<KEY> environment variable — e.g. SKAIDB_INFERENCE_URL, SKAIDB_INFERENCE_API_KEY, SKAIDB_INFERENCE_ENABLED=true, SKAIDB_INFERENCE_RERANK_URL — applied after the config file at startup (values are type-checked; an unknown or ill-typed variable fails startup loudly). Use this to keep secrets and per-host endpoints out of a shared config file.

Example — OpenAI:

[inference]
enabled = true
url = "https://api.openai.com/v1/embeddings"
model = "text-embedding-3-small"
dim = 1536
api_key = "sk-..."
tls_verify = "system"   # or "ca" + a tls_ca bundle until system roots land

Example — a local TEI/Ollama server (no auth, plaintext):

[inference]
enabled = true
url = "http://127.0.0.1:8080/v1/embeddings"
model = "BAAI/bge-base-en-v1.5"
dim = 768

The rerank endpoint (RERANK)

The same [inference] block also configures the cross-encoder reranker behind the SQL RERANK clause and the ES text_similarity_reranker retriever (see SEARCH.md). It is a separate endpoint (rerank_url) sharing api_key, timeout_secs, and the TLS settings. The wire contract (Cohere/Jina rerank API; the candidate texts are sent under both documents and texts so a TEI /rerank route works too):

POST <inference.rerank_url>

{ "model": "<model>", "query": "<query text>",
  "documents": ["candidate one", ...], "texts": ["candidate one", ...] }

expecting either shape (order-independent, index refers to the request order; relevance_score/score, higher = more relevant):

{ "results": [ {"index": 0, "relevance_score": 0.98}, ... ] }   // Cohere/Jina
[ {"index": 0, "score": 0.98}, ... ]                            // TEI

Reranking is invoked only by queries that say RERANK — never at ingest and never by ordinary searches — so the endpoint being down fails exactly those queries.

Model is pinned per index. The DIM you declare fixes the vector geometry; the [inference].model is the model actually called. Changing the model (or its dimension) means the stored vectors no longer match — DROP and re-CREATE the index (a fresh backfill re-embeds every row with the new model). Keep dim consistent across the config and every EMBED index.

When it runs (two moments): - Ingest — on INSERT/UPDATE of the text column, the row commits with the raw text and the vector is produced later (see below). Each node embeds its own shard's rows. - Query — a string NEAREST(text_col, 'some query', k) embeds the query string once (a single-input call to the same endpoint) and searches with the returned vector. A numeric-array NEAREST skips inference entirely.

Never blocks a write (out-of-band embedding). The write path never calls the model server. A write commits with the raw text (the source of truth); a background worker then drains a queue: it gathers a batch of pending texts under a brief lock, POSTs them to the endpoint off the engine lock, and inserts the returned vectors under a brief lock afterward. So if the model server is slow or down, rows simply stay queued and searchability lags — no write is ever blocked or failed, and the queue drains when the server returns. Each node embeds its own shard; a crash-window delta (rows written but not yet in the HNSW snapshot) is re-queued on restart, and a freshly created index backfills its existing rows the same way.

Searching (SQL)

SELECT id, _distance FROM docs NEAREST (embedding, [0.1, -0.2, 0.9], 10);
SELECT id FROM docs NEAREST (embedding, [0.1, -0.2, 0.9], 10) WHERE cat = 'news';

NEAREST (<path>, <query>, <k>) returns the k nearest rows ordered nearest-first with their distance exposed as _distance; <query> and <k> may be bind parameters. Full grammar in QUERY_SYNTAX.md.

NEAREST requires a vector index on that path — without one the query fails with no vector index on <table> (<path>) rather than falling back to a scan. Create it first (see above).

From an application

Every driver binds the query vector as a typed Array parameter over the prepared-statement path, so the floats never pass through SQL text. k binds too, and the results come back nearest-first with _distance. Placeholders differ per driver (? everywhere except Node.js and Ruby, which use $1) — see the matrix in HOWDOI.md.

SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?;

vec, k = [0.1, -0.2, 0.9], 10
cur = conn.cursor()
cur.execute("SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?",
            (vec, k, "news"))
for id_, distance in cur.fetchall():
    print(id_, distance)

const res = await client.query(
  'SELECT id, _distance FROM docs NEAREST (embedding, $1, $2) WHERE cat = $3',
  [[0.1, -0.2, 0.9], 10, 'news']);
for (const row of res.rows) console.log(row.id, row._distance);

// CheckNamedValue lets a slice through database/sql's type gate.
rows, err := db.Query(
    "SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?",
    []float64{0.1, -0.2, 0.9}, 10, "news")
defer rows.Close()
for rows.Next() {
    var id int
    var distance float64
    rows.Scan(&id, &distance)
}

Skaidb.ResultSet rs = conn.prepare(
        "SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?")
    .setObject(1, java.util.List.of(0.1, -0.2, 0.9))
    .setInt(2, 10)
    .setString(3, "news")
    .executeQuery();
while (rs.next()) System.out.println(rs.getLong("id") + " " + rs.getDouble("_distance"));

res = conn.exec_params(
  "SELECT id, _distance FROM docs NEAREST (embedding, $1, $2) WHERE cat = $3",
  [[0.1, -0.2, 0.9], 10, "news"])
res.each { |row| puts "#{row['id']} #{row['_distance']}" }

$stmt = $db->prepare('SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?');
$stmt->execute([[0.1, -0.2, 0.9], 10, 'news']);
foreach ($stmt->fetchAll() as $row) { echo $row['id'], ' ', $row['_distance'], PHP_EOL; }

using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?";
cmd.Parameters.Add(new object[] { 0.1, -0.2, 0.9 });
cmd.Parameters.Add(10);
cmd.Parameters.Add("news");
using var reader = cmd.ExecuteReader();
while (reader.Read()) Console.WriteLine($"{reader.GetInt64(0)} {reader.GetDouble(1)}");

use skaidb_proto::Response;
use skaidb_types::Value;
let vec = Value::Array(vec![Value::Float(0.1), Value::Float(-0.2), Value::Float(0.9)]);
let mut q = client.prepare(
    "SELECT id, _distance FROM docs NEAREST (embedding, ?, ?) WHERE cat = ?")?;
if let Response::Rows { rows, .. } =
    client.execute_prepared(&mut q, &[vec, Value::Int(10), Value::String("news".into())])?
{
    for row in rows { println!("{} {}", row[0], row[1]); }
}

With managed embeddings the query is a string instead of a vector — bind it the same way and the server embeds it for you:

SELECT id FROM docs NEAREST (body, ?, ?);   -- ? = 'natural language query'

Searching (API)

The SQL path above calls into the same embedded/cluster methods directly usable from Rust:

// Embedded single-node:
let hits = db.vector_search("docs_emb", &query_vec, 10, &None)?;        // (key, doc, distance)
let hits = db.vector_search("docs_emb", &query_vec, 10, &filter)?;      // filtered ANN

// Cluster coordinator (distributed): scatters to every node's local HNSW,
// merges the per-shard top-k by distance, then re-reads survivors at quorum.
let hits = node.vector_search("docs_emb", &query_vec, 10, &filter)?;

The embedded create_vector_index(name, table, path, metric, dim) also exists (pass dim = None to infer from existing rows — single-node only).

Distributed search

Similarity can't be routed to one shard, so distributed ANN broadcasts to all nodes and merges — the same scatter-gather skaidb uses for secondary-index pushdown. Each node runs its local HNSW top-k; the coordinator merges by distance, then re-reads the survivors at the read quorum (authoritative last-writer-wins vector) and applies the filter. The index is implicitly replicated/fault-tolerant because each replica derives its graph from the rows it already holds. Note: per-shard top-k merge means global recall depends on each shard's recall, so the coordinator over-fetches per shard (more so when a filter is present, since filtering happens after the re-read).

How it works

• HNSW (Hierarchical Navigable Small World): a layered proximity graph. A search descends with a small beam from a sparse top layer to the dense base layer, following edges toward the query, giving high recall at a fraction of a brute-force scan. Neighbor edges are chosen with the diversity heuristic (Malkov & Yashunin Algorithm 4), which preserves long-range links on clustered data — closest-only selection lets dense near-duplicate islands wire exclusively to each other and leaves whole regions unreachable. Metrics: cosine (vectors normalized on insert), squared L2, negative dot product. Verified at >90% recall vs. brute force on random data across all three metrics, plus dedicated self-recall tests on tightly clustered near-duplicate data.
• Filtered search evaluates the predicate against candidates surfaced by the graph; the graph is still traversed through filtered-out nodes for connectivity (the basic filtered-HNSW approach).

How vector DBs compare

ANN is a distinct index family from the B-tree / inverted indexes the OLTP and search engines use. Where vector search sits across the systems compared in docs/BENCHMARKS.md plus dedicated vector stores:

The dedicated vector DBs (Qdrant, Milvus, Weaviate; plus managed Pinecone) are specialists — superb at ANN and filtered ANN, but not transactional systems of record, so they usually run beside a primary DB. The general engines add vector search as a feature (pgvector, Mongo Atlas, ES dense_vector). skaidb sits in that second group: a durable, tunably-consistent store that can also do filtered, distributed ANN — for moderate vector sets (each node's graph is in-memory; see limitations).

Limitations

• In-memory by default — the whole graph (vectors included) lives in RAM (~1 GB resident for 182k×768 at exact f32). QUANTIZED (above) cuts the vector payload 4× by keeping int8 vectors in RAM and rescoring the top-k against the exact vectors re-read from the table; the graph adjacency itself stays in RAM — unless file-backed mode is on:

File-backed graphs (storage.vector_file_backed = true)

For indexes too large for RAM, the node can keep each frozen graph on disk and read it on demand instead of deserializing it at open: per base node, resident memory drops from vectors + adjacency (0.5–2 KB+) to an 8-byte offset plus the row key. Search traverses the file through a small per-index record cache; warm regions (the entry point, upper layers) stay cached, the cold tail reads through the OS page cache.

How it behaves:

• At-rest encryption seals it. On an encrypted node the frozen base is written sealed (SKHNSWE3): each record encrypted individually (keyed by its record index, so records cannot be transplanted between slots), the key table as one sealed blob, delta saves under a fresh random nonce. Consolidating a sealed base writes the sealed merge layout (SKHNSWE4), whose records carry their own never-reused nonces so the merge's in-place neighbor rewiring becomes a whole-record reseal — the full file-backed life cycle runs under at-rest with nothing accumulating.
• Writes keep working. HNSW cannot insert into a frozen structure, so rows written since the last freeze build an ordinary in-RAM delta graph; searches run both and merge by distance; deletions of frozen rows are tombstones. Each snapshot save rewrites only the small delta frame — the multi-GB base is never rewritten — and a crash mid-save falls back to the base watermark, with the ordinary open-time replay rebuilding the lost delta from the table.
• An index created before its data engages automatically: the first save after filling promotes the whole delta to the file-backed base (a plain freeze, no rebuild).
• The delta consolidates itself. When it reaches 10k nodes and a quarter of the base, a background merge folds it into a NEW base file without materializing the graph: base records are stream-copied, each delta node is HNSW-inserted against the new file (neighbor rewiring as in-place patches — the merged format uses fixed adjacency slots), and the work is paged under brief locks, chasing writes that land mid-merge until it can swap atomically. Searches and writes continue throughout; a crash leaves only a stale partial that the next open deletes. The merge's own memory is the traversal state plus a record cache — not the graph.
• The setting is node-local and restart-scoped; existing in-RAM snapshots convert at their first save (logged). Turning it back off makes file-backed snapshots unloadable — the index rebuilds in the background rather than loading.
• The graph build itself (backfill, rebuild) still happens in RAM before the freeze — plan the build's transient memory; the steady state after the freeze is what shrinks.
• Snapshots — each HNSW persists to <data>/vector/<name>.hnsw (written on build and graceful shutdown). A restart loads the snapshot and replays only rows stamped after its watermark — seconds, where the from-scratch build of a 182k×768 graph took 10–40 minutes per restart. A construction-parameter change or corrupt file falls back to a full rebuild — which the server runs in the background: the node starts and serves immediately, the affected index answers NEAREST with "rebuilding — retry shortly" (and reports building in SHOW INDEXES) until its pages complete and a fresh snapshot is written, with a log line at both ends. An index stopped mid-backfill deliberately has no snapshot (a partial graph must never load as a complete one), so a restart there re-queues the same background rebuild.
• ALTER VECTOR INDEX <name> SET (ef = <n>) retunes the search-time candidate-list size live (higher = better recall, slower queries; persisted, applies immediately). m/ef_construction shape the graph and need DROP + CREATE.
• Recall/latency beyond ef aren't tuned to production ANN libraries; large/high-dimensional workloads want a specialist.
• Vectors must be arrays of int/float of a single, consistent dimension.