Wishlist — similar documents via sqlite-vec

​

The pure-Go trade-off

• Option A: load as a runtime extension. Ship a SUCHI_VEC_PATH env pointing at libsqlite_vec.so; the operator installs the extension themselves. Binary stays pure-Go; feature is opt-in.
• Option B: fork modernc/sqlite to link vec statically. Enormous lift; not on the table.
• Option C: skip sqlite-vec, use pure-Go BM25 more-like-this on top FTS5 terms. Cheaper recall but zero new deps.

​

Data model

-- vec0 virtual table lives outside the main schema so a pure-Go
-- build (no extension) doesn't fail at open time. The pipeline
-- creates it lazily when SUCHI_VEC_PATH is set + loadable.
--
-- Dimension follows the endpoint's embedding model — 768 is the
-- Ollama nomic-embed-text default; operators overriding to a
-- different model set SUCHI_VEC_DIM to match.

-- Actual table CREATE happens in Go at boot, gated on
-- SUCHI_VEC_PATH loading successfully — a vec0 CREATE without
-- the extension errors.

-- The core row lives in a plain table so a vec-less build can
-- still see "which docs have embeddings":
CREATE TABLE document_embeddings (
    document_id INTEGER PRIMARY KEY REFERENCES documents(id) ON DELETE CASCADE,
    model       TEXT NOT NULL,           -- e.g. "nomic-embed-text:v1.5"
    dim         INTEGER NOT NULL,        -- match the vec0 dim
    generated_at INTEGER NOT NULL
) STRICT;

​

Ingest hook

1. Runs post-content (after OCR text lands in documents.content).
2. Takes the first ~4KB of content (embeddings on the full text waste tokens on boilerplate — the intro paragraph carries most of the signal).
3. Hits POST {LLM_ENDPOINT}/embeddings with the standard OpenAI shape.
4. On success:
   • INSERT OR REPLACE INTO document_embeddings(...).
   • INSERT OR REPLACE INTO vec_documents(rowid, embedding) VALUES (doc_id, ?) — only when vec0 loaded; skip otherwise.
5. On endpoint failure: log Warn, skip. Doc still ingests.

​

HTTP surface

GET /api/documents/{id}/similar?limit=10
→
{
  "results": [
    {"id": 91,  "title": "March rent",       "score": 0.87},
    {"id": 108, "title": "February rent",    "score": 0.83},
    ...
  ],
  "method": "vec" | "fts"    // which retrieval path answered
}

​

SPA integration

​

Config surface

• SUCHI_VEC_PATH — path to libsqlite_vec.so (empty disables).
• SUCHI_VEC_DIM — embedding dimension; must match the endpoint’s model. Default 768 (Ollama nomic-embed-text).
• SUCHI_VEC_MODEL — model identifier stored on document_embeddings.model for audit. Default "nomic-embed-text:v1.5".

​

Privacy invariants

• Embeddings never leave the server outside the request to the operator-acknowledged LLM endpoint. Stored embeddings + retrieval are 100% local.
• A doc marked sensitivity IN ('confidential','restricted') still gets embedded but is excluded from the retrieval pool by default (same rule as the ask endpoint in docs/wishlist/qa-over-archive).

​

What’s NOT in scope

• Chunk-level embeddings. One embedding per doc is enough for the “similar docs” affordance. Chunk-per-page + retrieval is the layer that makes ask (#120) work well; that lives in that doc.
• Semantic + FTS rank fusion. Follow-up when semantic search gets its own UI. v1 exposes vec-only or fts-only, not blended.
• Re-embed on doc.content change. Post-consume script edits + automations can rewrite content; we don’t re-embed for cost. A --force flag on the backfill CLI can re-embed one doc.

​

Rollout

1. Design doc lands ✅ (this file).
2. GET /api/documents/{id}/similar FTS5 fallback shipped ✅. Pure-Go BM25 more-like-this at core/api/documents_similar.go; returns method: "fts" on every response. Works today with no new dependency.
3. Migration + document_embeddings table. (Pending.)
4. core/pipeline/embed package with the LLM-endpoint call + vec0 insert. Behind SUCHI_VEC_PATH gate. (Pending.)
5. When vec0 loads at boot, the endpoint prefers vec0 and reports method: "vec"; missing extension falls back to the FTS5 path already in production. (Pending.)
6. suchi embeddings --backfill CLI. (Pending.)
7. SPA DocumentDetail “Similar” strip — can consume the FTS5 shape today; the strip’s quality improves silently when vec0 lands.
8. Docs: config.mdx new env rows, cli.mdx new verb.