Skip to main content
Suchi has one document index and several search and retrieval paths. This page explains how each path selects, ranks, limits, and authorizes results. Query syntax lives in Query language.

​
Search and retrieval paths

All surfaces restrict candidates to the selected system, require entry, and apply document visibility inside their database query. Admins bypass membership/ACLs, never explicit request/token systems. Name/code completion, counts, snippets, scores and pagination cannot return foreign-system rows. Public-demo exceptions apply only to the unnamed original demo archive. See Permissions. Whole canonical addresses such as S01.13.147 in Search/Omnibox use /api/jd/resolve; stale locations fail, not alias to a new category or fall back to broad text search. jd:13 remains a category filter within the current system. The final address component is an existing installation-global document ID.

​
FTS5 index

SQLite owns an external-content FTS5 table with two columns:
The index and its corpus statistics remain physically shared: system filtering does not promise separate FTS databases or independent rank statistics. Authorized results are constrained before hydration; this is application isolation, not physical search/storage separation.
  • title is document metadata.
  • content is extracted text from PDF, OCR, email, office-document, barcode, and other ingestion paths.
  • unicode61 provides Unicode tokenization.
  • remove_diacritics 2 makes diacritic-insensitive matching consistent for the supported scripts where SQLite can fold it.
  • porter supplies stemming for applicable terms.
Insert, delete, and title/content update triggers keep the FTS row synchronized with documents. The index stores searchable text, not original file bytes. Suchi currently has no vector index or embedding queue. “Semantic” behavior in the product comes from explicit metadata, approved extracted facts, lexical similarity, and bounded model synthesis after retrieval—not from hidden vector search. GET /api/search/?q=... compiles the rich query language into:
  1. one positive FTS5 MATCH expression for text clauses;
  2. fixed SQL predicates for negation and metadata filters;
  3. the caller’s visibility predicate; and
  4. live-document exclusion unless is:trash controls trash state.
Text results include a content snippet with <mark> around the matching span. The snippet window is 20 FTS tokens. Filter-only queries do not join FTS; they return an empty snippet and use newest-first ordering. Default recency-ranked searches first materialize only the requested page of document IDs and scores. A row-ID-restricted FTS pass then builds one snippet for each page result, so broad matches do not build display text for the whole candidate set. Raw recency=off searches stay in one FTS scan because a second bounded MATCH costs more than it saves for plain BM25. Both paths use score and document ID as a stable order. See SQLite WITH-clause materialization.

​
BM25 weighting

When text clauses exist, ranking begins with SQLite FTS5 BM25:
The first weight applies to title, the second to content. A title match is therefore three times as strong as the corresponding body-text contribution. SQLite’s BM25 rank is ordered ascending: a more negative value is a stronger match.

​
Recency decay

By default Suchi subtracts a bounded hyperbolic recency term:
2_592_000 seconds is 30 days: The term breaks ties and nudges close matches; lexical relevance remains the main signal. Hyperbolic decay avoids depending on optional SQLite math extensions. Use recency=off for raw BM25 ordering:

​
Pagination and wire score

Search defaults to 25 results and caps pages at 100. Each hit includes id, title, marked snippet, raw/blended rank, created_at, and mime_type. The response uses the standard {count,next,previous,results} envelope.

​
Documents

GET /api/documents/?q=... uses the same parser, resolver, and compiled predicates as ranked Search, so both endpoints select the same documents for the same rich query and authorization context. Documents does not apply BM25 or recency ordering. It supports list orders such as newest, oldest, updated, and title because the screen is also a bulk-action workspace. Legacy URL filters remain additive while older links and imported Views migrate to normalized queries. When a rich query contains positive text, Documents starts with the FTS match and joins the resulting row IDs to document metadata. Calendar’s source-document filter uses the same shape. This avoids scanning every document and running a correlated FTS probe for each row. Filter-only document lists stay on ordinary tables; the default live newest-first page uses a partial (created_at DESC, id DESC) index. Every supported list order includes ID as a deterministic tie-breaker, so equal timestamps or titles cannot move between pages.

​
Query completion and autocomplete

GET /api/autocomplete/?q=... first examines the active final token. If it is a recognized query qualifier, the endpoint returns full replacement queries for:
  • filter names;
  • Johnny.Decimal categories;
  • tags;
  • correspondents;
  • document types;
  • sensitivity values;
  • Calendar date roles; and
  • is: states.
The browser debounces requests by 160 ms and discards stale responses. Selecting a completion changes the query text; it does not run the query. When the input is not query-language completion, the endpoint retains its bounded taxonomy-prefix behavior for tags, correspondents, and document types. It caps combined suggestions at 20 by default and 50 explicitly.

​
Similar documents

GET /api/documents/{id}/similar uses FTS5 as a local “more like this” reader:
  1. Read the source title plus at most the first 4,096 content bytes.
  2. Extract alphanumeric tokens of length four or more.
  3. Remove a small deterministic stopword set.
  4. Select at most ten tokens by frequency, then alphabetically for ties.
  5. OR-join each token as a quoted FTS term.
  6. Exclude the source document and trash.
  7. Apply the caller’s visibility predicate.
  8. Rank with title 3.0, content 1.0, and no recency term.
The wire score is -bm25, so a higher positive value is more similar. The UI endpoint drops scores below 0.001; this removes the noise band produced by a single calendar-year overlap. Default limit is 10, maximum 50. A source with no content falls back to title tokens and sets matched_on_title_only so the UI can warn that results are noisier. Archive matching imports this same implementation. It changes thresholds and what happens after a match, not tokenization, SQL, ACLs, or ranking.

​
Saved Views and Calendar

A live View stores a normalized rich query. A snapshot View stores exact document_ids. New View visual controls compile into query clauses before persistence. Older/imported Views may use flat fields. Calendar sends view_id, never a client-reconstructed subset. The extracted-fact API loads the View only when owned or shared, applies its complete filter through the shared document-scope predicates, and then reapplies document ACLs. Calendar adds its approved-date range and role after that document scope. See Query language: Saved Views for serialization and Archive research snapshot behavior.

​
Archive research retrieval

Archive research is RAG evidence retrieval, not the rich query parser applied to the natural-language question. For each question Suchi:
  1. Normalizes up to 64 unique Unicode terms.
  2. Drops common question/document words when useful terms exist.
  3. Keeps at most 12 useful terms.
  4. Converts them to FTS prefixes and OR-joins them for recall.
  5. Ranks title 3.0, content 1.0, without recency.
  6. Selects at most six document IDs, keeping cited follow-up sources first and current BM25 results after them.
  7. Reloads up to three cited context sources directly in requested order, with fresh ACL, trash, sensitivity, and scope checks, then checks those authorized IDs for a current full-question match in one restricted FTS query. That match cannot be displaced by globally higher-ranked documents and applies in Focused mode too.
  8. Keeps the first full-question FTS passage for each matching document. For Balanced and Detailed, it runs each normalized term across all selected long document IDs in one batch and adds at most one passage per term.
  9. Removes whitespace-normalized duplicate or contained passages without changing primary-passage or source order, then appends a distinct document ending when the source-text ceiling has room.
  10. Adds approved extracted facts attached to those sources when permitted.
This is a two-stage document/passages flow: document authorization and ranking decide the stable citation numbers first; passage expansion operates only on those already-authorized IDs. All authorization, scope, document, FTS, and fallback reads share one read-only SQLite WAL snapshot, so a concurrent ACL, sensitivity, trash, or content change cannot mix old authorization with newer text. With at most 12 normalized terms, term expansion performs at most 12 bounded SQLite reads, each batching the term across the selected sources. It never issues one query per document. SQLite FTS5’s snippet() function returns at most 64 tokens. Suchi also caps each returned passage at 1,200 Unicode characters before assembly. Temporary markers locate the first match so the character cap is applied around it, then all markers are removed before the passage reaches the browser, logs, or model. Each already-ranked or row-ID-restricted result materializes its marked passage once before clipping, avoiding duplicate snippet() evaluation. Merely increasing one contiguous snippet would still inspect only the neighborhood chosen for that single FTS window, so it would not reliably recover far-apart clauses or repeated totals in a long document. Term-specific windows improve that coverage while the Focused, Balanced, and Detailed source ceilings remain hard bounds. See SQLite FTS5 snippets and Archive research: Research context presets. Short documents that fit the active ceiling are sent whole. A cited long document with no current-term match uses a bounded beginning-and-ending fallback. Source numbering remains stable, and the assembled source text returned by the API is exactly what the provider receives. Prefix-OR raises recall for the small evidence set. The model and citation validator then decide whether that evidence is sufficient. A research scope can still be a rich query, category, current document, exact document set, or the complete visible Documents/Search filter set. The shared document-scope predicate builder ANDs every field with the retrieval candidates. This distinction is why Archive research saves exact source snapshots instead of manufacturing a live rich query from question words. See Archive research for the model and citation contract.

​
MCP

The MCP adapter’s search_documents tool calls /api/search/; it does not have a second parser or ranking implementation. Query errors, pagination, scoring, and ACLs are therefore identical to HTTP and the web app. get_document retrieves one authorized result after selection. MCP never bypasses object authorization.

​
Performance and safety bounds

  • FTS content stays in SQLite; no network search service is required.
  • Queries are limited to 1 KiB, 64 tokens, and 256 bytes per value.
  • Search pages cap at 100; similar results at 50; Archive research sources at 6.
  • Archive research term-expansion reads cap at 12, passage windows at 64 FTS5 tokens and 1,200 Unicode characters, and per-source text at 1,600–4,800 characters.
  • SQL fragments are fixed; all user values use bound parameters.
  • Metadata filter names resolve to IDs before execution.
  • Snippets are returned only after visibility predicates match.
  • Archive evidence includes at most three extracted facts per source and twelve total, while fact summaries remain complete.
  • Fast route, search, and completion changes discard stale browser responses.

​
Maintainer map