Skip to main content
Suchi has one document index and several search and retrieval paths. This page explains how each path selects, ranks, limits, and authorizes results. Query syntax lives in Query language.

​
Search and retrieval paths

All surfaces restrict candidates to the selected system, require entry, and apply document visibility inside their database query. Admins bypass membership/ACLs, never explicit request/token systems. Name/code completion, counts, snippets, scores and pagination cannot return foreign-system rows. Public-demo exceptions apply only to the unnamed original demo archive. See Permissions. Canonical SYS.AC category addresses such as S01.21 in Search/Omnibox use /api/jd/resolve after system introduction. A successful resolution opens Documents filtered by the runtime category ID and resolved system. A canonical address with no category the user can enter runs as ordinary ranked search. Before introduction there are no addresses, so input such as 20.21 or a price such as 10.15 is ordinary search text. Former three-component input such as S01.21.147 is ordinary query text, not compatibility syntax. jd:21 remains a local category filter within the current selected system.

​
FTS5 indexes

SQLite owns two external-content FTS5 tables over the same title and content:
The index and its corpus statistics remain physically shared: system filtering does not promise separate FTS databases or independent rank statistics. Authorized results are constrained before hydration; this is application isolation, not physical search/storage separation.
  • title is document metadata.
  • content is extracted text from PDF, OCR, email, office-document, barcode, and other ingestion paths.
  • unicode61 provides Unicode tokenization.
  • remove_diacritics 2 makes diacritic-insensitive matching consistent for the supported scripts where SQLite can fold it.
  • porter supplies stemming for applicable terms.
  • trigram matches any substring of three or more characters, case- and diacritic-insensitive. It finds words inside German compounds (Versicherung in Krankenversicherung) and in Chinese, Japanese, Korean and Thai text, which has no spaces for unicode61 to split on.
Insert, delete, and title/content update triggers keep both FTS rows synchronized with documents. An update that writes back the same title and content (a missing text counts as empty) leaves both indexes untouched; the document’s revision counters still advance. The index stores searchable text, not original file bytes. Suchi currently has no vector index or embedding queue. “Semantic” behavior in the product comes from explicit metadata, approved extracted facts, lexical similarity, and bounded model synthesis after retrieval—not from hidden vector search. GET /api/search/?q=... compiles the rich query language into:
  1. one membership predicate per text clause: the word index, or the trigram index for terms of three or more characters, or a substring check of each candidate’s title and content for shorter terms in scripts written without spaces (東京), so a small system or narrow filter reads only its own rows. Each clause may match through a different index, and negation excludes matches in any of them;
  2. fixed SQL predicates for metadata filters;
  3. typed custom-field value predicates plus presence predicates, with target ACL/liveness inside a documentlink EXISTS;
  4. numeric/null predicates for archive serial numbers;
  5. the caller’s source-document visibility predicate;
  6. the caller-relative version currentness predicate; and
  7. live-document exclusion unless is:trash controls trash state.
Text results include a content snippet with <mark> around the matching span: a 20-token word-index snippet when the word index matched, otherwise a 240-character window of the trigram highlight. Filter-only queries do not join FTS; they return an empty snippet and use newest-first ordering. Currentness is evaluated inside the candidate query before count, ranking, and pagination. A version family contributes its greatest live document ID visible to that caller unless version:all or version:older changes the mode. Metadata filters stay outside the newer-version subquery: a newer visible revision suppresses an older revision even when only the older row matches another filter. This avoids presenting stale metadata as current. Exact-ID retrieval and Trash bypass family currentness. has-field: resolves a field ID in the selected system before compilation. Ordinary fields test for one stored value row. Document-link fields additionally join the exact target and require its intrinsic visibility and live state inside the EXISTS; negation wraps that complete predicate. Hidden target identity therefore cannot change a count or page in a way that claims the link is present. field: resolves the definition and data type in the selected system before compilation. Text-like equality checks the typed text column, multi-choice equality tests JSON membership, numbers/money compare value_number, dates compare value_date, and booleans compare value_bool. asn: compares the nullable archive serial number on the document row. These predicates stay inside the same visible/current candidate query used for counts and pages. Searches first materialize only the requested page of document IDs and scores. Row-ID-restricted FTS passes then build one snippet for each page result, so broad matches do not build display text for the whole candidate set. Word-index matches rank first, then documents found only by the trigram index, then documents found only by the short-term substring check. Tier, score and document ID form a stable order; recency=off drops the recency term. See SQLite WITH-clause materialization.

​
BM25 weighting

When text clauses exist, ranking begins with SQLite FTS5 BM25:
The first weight applies to title, the second to content. A title match is therefore three times as strong as the corresponding body-text contribution. SQLite’s BM25 rank is ordered ascending: a more negative value is a stronger match. Documents found only by the trigram index rank by bm25(documents_trigram, 3.0, 1.0) after every word-index match. The trigram index scores only documents the word index missed, since a word-index score always takes precedence.

​
Recency decay

By default Suchi subtracts a bounded hyperbolic recency term:
2_592_000 seconds is 30 days: The term breaks ties and nudges close matches; lexical relevance remains the main signal. Hyperbolic decay avoids depending on optional SQLite math extensions. Use recency=off for raw BM25 ordering:

​
Pagination and wire score

Search defaults to 25 results and caps pages at 100. Each hit includes id, title, marked snippet, raw/blended rank, created_at, and mime_type. The response uses the standard {count,next,previous,results} envelope.

​
Documents

GET /api/documents/?q=... uses the same parser, resolver, and compiled predicates as ranked Search, so both endpoints select the same documents for the same rich query and authorization context. Documents does not apply BM25 or recency ordering. It supports list orders such as newest, oldest, updated, and title because the screen is also a bulk-action workspace. Legacy URL filters remain additive while older links and imported Views migrate to normalized queries. When a rich query contains positive text, Documents starts from the row IDs the text indexes return and looks up document metadata by ID. Calendar’s source-document filter uses the same shape. This avoids scanning every document and running a correlated FTS probe for each row. Filter-only document lists stay on ordinary tables; the default live newest-first page uses a partial (created_at DESC, id DESC) index. Every supported list order includes ID as a deterministic tie-breaker, so equal timestamps or titles cannot move between pages.

​
Query completion and autocomplete

GET /api/autocomplete/?q=... first examines the active final token. If it is a recognized query qualifier, the endpoint returns full replacement queries for:
  • filter names;
  • filing categories;
  • tags;
  • correspondents;
  • custom fields for has-field: and field:;
  • sensitivity values;
  • Calendar date roles;
  • version modes;
  • asn:none; and
  • is: states.
The browser debounces requests by 160 ms and discards stale responses. Selecting a completion changes the query text; it does not run the query. When the input is not query-language completion, the endpoint retains its bounded taxonomy-prefix behavior for tags and correspondents. It caps combined suggestions at 20 by default and 50 explicitly.

​
Similar documents

GET /api/documents/{id}/similar uses FTS5 as a local “more like this” reader:
  1. Read the source title plus at most the first 4,096 content bytes.
  2. Extract alphanumeric tokens of length four or more. Runs in Chinese, Japanese, Korean, Thai and other scripts written without spaces yield overlapping three-character windows instead; windows of Hiragana alone are grammar and are dropped.
  3. Remove a deterministic set of low-information function, question and email boilerplate words.
  4. Select at most ten tokens by frequency, then alphabetically for ties.
  5. OR-join each token as a quoted FTS term: in the trigram index when any token is a character window, otherwise in the word index. One index per source keeps neighbour scores comparable.
  6. Exclude the source document and trash.
  7. Apply the caller’s visibility predicate.
  8. Rank with title 3.0, content 1.0, and no recency term.
The wire score is -bm25, so a higher positive value is more similar. The UI endpoint drops scores below 0.001; this removes the noise band produced by a single calendar-year overlap. Default limit is 10, maximum 50. A source with no content falls back to title tokens and sets matched_on_title_only so the UI can warn that results are noisier. Archive matching imports this retrieval implementation. Scalar category and correspondent suggestions require at least two supporting neighbours covering 30% of the returned sample; automatic application requires at least three. Configured confidence thresholds, ACLs and freshness checks still apply.

​
Saved Views and Calendar

A live View stores a normalized rich query. A snapshot View stores exact document_ids. New View visual controls compile into query clauses before persistence. Older/imported Views may use flat fields. Calendar sends view_id, never a client-reconstructed subset. The extracted-fact API loads the View only when owned or shared, applies its complete filter through the shared document-scope predicates, and then reapplies document ACLs. Calendar adds its approved-date range and role after that document filter. See Query language: Saved Views for serialization and snapshots saved from Ask your documents.

​
Document-question retrieval

Ask your documents uses broad lexical matching; it does not apply the rich query parser directly to a natural-language question. For each question Suchi:
  1. Normalizes up to 64 unique Unicode terms.
  2. Drops common question/document words when useful terms exist.
  3. Keeps at most 12 useful terms.
  4. Converts them to FTS prefixes and OR-joins them for recall; terms of three or more characters also OR-join as trigram substrings.
  5. Ranks title 3.0, content 1.0, without recency; word-index matches come before trigram-only matches.
  6. Selects at most six document IDs, keeping documents linked from the previous answer first and current BM25 results after them.
  7. Reloads up to three linked documents directly in requested order, with fresh ACL, Trash, sensitivity, and document-context checks, then checks those IDs for a current full-question match in one restricted FTS query. That match cannot be displaced by globally higher-ranked documents and applies in Focused mode too.
  8. Keeps the first full-question FTS section for each matching document. For Balanced and Detailed, it runs each normalized term across all selected long document IDs in one batch and adds at most one section per term.
  9. Removes whitespace-normalized duplicate or contained sections without changing primary-section or document order, then appends a distinct document ending when the text ceiling has room.
  10. Adds approved suggested details attached to those documents when permitted.
This is a two-stage document/text flow: document authorization and ranking decide the stable document numbers first; text expansion operates only on those already-authorized IDs. All authorization, document-context, document, FTS, and fallback reads share one read-only SQLite WAL snapshot, so a concurrent ACL, sensitivity, Trash, or content change cannot mix old authorization with newer text. With at most 12 normalized terms, term expansion performs at most 12 bounded SQLite reads, each batching the term across the selected documents. It never issues one query per document. SQLite FTS5’s snippet() function returns at most 64 tokens. Suchi also caps each returned section at 1,200 Unicode characters before assembly. Temporary markers locate the first match so the character cap is applied around it, then all markers are removed before the text reaches the browser, logs, or model. Each already-ranked or row-ID-restricted result materializes its marked section once before clipping, avoiding duplicate snippet() evaluation. Merely increasing one contiguous snippet would still inspect only the neighborhood chosen for that single FTS window, so it would not reliably recover far-apart clauses or repeated totals in a long document. Term-specific windows improve that coverage while the Focused, Balanced, and Detailed text ceilings remain hard bounds. See SQLite FTS5 snippets and Ask your documents: Reading-depth presets. Short documents that fit the active ceiling are sent whole. A linked long document with no current-term match uses a bounded beginning-and-ending fallback. Document numbering remains stable, and the assembled text returned by the API is exactly what the provider receives. Prefix-OR raises recall for the small document set. The model and document-link validator then decide whether that text is sufficient. The question’s document context can still be a rich query, category, current document, exact document set, or the complete visible Documents/Search filter set. The shared document-scope predicate builder ANDs every field with the matching documents. This distinction is why Ask your documents saves exact document snapshots instead of manufacturing a live rich query from question words. See Ask your documents for the model and numbered-document contract.

​
MCP

The MCP adapter’s search_documents tool calls /api/search/; it does not have a second parser or ranking implementation. Query errors, pagination, scoring, and ACLs are therefore identical to HTTP and the web app. get_document retrieves one authorized result after selection. MCP never bypasses object authorization.

​
Performance and safety bounds

  • FTS content stays in SQLite; no network search service is required.
  • The trigram index is about 1.6 times the size of the extracted text it covers, against about 0.3 times for the word index, and indexing a document costs roughly three times as much. go test ./core/api -run '^$' -bench 'IndexCorpus|SearchLanguages|ListDocumentsQuery|RewriteText' -benchtime 5x measures indexing, Search, Documents and text write-back on a fixed 20,000-document English, German and Japanese corpus. Queries run as an administrator in the 19,800-document system and the 200-document system beside it, and as a member whose access comes from ownership and a group.
  • Queries are limited to 1 KiB, 64 tokens, and 256 bytes per value.
  • Search pages cap at 100; similar results at 50; Ask your documents at 6 documents.
  • Document-question term-expansion reads cap at 12, text windows at 64 FTS5 tokens and 1,200 Unicode characters, and per-document text at 1,600–4,800 characters.
  • SQL fragments are fixed; all user values use bound parameters.
  • Metadata filter names resolve to IDs before execution.
  • Snippets are returned only after visibility predicates match.
  • Answers include at most three approved details per document and twelve total, while detail summaries remain complete.
  • Fast route, search, and completion changes discard stale browser responses.

​
Maintainer map