Skip to main content
suchi’s ingest pipeline branches by MIME type sniffed server-side from the first 1024 bytes of the upload. A PDF header preceded by a short binary prefix or leading whitespace within those bytes, as some e-filing downloads and invoice generators produce, is still routed as a PDF; qpdf repairs the working copy and the original stays unchanged. Every branch degrades gracefully: if the binary a branch needs isn’t installed, the doc is still stored and searchable, documents.content just lands empty until the tool shows up. core/mimeutil is the source of truth for document MIME aliases, filename refinement, output extensions, and the capabilities that select these branches. Adding a supported format there updates ingestion routing and linked/downloaded filenames together.

​
Format matrix

Anything else lands as a document with empty content. The original bytes remain safe in the CAS and deduplication and metadata still work, but text indexing requires an external converter.

​
PDF pipeline in detail

Email attachments with missing, generic, or malformed MIME headers (such as bin) are identified from their bytes. If an older PDF appears as a binary file, select it in Documents and choose Rescan after upgrading. Post-ingest corrects that MIME label before routing; password-protected PDFs then enter the normal decryption flow below. Original bytes are unchanged. The PDF path is the most involved because it fans out based on content: Detail per step (matching the boxes above):
  • preconsume — operator-defined script; see preconsume.
  • qpdf — normalize, strip restrictions, decrypt using empty password
    • candidates from the passwords file + learned passwords for this owner. If all attempts fail: mark encryption_state='encrypted' and halt — operator supplies a password via POST /api/documents/{id}/decrypt to resume.
  • docsplit — opt-in QR-scan pass. When separators fire, fan out one child doc per segment, soft-delete the parent, and exit (children carry the chain).
  • pageanalyze — optional. pdftoppm at 50 DPI → per-page mean intensity → qpdf --pages … to keep only non-blank pages in the working copy.
  • pdf-inspector — pdftotext -layout. ≥ 32 non-whitespace chars is the text-native shortcut (write documents.content directly); native text replaces provisional device text.
  • Device text — an image-only PDF retains accepted nonblank device OCR, skipping server OCR. This creates no new searchable-PDF archive, including with OCRmyPDF. See upload OCR metadata for confidence and validation rules. Without accepted text, use server OCR below.
  • OCR path — writes the text sidecar to documents.content. When the engine is ocrmypdf it also writes a searchable-PDF archive into documents.archive_blob.
  • zugferd — qpdf --list-attachments → --show-attachment → CII XML parse. A Factur-X / ZUGFeRD / XRechnung attachment populates the managed invoice_number, invoice_date, invoice_currency, invoice_total_gross, invoice_total_net, invoice_seller, invoice_seller_vat, and invoice_buyer custom fields. Their definitions cannot be renamed or deleted; document values remain editable.
Each step degrades independently. Missing pdftotext retains accepted device text or falls through to server OCR. Without accepted text or an available OCR engine, a scanned PDF’s content stays empty. Missing qpdf keeps the original bytes as-is (encrypted PDFs will then fail downstream).

​
OCR engine dispatch

Two engines are supported and picked at boot via OCR_ENGINE: tesseract is the smaller runtime path and does not produce a searchable-PDF archive — documents.archive_blob stays NULL for scanned docs. FTS + full-text search still work over documents.content. Pick ocrmypdf when you want the archive PDF (e.g. long-term retention where the searchable PDF matters more than image size), or set to auto and let the image you deployed decide. The lightweight engine retries a successfully processed page once with sparse text segmentation (--psm 11) only when normal OCR found no text. This helps isolated labels without replacing text already found. Rasterization, every page, and any retry share one SUCHI_TESSERACT_TIMEOUT deadline; extracted text is bounded by the configured output cap. It does not infer rotation from unreliable orientation guesses or correct photographed perspective. Configure OCR_LANGUAGES for the languages on the page; the default is English. For raster-image inputs, it also preserves the normal result and adds distinct lines from one sparse pass rasterized at a 2400-pixel longest edge. Only whole lines with mean Tesseract word confidence of at least 70 and at least four letters/digits are added, filtering common background noise. This pass shares the same deadline and bounded output; it changes neither stored pixels nor the PDF-input OCR path. It improves isolated photo headings, not every stylized font, curved package label, or perspective-distorted line. Check the extracted text before relying on exact model numbers or other critical details.

​
Password-protected PDFs

Encrypted PDFs go through a candidate-passwords loop before being parked as pending-decryption. The loop tries, in order:
  1. Empty password — the “owner restrictions only” case (very common for bank statements, invoices). qpdf --decrypt= with no password.
  2. Passwords file (optional) — INGEST_PASSWORDS_FILE points at a newline-separated list of candidates; blank lines and # comments are skipped. Chmod 600 recommended.
  3. Learned passwords — every password an operator supplies via POST /api/documents/{id}/decrypt with remember=true gets sealed with AES-256-GCM (.decrypt-key file) and stored owner-scoped in the decryption_passwords table. Hot passwords hit first.
If every candidate fails, the doc lands in encryption_state='encrypted' with the pipeline halted. Surface it via GET /api/documents/pending-decryption. The operator supplies a password:
On success suchi writes a decrypted CAS blob (originals are always preserved verbatim in original_blob), flips the state, and re-enqueues post-ingest — the doc flows through the rest of the pipeline (OCR, ZUGFeRD, automations, render) as if it had arrived unencrypted. Batch decrypt — for monthly statements where multiple accounts share a password, POST /api/documents/decrypt-batch accepts a doc-id array and one password; remember=true fires once per batch on first hit. qpdf warnings-are-fine gotcha. Some bank PDFs ship spec-nonconformant /Perms and produce qpdf exit code 3 (warnings). suchi treats exit 3 with non-empty output as success — the pragmatic convention for pre-consume decrypt.

​
Email (message/rfc822)

.eml files land as documents with the email body text as documents.content and headers projected onto standard fields:
  • title ← Subject (RFC 2047 encoded-word decoded)
  • created_at ← Date header (unix epoch)
  • email_message_id ← Message-Id provenance used for re-sync deduplication
  • Sender (From: Name <email>) upserts into the correspondents table and attaches to the doc under role=sender.
Subject parsing changes the title only while its owner is auto; a later reparse cannot overwrite a title renamed by a person or assigned by a rule. For Outlook .msg files, msgconvert creates a temporary RFC 822 working copy for this same parsing path. The original CFB bytes remain in original_blob, and downloads retain the application/vnd.ms-outlook MIME type. Attachments fan out into sibling docs. Each attachment gets its own documents row with email_parent_id pointing back at the .eml, its own CAS blob, and its own post-ingest job — so an attached PDF flows through the entire pipeline (qpdf, OCR, ZUGFeRD, automations, render) as if it had arrived directly. multipart/related inline images (referenced from an HTML body) are NOT ingested as separate docs — they’re kept inside the parent .eml’s CAS blob only. Attachment children inherit the parent’s correspondents (so the From: sender shows up on every child PDF), owner, and JD category. They do NOT inherit tags — the pipeline (automations and LLM classifier) reclassifies each child independently. Attachment children use the enclosing message’s valid Date header as created_at and retain the ingestion time as added_at; source_mtime keeps the same upstream date. Without a usable Date, created_at falls back to the ingestion time and source_mtime remains empty. How to feed it in. Any of the three supported ingest paths works; suchi doesn’t care what put the .eml on the disk:
  • INGEST_FS_DIR — read an existing folder recursively without changing source files. The reconciler polls, so it also works when a NAS mount does not forward local filesystem events. Files over BODY_LIMIT remain untouched; restarting Suchi with that limit raised or disabled retries the unchanged file. Pair the folder with mbsync/isync or another producer when needed.
  • Mailboxes UI — direct IMAP polling. Suchi seals the account credential and pulls new messages on the configured interval.
  • Upload API — POST a .eml file to /api/documents/ for one-off imports.
The parser is pure Go stdlib (net/mail + mime/multipart + mime/quotedprintable) — it works in both standard and full images with no external tools. RFC 2047 encoded-word subjects, quoted-printable + base64 bodies, and nested multipart trees all decode correctly.

​
Barcode-assisted physical filing with archive serial numbers

An archive serial number (ASN) is a stable integer printed on the paper and stored with its digital document. Suchi recognizes a case-sensitive QR payload made from SCAN_ASN_PREFIX followed immediately by decimal digits. With the default prefix, ASN000042 and ASN42 both store the integer 42; the web app renders the canonical chip ASN 42. Use one of two opt-in modes: Administrators can choose the mode, prefix, and analysis DPI under Settings > Archive configuration > Advanced > Scan processing. Those saved values apply to future processing jobs immediately. Deployment environment or config-file values take precedence and appear read-only without replacing the stored value; the equivalent operator configuration is:
  • assign leaves the PDF intact. An unlabelled PDF proceeds normally. If a prefix-looking label is present, the PDF must contain exactly one valid ASN QR code, on any page.
  • split also leaves an unlabelled PDF alone. Once an ASN label is detected, page 1 must carry exactly one, and every later ASN-labelled page starts the next document. The labelled pages remain as the first pages of their segments; they are not disposable separators.
For a feeder batch, put the label on the first page of each physical document, scan the stack into one PDF, and upload it normally. Suchi rasterizes every page once at SCAN_SPLIT_DPI, enumerates every visible barcode, validates the whole plan, then extracts the ranges. Each child gets the integer from its first page, its own immutable original and processing job. All child rows, inherited ACLs, sources and jobs commit in one writer transaction; only then is the combined parent moved to Trash. A transaction failure exposes no partial batch. ASN values are unique within one filing system across live documents, Trash and version history. Preflight therefore stops a conflicting batch before pre-consume or other mutable ingest hooks run. Malformed prefix-looking labels, multiple labels in assign mode, multiple ASN labels on one split page, repeated canonical values, a missing leading label, and a scan mixing ASN and legacy separator markers become terminal processing errors. The Upload screen explains the rejection on the file’s row, naming the page and number involved and, for a conflict, the document that holds the number when the uploader can open it; administrators also see the raw error in the Processing issues tab of Approvals. Correct the source or retained conflicting document, then ingest again. A replay after a completed split returns the existing children rather than creating another set; restoring or re-uploading the retired parent cannot silently resurrect it over those children. Blank-page cleanup never removes an ASN-labelled segment’s first page. The combined uploaded original remains in content-addressed storage and its retired parent remains recoverable from Trash during the normal retention window. Assignment mode likewise preserves the labelled page and original PDF. Print and scan a small calibration batch before committing to label stock. Start at 300 DPI for physical labels, then calibrate against the smallest printed code and the scanner’s scale, contrast, and compression; higher DPI costs more CPU and memory. Use a prefix that cannot overlap the legacy separator token, allocate each integer once, and keep the human-readable number beside the QR code so a filing mistake can be diagnosed without software.

​
Legacy disposable separator sheets

SCAN_SPLIT_ENABLED=on retains the earlier separator workflow. A barcode whose trimmed payload equals SCAN_SPLIT_TOKEN is removed, and each non-empty range between separators becomes a child document. The default payload is SUCHI-SPLIT:
Legacy separator and ASN detection may be configured together, but one PDF must not contain both marker families. Prefer ASN labels when the physical and digital copies need a shared durable identifier; use disposable separators when the pages should not become part of the child documents. See configuration for SCAN_ASN_MODE, SCAN_ASN_PREFIX, SCAN_SPLIT_ENABLED, SCAN_SPLIT_TOKEN, and SCAN_SPLIT_DPI. Run make smoke-asn to exercise a generated retained-label feeder PDF through watched-folder intake, atomic fanout, API projections, metadata conflicts, and completed-batch replay.

​
ZUGFeRD / Factur-X / XRechnung

PDF/A-3 documents with an embedded Cross-Industry Invoice XML get their structured data extracted and written to custom fields: The definitions are created lazily for that filing system on its first successful e-invoice extraction. The UI presents readable invoice labels while the invoice_* names above remain the stable API and storage contract. Document Detail shows extracted values on the matching invoice and allows corrections. Managed definitions without an extracted value stay out of the generic add-custom-field selector and do not appear as empty rows on other documents. Recognised attachment names (case-insensitive): factur-x.xml, zugferd-invoice.xml, xrechnung.xml.

​
Images (barcodes)

Every image upload runs QR / DataMatrix / Aztec decoding via gozxing (pure Go, in-process). If the Go QR reader cannot decode a symbol, optional zbarimg retries QR only against a locally re-encoded PNG. Both Docker images include it (zbar on Alpine, zbar-tools on Debian); bare-metal installations without it retain the Go readers. This handles some photographed QR codes that fail the Go reader’s checksum verification, without relaxing either decoder’s checksum checks. The fallback has a five-second timeout and a 64-KiB XML output limit; failures never discard successfully decoded Go results. Image dimensions above 64 megapixels skip barcode decoding before pixel allocation; the separate OCR conversion path remains available. Decoded links are indexed, never opened, and native output is not logged. Any decoded barcode value lands in documents.content as barcode:<value> tokens so the FTS5 index picks them up — search for barcode:INV-2026-0042 and the doc surfaces. Supported raster images use one imgpdf conversion path before OCR. PNG/JPEG produce one page, multipage TIFF/GIF retain their pages, and HEIC/HEIF sequences use the first still frame. The original bytes are never replaced. Barcode decoding also runs where the in-process decoder supports the source format; HEIC/HEIF text comes through ImageMagick and OCR, not a native Go HEIF decoder. Other image formats receive the barcode pass only. Camera EXIF orientation is applied before PDF conversion, including the quarter-turn metadata used by phone photos. The derived PDF uses 300-DPI page density without resizing the image, so the standard OCR rasterizer retains its original pixel resolution. This corrects camera metadata; it does not infer the reading direction of mixed labels or straighten photographed perspective.

​
Additional OCR languages

Both supported OCR engines use Tesseract language data. The default is English (eng); selecting another language does not install its data. Additional languages require both installed packs and an OCR language setting. The examples below combine English, Nepali, and German; other codes are listed in Tesseract’s language data. Orientation correction and QR decoding do not supply missing language data.

​
Docker installations

Language packs belong in a derived image: the supplied Compose service has a read-only root filesystem, and both runtime images run as user 65532:65532. Installing packages in a running container is not persistent across replacement. The standard image’s Alpine repository does not package Nepali separately. Its example uses the upstream tessdata_fast 4.1.0 model, downloaded into the build directory once:
For the Alpine-based standard image, Dockerfile.ocr contains:
For the Debian-based full image, the equivalent file contains:
Either file builds and verifies with:
The output should include eng, nep, and deu. --build-arg SUCHI_BASE=... can select a specific release tag or digest of the same image variant. The repository’s Compose file selects this derived image when its .env contains SUCHI_IMAGE=suchi-ocr:local.

​
Bare-metal installations

Language data is installed for the Tesseract installation used by the Suchi service, not on the browser or mobile device: For Alpine, the same pinned nep.traineddata download above can be verified before installation:
Suchi starts OCR subprocesses with an empty environment. Setting TESSDATA_PREFIX on the server alone therefore does not redirect model lookup. Package installation uses Tesseract’s default data directory. Manually supplied .traineddata files also need to be readable there by the service user. This command checks the selected executable without relying on a shell’s TESSDATA_PREFIX; a service with a different executable path needs that path checked instead:

​
Select languages and reprocess documents

Suchi uses comma-separated codes: OCR_LANGUAGES=eng,nep,deu. A TOML configuration uses ocr_languages = ["eng", "nep", "deu"]. Suchi converts the list to Tesseract’s eng+nep+deu syntax; installing a pack alone does not activate it. For Compose, this is an addition to the existing suchi service’s environment:
docker compose up -d suchi applies the selected image and environment changes. An entry in .env alone is not passed to the container unless Compose references it. Bare-metal file or environment changes take effect after a service restart. The web app’s Archive configuration → OCR and backups settings can select the same comma-separated codes without a restart when neither a config file nor the process environment pins OCR_LANGUAGES. Pinned values override saved web settings, including when a boot key is present but empty. Removing that key and restarting or recreating the server allows live UI control; see Configuration precedence. The current settings form checks code syntax, not installed packs, and rejects some valid Tesseract names such as chi_sim. Those names can be selected through the environment or configuration file. Language changes affect subsequent OCR jobs, not previously stored text. After a small new upload confirms recognition, selecting existing documents in Documents → Rescan reprocesses them from their originals. Language changes do not change the pipeline version, so a version-based stale rescan does not necessarily select affected documents. The empty-content filter also misses documents with partial OCR or decoded barcode text. Run suchi doctor in the same host or container as the server before enabling additional languages. Its OCR languages section invokes the selected tesseract binary with the same process sandbox used by OCR jobs, then prints the configured packs, every available pack, and any configured code that is missing. If a configured pack is missing:
  1. Install the exact package or trained-data file for the tesseract binary shown under pipeline binaries.
  2. Run suchi doctor again as the service user. Do not proceed until the missing code appears in the available list.
  3. Upload a small representative document and inspect its extracted content.
  4. Only then use Documents → Rescan on existing documents that need the new language.
A successful upload or job alone does not establish that all requested languages were read. Multiple enabled languages add CPU and memory work; detected document-language metadata does not choose OCR packs. OCR processing runs locally. The optional LLM classifier receives bounded extracted text rather than document images and does not act as an OCR fallback. Automatic remote vision OCR was not selected because it would disclose source image content to the configured provider and add image-inference costs. Local multilingual engines remain evaluation candidates rather than production OCR backends.

​
Image conversion requirements

ImageMagick must include a PDF encoder and allow PDF writing. On Alpine the encoder is in imagemagick-pdf, alongside imagemagick-heic for HEIF decoding. Alpine’s PDF package also brings its Ghostscript dependency; it is not a standalone lightweight encoder. Both images permit PDF writing while denying ImageMagick reads through PDF aliases and PostScript/EPS/XPS coders. Poppler handles PDF input. A .pdf filename alone does not prove the encoder worked: some installations silently write the source image format. Suchi explicitly requests PDF and rejects output without its signature. Suchi fixes the packaged encoder/policy and no longer uses the old HEIC-to-PNG workaround. Select affected photos/scans in Documents and Rescan to rebuild older missing or invalid image archives. See Pipeline versions for controlled processing upgrades and explicit rescans from originals.

​
EPUB

EPUB uses the same anydoc extractor as office documents. It preserves spine order and emits headings, links, and visible chapter text as Markdown.

​
DjVu

djvutxt from djvulibre-bin extracts the OCR text layer that archive.org and library-scan DjVus normally carry. If the DjVu has no text layer (unusual), the extraction returns empty content — suchi does not OCR DjVu itself; convert to PDF upstream if needed. Output cap: DJVU_MAX_CONTENT_BYTES (default 32 MiB). Truncation is logged.

​
Office documents

anydoc from Firecrawl (MIT, Rust) extracts text from EPUB, Word, PowerPoint, Excel, OpenDocument, RTF, and CSV into GitHub-flavored Markdown. The Markdown lands as documents.content and FTS5 indexes it just like OCR’d PDF text. Supported MIMEs (routed by core/pipeline/anydoc.Recognized): PDF deliberately remains on the qpdf → pdf-inspector → OCR path. Missing anydoc binary = documents in these formats still land, they just don’t get their content indexed. Same posture as missing ocrmypdf. Both Docker images (standard and full) bundle anydoc; bare-metal installs need to place an anydoc executable on PATH themselves — see source caveats below. Suchi runs converters with an empty environment. An npm-installed anydoc whose first line is #!/usr/bin/env node is found on PATH and reported by suchi doctor, but every extraction exits with env: node: No such file or directory and the document keeps empty content. Install a self-contained binary, or put a wrapper earlier on the server’s PATH that runs Node by absolute path:
Then select the affected Office documents and choose Rescan.

​
anydoc source caveats

Two things worth knowing about how suchi consumes anydoc: Upstream doesn’t ship a standalone CLI binary. anydoc is distributed as a Rust library (crates.io), Node napi binding (npm), and Python wheel (PyPI). Firecrawl’s CLI story is npx @firecrawl/anydoc — a Node wrapper around the napi binding. suchi doesn’t want Node in the runtime image, so the Dockerfile compiles examples/convert.rs from the anydoc repo into a static musl binary and installs it as /usr/local/bin/anydoc, avoiding a Node runtime dependency in the image. examples/convert.rs is not upstream-guaranteed as a stable CLI. It’s an example — the argv shape could shift between anydoc releases. As of v0.2.3 it’s convert <file> [-f <fmt>] [-o <out>] [--assets <dir>]. suchi shells out with just anydoc <file>, reading stdout for Markdown. If an upstream tag changes this shape, Suchi’s extractor at core/pipeline/anydoc/anydoc.go needs a matching update; symptoms would be “unrecognized argument” in res.StderrTail. The bump procedure — enforced by hack/pin-bumper.sh and the Dockerfile comment — is:
  1. Run hack/pin-bumper.sh from repo root. It reports when the pinned version is older than n-1 and suggests the previous stable release.
  2. Read the compare URL it prints. Scan examples/convert.rs for argv changes.
  3. Edit ARG ANYDOC_TAG= and its resolved ARG ANYDOC_COMMIT= in Dockerfile. Rebuild standard, run docker run --rm suchi:standard doctor to confirm anydoc is on PATH, and smoke-test a docx ingest.
  4. Commit as a standalone “bump” commit so it’s easy to roll back.
Alternative if the CLI drift becomes painful: vendor the convert example into suchi’s tree, stripping the language wrappers, and pin the anydoc library directly. Not done today because examples/convert.rs has been stable in shape since the crate’s first release and the maintenance burden isn’t worth it yet.

​
Unsupported formats

  • .zip / .tar bundles — no auto-explode. Ingest the individual documents.
Add support by dropping a new package under core/pipeline/<format>/ with an Extract(...) that returns (*Result, error), then wire it into core/pipeline/postingest/postingest.go behind a MIME check. Every existing extractor is a template.

​
Per-format tuning

All accept K/M/G suffixes or raw byte counts. Truncation is logged; the doc still lands and FTS indexes what fit.

​
JSON sidecar spec

Any producer dropping a file into the fs-watch staging dir (or a mail attachment carrying a Correspondent) can hand suchi structured metadata via a sibling <name>.json (or <path>.json) file. The sidecar shape below is the wire contract — additive fields don’t need a version bump; a required-field change bumps suchi_sidecar. The producer captures one filing system before intake. Filesystem watch uses INGEST_FS_SYSTEM (omitted means original system 1); mail uses its account’s system. Sidecars/document content cannot create or route to another system. Version, split and attachment documents inherit their source system.
Source of truth: core/ingest/sidecar/sidecar.go parses the wire shape; core/ingest/fswatch applies it. Sensitivity, metadata, ASN, the note, typed custom fields, and the ingest outbox commit together. An invalid sensitivity, unknown custom field, invalid typed value, invalid ASN value, bad JSON, or explicit-but-mismatched suchi_sidecar version rejects the intake without partial metadata. The read-only watcher retains the source file and records a blocked observation, retried hourly rather than every five minutes. Changing the sidecar retries that file at once; changing a custom-field definition retries blocked files in that filing system. An occupied ASN and transient filesystem/database failures retain their delayed retry because archive state can resolve them without changing the source. Producers that omit suchi_sidecar get the flat-JSON compatibility shape (title/title_owner/created/correspondent/tags only); fields outside that shape, including sensitivity, are ignored. The native suchi export sidecar is a partial takeout, not a complete restore contract. It carries title ownership with the title, source date, correspondents, tags, JD category, ASN, sensitivity, and restorable custom-field values, which fs-watch applies. Notes are not exported. Document-link and extraction-managed custom fields are omitted because document IDs are not portable and extraction reads managed values again from the original. mime_type and sha256 may appear for provenance, but the parser ignores them; MIME and SHA-256 are recomputed from the file. Native takeout manifest version "3" identifies one selected system and includes these optional source fields while sidecar format remains 1. jd_address is a potentially repeated category address; fs-watch ignores it for routing and identity, using jd_system and jd_category instead. Reingestion preserves available ASNs but can allocate new installation-global document IDs. Full DB/blob/key backup, not a sidecar, preserves membership and document relationships.

​
Docker image matrix

The standard image is the default and supports every format in this table. Pick full only when OCRmyPDF must write a text layer into scanned PDFs.