Supported file types

suchi’s ingest pipeline branches by MIME type sniffed server-side from the first 512 bytes of the upload. Every branch degrades gracefully: if the binary a branch needs isn’t installed, the doc is still stored and searchable, documents.content just lands empty until the tool shows up.

Format matrix

MIME	What happens	Binary needed	Slim image	Full image
`application/pdf`	qpdf normalize → pdf-inspector routes text-native vs OCR → ZUGFeRD invoice extraction	`qpdf`, `pdftotext`, tessocr (`pdftoppm`+`tesseract`) OR `ocrmypdf`	✅ full PDF chain (no OCR’d archive)	✅ full PDF chain (with OCR’d archive)
`image/*` (PNG, JPEG, GIF, TIFF, WebP)	QR / DataMatrix / Aztec barcodes decoded into `documents.content` as `barcode:<value>` tokens	none (pure-Go via gozxing)	✅	✅
`application/epub+zip`, `application/epub`	Pure-Go zip walker → OPF → spine → concatenated XHTML text into `documents.content`. Title + authors captured in logs.	none	✅	✅
`image/vnd.djvu`, `image/x-djvu`	`djvutxt` extracts the embedded OCR text layer	`djvutxt` (djvulibre-bin)	❌ (no binary)	✅
`image/heic`, `image/heif` (`.heic`, `.heif`)	ImageMagick converts to single-page PDF; the standard OCR path then produces a searchable archive — a photograph of a receipt becomes full-text searchable	`magick` (ImageMagick) with HEIF delegate	✅	✅
`message/rfc822` (`.eml` files)	Parse via net/mail; headers → title/date/correspondent, body text → content, each attachment becomes its own document linked via `email_parent_id`	none (pure Go)	✅	✅
`application/vnd.ms-outlook` (`.msg` files)	`msgconvert` transforms the CFB binary into RFC 822; falls through to the `.eml` path so Message-Id dedup + attachment fanout + correspondent inheritance work unchanged	`msgconvert` (libemail-outlook-message-perl)	❌ (no binary)	✅

Anything else lands as a doc with empty content — the original bytes are safe in the CAS, dedup + metadata still work, but there’s no text index. Add a converter (or wait for a future ingester) to fill in documents.content later.

PDF pipeline in detail

The PDF path is the most involved because it fans out based on content:

upload → preconsume (optional; operator script — see /preconsume)
      → qpdf (normalize, strip restrictions, decrypt using
                empty password + candidates from passwords file
                + learned passwords for this owner)
          → if all decrypt attempts fail: mark encryption_state='encrypted',
             bail out. Operator supplies password via
             POST /api/documents/{id}/decrypt to resume the pipeline.
      → docsplit (opt-in; pdftoppm+QR scan → separator pages)
          → if separators found: fan out one child doc per segment,
             soft-delete parent, exit (children carry the chain)
      → pageanalyze (optional; pdftoppm at 50 DPI → per-page mean intensity)
          → qpdf --pages … keep only non-blank pages in the working copy
      → pdf-inspector (pdftotext -layout)
          ├─ ≥32 non-whitespace chars → text-native shortcut
          │      → write content directly
          └─ otherwise → OCR path (see engine dispatch below)
                 → text sidecar (documents.content)
                 → optional searchable-PDF archive (documents.archive_blob),
                    when the engine is ocrmypdf
      → zugferd (qpdf --list-attachments → --show-attachment → CII XML parse)
             → invoice_* custom fields (when a Factur-X / XRechnung
                attachment is present)
      → rules engine → rendered-view symlink → optional LLM handoff

Each step degrades independently. Missing pdftotext sends every PDF to the OCR path (slower but correct). Missing OCR engine leaves the content empty for scanned PDFs. Missing qpdf keeps the original bytes as-is (encrypted PDFs will then fail downstream).

OCR engine dispatch

Two engines are supported and picked at boot via OCR_ENGINE:

`OCR_ENGINE`	Engine	Binaries	Extras
`auto` (default)	prefers tesseract, falls back to ocrmypdf	pdftoppm + tesseract, OR ocrmypdf	zero-config
`tesseract`	in-process pdftoppm → tesseract chain (`core/pipeline/tessocr`)	pdftoppm (poppler-utils) + tesseract	slim image default
`ocrmypdf`	ocrmypdf CLI wrapper (`core/pipeline/ocrmypdf`)	ocrmypdf + tesseract	full image default

tesseract is the smaller path (~4× less Docker image size vs ocrmypdf’s Python stack) but does not produce a searchable-PDF archive — documents.archive_blob stays NULL for scanned docs. FTS + full-text search still work over documents.content. Pick ocrmypdf when you want the archive PDF (e.g. long-term retention where the searchable PDF matters more than image size), or set to auto and let the image you deployed decide.

Password-protected PDFs

Encrypted PDFs go through a candidate-passwords loop before being parked as pending-decryption. The loop tries, in order:

Empty password — the “owner restrictions only” case (very common for bank statements, invoices). qpdf --decrypt= with no password.
Passwords file (optional) — INGEST_PASSWORDS_FILE points at a newline-separated list of candidates; blank lines and # comments are skipped. Chmod 600 recommended.
Learned passwords — every password an operator supplies via POST /api/documents/{id}/decrypt with remember=true gets sealed with AES-256-GCM (.decrypt-key file) and stored owner-scoped in the decryption_passwords table. Hot passwords hit first.

If every candidate fails, the doc lands in encryption_state='encrypted' with the pipeline halted. Surface it via GET /api/documents/pending-decryption. The operator supplies a password:

curl -X POST http://127.0.0.1:8000/api/documents/42/decrypt \
     -H "Authorization: Token $TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"password":"hunter2","remember":true,"label":"BofA 2024"}'

On success suchi writes a decrypted CAS blob (originals are always preserved verbatim in original_blob), flips the state, and re-enqueues post-ingest — the doc flows through the rest of the pipeline (OCR, ZUGFeRD, rules, render) as if it had arrived unencrypted. Batch decrypt — for monthly statements where multiple accounts share a password, POST /api/documents/decrypt-batch accepts a doc-id array and one password; remember=true fires once per batch on first hit. qpdf warnings-are-fine gotcha. Some bank PDFs ship spec-nonconformant /Perms and produce qpdf exit code 3 (warnings). suchi treats exit 3 with non-empty output as success — the pragmatic convention for pre-consume decrypt.

Email (message/rfc822)

.eml files land as documents with the email body text as documents.content and headers projected onto standard fields:

title ← Subject (RFC 2047 encoded-word decoded)
created_at ← Date header (unix epoch)
email_message_id ← Message-Id (for future dedup on re-syncs)
Sender (From: Name <email>) upserts into the correspondents table and attaches to the doc under role=sender.

Attachments fan out into sibling docs. Each attachment gets its own documents row with email_parent_id pointing back at the .eml, its own CAS blob, and its own post-ingest job — so an attached PDF flows through the entire pipeline (qpdf, OCR, ZUGFeRD, rules, render) as if it had arrived directly. multipart/related inline images (referenced from an HTML body) are NOT ingested as separate docs — they’re kept inside the parent .eml’s CAS blob only. Attachment children inherit the parent’s correspondents (so the From: sender shows up on every child PDF), owner, and JD category. They do NOT inherit tags or document type — the pipeline (rules, LLM classifier) reclassifies each child independently. How to feed it in. Any of the three canonical ingest paths works; suchi doesn’t care what put the .eml on the disk:

INGEST_FS_DIR — a fs-watch on a Maildir or archive folder. Pair with mbsync/isync against your IMAP server (or a Bridge like Proton’s) to keep the folder fresh. Recommended for setups that already have a mail sync toolchain.
INGEST_IMAP_URL — direct IMAP polling. No external mail sync needed; suchi holds the credentials and pulls new messages on an interval.
Upload API — POST a .eml file to /api/documents/ for one-off imports.

The parser is pure Go stdlib (net/mail + mime/multipart + mime/quotedprintable) — it works in both slim and full images with no external tools. RFC 2047 encoded-word subjects, quoted-printable + base64 bodies, and nested multipart trees all decode correctly.

Multi-doc splitting on QR separator sheets

Feeder-scanning a stack of unrelated documents produces one large PDF that should really be N docs. Opt in with SCAN_SPLIT_ENABLED=on and suchi will detect separator sheets — pages carrying a QR code with a specific payload — and fan out one document per segment. Setup:

Generate a separator sheet:
qrencode -s 20 -o separator.png SUCHI-SPLIT
Print several copies. Any QR encoder works; the default token is SUCHI-SPLIT (override via SCAN_SPLIT_TOKEN).
Enable in the environment:
export SCAN_SPLIT_ENABLED=on
Insert a separator between each document in your feeder stack and scan the whole pile as one PDF. Upload as usual.

What happens on ingest:

docsplit rasterizes each page at 150 DPI and scans for QR codes.
Separator pages are dropped; the ranges between them become segments.
Each segment becomes a fresh document (with its own original_blob and post-ingest job), linked via split_parent_id.
The parent doc is soft-deleted so the workspace only shows the children. Undelete brings the parent (original combined scan) back if the split was wrong.

Why QR-only: blank-page splitting sounds ergonomic but silently breaks legit multi-page docs that happen to contain a mostly-empty page. QR is explicit — the user prints separator sheets on purpose. Config knobs: see config for SCAN_SPLIT_ENABLED, SCAN_SPLIT_TOKEN, SCAN_SPLIT_DPI.

ZUGFeRD / Factur-X / XRechnung

PDF/A-3 documents with an embedded Cross-Industry Invoice XML get their structured data extracted and written to custom fields:

Custom field	CII source	Type
`invoice_number`	`ExchangedDocument/ID` (BT-1)	text
`invoice_date`	`ExchangedDocument/IssueDateTime` (BT-2)	date
`invoice_currency`	`InvoiceCurrencyCode` (BT-5)	text
`invoice_total_gross`	`GrandTotalAmount` (BT-112)	monetary
`invoice_total_net`	`LineTotalAmount` (BT-109)	monetary
`invoice_seller`	`SellerTradeParty/Name` (BT-27)	text
`invoice_seller_vat`	`SellerTradeParty/SpecifiedTaxRegistration[VA]` (BT-31)	text
`invoice_buyer`	`BuyerTradeParty/Name` (BT-44)	text

The custom field rows are created lazily on the first successful extraction — a stock install without any e-invoices doesn’t clutter the fields table. Recognised attachment names (case-insensitive): factur-x.xml, zugferd-invoice.xml, xrechnung.xml.

Images (barcodes)

Every image upload runs QR / DataMatrix / Aztec decoding via gozxing (pure Go, in-process, no external binary). Any decoded barcode value lands in documents.content as barcode:<value> tokens so the FTS5 index picks them up — search for barcode:INV-2026-0042 and the doc surfaces. The image path skips qpdf / pdf-inspector / ocrmypdf entirely because those tools would fail on non-PDF input. There is no OCR of image uploads today — the design assumes you shot a barcode or a photo of a receipt whose barcode carries the useful key.

EPUB

Pure-Go zip walker under core/pipeline/epub/. Reads META-INF/container.xml → OPF manifest → spine, then concatenates the visible text from each XHTML in reading order into documents.content. Naive tag stripping:

<script> and <style> bodies are dropped.
All other tags become whitespace so words don’t get glued together.
Common named entities (&,  , < etc.) are decoded.

Title (dc:title) and authors (dc:creator) are parsed and logged today; wiring them into custom fields is a follow-up. The output cap is configurable via EPUB_MAX_CONTENT_BYTES — hitting it truncates silently and logs at Warn (epub.truncated).

DjVu

djvutxt from djvulibre-bin extracts the OCR text layer that archive.org and library-scan DjVus normally carry. If the DjVu has no text layer (unusual), the extraction returns empty content — suchi does not OCR DjVu itself; convert to PDF upstream if needed. Output cap: DJVU_MAX_CONTENT_BYTES (default 32 MiB). Same silent-truncate

Warn behaviour as EPUB.

Office documents

anydoc from Firecrawl (MIT, Rust) extracts text from Word, PowerPoint, Excel, OpenDocument, RTF, and CSV files into GitHub-flavored Markdown. The Markdown lands as documents.content and FTS5 indexes it just like OCR’d PDF text. Supported MIMEs (routed by core/pipeline/anydoc.Recognized):

Family	Extensions
Word	`.doc`, `.docx`, `.docm`
PowerPoint	`.ppt`, `.pptx`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`
Excel	`.xls`, `.xlsx`, `.xlsm`, `.xlsb`
OpenDocument	`.odt`, `.ods`, `.odp`
Rich Text Format	`.rtf`
CSV	`.csv`

PDF and EPUB are deliberately NOT routed through anydoc — the PDF path (qpdf → pdf-inspector → ocrmypdf) and the pure-Go EPUB extractor own those MIMEs. anydoc’s own coverage of them isn’t used. Missing anydoc binary = documents in these formats still land, they just don’t get their content indexed. Same posture as missing ocrmypdf. Both Docker images (slim and full) bundle anydoc as of Phase 3.5; bare-metal installs need to place an anydoc executable on PATH themselves — see source caveats below.

anydoc source caveats

Two things worth knowing about how suchi consumes anydoc: Upstream doesn’t ship a standalone CLI binary. anydoc is distributed as a Rust library (crates.io), Node napi binding (npm), and Python wheel (PyPI). Firecrawl’s CLI story is npx @firecrawl/anydoc — a Node wrapper around the napi binding. suchi doesn’t want Node in the runtime image, so the Dockerfile compiles examples/convert.rs from the anydoc repo into a static musl binary and installs it as /usr/local/bin/anydoc. Cost: ~10 MB, no runtime dependency. examples/convert.rs is not upstream-guaranteed as a stable CLI. It’s an example — the argv shape could shift between anydoc releases. As of v0.1.3 it’s convert <file> [-f <fmt>] [-o <out>] [--assets <dir>]. suchi shells out with just anydoc <file>, reading stdout for Markdown. If a future upstream tag changes this shape, suchi’s extractor at core/pipeline/anydoc/anydoc.go needs a matching update; symptoms would be silent Skipped=true with non-zero exit and “unrecognized argument” in res.StderrTail. The bump procedure — enforced by hack/pin-bumper.sh and the Dockerfile comment — is:

Run hack/pin-bumper.sh from repo root. It reports when the pinned version is more than 2 releases behind and suggests a bump to n-1 (never latest — soak time matters).
Read the compare URL it prints. Scan examples/convert.rs for argv changes.
Edit ARG ANYDOC_TAG= in Dockerfile. Rebuild slim, run docker run --rm suchi:slim doctor to confirm anydoc is on PATH, and smoke-test a docx ingest.
Commit as a standalone “bump” commit so it’s easy to roll back.

Alternative if the CLI drift becomes painful: vendor the convert example into suchi’s tree, stripping the language wrappers, and pin the anydoc library directly. Not done today because examples/convert.rs has been stable in shape since the crate’s first release and the maintenance burden isn’t worth it yet.

Not yet supported

.zip / .tar bundles — no auto-explode. Ingest the individual documents.

Add support by dropping a new package under core/pipeline/<format>/ with an Extract(...) that returns (*Result, error), then wire it into core/pipeline/postingest/postingest.go behind a MIME check. Every existing extractor is a template.

Per-format tuning

Env var	Default	Meaning
`PDF_MAX_CONTENT_BYTES`	`8M`	Cap on `pdftotext` output length.
`EPUB_MAX_CONTENT_BYTES`	`32M`	Cap on concatenated XHTML text. Ebooks routinely exceed the PDF-oriented default.
`DJVU_MAX_CONTENT_BYTES`	`32M`	Cap on `djvutxt` output.
`OCR_LANGUAGES`	`eng`	Comma-separated Tesseract language codes. `eng,deu,hin` etc.

All accept K/M/G suffixes or raw byte counts. Truncation is silent + logged at Warn — the doc still lands, FTS still indexes what fit.

JSON sidecar spec

Any producer dropping a file into the fs-watch staging dir (or a mail attachment carrying a Correspondent) can hand suchi structured metadata via a sibling <name>.json (or <path>.json) file. The sidecar shape below is the wire contract — additive fields don’t need a version bump; a required-field change bumps suchi_sidecar.

{
  "suchi_sidecar": 1,
  "title":         "Electricity bill Mar 2026",
  "created":       "2026-03-02",
  "correspondent": "BESCOM",
  "correspondents": [
    { "name": "BESCOM",       "role": "sender"    },
    { "name": "Ritesh S.",    "role": "recipient" }
  ],
  "tags":          ["utilities", "source:fs-watch"],
  "jd_category":   31,
  "notes":         "auto-picked-up",
  "custom_fields": { "amount_inr": "1420", "billing_period": "March" }
}

Field	Type	Semantics
`suchi_sidecar`	int	Must equal `1`. Producers that omit it get the flat-JSON compat path (title/created/correspondent/tags only).
`title`	string	Overrides the filename-derived title.
`created`	string (ISO 8601)	Source date. Stored as `documents.added_at` — the date the doc was created upstream, not the moment suchi ingested it.
`correspondent`	string	Singular sender name. Backwards-compat with pre-Phase-2 producers. When `correspondents[]` is also set, the array wins.
`correspondents[]`	array	Multi-party form. Each entry is `{name, role}`; role is `sender \| recipient \| cc \| other` (`sender` when omitted). Applied via `document_correspondents`.
`tags[]`	array	Tag slugs. Auto-created on ingest.
`jd_category`	int	JD code (e.g. `31` = “Utilities” under area 30–39). Falls through to inbox when unresolved.
`notes`	string	Free-form; stored on the document row.
`custom_fields`	object	Map of custom-field name → value. Field must exist (`/api/custom_fields/`); type coercion follows the field’s declared type.

Source of truth: core/ingest/sidecar/sidecar.go. Parser accepts the native shape and a flat-JSON compat shape (no suchi_sidecar key, top-level title/created/correspondent/tags). Bad JSON or an explicit-but-mismatched suchi_sidecar version fails ingest; the file moves to <dir>/errors/ with a .err companion.

Docker image matrix

Feature	Slim image	Full image
Text-native PDF via `pdftotext`	✅	✅
Scanned PDF via tessocr (pdftoppm + tesseract)	✅	✅
Scanned PDF via ocrmypdf (writes text layer inside the PDF)	❌	✅
PDF normalization via `qpdf`	✅	✅
ZUGFeRD attachment extraction	✅	✅
Image barcode decoding	✅	✅
EPUB extraction	✅	✅
Office documents via anydoc (docx, xlsx, pptx, odt, rtf, csv)	✅	✅
DjVu extraction	❌	✅
Outlook `.msg` extraction	❌	✅
Image size (approx)	~80 MB	~400 MB
Default `OCR_ENGINE`	`tesseract`	`ocrmypdf`

The slim image is the recommended default — it covers PDF end-to-end including OCR of scanned pages, plus office document text extraction via anydoc. Pick full when you need text-selectable scanned-PDF archives (ocrmypdf writes the OCR layer inside the PDF), DjVu ingest, or Outlook .msg parsing.

Overview

Getting started

Reference

Concepts

Operations

Guides

Development

Alternatives

Supported file types

Format matrix

PDF pipeline in detail

OCR engine dispatch

Password-protected PDFs

Email (message/rfc822)

Multi-doc splitting on QR separator sheets

ZUGFeRD / Factur-X / XRechnung

Images (barcodes)

EPUB

DjVu

Office documents

anydoc source caveats

Not yet supported

Per-format tuning

JSON sidecar spec

Docker image matrix

​Format matrix

​PDF pipeline in detail

​OCR engine dispatch

​Password-protected PDFs

​Email (message/rfc822)

​Multi-doc splitting on QR separator sheets

​ZUGFeRD / Factur-X / XRechnung

​Images (barcodes)

​EPUB

​DjVu

​Office documents

​anydoc source caveats

​Not yet supported

​Per-format tuning

​JSON sidecar spec

​Docker image matrix

Format matrix

PDF pipeline in detail

OCR engine dispatch

Password-protected PDFs

Email (message/rfc822)

Multi-doc splitting on QR separator sheets

ZUGFeRD / Factur-X / XRechnung

Images (barcodes)

EPUB

DjVu

Office documents

anydoc source caveats

Not yet supported

Per-format tuning

JSON sidecar spec

Docker image matrix