Skip to main content

​
Layout

Monorepo, multiple Go modules glued by go.work. Module boundaries are the contract, not repo count.
Plugins never touch the core schema directly — every plugin gets plugin_kv(plugin_name, key, value_json) for its scratch space.

​
Where does new code go? plugins/ vs core/pipeline/

Two homes, one rule:
  • plugins/<name>/ — a separate Go module with its own go.mod, imported into distro/cmd/suchi/main.go via a blank import. Meant for things that (a) register with the plugin-api (Subscriber, Authenticator, AuditSink), (b) have their own lifecycle (New(), boot-time configuration), or (c) could plausibly be replaced by an external contributor (an alternative auth backend, a different classifier). Examples: plugins/local-auth, plugins/oidc, plugins/llm-classifier.
  • core/pipeline/<name>/ — a package inside the core module. A pure library that other core code calls synchronously. No lifecycle, no registration, no imports from plugins/. Meant for file-format extractors and ingest primitives: exposes Recognized(mime) and Extract(ctx, src, log, opts) shaped like core/pipeline/djvu. Examples: pdf, pdfinspector, qpdf, tessocr, ocrmypdf, epub, djvu, msg, heic, barcode, eml, anydoc.
Rule of thumb: if it lives on the outbox as a Subscriber or gets constructed once at boot and passed a handle, it’s a plugin. If it’s a pure function called from postingest.go, it’s a core/pipeline/ package. Direction of imports: core never imports from plugins/. distro imports from both. Getting this backwards (a core/pipeline/ package depending on a plugins/ module) creates a circular go.work headache and blurs the “core is the platform, plugins are the extensions” architecture.

​
Data flow

​
Non-negotiable design principles

Every architectural decision is tested against these.
Idle RAM ≤ ~100 MB; ~35 MB slim image; runs on a $5 VPS or a Pi.
One binary, one config, one data dir. suchi --help fits on a screen. Every knob has an env-var equivalent.
stdlib where possible, SQLite by default, no message broker, no cache tier. Modern is adopted only when it makes the stack more boring.
Every integration point is an interface in plugin-api/.
No in-memory-only work queues; the durable outbox table is the truth.
Blobs + SQLite are the source of truth. Rendered file tree, thumbnails, FTS index are all regenerable.
PDFs are the malware delivery format. Every parser runs as a subprocess with timeouts, rlimits, output caps, no network. See core/sandbox/.
No telemetry, ever. Every egress opt-in, visible in config, logged. See privacy.
Not an add-on. Ships in every default image alongside filesystem watch and upload.

​
Durable job outbox

Every unit of async work is a row in jobs. The dispatcher polls jobs_ready (partial index on state='pending' AND next_run_at<=now) and hands rows to registered subscribers.
  • Handlers are idempotent — retry semantics assume this.
  • Ingest writes the document row AND its post-ingest job in the same transaction. There is no window where a doc exists but its work is lost.
  • Failures back off exponentially, cap at attempts=5, then move to state='dead'. Dead rows are visible via /api/tasks/ — the same table doubles as the mobile-app tasks endpoint in Phase 4, free.

​
SQLite discipline

Two connection pools on the same file:
  • Write pool: MaxOpenConns=1 + BEGIN IMMEDIATE. One writer, serialized at the pool level so SQLITE_BUSY is structurally impossible.
  • Read pool: ordinary WAL readers, concurrent with the writer.
Pragmas set in the DSN so the first connection is already correctly configured: journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, foreign_keys=ON, mmap_size=256M, temp_store=MEMORY. Migrations are one-way and gated by PRAGMA user_version. Down migrations aren’t supported — production rollbacks restore from the pre-migration snapshot backups take before each schema change.

​
Post-ingest pipeline

Every uploaded PDF flows through a fixed chain, hosted by the core/pipeline/postingest Handler and dispatched via the durable outbox: Every subprocess in the chain runs via core/sandbox — empty env, no network, hard timeout, bounded output, fresh workdir removed on return. Every step degrades gracefully: if a binary isn’t installed (or the input is unrecognized), the step logs a skip and the pipeline continues with the previous state. The design principle: ingest completes even when some tools aren’t installed.
  • qpdf strips restrictions + decrypts empty-user-password PDFs before ocrmypdf sees them. Absorbs decompression bombs.
  • pdf-inspector / pdftotext decides whether the PDF has enough embedded text to skip OCR entirely (threshold: 32 non-whitespace chars). If yes → shortcut writes text into documents.content.
  • ocrmypdf runs only when the shortcut fails. Emits a searchable-PDF archive (stored in the CAS as archive_blob) and a plain-text sidecar (stored in documents.content). The FTS5 trigger picks up the content change automatically.
  • zugferd (Factur-X / XRechnung / ZUGFeRD) inspects the PDF for an embedded invoice XML attachment and, if present, populates invoice_number, invoice_date, invoice_currency, invoice_total_gross, invoice_total_net, invoice_seller, invoice_seller_vat, invoice_buyer custom fields. The custom_fields rows are created lazily on the first extraction.
  • epub ingest is a pure-Go zip walker: reads META-INF/container.xml → OPF → spine, concatenates the visible text from each XHTML in reading order into documents.content. No qpdf, no ocrmypdf, no external binary — works in the slim image too.
  • djvu ingest routes through djvutxt (djvulibre-bin) which pulls the embedded text layer from OCR’d DjVu scans (archive.org / library archive workflows). Missing binary → skipped, doc lands with empty content and can be re-ingested later.
Rules-engine classification runs after this (Phase 2 later); LLM / agent classification is Phase 3.

​
Workflow engine

Post-ingest pushes a document into a consistent state. What happens next — the manager sign-off, the finance approval, the escalation when nobody answers in 24 h — lives in the workflow engine. Definitions are versioned JSON in approval_defs; each run keeps one current_state cursor in approval_runs; every transition appends to approval_transitions (audit log); human steps land in approval_tasks. Advancement rides the same durable outbox as everything else via approval:advance / approval:resume / approval:timeout-sweep — no second scheduler. See approvals for the spec shape, the six HTTP endpoints, and a worked invoice-approval example. For “when doc lands, do X” rules that don’t need a human click, see automations.

​
Permissions

The multi-user data model has been there since Phase 0 — every document has an owner_id. Phase 6 adds:
  • groups — named collections of users
  • group_members — user ↔ group junction
  • object_acls — polymorphic grants keyed on (object_kind, object_id, principal_kind, principal_id, perm_bits)
The authz.Authorizer interface owns the decision. Two shipped implementations:
  • RoleAuthorizer — legacy owner+admin gate. Zero ACL awareness.
  • ACLAuthorizer — folds ACL grants over the owner/admin fallback. Wired as the default at api.New(); empty ACLs behave identically to RoleAuthorizer, so this is backward-compatible.
Perm bits are view=1, change=2, delete=4; grants OR together. Admin bypasses ACLs entirely; owner has the full mask on their own objects. See permissions for the full model, REST surface, and worked examples. Enterprise builds swap the interface for a SAML/SCIM-aware implementation without touching handlers.

​
Automations vs Approvals

Two distinct engines with well-separated concerns:
  • Automations (core/automations/) — trigger→conditions→actions rules that fire on document events. No human involvement; matches Paperless-ngx’s “workflows” shape in the mobile-compat plane but named for what it does. See automations.
  • Approvals (core/approvals/) — human-in-the-loop state machines with timeouts. Doc 17 needs finance sign-off before landing in the vault. See approvals.
The rules engine (core/classify/rules/) sits below both — it’s the one-shot deterministic classifier that runs first in post-ingest.

​
Headless mode

SUCHI_UI_DISABLED=1 skips registration of every browser-facing route at boot: / (SPA redirect), /app/* (embedded Svelte SPA), /preview/{id}, /download/{id}, /login, /bootstrap, and /assets/*. What stays: /api/* (the JSON surface), /healthz, /readyz, /metrics. Blob access for headless callers keeps working via the /api/documents/{id}/{preview,download} mirrors. Intended for deployments where a separate frontend — an external SPA built against /api/, an internal-tools dashboard, a mobile app — is the primary UX. suchi runs as an API server; the frontend is whatever the operator hosts. The three auth chains (Token, Bearer, OIDC) are unaffected. The first-boot bootstrap flow (POST /setup) is unaffected — it’s an API endpoint, not a UI page — but you’ll need to POST the token manually since the /bootstrap browser wizard is gone.

​
Content-addressed storage

Blobs live at $DATA_DIR/blobs/sha256/{aa}/{bb}/{full-hash}. Puts stream through SHA-256 into a per-put temp file in the destination shard, fsync, atomic rename. Same-device by construction so the rename is POSIX-atomic. Duplicate puts return the existing ref without rewriting — free dedup. Documents carry two blob references:
  • original_blob (NOT NULL) — the bytes as ingested. Dedup key.
  • archive_blob (nullable) — the OCR-rewritten PDF. Regenerable in principle; stored because OCR is expensive.