Skip to main content
Read this before adding a package to core/ or a subcommand to distro/. Companion to SPA architecture (the client) and SPA backend (the endpoint-by-endpoint contract).

​
1. What the backend is, in one paragraph

A single Go binary — distro/cmd/suchi — that owns HTTP, SQLite, blob storage, ingest, the post-ingest pipeline, the durable job dispatcher, and the embedded Svelte SPA. One process, one file tree, one .db. CGO is off (modernc SQLite). Everything the caller sees comes through /api/* (JSON) or /app/* (embedded SPA); there is no second service, no message broker, no external cache.

​
2. The constraints that explain everything else

  1. One binary, one file tree. No sidecars, no daemons, no external services required to run. Anything you’d reach for a Redis or a queue broker for lives in the SQLite jobs table.
  2. Contributors clone and go build. No CGO, no code generators in the critical path, no pre-build steps beyond make ui (SPA-only, and the built dist is committed).
  3. The JSON API is the only contract. The SPA, the mobile apps, the agent surface, and the ingest producers all go through /api/*. If a feature needs a private door, it’s an API bug.
  4. Plugins are additive, versioned, and blank-imported. plugin-api is the only module every other module depends on; it stays small on purpose. New plugin kinds go there; new business logic doesn’t.

​
3. Repo layout

​
4. The SQLite discipline (this is the important one)

core/db opens two pools against the same file:
  • DB.Write — MaxOpenConns=1. Every write goes through it. Serializes at the process level so SQLITE_BUSY cannot happen from our own code.
  • DB.Read — ordinary pool. WAL mode lets readers run concurrently with the writer.
Boot pragmas live in the DSN (journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000, foreign_keys=ON, mmap_size=256MiB). foreign_keys is per-connection in SQLite, so the write pool re-applies it — do not remove that. Rules for handlers:
  • Reads use DB.Read. Writes use DB.WriteTx(ctx, func(tx)).
  • A read that races an in-flight write is fine (WAL); a write that reads is fine (single writer sees its own state).
  • Do not open a third pool. Do not run raw BEGIN on DB.Read.
  • foreign_keys=ON is load-bearing. New migrations must not add cycles that only survive with it OFF.
Migrations live in core/db/migrations/*.sql, embedded via go:embed and applied at boot by db.Migrate. Numbered NNNN_*, apply-once, no down-migrations by design. If you need to change a prior migration’s shape, add a new one.

​
5. HTTP request lifecycle

  • auth.Chain returns the first non-nil *pluginapi.Principal from its authenticators. Order matters (config-defined). A bad-token error stops the chain — no silent downgrade to anonymous.
  • auth.RequireScope(name, scope)(handler) gates a handler on a token scope (e.g. documents:read, events:read).
  • authz.Authorizer.Can(ctx, principal, kind, id, perm) is the per-object decision. Two shipped implementations: RoleAuthorizer (owner-or-admin) and ACLAuthorizer (owner-or-admin plus object_acls + group_members). ACLAuthorizer is the default — empty ACL table behaves like Role, so it’s backward-compatible.
  • List handlers do not call Can per row. They splice authz.DocVisibilityWhere(userID, groups) into the SQL and let the database do the filter. Never post-filter in Go what the DB can filter in SQL — the pagination envelope depends on it.

​
6. The durable outbox (ingest → pipeline)

Every asynchronous unit of work is a row in jobs. The dispatcher (core/jobs) polls that table, hands rows to registered subscribers, and updates state on success/failure. In-process nudge shortens the poll window; the table remains the source of truth. Handler contract:
  • Idempotent. Backoff retries do not distinguish duplicates.
  • Respect ctx. Cancellation must return quickly.
  • Payload is per-kind JSON. Document the schema next to the subscriber.
Subscribers wired in main.go (see runServe): At boot, disp.ReclaimOrphaned(ctx) resets state='running' rows back to pending — a crash mid-handler otherwise strands the job forever. agent:* kinds are excluded because their workers are external processes with their own lease-deadline model. The producer contract: enqueue the job in the same transaction as the row insert. core/pipeline/* upload paths do this; follow the pattern. A doc row without its follow-up job = a document stuck in pre-ingest, and no dispatcher tick will save it.

​
7. Ingest producers

Three canonical entry points, all landing rows in documents and enqueuing postingest:*:
  • POST /api/documents/ — multipart upload (SPA, mobile, agents). Handler in core/api/documents.go.
  • core/ingest/fswatch — staging-directory watcher. Config-driven; owner-email is the settings-first lookup with an env fallback.
  • core/ingest/emailwatch — IMAP polling loop (canonical ingest path #3). Attachments become child docs via the eml pipeline package.
A fourth (core/ingest/sidecar) exists for the paperless-native compat parse and mail-intake compose bundle — it’s a consumer of files another process drops in, still going through the same doc row + job enqueue.

​
8. API package conventions

core/api is one handler per file, no framework, no middleware magic:
  • One handler per file. documents_list.go, groups.go, agent.go. Grep for the endpoint, get the file.
  • s.writeJSON / s.writeError. Everything JSON, no template rendering. Errors are {code, message}; codes are stable strings the SPA and mobile clients switch on.
  • Pagination envelope. List endpoints return {count, next, previous, results} — DRF-shaped for mobile compat. Helpers in httpx (ParsePageParams, BuildEnvelope).
  • Ordering allow-lists. Never pass user-supplied ordering strings into SQL. Each list handler declares an orderingAllow map of accepted keys → SQL fragment. See documents_list.go.
  • Filters use bind params. sqlbuilder-free by design; if you need dynamic clauses, strings.Builder + ? placeholders, and pass every value as an interface. Nothing is interpolated.
  • Audit-write every state change. audit.Log(ctx, Event{...}) in the same transaction as the mutation. before/after JSON is metadata only — never content, never secrets.
The SPA backend doc lists the endpoint table. docs/api.mdx and core/api/schema.json are the operator-facing reference; keep both in sync when adding an endpoint.

​
9. Blob storage

core/blob.CAS is content-addressed by sha256, sharded three levels deep:
  • Duplicate Put is cheap: same hash → same path → return the existing ref.
  • Writes stream through sha256 and land via atomic rename from a per-put temp file in the same sharded directory (same-device).
  • The interface stays concrete for now. When an S3 backend or blob-crypt lands, Put/Get/Stat/Delete lifts into plugin-api and this promotes to plugins/fs-cas. Do not preempt that.
  • Original blobs are immutable. Every derived artifact (archive PDF, thumbnail, decrypted variant) is a separate CAS entry referenced by document_blobs.

​
10. Search (FTS5)

documents_fts is an FTS5 external-content table over documents.content + documents.title. Migration 0003_fts5.sql sets it up; the pipeline writes the content column and the FTS index refreshes via triggers.
  • Query surface: GET /api/search/ uses BM25 ranking + <mark> snippet extraction. Snippets pass through safeSnippet() on the client; server side, the SQL uses snippet() with a fixed token count.
  • Autocomplete: GET /api/autocomplete/ is a prefix scan against the same table.
  • ACL: the search handler splices DocVisibilityWhere before the FTS join — the raw FTS query never sees rows the caller can’t view.

​
11. Auth chain, in more detail

plugins/local-auth is the built-in authenticator. It handles:
  • Authorization: Token … — hashed API tokens. Scopes on the row.
  • Session cookie — set at login, carries the user_id.
  • Password login — POST /api/login mints a token and sets the cookie (dual-channel; the SPA needs both).
plugins/oidc layers on top when configured — its Authenticate runs first if it’s in the chain, and issues a Principal from the verified id_token. plugins/local-auth still gets a look for Token/Bearer. The plugin surface (plugin-api/principal.go) is deliberately minimal: Kind, UserID, Email, Scopes, AuthNBy. Anything more is a role or a permission — those live in authz, not in the Principal.

​
12. Plugin seam

plugin-api is the shared surface. Every other module imports it; it imports nothing from suchi. When something needs to be pluggable, it lands here:
  • Authenticator (auth chain)
  • Subscriber (jobs dispatcher)
  • AuditSink (SIEM export)
  • Value types: Principal, Event, DocRef, BlobRef
Kinds live in plugin-api/kinds.go as string constants — enum by convention, because distro’s plugin index blank-imports one package per plugin and we do not need type safety at the plugin boundary, only vocabulary. New plugins live under plugins/<name>/ with their own go.mod, added to go.work, and blank-imported (or wired explicitly) from main.go. Cross-plugin calls are not a thing — plugins talk to core through plugin-api, and to each other only through jobs + audit events.

​
13. How to add a feature

  1. Pick the surface. JSON in-and-out ⇒ core/api. Async work ⇒ new job kind + subscriber. Cross-cutting ⇒ new package under core/, or a plugin if it should be opt-in.
  2. Migration first. If the schema changes, add NNNN_short_slug.sql. Do not edit an applied migration.
  3. Write the handler and the audit event together. Every state change writes an audit row in the same transaction as the mutation. Not doing so is a bug caught in review.
  4. Wire authz. authz.Can for object-level, RequireScope for token scopes, DocVisibilityWhere for list SQL. Never post-filter in Go what the DB can filter in SQL.
  5. Test the handler. httptest.NewRecorder, real SQLite in t.TempDir, real migrations. Mock nothing you can seed. See documents_list_test.go for the pattern.
  6. Docs update in the same commit. New endpoint ⇒ docs/api.mdx + core/api/schema.json. New env var ⇒ docs/config.mdx. New CLI flag ⇒ docs/cli.mdx. New importer flag ⇒ docs/importer.mdx.
  7. make fmt && go test ./... && gofmt -l core/ before pushing. CI’s gofmt gate is not a suggestion.

​
14. Known trade-offs

  • SQLite + one writer. Not a limitation — a discipline. If you find yourself wanting Postgres, we probably want a worker-tier plugin first. Bringing in a second database engine is a design review, not an import.
  • No ORM. database/sql + hand-written SQL. Grep-ability and query control beat repository patterns at this scale. If a query is repeated enough to want a helper, promote it to a named function in the same file.
  • No framework. net/http.ServeMux + http.HandlerFunc. Method-scoped routes ("GET /api/documents/") are Go 1.22+; use them. Middleware is a function that wraps a handler; there is no chain type.
  • Durable outbox over Kafka/NATS/Redis. SQLite jobs table is the queue. Delivery-once semantics come from the write-pool serialization + state='running' reclaim at boot. If you need fan-out or cross-process coordination, that’s a design review.
  • No generated code in the critical path. No sqlc, no protobuf codegen for the JSON API. If you catch yourself reaching for it, ask why the hand-written version is not good enough.

​
15. Glossary

CAS — content-addressed store; blobs indexed by sha256. Outbox — the jobs table treated as a queue: producers write rows in the same tx as the state they describe, consumers (the dispatcher) read those rows and update state. Post-ingest — the first job the pipeline runs after a doc row lands; the qpdf → text-native → OCR chain lives here. Rendered view — the symlink tree under $DATA_DIR/rendered/ that presents documents by Johnny.Decimal path; every metadata mutator enqueues a render:* job to keep it consistent. Principal — the authenticated caller: user id, email, scopes, authenticated-by plugin name. Anonymous = nil. Authz — the per-object decision layer (Can). Scope — a token-level permission (documents:read, events:read). Scopes gate handlers; authz gates rows. Audit event — append-only row in audit_events describing a state change; the request-id links it back to the log line.