One process per archive
One Suchi process owns an archive at a time.core/archivelock holds an
exclusive, nonblocking operating-system lock on a mode-0600 .suchi.lock file
in DATA_DIR, and core/db.Open takes it before opening SQLite and keeps it
for the database lifetime. A second suchi serve, or an offline command such
as export or GC, against the same directory fails before touching the database
with the stable error archive_in_use. The lock file must be a regular file;
a symlink is refused. It is released after both pools close.
The lock belongs to the process, not the file. If Suchi exits or crashes, the
operating system releases it, so the .suchi.lock file left behind is harmless
and never needs deleting. This process boundary protects the single writer
pool, startup reclamation, jobs and intake loops; it does not make shared
network filesystems supported storage.
The single writer
DB.Write has one connection; DB.Read has a small pool. Both use the same WAL
SQLite file with foreign keys, busy timeout and bounded mmap. Mutations use
WriteTx; participating helpers receive its *sql.Tx. Re-entering the writer
from its transaction deadlocks on the held connection. Deferred rollback releases
it after errors or panics, including when callers recover.
Keep network calls, subprocesses and long processing outside write transactions.
Enqueue required work with the state change, never as a later best-effort write.
Ordinary audit persistence/delivery remains best-effort and is not the
transactional job outbox. Identity binding, account-email changes and capability
transitions are narrow exceptions: core/audit.RecordInTx is required inside
their state transaction. An insertion failure rolls back the identity or
capability change, dependent resource revocations, queued render work and session
rotation. Sink fan-out still happens only after commit and cannot undo it.
Filing systems, tags, and accounts
core/jd/systems is the leaf owner of identity, membership, target resolution and
address syntax. System 1 stays hidden until first prefixed import atomically names
it or preserves it separately. Codes and document system ownership are immutable;
names/tree/provenance belong to the system. JD owns presentation/import rules.
The taxonomy upgrade retains surviving global document IDs and introduces
AUTOINCREMENT. IDs purged before upgrade have no retained allocation history and
may be reused; nonreuse starts at upgrade. Categories, metadata, views, automations,
mailboxes, passwords, shares and approval definitions/runs are system-owned.
Composite constraints/targeted guards reject foreign-system relationships.
Authenticated custom-field document links are the exception described above;
authorization is checked at both endpoints, while all projections and reverse
lookups redact inaccessible rows before counts or pagination.
core/api/tags.go owns tag catalog writes, independently of document
assignment via core/api/documents_bulk.go. A document assignment uses an
existing tag ID and document change permission; it neither creates catalog
entries nor grants catalog administration. Catalog single/batch deletion
requires an admin browser session in the selected system. The batch handler
validates every ID inside one DB.WriteTx, detaches surviving children, then
deletes the selected IDs in a bounded statement; missing/foreign IDs roll back
without deleting document-tag links. Foreign keys remove only links to deleted
tags. One post-commit batch audit records IDs/count; no background job, network
egress, or plugin contract is involved. The current schema declares
ON DELETE SET NULL for tag parents; the handlers still detach children
explicitly so archives adopted from older schemas with a cascading foreign key
behave the same.
Verify database and wire behavior with go test ./core/api -run TestDeleteTagsAtomicAndReroots and go test ./core/db -run TestTagParentDeleteReroots; API reference owns status/body details.
SQLite/FTS/CAS remain shared. Document deduplication is system/owner scoped;
user-global idempotency keys bind fingerprints to the system and never exempt
receipt replay from authorization. Users/groups, roles/capabilities, sessions,
built-in catalog, server/OCR/LLM settings and full backups are instance-wide.
Admins and host/backup operators remain trusted; system codes do not create
separate infrastructure or keys.
Account identity also remains instance-wide. Numeric users.id is the durable
owner/principal key; email is a mutable sign-in attribute, and OIDC returning
identity is the unique non-null (oidc_issuer, oidc_subject) pair stored directly
on that row. Email changes do not rewrite document owners, memberships, ACLs,
tokens, or other numeric user references.
Local two-factor state is also instance-wide and keyed directly to users.id.
local_auth_totp stores one sealed seed, its exact last accepted time step, the
consecutive-failure count and block deadline, and enable time; local_auth_recovery_codes stores only SHA-256 digests of unused
codes. Both cascade with the user. The .decrypt-key opens the seed during
online authentication, while recovery-code comparison and offline factor reset
never require plaintext seed access. The security architecture
guide owns
authentication, rotation, replay, and recovery transactions.
Blobs and rendered paths
Original/derived bytes are SHA-256-addressed:suchi doctor --scrub-cas explicitly
rehashes stored objects when an operator wants an integrity check.
CAS.Put streams to a temporary file at the CAS root, then renames on the same
filesystem. A failed subsequent database transaction may leave an unreferenced
blob. Never delete by hash on request failure: another document or in-flight
publisher may use it. Online purge, including demo expiry, never deletes CAS;
only offline core/gc/ with all archive writers stopped reclaims bytes.
Blob age cannot establish concurrency safety.
Preserve original_blob and original MIME; decryption, normalization and OCR
produce derived objects. Rendered paths are rebuildable. Trash/retention purge
snapshots candidate paths plus document/journal hashes in the writer before
removing rows. Post-commit cleanup pins real parent directories and, per path in
its own writer transaction, skips paths another document still claims and
removes only matching CAS symlinks; foreign artifacts remain and count as
cleanup failures.
Rendering prepares directories outside the writer, then shares a short liveness/
ownership check and atomic link replacement with purge’s snapshot transaction.
A paused render cannot recreate a deleted document’s link. core/render/paths
owns CAS-link validation shared with Trash; targets come from
validated CAS.Path, not duplicated shard logic. Same-path jobs repair missing/
stale links to the selected archive/original blob. Startup does not rewrite the
whole document projection; repair occurs on render/refile.
core/render/view derives category address and path at render time. Default JD
and flat templates always retain __{{ doc_pk }} for uniqueness; introduction
adds the immutable <SYS>/ root but never substitutes jd.address for document
identity. Explicit templates may omit document identity and collide.
core/render/view/pathclaim derives path ownership from each unpurged
document’s pending and latest-applied render_moves, comparing a Unicode
case-folded NFC key (*_path_key, indexed) so case/normalization variants
collide on every platform. Trashed documents keep their claim until purge, so
restore never meets a second owner. View checks claims in the writer before
journaling, publishing or deleting a link, and Trash before unlinking; a
matching CAS hash never lets one document claim or remove another’s projection.
A refused document journals nothing, keeps any earlier applied link, and its
render job fails until a later render finds the path free. Foreign files and symlinks remain
untouched. The address cutover does not rewrite stored
templates, historical links, or the whole archive.
core/render/index refreshes a current tree-only core/taxonomy snapshot at
startup/tree changes. After introduction its path is
<SYS>/00-09 System index/00.00 archive.huml; before introduction it uses the old
root. presetfile.Marshal serializes it; a unique same-directory temporary file
is written, flushed and closed before atomic replacement. Refuse symlinked
parents/targets and unexpected files; failures preserve the previous complete
index. It creates no area/category/document/blob rows. Reserve this namespace
from document templates. View cannot import taxonomy because taxonomy depends
on view.
Index snapshot/publication and view journal/publication each use their owner’s
mutex. Initial renders, moves and same-path blob changes journal previous/intended
path/blob pairs before publication. Keep superseded evidence until current
publication and old-path cleanup finish transactionally. Recovery targets the
current system destination and cleans only verified links to retained document
blobs or that path’s journaled blobs; same-path replacement also proves ownership.
Never republish a superseded destination merely because it was once valid.
View resolves system/mode per document, enforcing a code-only root outside explicit
templates; display-name edits do not move roots. Generated paths
own filename/default-mode details. Only system 1 removes a proved-managed legacy
index after its named replacement succeeds. Unknown files stay untouched. An old
template link occupying a future index namespace may be relocated only when its
journaled CAS target verifies through the pinned parent; new destinations still
refuse that namespace. Index retry can then converge without deleting foreign
files or deadlocking recovery.
Pre-upgrade journal hashes may be unknown. Recovery reports and preserves a link
to an unreferenced historical archive when ownership cannot be established.
Restored absolute links with a known hash and the pre-v0.1.0 two-shard or
current three-shard CAS layout may be repaired. Render-only refile changes projections
without OCR/classifier/filing reruns. Backup and restore requires
all of DATA_DIR, including credential keys, with writers stopped or an atomic
whole-directory snapshot; a database-only copy is incomplete.
Schema migrations
The active catalog incore/db/migrations/ starts from an immutable baseline
for stable schema 1 (v0.1.0) and adds each release’s migrations in order; the
current checkout reaches schema version 5. The v0.2.0 migrations convert
document types to namespaced tags, add version-family currentness, and add
classifier-backed automation questions. The v0.3.0 migration adds title
ownership, conservatively marks existing curated and imported titles as
human-owned, and extends title revision tracking to owner changes. The v0.4.0
migration adds two-factor sign-in, source file names and client labels, and
stops rewriting the search indexes when a write leaves text unchanged. The one-row
schema_lineage = 'stable-v1' identity is unchanged. Frozen pre-v0.1.0 SQL and
the final account-identity rebuild live separately as compatibility migrations;
the active catalog cannot discover them by path. Migration file names and
grouping are implementation detail; the release process compatibility
table records which schemas
each release accepts.
The v0.2.0 document-type migration removes the duplicate vocabulary after
mapping each row and every live document, ACL, saved-view, automation, and
template reference to a type:<name> tag. Unrepresentable tag-and-type
automation conjunctions suspend rather than broadening. Source-archive types
are translated at the importer boundary; no native type schema or API remains.
migrations.Prepare delegates core admission to db.PrepareStable, which
classifies the database before writing. Empty databases receive all active
migrations. An exact current schema-5 database is a no-op. Exact stable schemas
1, 2, 3, and 4 first pass integrity and foreign-key checks, then apply only their
remaining active migrations and must match the current manifest. Every accepted
pre-stable state must match its complete canonical sqlite_schema fingerprint
and lineage, not merely a user_version. Unknown, partial, spoofed, corrupt and
future states fail without mutation. The exact pre-identity schema-4 fingerprint
is a separate explicit compatibility boundary; a schema claiming stable lineage
with an unrecognized fingerprint is deliberately rejected. The release process
compatibility table is the
sole authority for supported incoming states, frozen steps and the required
release test matrix.
Every admitted pre-stable archive passes integrity and foreign-key checks and
gets an exclusive mode-0600 SQLite snapshot before adoption. Remaining
compatibility steps, the remaining active migrations, stable lineage and the
current schema version commit in one immediate transaction through the sole
writer. The final compatibility step rebuilds users, documents and
audit_events; adds
provider identity, a durable development-account marker and retained-event
metadata; moves the singular correspondent into the role-bearing junction; and
preserves primary keys, revision values and dependent references. It also
resolves watched-folder ownership to a numeric user ID, drops the empty obsolete
plugin_kv table and redundant ACL index, and removes successful jobs older
than seven days. The final integrity, foreign keys and canonical schema-5
fingerprint must match a fresh archive; failures roll back while retaining the
snapshot and restoring foreign-key enforcement.
Extension objects created after _suchi_extension_migrations are outside the
core fingerprint. Core objects rebuilt during adoption remain explicitly
included even when SQLite places their replacement rows after that extension
boundary.
Core migration history is bounded at major-version schema epochs. The outgoing
major’s bridge validates its last canonical schema and writes an
application-owned epoch/version marker; the incoming major creates one
declarative baseline and carries only migrations from that epoch. Direct older
conversion stays in the bridge or a separate offline command, not steady-state
server startup. The current pre-1.0 admission machinery remains required until
the v1 bridge ships; Release process
owns the cutover policy.
Extension migrations
Compiled extensions usecore/db.MigrationSet and MigrateSet, independently of
PRAGMA user_version. Each component supplies its complete ordered history from
version 1. The reserved _suchi_extension_migrations table stores
(component, version) as its primary key, a SHA-256 checksum of exact SQL plus
rebuild mode, and Unix-second applied_at. Component names are bound data, never
SQL identifiers. Invalid names, duplicates/gaps/out-of-order versions, empty SQL,
missing applied history and checksum drift fail explicitly. Unchanged reruns do
not rewrite ledger rows. Each migration and its ledger record commit together
through the sole writer; SQL and foreign-key failures roll back DDL/data as well
as the record. RebuildTables uses the same foreign-key restoration/checking
boundary as core migrations. Extension SQL is trusted compiled code and must not
control transactions, core user_version, or the reserved _suchi_ namespace.
Core migrations and published SQL retain their existing versioning and behavior.
Extension tables remain owned by the component that declared them; core provides
no generic plugin key/value table. Before a compatibility migration rebuilds a
core table, adoption temporarily removes extension-created indexes and triggers
from after the extension-ledger boundary, then replays their exact DDL. A
dependency on a retired object such as plugin_kv, or an incompatible object
recreated by core, aborts the migration atomically instead of silently discarding
extension state.
Device OCR provenance, split origins, idempotency, pairing/token provenance precede
taxonomy rebuilds. Existing credential hashes remain; unknown pairing provenance
stays empty; default memberships/token bindings move to system 1 without issuing
credentials. Tests pin published SQL, populated adoption behavior, rollback and
writer enforcement; release acceptance remains owned by the linked compatibility
matrix.