Files
pages/docs/design/projection-metadata-cache.md
T
Anya Tokugawa c1175d389f Implement Knowledge City platform, organizer and DAV integration
Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming.

Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
2026-09-27 14:44:43 +09:00

110 lines
6.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Deferred PDF and archive metadata
Implemented: 2026-09-14. PDF page counts and non-ZIP archive listings no longer
run `pdfinfo` or `7z` inside a Viewer, preview, print, or metadata HTTP request.
## Request and worker contract
The existing authorized routes expose a metadata mode:
| Request | Resolution and authorization |
| --- | --- |
| `/v/pdf-page?rev=<UUID>&metadata=1` | immutable Revision → owning Object's current read authority |
| `/v/archive?path=<Path>&rev=<UUID>&metadata=1` | current Domain Path read authority; supplied Revision must belong to that Path's Object |
The cache is never addressed by an unauthenticated hash-only public endpoint.
Authorization is checked even for ready/failed cache hits. Responses vary on
`Authorization, Cookie` and use `private, no-store`, so revoked access cannot
be bypassed through a browser-cached metadata response.
- `202`: `{"status":"pending"}`, with `Retry-After: 2`.
- `200`: `{"status":"ready", "pdf_pages": N}` or bounded archive `entries`.
- `422`: a cached failed projection, with a retry timestamp and `Retry-After`.
- `503`: queue/storage unavailable; raw download is still an independent path.
A miss enqueues `projection.metadata.generate`, keyed by
`<parser version>:<kind>:<immutable Document SHA-256>`. Its payload contains only
the hash, kind (`pdf` or `archive`), and parser version. Active queue entries are
idempotent across requests. Metadata uses the existing scheduler concurrency,
pause, foreground-work reservation, cancellation, logs, and manual retry UI;
it does not create an independent worker pool. Its priority is currently -5,
below ordinary preview jobs, and its max attempts is 1. A failed decoder result
is cached for five minutes to prevent repeated visits from creating a retry
storm; a scheduler manual retry can run it earlier.
The semantic Viewer and preview use a metadata-only fast path for PDFs and
non-ZIP archives. They do not load the entire Revision body before queueing.
The worker opens the immutable Document Store file directly; there is no
request-time archive copy into a temporary file. Print uses the same metadata
cache for archive tables. Raw PDF iframe delivery remains independent.
## Persistent cache and bounds
`projection_metadata` is keyed by `(document_hash, kind, version)`. The payload
contains no semantic Path, Revision UUID, generated entry URL, principal, or
credential. Request-specific archive links are created after authorization and
pin the viewed Revision, so replacing the Page cannot silently extract an entry
from a different archive revision.
| Boundary | Current limit |
| --- | --- |
| Input Document | 512 MiB, regular Document Store file |
| Decoder wall time | 30 seconds; one-second process wait grace |
| PDF stdout / accepted count | 64 KiB / 1–10,000 pages |
| `7z` stdout | 4 MiB, with partial listing explicitly marked truncated |
| Archive entries | 2,000 |
| Entry name / total name bytes | 4 KiB / 256 KiB; UTF-8, no control characters |
| One persisted result | 1 MiB |
| Aggregate cache | 64 MiB payload/error budget and 2,048 records |
External commands use `nice -n 15`; `7z` has one decoder thread, no interactive
password prompt, and no shell interpolation. Output is bounded by a writer
that does not expose `bytes.Buffer.ReadFrom` (which would bypass the writer
limit through `io.Copy`). Excess output cancels the process. Cache eviction is
performed transactionally by the background writer, oldest generation first;
foreground hits do not update timestamps or trigger global eviction scans.
Evicted entries can be reconstructed from immutable Documents.
## Browser behavior
Pending and failed states are explicit and localized in Japanese and English.
The client polls every 2–10 seconds, stops after 60 pending polls, pauses while
the tab is hidden, and aborts on page exit. A retry button handles a paused queue,
network failure, or cached decoder failure. It does not reload the whole page.
PDF navigation picks up the total as soon as metadata arrives. An asynchronously
loaded archive table is built using DOM text nodes, never filename HTML. Comic
mode initializes once its images arrive and preserves single/spread pages,
keyboard navigation, hidden inactive images, and browser-history behavior.
ZIP remains the existing in-process central-directory Viewer and comic mode;
its displayed listing is capped at 2,000 entries and its entry URLs now also pin
the Revision. Large ZIP central-directory parsing still needs a later metadata
worker migration if such archives become common.
## Explicitly deferred work
- Non-ZIP **entry extraction** still streams through `7z` in the request path,
capped at 128 MiB and 60 seconds. It now treats entry names literally (no
wildcard or `@listfile` expansion), safely formats response filenames, and
cancels the child when the output cap/client disconnect stops draining it.
Persisted, seekable entry projections require their own storage quota,
leases, Range/cache semantics, and archive-bomb policy.
- Listing is the top-level listing reported by `7z`, not an unlimited recursive
traversal of nested archives. Compressed TAR envelopes and nested containers
require a separate bounded expansion/index contract.
- Per-child-process memory/cgroup quotas and hardened decoder isolation remain
deployment work. Input/output/thread/time limits do not imply a hard peak
memory bound inside a native decoder.
- Multi-instance worker leases and cancellation fencing remain scheduler-wide
work; this feature does not claim distributed exactly-once execution.
## Tests
Store tests cover key/payload validation, parser-version separation, persistence,
and eviction. Server tests prove that cold Viewer/print/metadata calls do not
invoke tools, repeated requests enqueue one job, cached authorization denial
does not leak metadata or enqueue work, decoder errors are cached, limits are
enforced, and ZIP links retain immutable revisions. Opt-in Chromium tests cover
pending → ready PDF/table/comic UI, cached failure/retry presentation, and
hostile archive filenames rendered as text.