Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming. Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
110 lines
6.0 KiB
Markdown
110 lines
6.0 KiB
Markdown
# Deferred PDF and archive metadata
|
||
|
||
Implemented: 2026-09-14. PDF page counts and non-ZIP archive listings no longer
|
||
run `pdfinfo` or `7z` inside a Viewer, preview, print, or metadata HTTP request.
|
||
|
||
## Request and worker contract
|
||
|
||
The existing authorized routes expose a metadata mode:
|
||
|
||
| Request | Resolution and authorization |
|
||
| --- | --- |
|
||
| `/v/pdf-page?rev=<UUID>&metadata=1` | immutable Revision → owning Object's current read authority |
|
||
| `/v/archive?path=<Path>&rev=<UUID>&metadata=1` | current Domain Path read authority; supplied Revision must belong to that Path's Object |
|
||
|
||
The cache is never addressed by an unauthenticated hash-only public endpoint.
|
||
Authorization is checked even for ready/failed cache hits. Responses vary on
|
||
`Authorization, Cookie` and use `private, no-store`, so revoked access cannot
|
||
be bypassed through a browser-cached metadata response.
|
||
|
||
- `202`: `{"status":"pending"}`, with `Retry-After: 2`.
|
||
- `200`: `{"status":"ready", "pdf_pages": N}` or bounded archive `entries`.
|
||
- `422`: a cached failed projection, with a retry timestamp and `Retry-After`.
|
||
- `503`: queue/storage unavailable; raw download is still an independent path.
|
||
|
||
A miss enqueues `projection.metadata.generate`, keyed by
|
||
`<parser version>:<kind>:<immutable Document SHA-256>`. Its payload contains only
|
||
the hash, kind (`pdf` or `archive`), and parser version. Active queue entries are
|
||
idempotent across requests. Metadata uses the existing scheduler concurrency,
|
||
pause, foreground-work reservation, cancellation, logs, and manual retry UI;
|
||
it does not create an independent worker pool. Its priority is currently -5,
|
||
below ordinary preview jobs, and its max attempts is 1. A failed decoder result
|
||
is cached for five minutes to prevent repeated visits from creating a retry
|
||
storm; a scheduler manual retry can run it earlier.
|
||
|
||
The semantic Viewer and preview use a metadata-only fast path for PDFs and
|
||
non-ZIP archives. They do not load the entire Revision body before queueing.
|
||
The worker opens the immutable Document Store file directly; there is no
|
||
request-time archive copy into a temporary file. Print uses the same metadata
|
||
cache for archive tables. Raw PDF iframe delivery remains independent.
|
||
|
||
## Persistent cache and bounds
|
||
|
||
`projection_metadata` is keyed by `(document_hash, kind, version)`. The payload
|
||
contains no semantic Path, Revision UUID, generated entry URL, principal, or
|
||
credential. Request-specific archive links are created after authorization and
|
||
pin the viewed Revision, so replacing the Page cannot silently extract an entry
|
||
from a different archive revision.
|
||
|
||
| Boundary | Current limit |
|
||
| --- | --- |
|
||
| Input Document | 512 MiB, regular Document Store file |
|
||
| Decoder wall time | 30 seconds; one-second process wait grace |
|
||
| PDF stdout / accepted count | 64 KiB / 1–10,000 pages |
|
||
| `7z` stdout | 4 MiB, with partial listing explicitly marked truncated |
|
||
| Archive entries | 2,000 |
|
||
| Entry name / total name bytes | 4 KiB / 256 KiB; UTF-8, no control characters |
|
||
| One persisted result | 1 MiB |
|
||
| Aggregate cache | 64 MiB payload/error budget and 2,048 records |
|
||
|
||
External commands use `nice -n 15`; `7z` has one decoder thread, no interactive
|
||
password prompt, and no shell interpolation. Output is bounded by a writer
|
||
that does not expose `bytes.Buffer.ReadFrom` (which would bypass the writer
|
||
limit through `io.Copy`). Excess output cancels the process. Cache eviction is
|
||
performed transactionally by the background writer, oldest generation first;
|
||
foreground hits do not update timestamps or trigger global eviction scans.
|
||
Evicted entries can be reconstructed from immutable Documents.
|
||
|
||
## Browser behavior
|
||
|
||
Pending and failed states are explicit and localized in Japanese and English.
|
||
The client polls every 2–10 seconds, stops after 60 pending polls, pauses while
|
||
the tab is hidden, and aborts on page exit. A retry button handles a paused queue,
|
||
network failure, or cached decoder failure. It does not reload the whole page.
|
||
PDF navigation picks up the total as soon as metadata arrives. An asynchronously
|
||
loaded archive table is built using DOM text nodes, never filename HTML. Comic
|
||
mode initializes once its images arrive and preserves single/spread pages,
|
||
keyboard navigation, hidden inactive images, and browser-history behavior.
|
||
|
||
ZIP remains the existing in-process central-directory Viewer and comic mode;
|
||
its displayed listing is capped at 2,000 entries and its entry URLs now also pin
|
||
the Revision. Large ZIP central-directory parsing still needs a later metadata
|
||
worker migration if such archives become common.
|
||
|
||
## Explicitly deferred work
|
||
|
||
- Non-ZIP **entry extraction** still streams through `7z` in the request path,
|
||
capped at 128 MiB and 60 seconds. It now treats entry names literally (no
|
||
wildcard or `@listfile` expansion), safely formats response filenames, and
|
||
cancels the child when the output cap/client disconnect stops draining it.
|
||
Persisted, seekable entry projections require their own storage quota,
|
||
leases, Range/cache semantics, and archive-bomb policy.
|
||
- Listing is the top-level listing reported by `7z`, not an unlimited recursive
|
||
traversal of nested archives. Compressed TAR envelopes and nested containers
|
||
require a separate bounded expansion/index contract.
|
||
- Per-child-process memory/cgroup quotas and hardened decoder isolation remain
|
||
deployment work. Input/output/thread/time limits do not imply a hard peak
|
||
memory bound inside a native decoder.
|
||
- Multi-instance worker leases and cancellation fencing remain scheduler-wide
|
||
work; this feature does not claim distributed exactly-once execution.
|
||
|
||
## Tests
|
||
|
||
Store tests cover key/payload validation, parser-version separation, persistence,
|
||
and eviction. Server tests prove that cold Viewer/print/metadata calls do not
|
||
invoke tools, repeated requests enqueue one job, cached authorization denial
|
||
does not leak metadata or enqueue work, decoder errors are cached, limits are
|
||
enforced, and ZIP links retain immutable revisions. Opt-in Chromium tests cover
|
||
pending → ready PDF/table/comic UI, cached failure/retry presentation, and
|
||
hostile archive filenames rendered as text.
|