Files
pages/docs/design/projection-adapters.md
Anya Tokugawa c1175d389f Implement Knowledge City platform, organizer and DAV integration
Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming.

Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
2026-09-27 14:44:43 +09:00

468 lines
20 KiB
Markdown

# Projection Adapter Design
## Purpose
A document address identifies content. It does not select its renderer.
Viewer chooses a projection adapter from document metadata and runtime
capabilities:
```text
internal address
-> revision metadata
-> effective MIME detection
-> projection adapter selection
-> bounded data delivery
-> Viewer component
```
The adapter boundary must remain independent from Semantic Path, Page UUID,
Revision UUID, and raw payload delivery.
## Current Adapter Contract
The server registry in `internal/server/content_adapter.go` describes:
| Field | Meaning |
| --- | --- |
| `ID` | Stable adapter identifier exposed as Viewer metadata |
| `Kind` | Template component used by the current Viewer |
| `Delivery` | HTML, decoded text, raw media, raw frame, or download |
| `NeedsText` | Whether safe character decoding is required |
| `WindowableText` | Whether large input must move to bounded retrieval |
| `Match` | MIME and filename matching predicate |
Selection is ordered. Specific structured formats must precede generic JSON,
text, and download adapters.
The initial registry reserves these stable IDs:
| Adapter | Current projection | Intended projection |
| --- | --- | --- |
| `html` | rendered HTML | rendered HTML |
| `text` | CodeMirror text | windowed text |
| `csv` | bounded table with sticky header | paginated virtualized table |
| `excalidraw` | validated interactive canvas + source fallback | canvas + editor action |
| `nwdiag` | bounded network diagram projection | static SVG preview + hosted/editor action |
| `archive` | ZIP/RAR/7z/TAR entry table + comic mode | archive directory |
| `eml` | decoded headers and body | multipart mail viewer |
| `debian-package` | ar/control metadata | package metadata |
| `pe` | PE/COFF headers and sections | executable metadata |
| `image` | raw media | raw media |
| `audio` | raw media | raw media |
| `video` | raw media | raw media |
| `pdf` | raw frame | PDF viewer |
| `download` | download | download |
These adapters are built in and versioned with Pages Wiki. They are always
listed by `GET /_/api/v1/adapter`; they are not configuration records and
cannot be deleted by an administrator.
## Integration Adapter Plugins
Additional hosted viewers and editors are Domain-scoped integration adapter
plugins. They may run at another FQDN or in a separately deployed container.
They supplement the built-in projection and do not replace document identity,
Raw delivery, or immutable Revision storage.
Each plugin declares:
| Field | Meaning |
| --- | --- |
| `id` | Stable lowercase plugin identifier |
| `name` | User-facing name in “Open with” |
| `mode` | hosted `viewer` or hosted `editor` |
| `media_types` | Exact MIME types or suffix-wildcards such as `image/*` |
| `extensions` | Lowercase filename extensions |
| `url_template` | Absolute HTTP(S) URL for the hosted service |
| `enabled` | Whether it is offered at runtime |
Templates can use `{url}`, `{path}`, and `{filename}`. Substitutions are
URL-encoded. `{url}` is an immutable Revision Raw URL; it never carries a
third-party token. Only `http:` and `https:` templates are accepted.
Native application launch is not implemented. `vim.exe`, `code.exe`,
`/usr/bin/vim`, custom URL schemes, a local workspace cache, and a Box Edit
style helper remain a future TODO. Legacy `native` records remain visible but
are disabled and are never returned as runtime actions.
Management and runtime discovery use:
```text
GET /_/api/v1/adapter
POST /_/api/v1/adapter
POST /_/api/v1/adapter/delete
GET /_/api/v1/adapter?path=/semantic/path
GET /v/open-with?adapter=<id>&path=/semantic/path
```
The server revalidates MIME/extension compatibility when `/v/open-with` is
called. A context-menu URL is therefore not an authorization bypass.
### Hosted Deployment
A hosted adapter can be deployed beside Pages Wiki without being part of the
Pages Wiki process:
```text
pages.example.test -> Pages Wiki
code.pages.example.test -> code-server + Pages Wiki adapter gateway
vim.pages.example.test -> vim.wasm adapter
```
The current integrated deployment uses one Pages Wiki origin and an internal
Gateway service instead of publishing the Gateway port:
```text
pages.anya.jp/_/adaptor/code/* -> adaptor-gateway:3000 -> code-server:8080
pages.anya.jp/_/adaptor/vim/* -> adaptor-gateway:3000 -> vim.wasm assets/API
```
`adaptor-gateway:3000` is available only on the Compose network. Pages Wiki
owns the public `/_/adaptor/` route, forwards the prefix to the Gateway, and
retains the prefix in redirects and cookie paths.
The first implementation opens the configured service in a top-level browser
navigation. Cross-origin iframe embedding is not assumed; a service may opt in
later with an explicit CSP `frame-ancestors` policy.
`vim.wasm` or a purpose-built Web editor can consume the immutable `{url}`
directly for read-only use. A general code-server container cannot edit an HTTP
Raw URL as a local workspace by itself. It requires an adapter gateway that:
1. exchanges a short-lived, one-use launch ticket;
2. materializes the selected Revision inside a per-session workspace;
3. opens that workspace in code-server;
4. submits changed bytes as a new Revision with optimistic conflict checking;
5. destroys or expires the workspace.
This launch and write-back contract is implemented by the reference
`adaptor-gateway` container. An editor URL template must contain `{ticket}`;
the ticket expires after 60 seconds and can be exchanged only once. The
gateway stores the resulting eight-hour session bearer server-side and sends
it in an Authorization header. Saving creates a new Revision. If the base
Revision is no longer current, Pages Wiki returns HTTP 409 and preserves both
versions without silently overwriting either one.
The reference gateway materializes the selected file and its descendant
documents in an isolated session directory
on a dedicated workspace volume and opens that directory through an internal
code-server instance. code-server is not published on a host port and runs
with authentication disabled only inside the Compose network; the Gateway
requires its HttpOnly adapter session before proxying `/code/`, including
WebSocket traffic. A watcher submits changed target bytes as a new Revision.
Relative hierarchy and a separate optimistic base Revision are retained for
every file. Hosted plugin settings can request a new browser tab and an
HTML/Markdown round trip: HTML becomes Markdown in the workspace and is
rendered back to canonical HTML when saved.
vim.wasm integrates behind this same Gateway contract rather than receiving
Pages Wiki database or Document Store access. The Gateway loads the session
workspace into vim.wasm MEMFS. A `BufWritePost` hook exports the written file,
and the browser sends those bytes through the scoped session API to create a
new Revision.
vim.wasm, its Worker, Wasm binary, and data archive are served with
`Cross-Origin-Opener-Policy: same-origin`,
`Cross-Origin-Embedder-Policy: require-corp`, and
`Cross-Origin-Resource-Policy: same-origin`. The deployment must also use a
secure context. A trusted HTTPS adapter URL is preferred; loopback development
may use `http://localhost`, which browsers treat as potentially trustworthy.
```text
POST /_/api/v1/adapter/session/exchange
GET /_/api/v1/adapter/session/workspace
GET /_/api/v1/adapter/session/content
PUT /_/api/v1/adapter/session/content
```
Recommended container boundaries are:
- a dedicated origin and cookie namespace;
- an allowlist of adapter origins in Pages Wiki;
- no direct mount of the Pages Wiki Document Store or SQLite database;
- CPU, memory, process, and workspace quotas per adapter container;
- immutable Revision download through Pages Wiki APIs only;
- write-back through the narrowly scoped adapter session capability only.
- code-server terminal and extension privileges confined to its container;
- only the adapter workspace volume shared with the Gateway.
## File Context Menu
Directory, search, and file-manager projections expose the same contextual
operations: open, rename, copy, paste, move, recursive delete, and “Open
with”. Copy/paste uses a browser-local clipboard containing Semantic Paths;
the server performs the actual Domain-scoped copy through
`POST /_/api/v1/path/copy`.
A copied Page receives new Object and Revision UUIDs. Its immutable Document
hashes are reused, and its Revision history and metadata are preserved. Copy
therefore means an independent Page, not a new Path link to the source.
“Open with” is shown only for adapters matching the resolved latest Revision.
Fallback rendering is intentional. Recognizing a format and providing an
interactive renderer are separate capabilities.
PDF page counts and non-ZIP archive listings now use the persistent, bounded
[`projection.metadata.generate` cache](projection-metadata-cache.md). Viewer
requests enqueue missing metadata and render explicit pending/retry states;
decoder subprocesses execute only in the scheduler. Cached entry names remain
plain data and receive authorized, immutable-Revision links at response time.
## Metadata UI
Viewer must not reserve a full horizontal row for MIME information.
Document metadata is exposed from the information control at the right of the
breadcrumb bar. It contains:
- filename
- effective MIME type
- selected adapter
- payload size
The control must work with both pointer hover and keyboard focus.
## Web Upload
`/v/upload` accepts a destination Semantic Path and one file. A destination
ending in `/` appends the uploaded filename. An exact existing Path creates a
new Revision of the existing Page; it never destroys history. Upload stores the
original filename and detected MIME type as Revision metadata.
The request is bounded to 256 MiB. Specialized resumable upload is a separate
future adapter and must not silently raise this synchronous request limit.
## Media Interaction
Image projection opens a modal full-viewport lightbox from the inline preview.
Audio and video projections retain native controls and add a common toolbar for
elapsed time and playback rate. Video additionally exposes Picture-in-Picture
when supported and full-screen projection. Raw media remains Range-capable.
ZIP entries are exposed through a scoped archive-entry projection. It resolves
the archive by Domain and semantic Path and never accepts a host filesystem
path. Comic mode includes recognized image entries and lazy-loads them in
archive order.
## Folder Search
The topbar search form sends the current semantic Path as its scope. SQLite
searches effective Paths and latest Revision filenames inside that segment
boundary and returns at most 100 results. The scope parameter is retained so a
later Domain-wide FTS index can add “all” search without changing the topbar
contract.
## Children Macro Views
Macro processing has explicit modes:
| Mode | `children` behavior |
| --- | --- |
| Authoring HTML → Markdown | Preserve the macro source verbatim |
| Markdown editor preview | Render Markdown to HTML, then expand the macro |
| Saved Viewer projection | Expand from canonical stored HTML |
| Markdown source projection | Preserve authoring source |
Expansion must never run before an authoring round trip: generated async
projection markup is runtime output and must not replace the macro in the
editor or a subsequent Revision.
The macro only selects the child Path. Presentation is viewer state rather
than document content:
```text
{{children}}
{{children:/gallery}}
{{children:/downloads exclude=/downloads/private}}
```
The breadcrumb action area exposes thumbnail, detail, and normal-list views.
The selected view is stored locally and applies to every `children` projection
on the page. Legacy `view=`, `mode=`, and `compact` arguments are ignored.
`exclude=` remains part of the macro because it changes the projected set.
Ctrl/Command-click toggles selection without following the link. Right-clicking
a selected item opens bulk move and recursive-delete actions. Moving a virtual
folder renames its Path prefix, preserving Object and Revision identity.
The macro initially emits a lightweight placeholder. Viewer fetches the
Domain-scoped `/v/children` projection asynchronously, so Page HTML and other
macros are not blocked by metadata or thumbnail discovery.
Children are delivered in bounded pages. There is no explicit “More” action:
an IntersectionObserver requests the next page when its sentinel approaches
the viewport, including touch/drag scrolling. The item grid and labels are
painted before any preview payload is requested.
Icon projection uses generated, immutable thumbnails for images, first-page
PDF previews, video frames, and Excalidraw scenes. Folders, ZIP archives,
Debian packages, audio, plain text, and source code use format-specific SVG
icons. Cache checks are viewport-prioritized. Mutable `path:` raw projections
use a short private cache; immutable `rev:` projections use a one-year
immutable cache and ETag.
Preview I/O is a background workload. The browser:
- paints item frames and labels before scheduling preview work;
- starts work during an idle period;
- allows at most two concurrent previews, or one on data-saver/2G links;
- observes only items within 250 px of the viewport;
- cancels queued fetches and active Excalidraw requests on page departure.
Preview Raw URLs carry `preview=1`. The server isolates those requests behind a
single global preview slot. Normal Viewer navigation, APIs, and non-preview Raw
delivery never wait for this slot.
File-manager thumbnails no longer use Raw PDF frames or browser video
decoders. They consume only immutable files produced by the persistent
[event scheduler](event-scheduler.md). Until generation succeeds, the icon
projection remains a lightweight format icon.
## Raw and Derived Projection URLs
Adapters never construct an alternative identity URL. Raw bytes continue to
use an explicit internal address:
```text
/v/raw/path:/reports/annual.pdf
/v/raw/uuid:<page-uuid>
/v/raw/rev:<revision-uuid>
```
Structured viewers use derived endpoints below `/v/<adapter>/` and carry the
same address forms. Proposed examples:
```text
/v/text/path:/logs/server.log?offset=0&limit=65536
/v/csv/rev:<revision-uuid>?row=0&limit=200
/v/archive/rev:<revision-uuid>?path=/usr/bin
/v/deb/rev:<revision-uuid>
/v/excalidraw/rev:<revision-uuid>
```
Printable projections use the address-model route
`/_/view/print/{path:|uuid:|rev:}` and pin every Raw dependency to the selected
immutable Revision. Text and CSV print output is bounded to 2 MiB; media is
printed from metadata plus its immutable Raw URL without hydrating its body.
Structured formats above the synchronous hydration ceiling degrade to a
download link instead of blocking foreground navigation.
A mutable `path:` request may resolve to a revision and return its UUID in the
response. Paging requests after the first response must use `rev:` so a
changing current document cannot mix two revisions in one Viewer session.
## Large Text and JSON
The current inline text threshold is 2 MiB. Crossing the threshold is recorded
as a requirement for windowed delivery. The complete scalable implementation
must not hydrate or embed the full document merely to render its first screen.
The text adapter will use:
1. metadata-only revision lookup;
2. a seekable Document Store reader;
3. byte-window retrieval with a fixed server maximum;
4. UTF boundary repair and explicit byte offsets;
5. line-index checkpoints built asynchronously or lazily;
6. client virtualization for visible lines and minimap summaries.
JSON uses bounded structural streaming at `/v/json/rev:<revision-uuid>`.
Expansion fetches a capped direct-child range selected by JSON Pointer. The
request handler streams tokens and never unmarshals the complete payload;
nested containers are revisited lazily from the immutable Document. A future
persistent checkpoint index may accelerate deep pointers without changing the
API or Revision identity.
Responses should include:
```text
revision_uuid
document_hash
content_type
total_bytes
offset
next_offset
line_start
line_end
truncated
```
Range limits, parse depth, row count, archive entries, and decompressed bytes
must be capped server-side.
## CSV Viewer
CSV detection uses MIME type or `.csv`. The derived adapter must:
- sniff delimiter and quoting from a bounded prefix;
- expose encoding and dialect in metadata;
- paginate rows;
- virtualize columns and rows in the browser;
- preserve the immutable Revision during paging;
- fall back to the text adapter on malformed input.
## nwdiag network diagrams
`nwdiag` is a source/document format, not an instruction to execute arbitrary
code. The adapter parses a bounded, version-pinned grammar and produces a
static SVG (or PNG) projection. It must reject oversized graphs, recursive
includes, external image/font imports, unsafe SVG attributes, and unknown
directives rather than forwarding them to a renderer. The source Revision is
immutable and remains available through the plain/source view.
An optional hosted editor is a separate `viewer`/`editor` adapter. It receives
only an immutable Revision URL and a one-use save ticket, never the browser
session token. Save uses optimistic Revision checks and creates a new source
Revision; rendering is queued and cached by `(document hash, parser version,
theme, renderer version)`. If the renderer or editor is unavailable, the Viewer
shows escaped source and the last verified static projection. A future Kroki
provider may render nwdiag, but network access, origin allowlists, timeout,
resource quotas, and provider version must be explicit.
## ZIP Viewer
ZIP projection lists entries without extracting them into the data directory.
Entry paths must be normalized and zip-slip paths rejected. Limits apply to
entry count, nested archives, compression ratio, total expanded size, and
individual previews. Entry download is a derived projection of the containing
Revision and entry path.
## Debian Package Viewer
The DEB adapter reads the outer `ar` members and bounded control archive only.
It exposes package name, version, architecture, dependencies, description,
installed size, checksums, and a bounded file list. Maintainer scripts render
as escaped text. No package script is executed.
## Excalidraw
`.excalidraw` files are recognized before generic JSON. The renderer validates
a bounded scene graph and produces an isolated canvas projection.
The preferred projection is the bundled `org.pageswiki.excalidraw-viewer`
plugin. It runs `@excalidraw/excalidraw` in read-only view mode in a separate
container and is routed through `/_/adaptor/excalidraw/`. Pages Wiki passes
only an immutable same-origin `rev:` Raw URL. The plugin rejects mutable,
cross-origin, and over-2-MiB inputs, is framed with a restrictive CSP, and
reports readiness through `postMessage`. A timeout, load failure, or invalid
scene activates the dependency-free built-in Canvas renderer.
Configuration may provide an editor URL template. The action receives a
read-capable document URL and, only when explicitly authorized, a save callback
or capability. Tokens must not be placed in a third-party URL query string.
Without a configured editor, Viewer offers the internal canvas and source
fallback.
## Security and Failure Rules
- Normal HTML displays use the common `controlled-v2` allowlist and application CSP;
saved source remains unchanged. Raw HTML attachment, raw/diagram SVG sandbox,
and registered application frames retain their separate response contracts.
There is no executable authored-HTML mode. See
[HTML projection trust and compatibility](html-projection-trust.md); public
shares and anonymous viewing still require their own authorization gates.
- Archive and package adapters never execute contained files.
- Derived adapters operate on immutable revisions after initial resolution.
- MIME detection is advisory; parser validation is authoritative.
- An unavailable or failed specialized adapter falls back to safe text or
download behavior.
- Adapter errors do not change document identity or stored content.