Files
pages/docs/design/object-list-pagination.md
T
Anya Tokugawa c1175d389f Implement Knowledge City platform, organizer and DAV integration
Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming.

Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
2026-09-27 14:44:43 +09:00

98 lines
5.1 KiB
Markdown

# Object metadata list: bounded cursor pagination
Implemented: 2026-09-14. Endpoint: `GET` / `HEAD /_/api/v1/object/list`.
## Compatible response and continuation
The body remains a JSON array of the existing `Object` representation
(`UUID`, `Kind`, `CreatedAt`). No Path, revision body, aggregate count, or
unreadable Object metadata is added. Empty results serialize as `[]`.
Query parameters:
- `limit`: default 100, positive integers, capped at 200.
- `cursor`: opaque continuation token from the preceding response.
When continuation is possible, responses include both `X-Next-Cursor` and an
RFC-style `Link: <...>; rel="next"` header containing the same cursor. HEAD
returns the corresponding metadata/continuation headers without an entity
body. Responses are `private, no-store`; unsupported methods return 405.
Clients **must continue when a next cursor exists, even if the current array
is empty**. A page may examine only unreadable Objects and still have later
readable Objects. Absence of a next cursor is the end-of-scan condition. There
is deliberately no `total`, denied count, or public scanned-row count.
Ordering is `(COALESCE(created_at, 0) DESC, uuid ASC)`. The keyset position does
not require the boundary Object to continue existing, and handles identical
timestamps and legacy NULL timestamps without OFFSET or duplicates. New
Objects/links created between requests may appear before or after the current
position; pagination is not a multi-request immutable snapshot.
The continuation query separately seeks the same-timestamp UUID suffix and
then older timestamps, keeping large same-second imports from repeatedly
rescanning UUIDs before the cursor.
## Cursor confidentiality and authority
The cursor contains its purpose/version, Domain, principal, fifteen-minute
expiry, and `(created_at, UUID)` position. The existing instance authentication
key encrypts/authenticates it with AES-GCM; the transport encoding is base64url.
A base64-only bookmark would disclose the last denied Object's UUID and time
on an empty page, so it is explicitly not used.
Tampered, expired, wrong-purpose, cross-Domain, and cross-principal cursors
return a generic 400. Cursors do not carry permissions: current session,
membership and grants are reevaluated on every request. They survive a process
restart when the authentication key is unchanged. Replicas need the same key
to interchange cursors; key distribution itself is not implemented here.
## Bounded selection and authorization
The API retains its existing scope: only Objects with a **live semantic Path
binding in the current Domain** are candidates. Unbound Objects and Objects
bound exclusively in another Domain do not become listable merely because
their global UUID or creation origin is known. Direct Object grants authorize
metadata among these candidates; they do not widen this listing's scope.
The handler installs a two-second request context, loads exact Object / broad
Domain-or-Object read grants in one bounded query, and uses the shared
request-scoped Path-read filter. Principal and Domain membership must be active
(readonly Principals may read). Candidate metadata and binding inspection share
a read-only SQLite transaction.
- At most 400 candidates, plus one lookahead row, are selected in SQL.
- Direct Object and broad read grants are applied before looking at Paths.
- At most 8,000 binding rows are inspected in one batch for the remaining
candidates. No per-Object SQL authorization loop remains.
- Exact Object / Domain grant loading is capped at 4,096 grants.
- If a binding/grant budget is exceeded, the request fails closed with a
generic 503. It does not incorrectly skip insufficiently inspected Objects.
- A visible page ending early resumes after its last returned Object; a
denied-only candidate batch resumes after its last inspected Object.
Schema initialization adds metadata-order and Object/Domain/Path indexes once.
The helper tolerates legacy Domain-only migrations where Object tables have
not yet been installed. No foreground listing performs DDL.
## Remaining scaling work
Very many cross-Domain Objects can still require scanning the metadata-order
index before finding in-Domain candidates; the deadline bounds this rather than
a claim of constant query time. Very highly aliased Objects can exceed the
binding budget for a Path-only reader. Domain-specific listing projections and
SQL-side joins to a compiled, bounded ACL projection would address these cases.
The older trusted store listing methods remain available to their other callers;
this change specifically bounds the HTTP metadata-list endpoint.
## Regression tests
Tests cover default/max limits, stable tied/NULL timestamps, deleted boundary
Objects, HEAD behavior, malformed/tampered/expired/scoped cursors, empty pages
with continuation, exact Object and Path grants, grant revocation, hidden Path
and cursor confidentiality, Domain/unbound exclusion, cancellation, extreme
binding budgets, and the legacy Domain-only migration fixture.
```sh
go test -race ./internal/server ./internal/store -run 'TestObjectList|TestObjectMetadata|TestLegacyDistrictsMigrateToDomains' -count=1
```