Files
pages/docs/design/search-results.md
T
Anya Tokugawa c1175d389f Implement Knowledge City platform, organizer and DAV integration
Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming.

Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
2026-09-27 14:44:43 +09:00

248 lines
15 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Bounded, permission-filtered search results
Implemented: 2026-09-14.
## City-first results and search options
Search now places matching Quartiers, Streets and Spots above Page/file hits.
The compact City section remains visible in list/detail/icon modes and shows
the locality kind, highlighted title, literal boundary Path, readable ancestor
context, a description excerpt, matching-source badges and update time. It is
not a file-selection target: Page copy/rename/delete actions cannot accidentally
operate on a City result. The title opens City details; the Path opens its portal
boundary. Quartier/Spot links use their unique names. Street links use
`/v/city/s/{quartier-name}/{street-name}` so repeated names remain unambiguous.
Explicit `uuid:` selectors remain supported. Map locality search submits that
explicit selector and requires read access to the locality and its ancestors.
`SearchCityInDomain` reads metadata from three bounded SQL streams; it never
loads the full City model, expands Page placements or counts descendants.
Only current-Domain, non-archived localities with valid absolute boundaries and
active same-Domain ancestors are eligible. Scope contains the locality's own
Path; ancestors outside a narrower scope are not added automatically. Every
boundary in the parent chain must pass the same request-scoped read policy
before names or descriptions are exposed. Limits are 1,000 candidates per kind
and 30 readable City results. Ordering is Quartier, Street, Spot; within each
kind name/title matches precede Path matches, followed by stable Path/UUID ties.
Matching is literal substring search over name/title/description/Path, with
SQLite ASCII case-folding and literal Unicode/punctuation handling.
The HTTP handler shares a two-second deadline between the City and Page queries;
their internal snapshots remain separate read transactions. Existing bounded
Page candidate and optional-excerpt behavior is retained. City/Page results may
therefore observe separate commits during a concurrent edit, not a single
cross-model snapshot.
Options are a collapsed, keyboard-accessible disclosure. Active/error filters
open it. Inputs outside the topbar associate with the same GET form, so editing
the keyword, pressing Enter or applying filters preserves one copy of each
parameter, scope and locality. Clear filters retains the keyword/scope/locality;
timeout retry retains the selected filters too. No new JavaScript/storage state
or cookie is required for these shareable URL options.
| Parameter | Meaning |
| --- | --- |
| `file_type` | Empty for all, or `html`, `text`, `image`, `audio`, `video`, `pdf`, `archive`, `csv`, `json`, `code`, `office`, `other` |
| `ext` | One extension, e.g. `py`, `.PDF`, `tar.gz`; normalized to lower case without the leading dot |
| `date_field` | `updated` (default) or `created` |
| `from` | Inclusive starting calendar date, `YYYY-MM-DD` |
| `to` | Inclusive ending calendar date, `YYYY-MM-DD` |
Specific MIME metadata takes precedence over suffix guesses; generic MIME may
use a known filename suffix, falling back to Path when filename is absent.
Type and extension combine with AND. The classification never loads/sniffs the
Document. Extension filtering uses the stored original filename when present:
an HTML canonical Page with retained `notes.md` metadata can match `ext=md`,
while its type is still HTML. Selecting a type is not selecting an editor mode.
Page update dates use the latest Revision timestamp, and creation dates use
Object creation, not the time a Path alias was added. City dates use the locality's
own timestamps, not descendant Page activity. Dates use the Domain timezone,
shown alongside the inputs. The ending bound is the next local midnight using
calendar arithmetic, including 23/25-hour daylight-saving dates. Invalid types,
extensions, repeated filter parameters or invalid/reversed dates return a
localized `400` without interpolating user input into SQL.
File type/extension filters exclude City elements (a Spot containing PDFs is
not itself a PDF). Date filters apply to both classes. Keywords are optional
when a type, extension or date boundary is supplied; selecting only a date-field
kind does not enumerate everything. Every filter runs before each Page
candidate LIMIT, rather than discarding already-limited results afterward.
Current limits remain deliberate: one file type and one extension at a time,
metadata-based classification, no historical content search, no cross-Domain
City search, and no derived descendant file-type/date facet on a City element.
## Presentation
`/v/search` retains the shared list / detail / icon controls, preference storage,
Ctrl/Command selection, and file context menu. Each mode now presents the same
information:
- Page display name: the current Revision filename's final component, falling
back to the current Path's final component. There is no separate Page-title
field in the current Object schema.
- The complete matching semantic Path, with keywords emphasized independently
of the display name. Link URLs encode literal spaces, `?`, `#`, and `%` as
Path characters rather than treating them as query/fragment syntax.
- Explicit Path / filename / body match-source labels. Multiple sources may
apply to one result. Each readable alias remains a distinct navigable result.
- A short contextual body excerpt for full-text matches. Name-only matches
receive a bounded opening summary when the Revision contains textual data.
- Large or expensive body matches retain their name, Path and match-source
badge, with a localized explanation when the excerpt is omitted.
- In detail mode, content type and update time. Unknown file sizes are no
longer misleadingly displayed as zero.
List mode is a compact stacked overview; icon mode adds the existing cached /
asynchronous preview; detail mode keeps the dense table. The search submit
button is an accessible icon aligned with its input. Styles use existing theme
variables and adapt to narrower viewports.
Keywords become typed text segments, rendered using `html/template`-owned
`<mark>` elements. No search query, filename, Path, FTS marker, or excerpt is
cast to trusted HTML. HTML excerpts are tokenized to plain text, excluding
script/style content, before Unicode-safe windowing and highlighting.
## Query and authorization boundary
The handler builds one request-scoped Path-read filter and validates the scope
with it. `SearchPathsInDomainWithOptions` (or its no-options compatibility wrapper
`SearchPathsInDomainFiltered`) uses that immutable filter during
candidate collection, before any excerpt lookup or preview construction.
Global counts, denied Paths, denied filenames, and denied excerpts are never
included in the rendered data. An all-denied candidate set uses the same empty
state as an ordinary no-match query.
All search statements use one read-only SQLite transaction, preserving a
consistent current Path / latest Revision snapshot. The three candidate streams
are:
1. Literal substring matching against current Paths and current filenames.
2. FTS5 body-prefix matching, joined to the latest Revision projection.
3. FTS5 filename-prefix matching, joined to the same projection.
Each stream is limited to 1,000 candidate Paths in SQL. Full-text streams do not
materialize an unbounded global `IN` list or globally sort all matching
historical Revisions. An index on `latest_revision_index(revision_uuid)` supports
the latest-only join. Explicit `CROSS JOIN` keeps FTS outermost: scan MATCH
once, then seek current Revision metadata and Path bindings through B-tree
indexes. An ordinary JOIN can instead put Paths first and repeat the entire
FTS scan for every Path because the FTS UUID payload column is unindexed.
Domain and segment-aware scope filtering precede each
candidate limit. Prefix bounds use binary Path ranges, not wildcard `LIKE`;
literal `%` and `_` in folder names and queries cannot widen the scope.
Authorized candidates are deduplicated by Path, then ordered with direct
name/Path matches first and stable length/Path ties. The normal UI displays at
most 100; trusted store callers may request up to 200. Only authorized displayed
rows receive a byte-size preflight, using their FTS integer rowids from the same
snapshot. `octet_length(body)` counts bytes, not Unicode characters; see
[SQLite's function contract](https://www.sqlite.org/lang_corefunc.html#octet_length).
Eligible body excerpts are fetched together with one FTS snippet query restricted
by those rowids, not the unindexed UUID payload column. Name-only textual
summaries use one additional bounded Revision-prefix query. There are no
per-result authorization queries or Document-file stat calls. Existing preview
cache checks run only for displayed authorized results, without scheduling
foreground generation.
## Explicit bounds and failure behavior
- Query input: at most 512 bytes and 16 terms.
- Search transaction: two-second context deadline; caller cancellation applies.
- Candidate rows: at most 3,000 total; displayed rows: normally 100.
- Snippet transport: at most 4,096 characters per body excerpt, 1,024 per
name-only summary; display window: approximately 260 Unicode characters.
- Snippet input: at most 64 KiB per document and 1 MiB across unique displayed
Revisions. Multiple authorized aliases share an excerpt and consume its
input budget only once. An oversized body still participates in MATCH.
- Optional snippet time: 400 ms within the overall search context. If spent,
omit the snippet batch but retain the already authorized results. Parent
cancellation still aborts the search; unrelated SQL errors are not swallowed.
These are cooperative SQLite deadlines, not hard process-kill guarantees.
- No total-count query. The 100-result notice depends only on visible rows,
not on hidden candidate counts. A general scope-narrowing hint is always
shown for a submitted query.
A search-budget timeout returns a themed, localized `503` page with
`Retry-After: 5` and a retry link preserving query/scope/locality. Unexpected
database errors return a generic `500`, without SQL/schema/connection details.
Canceled HTTP requests stop without further rendering. Snippet-only timeout
is a successful results page with explicit omission, not a search failure.
## Timeout regression and correction (2026-09-14)
The initial small-fixture tests did not catch a production join-order problem.
Read-only investigation of the current 943 Paths / 1,632 indexed Revisions
reproduced it: a native SQLite body candidate query took 8.21 seconds with the
old Path-first plan and 0.10 seconds with the FTS-first plan, returning the same
36 candidates. Another common prefix exceeded a ten-second diagnostic bound.
There was a second bottleneck: FTS bodies reach tens of megabytes.
`snippet(...,36)` and a `substr` on its output do **not** limit the tokenizer's
input work. Even smaller documents can have costly prefix-position scans;
the separate input and optional time budgets address both cases.
The read-only Go Store diagnostic reproduced all four initial query failures
before the correction. With the final code the representative cases returned
57 / 100 / 100 / 7 rows in approximately 86 / 128 / 506 / 72 ms, respectively.
The short, broad prefix used the optional-excerpt fallback. Timings are local
measurements, not service-level guarantees; no production content or credentials
were printed or changed. No schema migration or index rebuild is required by
this correction.
## Remaining work
These bounds intentionally trade exhaustive recall for predictable work. A
broad search or many inaccessible leading candidates may omit readable hits
beyond a stream's candidate budget; narrowing the scope is the current remedy.
SQLite may still inspect many FTS postings / historical rows before locating
current in-scope candidates, but the deadline bounds this work. A current-only,
Domain-partitioned FTS projection with a persisted index watermark and genuine
keyset pagination remains a future scaling step, not an implemented claim.
The current `unicode61` tokenizer and raw-HTML search index are unchanged:
Japanese infix search and exclusion of markup-only matches need a separate
plain-text / language-aware indexing migration. Excerpts already remove HTML
markup safely, but an indexed markup token can still cause a body-source hit
with no visible text excerpt. Independent Page titles and relevance tuning are
also separate features.
## Regression coverage
Store tests cover latest-only bodies and current renamed Paths, literal scope
and query punctuation, body/filename source distinction, Domain isolation,
ACL-before-limit behavior, bounded candidates, query bounds, and cancellation.
Query-plan tests require FTS-first candidate collection and rowid-constrained
snippets. A 600-Page / 1,200-Revision fixture checks current-only results, both
match sources and excerpt association under the real search budget. Input-budget
tests cover Unicode byte lengths, total bytes, aliases and retained hits;
deadline tests distinguish optional enrichment from parent cancellation/errors.
Server tests cover hidden-only results, denied scopes, encoded Path links,
Unicode keyword segments, contextual clipping, and malicious source HTML.
An opt-in Chromium test exercises all three display modes, safe highlights,
Ctrl selection, and absence of horizontal page overflow.
HTTP tests also cover themed GET/HEAD failures, sanitized errors, retained retry
parameters and a large-body match beyond the bounded prefix in Japanese/English.
City/filter regressions cover hidden/foreign/archived ancestors, duplicate Street
names and encoded links, metadata-only single-connection reads, literal scopes,
MIME priority/fallback, combined filters before all three candidate caps,
Object-vs-alias creation times, inclusive millisecond bounds and Japanese/DST
calendar conversion. Chromium checks desktop (1440px) and mobile (390px), with
real City results above Pages, no file selection on City rows, and one set of
form parameters while changing views/options.
Read-only verification against the existing main DB measured the additional
filter-only cases at approximately 1–6 ms and City metadata lookup at 2 ms;
representative keyword Page searches remained approximately 79–515 ms. These
are local observations, not latency guarantees or authenticated HTTP timings.
This feature adds no schema migration or index rebuild.
```sh
go test -race ./internal/server ./internal/store -run '^TestSearch' -count=1
PAGESWIKI_BROWSER_TEST=1 go test ./internal/server -run '^TestSearchResultsChromium$' -count=1
# Optional: opens an existing DB with mode=ro and query_only, with no migrations.
PAGESWIKI_SEARCH_DIAGNOSTIC_DB=/absolute/path/pages.db go test ./internal/store -run '^TestSearchReadOnlyDatabase$' -count=1 -v
```