Extract Grammaratur, Dregarnuhr, Geduldh, Mestionora, Ordoschnellic, Schutzaria, Kunstzeal and Seheweit behind compatible application contracts. Include accumulated editor resilience, navigation and theme UX, remote CLI, storage/schema and DR groundwork, tests and design/operations documentation. Verified: uncached full Go suite; go vet ./...; race tests for all eight components; 11 targeted Chromium tests; Node suites; 19 Python CLI tests; reproducible editor build. No push, release tag or deployment in this checkpoint.
256 lines
15 KiB
Markdown
256 lines
15 KiB
Markdown
# Bounded, permission-filtered search results
|
||
|
||
Implemented: 2026-09-14.
|
||
|
||
2026-11-01: SQLite search/index maintenance and City metadata lookup are extracted
|
||
into `internal/mestionora`. Store APIs/types remain compatibility facades; HTTP
|
||
search and suggestions call Mestionora with the existing request read policy.
|
||
No new index schema or migration is introduced. See [Mestionora's contract](ja/mestionora.md).
|
||
|
||
## City-first results and search options
|
||
|
||
Search now places matching Quartiers, Streets and Spots above Page/file hits.
|
||
The compact City section remains visible in list/detail/icon modes and shows
|
||
the locality kind, highlighted title, literal boundary Path, readable ancestor
|
||
context, a description excerpt, matching-source badges and update time. It is
|
||
not a file-selection target: Page copy/rename/delete actions cannot accidentally
|
||
operate on a City result. The title opens City details; the Path opens its portal
|
||
boundary. Quartier/Spot links use their unique names. Street links use
|
||
`/v/city/s/{quartier-name}/{street-name}` so repeated names remain unambiguous.
|
||
Explicit `uuid:` selectors remain supported. Map locality search submits that
|
||
explicit selector and requires read access to the locality and its ancestors.
|
||
|
||
`SearchCityInDomain` reads metadata from three bounded SQL streams; it never
|
||
loads the full City model, expands Page placements or counts descendants.
|
||
Only current-Domain, non-archived localities with valid absolute boundaries and
|
||
active same-Domain ancestors are eligible. Scope contains the locality's own
|
||
Path; ancestors outside a narrower scope are not added automatically. Every
|
||
boundary in the parent chain must pass the same request-scoped read policy
|
||
before names or descriptions are exposed. Limits are 1,000 candidates per kind
|
||
and 30 readable City results. Ordering is Quartier, Street, Spot; within each
|
||
kind name/title matches precede Path matches, followed by stable Path/UUID ties.
|
||
Matching is literal substring search over name/title/description/Path, with
|
||
SQLite ASCII case-folding and literal Unicode/punctuation handling.
|
||
|
||
The HTTP handler shares a two-second deadline between the City and Page queries;
|
||
their internal snapshots remain separate read transactions. Existing bounded
|
||
Page candidate and optional-excerpt behavior is retained. City/Page results may
|
||
therefore observe separate commits during a concurrent edit, not a single
|
||
cross-model snapshot.
|
||
|
||
Options are a collapsed, keyboard-accessible disclosure. Active/error filters
|
||
open it. Inputs outside the topbar associate with the same GET form, so editing
|
||
the keyword, pressing Enter or applying filters preserves one copy of each
|
||
parameter, scope and locality. Clear filters retains the keyword/scope/locality;
|
||
timeout retry retains the selected filters too. No new JavaScript/storage state
|
||
or cookie is required for these shareable URL options.
|
||
|
||
| Parameter | Meaning |
|
||
| --- | --- |
|
||
| `file_type` | Empty for all, or `html`, `text`, `image`, `audio`, `video`, `pdf`, `archive`, `csv`, `json`, `code`, `office`, `other` |
|
||
| `ext` | One extension, e.g. `py`, `.PDF`, `tar.gz`; normalized to lower case without the leading dot |
|
||
| `date_field` | `updated` (default) or `created` |
|
||
| `from` | Inclusive starting calendar date, `YYYY-MM-DD` |
|
||
| `to` | Inclusive ending calendar date, `YYYY-MM-DD` |
|
||
|
||
Specific MIME metadata takes precedence over suffix guesses; generic MIME may
|
||
use a known filename suffix, falling back to Path when filename is absent.
|
||
Type and extension combine with AND. The classification never loads/sniffs the
|
||
Document. Extension filtering uses the stored original filename when present:
|
||
an HTML canonical Page with retained `notes.md` metadata can match `ext=md`,
|
||
while its type is still HTML. Selecting a type is not selecting an editor mode.
|
||
|
||
Page update dates use the latest Revision timestamp, and creation dates use
|
||
Object creation, not the time a Path alias was added. City dates use the locality's
|
||
own timestamps, not descendant Page activity. Dates use the Domain timezone,
|
||
shown alongside the inputs. The ending bound is the next local midnight using
|
||
calendar arithmetic, including 23/25-hour daylight-saving dates. Invalid types,
|
||
extensions, repeated filter parameters or invalid/reversed dates return a
|
||
localized `400` without interpolating user input into SQL.
|
||
|
||
File type/extension filters exclude City elements (a Spot containing PDFs is
|
||
not itself a PDF). Date filters apply to both classes. Keywords are optional
|
||
when a type, extension or date boundary is supplied; selecting only a date-field
|
||
kind does not enumerate everything. Every filter runs before each Page
|
||
candidate LIMIT, rather than discarding already-limited results afterward.
|
||
|
||
Current limits remain deliberate: one file type and one extension at a time,
|
||
metadata-based classification, no historical content search, no cross-Domain
|
||
City search, and no derived descendant file-type/date facet on a City element.
|
||
|
||
## Presentation
|
||
|
||
`/v/search` retains the shared list / detail / icon controls, preference storage,
|
||
Ctrl/Command selection, and file context menu. Each mode now presents the same
|
||
information:
|
||
|
||
- Page display name: the current Revision filename's final component, falling
|
||
back to the current Path's final component. There is no separate Page-title
|
||
field in the current Object schema.
|
||
- The complete matching semantic Path, with keywords emphasized independently
|
||
of the display name. Link URLs encode literal spaces, `?`, `#`, and `%` as
|
||
Path characters rather than treating them as query/fragment syntax.
|
||
- Explicit Path / filename / body match-source labels. Multiple sources may
|
||
apply to one result. Each readable alias remains a distinct navigable result.
|
||
- A short contextual body excerpt for full-text matches. Name-only matches
|
||
receive a bounded opening summary when the Revision contains textual data.
|
||
- Large or expensive body matches retain their name, Path and match-source
|
||
badge, with a localized explanation when the excerpt is omitted.
|
||
- In detail mode, content type and update time. Unknown file sizes are no
|
||
longer misleadingly displayed as zero.
|
||
|
||
List mode is a compact stacked overview; icon mode adds the existing cached /
|
||
asynchronous preview; detail mode keeps the dense table. The search submit
|
||
button is an accessible icon aligned with its input. Styles use existing theme
|
||
variables and adapt to narrower viewports.
|
||
|
||
Keywords become typed text segments, rendered using `html/template`-owned
|
||
`<mark>` elements. No search query, filename, Path, FTS marker, or excerpt is
|
||
cast to trusted HTML. HTML excerpts are tokenized to plain text, excluding
|
||
script/style content, before Unicode-safe windowing and highlighting.
|
||
|
||
## Query and authorization boundary
|
||
|
||
The handler builds one request-scoped Path-read filter and validates the scope
|
||
with it. `SearchPathsInDomainWithOptions` (or its no-options compatibility wrapper
|
||
`SearchPathsInDomainFiltered`) uses that immutable filter during
|
||
candidate collection, before any excerpt lookup or preview construction.
|
||
Global counts, denied Paths, denied filenames, and denied excerpts are never
|
||
included in the rendered data. An all-denied candidate set uses the same empty
|
||
state as an ordinary no-match query.
|
||
|
||
Page candidate and excerpt statements use one read-only SQLite transaction, preserving a
|
||
consistent current Path / latest Revision snapshot. The three candidate streams
|
||
are:
|
||
|
||
1. Literal substring matching against current Paths and current filenames.
|
||
2. FTS5 body-prefix matching, joined to the latest Revision projection.
|
||
3. FTS5 filename-prefix matching, joined to the same projection.
|
||
|
||
Each stream is limited to 1,000 candidate Paths in SQL. Full-text streams do not
|
||
materialize an unbounded global `IN` list or globally sort all matching
|
||
historical Revisions. An index on `latest_revision_index(revision_uuid)` supports
|
||
the latest-only join. Explicit `CROSS JOIN` keeps FTS outermost: scan MATCH
|
||
once, then seek current Revision metadata and Path bindings through B-tree
|
||
indexes. An ordinary JOIN can instead put Paths first and repeat the entire
|
||
FTS scan for every Path because the FTS UUID payload column is unindexed.
|
||
Domain and segment-aware scope filtering precede each
|
||
candidate limit. Prefix bounds use binary Path ranges, not wildcard `LIKE`;
|
||
literal `%` and `_` in folder names and queries cannot widen the scope.
|
||
|
||
Authorized candidates are deduplicated by Path, then ordered with direct
|
||
name/Path matches first and stable length/Path ties. The normal UI displays at
|
||
most 100; trusted store callers may request up to 200. Only authorized displayed
|
||
rows receive a byte-size preflight, using their FTS integer rowids from the same
|
||
snapshot. `octet_length(body)` counts bytes, not Unicode characters; see
|
||
[SQLite's function contract](https://www.sqlite.org/lang_corefunc.html#octet_length).
|
||
Eligible body excerpts are fetched together with one FTS snippet query restricted
|
||
by those rowids, not the unindexed UUID payload column. Name-only textual
|
||
summaries use one additional bounded Revision-prefix query. There are no
|
||
Document-file stat calls. The normal Path policy is preloaded; lazy `/u/` policy
|
||
lookups can still borrow another DB connection during authorization. Do not
|
||
combine that callback with a single-connection pool until the policy preloading
|
||
follow-up is implemented. Existing preview
|
||
cache checks run only for displayed authorized results, without scheduling
|
||
foreground generation.
|
||
|
||
## Explicit bounds and failure behavior
|
||
|
||
- Query input: at most 512 bytes and 16 terms.
|
||
- Search transaction: two-second context deadline; caller cancellation applies.
|
||
- Candidate rows: at most 3,000 total; displayed rows: normally 100.
|
||
- Snippet transport: at most 4,096 characters per body excerpt, 1,024 per
|
||
name-only summary; display window: approximately 260 Unicode characters.
|
||
- Snippet input: at most 64 KiB per document and 1 MiB across unique displayed
|
||
Revisions. Multiple authorized aliases share an excerpt and consume its
|
||
input budget only once. An oversized body still participates in MATCH.
|
||
- Optional snippet time: 400 ms within the overall search context. If spent,
|
||
omit the snippet batch but retain the already authorized results. Parent
|
||
cancellation still aborts the search; unrelated SQL errors are not swallowed.
|
||
These are cooperative SQLite deadlines, not hard process-kill guarantees.
|
||
- No total-count query. The 100-result notice depends only on visible rows,
|
||
not on hidden candidate counts. A general scope-narrowing hint is always
|
||
shown for a submitted query.
|
||
|
||
A search-budget timeout returns a themed, localized `503` page with
|
||
`Retry-After: 5` and a retry link preserving query/scope/locality. Unexpected
|
||
database errors return a generic `500`, without SQL/schema/connection details.
|
||
Canceled HTTP requests stop without further rendering. Snippet-only timeout
|
||
is a successful results page with explicit omission, not a search failure.
|
||
|
||
## Timeout regression and correction (2026-09-14)
|
||
|
||
The initial small-fixture tests did not catch a production join-order problem.
|
||
Read-only investigation of the current 943 Paths / 1,632 indexed Revisions
|
||
reproduced it: a native SQLite body candidate query took 8.21 seconds with the
|
||
old Path-first plan and 0.10 seconds with the FTS-first plan, returning the same
|
||
36 candidates. Another common prefix exceeded a ten-second diagnostic bound.
|
||
|
||
There was a second bottleneck: FTS bodies reach tens of megabytes.
|
||
`snippet(...,36)` and a `substr` on its output do **not** limit the tokenizer's
|
||
input work. Even smaller documents can have costly prefix-position scans;
|
||
the separate input and optional time budgets address both cases.
|
||
|
||
The read-only Go Store diagnostic reproduced all four initial query failures
|
||
before the correction. With the final code the representative cases returned
|
||
57 / 100 / 100 / 7 rows in approximately 86 / 128 / 506 / 72 ms, respectively.
|
||
The short, broad prefix used the optional-excerpt fallback. Timings are local
|
||
measurements, not service-level guarantees; no production content or credentials
|
||
were printed or changed. No schema migration or index rebuild is required by
|
||
this correction.
|
||
|
||
## Remaining work
|
||
|
||
These bounds intentionally trade exhaustive recall for predictable work. A
|
||
broad search or many inaccessible leading candidates may omit readable hits
|
||
beyond a stream's candidate budget; narrowing the scope is the current remedy.
|
||
SQLite may still inspect many FTS postings / historical rows before locating
|
||
current in-scope candidates, but the deadline bounds this work. A current-only,
|
||
Domain-partitioned FTS projection with a persisted index watermark and genuine
|
||
keyset pagination remains a future scaling step, not an implemented claim.
|
||
|
||
The current `unicode61` tokenizer and raw-HTML search index are unchanged:
|
||
Japanese infix search and exclusion of markup-only matches need a separate
|
||
plain-text / language-aware indexing migration. Excerpts already remove HTML
|
||
markup safely, but an indexed markup token can still cause a body-source hit
|
||
with no visible text excerpt. Independent Page titles and relevance tuning are
|
||
also separate features.
|
||
|
||
## Regression coverage
|
||
|
||
Store tests cover latest-only bodies and current renamed Paths, literal scope
|
||
and query punctuation, body/filename source distinction, Domain isolation,
|
||
ACL-before-limit behavior, bounded candidates, query bounds, and cancellation.
|
||
Query-plan tests require FTS-first candidate collection and rowid-constrained
|
||
snippets. A 600-Page / 1,200-Revision fixture checks current-only results, both
|
||
match sources and excerpt association under the real search budget. Input-budget
|
||
tests cover Unicode byte lengths, total bytes, aliases and retained hits;
|
||
deadline tests distinguish optional enrichment from parent cancellation/errors.
|
||
Server tests cover hidden-only results, denied scopes, encoded Path links,
|
||
Unicode keyword segments, contextual clipping, and malicious source HTML.
|
||
An opt-in Chromium test exercises all three display modes, safe highlights,
|
||
Ctrl selection, and absence of horizontal page overflow.
|
||
HTTP tests also cover themed GET/HEAD failures, sanitized errors, retained retry
|
||
parameters and a large-body match beyond the bounded prefix in Japanese/English.
|
||
|
||
City/filter regressions cover hidden/foreign/archived ancestors, duplicate Street
|
||
names and encoded links, metadata-only single-connection reads, literal scopes,
|
||
MIME priority/fallback, combined filters before all three candidate caps,
|
||
Object-vs-alias creation times, inclusive millisecond bounds and Japanese/DST
|
||
calendar conversion. Chromium checks desktop (1440px) and mobile (390px), with
|
||
real City results above Pages, no file selection on City rows, and one set of
|
||
form parameters while changing views/options.
|
||
|
||
Read-only verification against the existing main DB measured the additional
|
||
filter-only cases at approximately 1–6 ms and City metadata lookup at 2 ms;
|
||
representative keyword Page searches remained approximately 79–515 ms. These
|
||
are local observations, not latency guarantees or authenticated HTTP timings.
|
||
This feature adds no schema migration or index rebuild.
|
||
|
||
```sh
|
||
go test -race ./internal/mestionora ./internal/server ./internal/store -run '^TestSearch' -count=1
|
||
PAGESWIKI_BROWSER_TEST=1 go test ./internal/server -run '^TestSearchResultsChromium$' -count=1
|
||
# Optional: opens an existing DB with mode=ro and query_only, with no migrations.
|
||
PAGESWIKI_SEARCH_DIAGNOSTIC_DB=/absolute/path/pages.db go test ./internal/store -run '^TestSearchReadOnlyDatabase$' -count=1 -v
|
||
```
|