Files
pages/docs/design/city-review-and-reconciliation.md
T
Anya Tokugawa c1175d389f Implement Knowledge City platform, organizer and DAV integration
Capture accumulated application, UI, CLI, regression tests and design documentation changes, including task project mapping and default CalDAV list naming.

Keep deployment data, backups, local credentials and build artifacts out of Git. Move the Compose enrollment token to an ignored local environment file.
2026-09-27 14:44:43 +09:00

18 KiB
Raw Blame History

City module review and URL-driven reconciliation

Reviewed: 2026-09-13; implementation follow-up 2026-09-14. Status: incremental repairs landed; automatic reconciliation remains design-only.

Scope: City stores/APIs, URL autoplan, SVG UI, the bundled gen-city 1.4.6 adapter, and scheduler integration. This supplements Knowledge City Model and the platform outbox contract. Phase 2 checkboxes represent first slices, not completion of these repairs.

Landed follow-ups

  • CITY-05 named-route slice: parent-qualified Streets, bounded cancellable detail lookup with active same-Domain ancestors and ambiguity detection; shared links across search, map, inspector, context menu and macros. Detail views no longer load the whole City; map focus and locality search support all three levels. Spot autoplan name collision handling remains open.
  • Per-result map/detail/Page/macro/badge authorization and private responses; visible portal/count/relation recomputation.
  • One-connection-safe bounded City/Spot read transactions and binary literal Path containment/overlap/context/health matching; see read integrity and remaining limits.
  • Stable Domain fabric seed, local Worker generation, bounded geometry cache, deadline/cancellation and directional Arrow/WASD/HJKL navigation; see Worker contract.
  • Horizontal/resting labels, manual organization, nonoverlapping bounded Spot courtyards with Page buildings and a modal utility menu; see immersive projection.

Lazy Page APIs, connected road allocation, deterministic generated Spot names and the outbox/provenance/idempotent inference pipeline are not implied by these checks.

Original findings, in repair order

ID / priority Evidence at audit Correction
CITY-01 / P0 city_handler.go returns GetCityMap and ListSpotPages without per-member authorization. Middleware gates the endpoint, not each returned Page/locality. Batch-authorize nodes, Page links, portals, relations and counts. Audit details/macros too; Domain read must not expose denied descendants.
CITY-02 / P1 buildCityAutoPlan ignores Paths shorter than three segments despite its UI's shallow-Page description. AddSpotPage increments topology version even on identical repeated UPSERTs. Explicit root/portal/leaf rules and no-op reconciliation for unchanged inputs. Separate inferred membership from manual edges.
CITY-03 / P1 POST /city/autoplan synchronously scans all Paths and applies separate node/edge transactions; the first error leaves a partially applied plan. No generation ownership/checkpoint exists. Preview/adoption, bounded resumable batches, source provenance and itemized conflicts before automatic jobs.
CITY-04 / P1 CityNodeOverlappingPath merges exact/ancestor/descendant candidates with LIMIT 1; it and ListSpotPages use unescaped LIKE patterns. Autoplan treats lookup errors as absence. Literal normalized Path segments, compatible parent/identity matching, explicit ambiguity and DB errors. Test %, _, case and non-ASCII Paths.
CITY-05 / P1 Streets allow repeated names under different parents but CityNode/detail URLs resolve by Domain/name. Spots require Domain-unique names, while autoplan derives only a URL segment. Server-provided unambiguous hrefs, parent-qualified readable Street routes, existing UUID fallback, deterministic Spot name collision handling without altering display titles.
CITY-06 / P1 GetCityMap nests open cursors and queries per Quartier/Street/Spot, returning every Spot's entire Page list. Autoplan also queries while the Paths cursor is open. No consistent read snapshot. Bounded bulk queries, snapshots, closed cursors, cancellation and lazy locality/Page APIs. Test with a one-connection pool.
CITY-07 / P1 render() seeds gen-city with domain_uuid:topology_version on every reload. Cached city-geometry.js output is assigned to geometry but never used by this renderer. Stable base-fabric seed and one real geometry cache; remove obsolete generation after migration/tests.
CITY-08 / P1 gen-city's async generate() runs synchronous matrix initialization, recursive path generation and building generation. Layout/SVG construction also runs on the main thread. Worker generation/layout with cancellation and budgets; separately bound DOM work. await alone is not CPU isolation.
CITY-09 / P1 cityLayout sorts/slices roads by coordinates, not connectivity; early Streets can consume all roads. Spots without buildings are omitted. Proximity uses infinite-line rather than segment distance. Connected/fair allocation, finite segment projection and fallback picking targets. Geometry scarcity must never hide a logical node.
CITY-10 / P1 Street weights sum Spot weights (duplicate Page counting), unlike Quartier weights. Allocation repeats on navigation; a MutationObserver repairs Spot positions afterward. Deduplicate at every level, cache allocation, place directly, manage label collisions and remove post-render repair.
CITY-11 / P2 Three embedded creation panels plus inspector move/binding/relation forms; many dynamic English literals. Contextual icon actions, shared modal forms, localized errors and progressive disclosure.
CITY-12 / P2 Only Enter/Space node keys; no map navigation stack. showSpot can append a late response to a different selection. Deleted current nodes can leave an invalid view. Directional focus, explicit view state, stale-response guards, valid-parent fallback and separate in-map history.

CITY-01 describes a confirmed missing check in the inspected call chain, not a production exploit test. CITY-06 is a code-level scaling/pool risk, not a measurement of a production DB lock. Additional correctness requirements:

  • Path/Revision/policy changes can change map output without changing its topology-only ETag. Model and authorization generations belong in validators.
  • Explicit edges resolve a Page's primary Path, which may be outside the inferred locality or unreadable. Preserve locality-specific safe portal links.
  • Undefined classification currently guesses from name/path substrings. Add an explicit kind instead of interpreting a user's spelling as system metadata.
  • Deleting a generated node requires an exclusion/tombstone; otherwise the next sweep recreates it. Offer explicit re-enablement.
  • Recursive rename already updates City prefixes transactionally. Reuse this identity-preserving behavior rather than recreating locality UUIDs.

Component boundaries

  1. URL topology reconciler: canonical Path bindings → derived memberships and generated node proposals, running in the existing event scheduler.
  2. Authorized City projection: topology + metadata → a caller-specific, bounded navigation model. Geometry never grants access.
  3. Geometry adapter: stable fabric + permitted model → blocks, connected Streets, anchors, hit targets and labels. gen-city does not own identity.

Retain gen-city for the first repair and evaluate continuity/capacity/performance fixtures before replacing it or maintaining a fork. Preserve its license and rebuild the committed bundle from assets/city-generator, not vendor edits.

URL inference and preservation

  • Infer from canonical Domain-scoped Paths, never display slugs, host strings or repeated URL decoding. Use shared reserved-address rules to exclude user, group and system routes instead of an incomplete local exclusion list.
  • First directory → Quartier; second → Street; third → Spot; deeper descendants → that Spot. A directory has descendants and may be virtual (no own Page).
  • A locality-boundary Page is its portal, not a child of itself. For example, /life/home/kitchen with descendants is the Kitchen Spot portal; /life/home/readme without descendants is an undefined Page on Home Street.
  • A leaf gaining descendants may gain a locality without changing Page UUID. Change only derived membership, not curated semantic edges.
  • /, /life, /life/home remain Domain/Quartier/Street portal Pages. Show shallow/unassigned Pages at their nearest valid level; do not manufacture strict-descendant boundaries to force them into a Spot.
  • Undefined Pages stay individual yellow-green points. An internal bucket must not become a fabricated public /_undefined portal.
  • Store source/rule version and manual overrides separately from descriptions. Existing nodes and source=local edges start as manual/unknown: never adopt automatically by matching “Generated from URL Paths” text.
  • Preview adoption. Reuse exact compatible boundaries; ambiguous ancestors or descendants need explicit mapping. Keep exclusions, UUIDs and Page content.
  • Aliases may infer multiple memberships, but count each Page UUID once in each aggregate; select readable portal Paths within the locality.

Event/queue design

Extend the shared transactional outbox; do not install expensive callbacks in individual HTTP handlers or build a second scheduler. Cover WebUI, CLI, import, hosted editors, restore, link/unlink, rename and deletion through store writes.

Committed change City invalidation
Path bind/create/unlink/delete; Object deletion Domain/prefix membership and metadata.
Rename/move, including recursive operations Old and new prefixes; retain locality UUIDs.
Revision/edit/restore; filename, MIME or title change Metadata in Domains binding this Page; unchanged Paths do not regenerate fabric.
Topology/manual rules Affected mapping/layout.
Permission/group inheritance change Authorized projections, not canonical geometry seed.
Periodic or manual repair Same bounded reconciler/checkpoint.

Proposed persistent state, not yet migrated:

  • Shared activity_outbox: instance/event UUID, Domain sequence, event version, actor/source, typed Page/Revision target and old/new Path data. Atomic with the mutation; no document bodies or credentials.
  • city_reconcile_state: Domain, desired/applied sequence, rule version, debounce deadline, continuation cursor and last error/job UUID.
  • Generated node/membership provenance: stable source key, rule version and manual override/exclusion; independent of explicit SpotGraph edges.
  • Completed model generation: publish finished snapshots, not half-built reconciliation as a completed map.

Delivery protocol:

  1. Commit change and outbox event together; rollback emits no work. SQLite invalidation triggers may cover legacy writers, but cannot replace actor-aware events. Unsupported raw SQL edits rely on periodic repair.
  2. Consume bounded batches, advance desired sequence and coalesce import/rename bursts by prefix/Domain. Do not create a full-scan job for every file.
  3. Enqueue city.reconcile, keyed by Domain. Initial proposed debounce: 2 seconds, maximum coalescing delay: 10 seconds, configurable to avoid starvation under continuous editing. These are starting values, not measured tuning results.
  4. Capture watermark N and process bounded batches (initial target: 200 Paths, short transactions). Persist derived changes and progress together. Reconcile current state; do not replay stale rename payloads as commands. Recheck source generation before publishing a completed snapshot.
  5. Acknowledge only processed N. If desired advanced while queued/running, enqueue a successor after the active job finishes, or let the dispatcher do so. Current active-key queue deduplication must not acknowledge/drop newer edits.
  6. Retries are idempotent; unchanged data does not increment topology version. Keep itemized conflicts/cursors. Queue-full, pause, crash and cancellation preserve pending invalidations and unprocessed events.
  7. city.reconcile.sweep is a fair, bounded recovery pass across Domains using the same algorithm. Compact outbox only after consumers' checkpoints/retention requirements allow it.

Use existing /_/api/v1/event/ and /_/api/v1/queue definitions, trigger, job and log mechanisms. Proposed City event types are not registered yet. When POST /city/autoplan becomes asynchronous, return 202 with a job UUID and update its UI simultaneously. Dry-run needs bounded results and a base/rule-version token; stale approval must not silently apply a newly recomputed plan.

Settings are Domain-scoped. Existing Domains start manual; enable URL-auto after an adoption preview. New-Domain URL-auto defaults can follow once gates pass. Manual/scheduled triggers share settings and permission checks. Workers may mutate derived City state only, not arbitrary Pages using an administrator token.

Use low priority, one City writer per Domain, shared preview/import resource budgets, cancellation and backoff. Keep geometry out of write transactions and request handling. This is single-writer operation: current startup requeue of all running jobs is not an HA lease; multi-instance workers require fencing.

Geometry and read scalability

  • Fabric key = Domain UUID + explicit seed + generator/version/options. Ordinary edits and topology increments must not change the seed.
  • Layout key = fabric + structural/membership generation + layout version; metadata/policy generations are separate. Preserve prior assignments with bounded growth/shrink hysteresis to avoid global rearrangement for one Page.
  • Build a connected road graph, clip at Quartier boundaries, allocate fairly, project anchors onto bounded segments. Keep every locality pickable with explicit fallback/capacity warnings when geometry is scarce.
  • Deduplicate weights at every level; leave unoccupied space when membership is small. Use zoom-dependent labels/aggregation for dense points.
  • Worker generates/allocates with size/time/cancellation limits; viewport/zoom bounds main-thread rendering separately. Superseded results cannot replace newer views. Remove observer-driven placement repair after regression tests.
  • Top-level API returns Quartier summaries, not every descendant Page. Lazy bounded Street/Spot/Page APIs use bulk queries, snapshots and cancellation.
  • Cache authorized models by Domain, principal/policy and data generation with private HTTP validators. Invalidate on logout/policy change; don't put restricted labels/coordinates in a shared origin-wide cache. Topology-independent fabric can be cached separately. Remove unused city-geometry.js storage work.
  • During regeneration keep the last valid authorized map, with a small status icon. Failures provide a directory fallback/retry, never block Page navigation.

Compact UI and keyboard contract

  • Toolbar: breadcrumbs, search, zoom/reset, add icon, settings icon. Generation status tooltip links to details and existing job monitor, not verbose inline text.
  • Add opens one modal for Quartier/Street/Spot with contextual parent/Path defaults and ancestry/conflict validation. Page bindings/relations use the same modal shell. Preserve inputs on failure, prevent duplicate submits and scope parents.
  • Settings modal: manual/URL-auto, adoption preview, rules/exclusions, event timing and explicit regeneration. Changing fabric seed is a separate confirmed action.
  • Accessible icon names/tooltips, adequate hit areas, visible focus, Japanese and English strings, trapped/restored modal focus. Escape closes innermost overlay before outer map popup. No global form-input shortcuts.
Key, only while map navigation is focused Action
Arrows; W/A/S/D; K/H/J/L Focus nearest visible selectable node up/left/down/right, without activation.
Enter / Space Open locality or activate Spot portal/Page, matching click behavior.
Backspace Parent City level, without leaving the page.
+ / -; 0 Zoom in/out; reset viewport.
Shift+F10 / Context Menu Context menu on focused node.
Escape Close menu/modal/map popup from inside outward.
Tab / Shift+Tab Native accessible traversal, roving tab stop within map nodes.

Do not intercept input/textarea/select/contenteditable, IME or Ctrl/Alt/Meta shortcuts. Pointer drag pans; directional keys navigate nodes, with defined offscreen/empty-view fallback. Refresh preserves selection/zoom; deletion falls back to nearest valid parent. Guard async inspector/load responses by selection and request generation, and cancel obsolete requests.

Locality history is an in-memory stack, not a browser entry for each movement. Existing back/forward buttons invoke browser history; they cannot provide per-locality traversal without entries. Provide distinct map back/forward controls instead of pretending these two requirements are equivalent.

Delivery and acceptance checklist

  1. CITY-A — correctness gate: denied descendants, literal Paths, scoped names/links, shallow/portal rules, source-aware idempotent plan/apply.
  2. CITY-B — durable events: outbox and watermarks, consumer, CLI/import/ adapter coverage, concurrent-edit/rollback/retry tests, manual/scheduled/auto UI.
  3. CITY-C — responsive map: stable seed/cache, Worker, connected allocation, capacity fallbacks, lazy authorized reads and continuity/stale-result tests.
  4. CITY-D — compact navigation: icons/modals, keyboard/i18n/focus/history, popup and error recovery browser tests.

CITY-C/D may proceed independently of CITY-B once CITY-A's read/identity contracts are established. Do not enable automatic production reconciliation before CITY-A/B pass. Main-roadmap P0 authentication work remains open.

Fixtures: 500 / 5,000 / 50,000 Paths, aliases, many tiny Streets, one crowded Street, virtual folders, repeated names, %/_/Unicode, manual/archived mappings, denied descendants, rename/edit during generation, crash/queue-full/retry/cancel. Record API p50/p95, response size, query count, SQLite write-lock duration, generation CPU/memory and browser long tasks before fixing production budgets.

Audit checks performed locally:

  • Passed existing tests: GOCACHE=/tmp/pageswiki-go-cache go test ./internal/store ./internal/server -run 'TestCity|Test.*City|Test.*Topology' -count=1. These do not cover all findings above.
  • Ran the checked-in gen-city bundle in a Node VM three times, seed key audit-city:1, dimensions 240×160. Synchronous call times: about 39/32/29 ms; 259 roads and 279 buildings each. A zero-delay timer did not run before generation completed. This supports the source finding of synchronous work, not a browser/production latency benchmark.
  • No live DB mutation, schema migration, runtime/UI change or deployment in this audit.