Files
pages/docs/design/event-scheduler.md
Anya Tokugawa d5d21b7841
Build and publish container images / images (push) Successful in 9m34s
Release container images / release (push) Successful in 36s
Refactor architecture into eight components and consolidate reliability work
Extract Grammaratur, Dregarnuhr, Geduldh, Mestionora, Ordoschnellic, Schutzaria, Kunstzeal and Seheweit behind compatible application contracts.

Include accumulated editor resilience, navigation and theme UX, remote CLI, storage/schema and DR groundwork, tests and design/operations documentation.

Verified: uncached full Go suite; go vet ./...; race tests for all eight components; 11 targeted Chromium tests; Node suites; 19 Python CLI tests; reproducible editor build. No push, release tag or deployment in this checkpoint.
2026-11-01 15:23:31 +09:00

4.5 KiB

Event Scheduler

Purpose

CPU-, memory-, and decoder-heavy work must not execute in Viewer requests. PagesWiki uses a persistent SQLite event queue for deferred work.

event_jobs records:

  • event UUID and type
  • idempotency key
  • JSON payload
  • queued, running, succeeded, or failed status
  • scheduled time and attempt count
  • latest error
  • creation and update timestamps

The current implementation is a single scheduler loop that may run up to the configured concurrency in goroutines. It checks foreground request activity between claims and keeps one reserved background slot so a long-lived editor or stream cannot starve all jobs. Cross-instance leases are still future work. The transactional Domain outbox now records Page/Revision creation and Path/Object mutations. Domain-local consumer watermarks are durable and monotonic; notification delivery remains future work. External preview tools run through bounded commands with timeouts.

The in-process clock, loop, admission control and retry policy now live in internal/dregarnuhr. Application handlers are registered in internal/server/event_handlers.go; SQL ownership remains in store through the server queue adapter. The existing API, JSON, IDs and schema are unchanged. Admission is serialized within one runner, not across processes. Atomic claims are not expiring leases, and startup recovery must not run alongside another active writer. See Dregarnuhr contracts and pending HA work.

event_logs remains the recent, drill-down log. Each record is associated with its event type when the job is known. event_log_policies provides per-event retention, aggregation window, bucket size, and maximum raw-row settings; * is the safe default. event_log_buckets stores compact counts and first/last timestamps by event type, level, and time bucket, so trend charts do not scan the raw log table. The API exposes /event/logs/trends and /event/policies.

Retention removes old rows in place and keeps bounded aggregate buckets. A future partition/rotation implementation may replace the physical table, but must use an atomic rename-and-recreate procedure; an online DROP TABLE is never used while workers can append logs.

Preview Events

preview.generate uses the immutable Document hash as its idempotency key. Viewer never generates a thumbnail. A children projection:

  1. checks the immutable preview cache;
  2. emits the cached thumbnail URL when present;
  3. otherwise enqueues the event and immediately renders a format icon.

The browser polls only the lightweight cache URL for visible entries. Once the job succeeds, it replaces the icon with the cached image without reloading the Page.

Current generators:

Input Generator Cached form
PNG/JPEG/GIF bounded Go image resampler JPEG
PDF pdftoppm, first page JPEG
Video ffmpeg, one frame JPEG
Excalidraw bounded server SVG renderer sandboxed SVG

Individual PDF pages use the same queue. A cache miss returns 202 with Retry-After; the browser retries until the scheduler writes the immutable <document-hash>-p<page>.jpg file. PDF page counts and non-ZIP archive listings are persisted output of projection.metadata.generate, not Viewer work. See deferred projection metadata for cache quotas, pending/retry responses, authorization, and decoder limits.

Generated files live beside the Document Store in previews/ and are addressed by immutable Document hash. Cache responses are immutable for one year.

Scheduling

  • Missing previews are enqueued when an icon projection encounters them.
  • A periodic sweep runs every 15 minutes.
  • The first sweep runs shortly after startup.
  • Configuration → Execution monitor combines resource charts with running jobs only. Job history & logs contains paginated completed jobs and per-job logs. Event definitions exposes the manual “Generate missing previews” trigger.

See Configuration panels for polling, indexed queue listing, lazy log retrieval and legacy resource-panel URL compatibility.

Failures remain visible in scheduler statistics and logs. Retry/backoff and cancellation are persisted in the queue; a later sweep or manual trigger may enqueue a missing cache again. Preview requests never execute the generator in the normal Viewer request path. Non-ZIP entry extraction remains a bounded foreground stream pending its separate cached-entry design; metadata listing and PDF page counts have migrated to deferred projections.