Extract Grammaratur, Dregarnuhr, Geduldh, Mestionora, Ordoschnellic, Schutzaria, Kunstzeal and Seheweit behind compatible application contracts. Include accumulated editor resilience, navigation and theme UX, remote CLI, storage/schema and DR groundwork, tests and design/operations documentation. Verified: uncached full Go suite; go vet ./...; race tests for all eight components; 11 targeted Chromium tests; Node suites; 19 Python CLI tests; reproducible editor build. No push, release tag or deployment in this checkpoint.
100 lines
4.5 KiB
Markdown
100 lines
4.5 KiB
Markdown
# Event Scheduler
|
|
|
|
## Purpose
|
|
|
|
CPU-, memory-, and decoder-heavy work must not execute in Viewer requests.
|
|
PagesWiki uses a persistent SQLite event queue for deferred work.
|
|
|
|
`event_jobs` records:
|
|
|
|
- event UUID and type
|
|
- idempotency key
|
|
- JSON payload
|
|
- `queued`, `running`, `succeeded`, or `failed` status
|
|
- scheduled time and attempt count
|
|
- latest error
|
|
- creation and update timestamps
|
|
|
|
The current implementation is a single scheduler loop that may run up to the
|
|
configured concurrency in goroutines. It checks foreground request activity
|
|
between claims and keeps one reserved background slot so a long-lived editor or
|
|
stream cannot starve all jobs. Cross-instance leases are still future work. The
|
|
transactional Domain outbox now records Page/Revision creation and Path/Object
|
|
mutations. Domain-local consumer watermarks are durable and monotonic;
|
|
notification delivery remains future work.
|
|
External preview tools run through bounded
|
|
commands with timeouts.
|
|
|
|
The in-process clock, loop, admission control and retry policy now live in
|
|
`internal/dregarnuhr`. Application handlers are registered in
|
|
`internal/server/event_handlers.go`; SQL ownership remains in `store` through
|
|
the server queue adapter. The existing API, JSON, IDs and schema are unchanged.
|
|
Admission is serialized within one runner, not across processes. Atomic claims
|
|
are not expiring leases, and startup recovery must not run alongside another
|
|
active writer. See [Dregarnuhr contracts and pending HA work](ja/dregarnuhr.md).
|
|
|
|
## Event log retention and trends
|
|
|
|
`event_logs` remains the recent, drill-down log. Each record is associated with
|
|
its event type when the job is known. `event_log_policies` provides per-event
|
|
retention, aggregation window, bucket size, and maximum raw-row settings; `*`
|
|
is the safe default. `event_log_buckets` stores compact counts and first/last
|
|
timestamps by event type, level, and time bucket, so trend charts do not scan
|
|
the raw log table. The API exposes `/event/logs/trends` and `/event/policies`.
|
|
|
|
Retention removes old rows in place and keeps bounded aggregate buckets. A
|
|
future partition/rotation implementation may replace the physical table, but
|
|
must use an atomic rename-and-recreate procedure; an online `DROP TABLE` is
|
|
never used while workers can append logs.
|
|
|
|
## Preview Events
|
|
|
|
`preview.generate` uses the immutable Document hash as its idempotency key.
|
|
Viewer never generates a thumbnail. A children projection:
|
|
|
|
1. checks the immutable preview cache;
|
|
2. emits the cached thumbnail URL when present;
|
|
3. otherwise enqueues the event and immediately renders a format icon.
|
|
|
|
The browser polls only the lightweight cache URL for visible entries. Once the
|
|
job succeeds, it replaces the icon with the cached image without reloading the
|
|
Page.
|
|
|
|
Current generators:
|
|
|
|
| Input | Generator | Cached form |
|
|
| --- | --- | --- |
|
|
| PNG/JPEG/GIF | bounded Go image resampler | JPEG |
|
|
| PDF | `pdftoppm`, first page | JPEG |
|
|
| Video | `ffmpeg`, one frame | JPEG |
|
|
| Excalidraw | bounded server SVG renderer | sandboxed SVG |
|
|
|
|
Individual PDF pages use the same queue. A cache miss returns `202` with
|
|
`Retry-After`; the browser retries until the scheduler writes the immutable
|
|
`<document-hash>-p<page>.jpg` file. PDF page counts and non-ZIP archive listings
|
|
are persisted output of `projection.metadata.generate`, not Viewer work. See
|
|
[deferred projection metadata](projection-metadata-cache.md) for cache quotas,
|
|
pending/retry responses, authorization, and decoder limits.
|
|
|
|
Generated files live beside the Document Store in `previews/` and are addressed
|
|
by immutable Document hash. Cache responses are immutable for one year.
|
|
|
|
## Scheduling
|
|
|
|
- Missing previews are enqueued when an icon projection encounters them.
|
|
- A periodic sweep runs every 15 minutes.
|
|
- The first sweep runs shortly after startup.
|
|
- Configuration → Execution monitor combines resource charts with running jobs
|
|
only. Job history & logs contains paginated completed jobs and per-job logs.
|
|
Event definitions exposes the manual “Generate missing previews” trigger.
|
|
|
|
See [Configuration panels](configuration-panels.md) for polling, indexed queue
|
|
listing, lazy log retrieval and legacy resource-panel URL compatibility.
|
|
|
|
Failures remain visible in scheduler statistics and logs. Retry/backoff and
|
|
cancellation are persisted in the queue; a later sweep or manual trigger may
|
|
enqueue a missing cache again. Preview requests never execute the generator in
|
|
the normal Viewer request path. Non-ZIP entry extraction remains a bounded
|
|
foreground stream pending its separate cached-entry design; metadata listing
|
|
and PDF page counts have migrated to deferred projections.
|