Rawman Grey
c8f8bbc5ac
Bundle extension (upload hook + API endpoint + admin module) that warms Directus image transform cache in-process with an explicit acceptFormat, so the generated file matches the browser's cache key byte-for-byte. - resumable, window-aware backfill with progress + cancel - system + custom presets, effective-format reporting, shadow warnings - storage breakdown (previews vs originals), human-readable API fields - diagnostics: resolve-plan prediction, per-file generate, ops ring buffer - isolated dev/staging compose stack + deploy/rollback scripts Verified on Directus 11.17.4: 40/40 tests, hash-match gate passed, idempotent backfill, custom fixed-format presets, error surfacing.
206 lines
14 KiB
JSON
206 lines
14 KiB
JSON
{
|
||
"$schema": [
|
||
"/home/rawman/.config/opencode/schemas/feature_list.schema.json"
|
||
],
|
||
"features": [
|
||
{
|
||
"feature_id": "F01",
|
||
"title": "Bundle scaffold (module + hook + endpoint), build pipeline, and local load",
|
||
"acceptance_steps": [
|
||
"Run `npm install` → resolves the pinned devDependencies including vue.",
|
||
"Run `npm run build` → emits `dist/app.js` and `dist/api.js`; `npm run typecheck` reports 0 errors.",
|
||
"`directus-extension link` (or the dev compose stack) loads the bundle into a local Directus 11.17.4 with no error.",
|
||
"The `image-previews` module appears in the admin sidebar, `GET /image-previews/status` answers, and a test upload emits the hook debug line.",
|
||
"`npm run publish` copies `dist/` + `package.json` to the configured deploy target (staging only at this stage)."
|
||
],
|
||
"passes": false,
|
||
"notes": [
|
||
"Manifest is a bundle: `path:{app,api}`, `entries[{type:module|hook|endpoint,name,source}]`, `host:^10.10.0 || ^11.0.0`.",
|
||
"Module sidebar fields (id/name/icon/routes) live in the entrypoint object, not package.json."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F02",
|
||
"title": "Staging environment harness + deploy/rollback scripts (safe testing)",
|
||
"acceptance_steps": [
|
||
"`dev/compose.dev.yml` brings up Directus 11.17.4 with its own postgres + redis on a dedicated port, with the extension mounted.",
|
||
"The stack seeds from a **restored anonymized DB dump** and a **copy of uploads** (`dev/seed/`), so previews can be tested against realistic data.",
|
||
"A documented script builds and deploys to the staging instance, and a rollback script restores the previous `dist/` and disables the extension.",
|
||
"Confirmed the local/staging stack shares NOTHING with prod (separate DB, separate storage, separate ports).",
|
||
"README documents how to start/stop/seed the stack and the exact anonymization rules used for the dump."
|
||
],
|
||
"passes": false,
|
||
"blocked_by": [
|
||
"F01"
|
||
],
|
||
"notes": [
|
||
"This must land before any feature is tested on real-ish data; never test on the prod instance.",
|
||
"Decide staging placement (second compose project on land-nest vs a separate dev box) — goal.md open item #5.",
|
||
"Dev harness (dev/compose.dev.yml + sync-extension.sh) verified; staging compose/deploy/rollback templates added but not executed remotely."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F03",
|
||
"title": "Persisted settings + progress-state store and the config/status API endpoints",
|
||
"acceptance_steps": [
|
||
"`src/lib/settings.ts` reads/writes a JSON settings file under `IMAGE_PREVIEWS_STATE_DIR` (default `/directus/extensions/.image-previews`), merged over env defaults; keys: `enabled` (default false), `windowStart`, `windowEnd`, `concurrency`, `presets`, `formats` (`avif,webp`), `onUpload`, `debug`.",
|
||
"`GET /image-previews/config` returns effective settings; `PUT /image-previews/config` validates and persists them; invalid payloads return 400 without corrupting the file.",
|
||
"`src/lib/state.ts` appends completed `{file_id,preset_key,format}` records to `state.jsonl` and loads them back; a resume helper returns completed pairs + totals.",
|
||
"`GET /image-previews/status` returns `{ running, total, done, remaining, generated, hits, failed, rate, eta, lastRun, nextWindow }`.",
|
||
"Unit tests cover settings merge/defaults/validation, config round-trip, state append/resume, status aggregation.",
|
||
"`npm test` green and `npm run typecheck` 0 errors."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F02"
|
||
],
|
||
"notes": [
|
||
"No native Directus extension-settings store exists; a JSON file under a persisted writable dir is the pragmatic choice.",
|
||
"All state is file-backed (PM2 cluster, 8 workers — never in-memory)."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F04",
|
||
"title": "Preset registry (system + settings, dynamic) and the in-process warmer core",
|
||
"acceptance_steps": [
|
||
"`src/lib/presets.ts` returns the merged preset list in controller order (`[...SYSTEM_ASSET_ALLOW_LIST, ...storage_asset_presets]`) with first-match-wins on duplicate keys.",
|
||
"System preset objects come from an `@directus/api` constant import if available, else the version-pinned copy in `src/lib/system-presets.ts`; the mechanism used is recorded in progress.txt.",
|
||
"`src/lib/warmer.ts` calls `new services.AssetsService({ schema }).getAsset(fileId, { transformationParams: <verbatim preset object>, acceptFormat })` then destroys/unpipes the returned stream.",
|
||
"For a `format:'auto'` preset the warmer emits BOTH `avif` and `webp`; a fixed-`format` preset emits one; a resize-only preset emits none.",
|
||
"Concurrency honours the configured limit; a per-preset/format failure (concurrency error, `IllegalAssetTransformation`, missing dims) is caught, logged, and doesn't abort the rest; the result summarises `{ requested, ok, failed }`.",
|
||
"Unit tests assert: merged preset order + first-match-wins; variant selection per format; exact `getAsset` args; concurrency cap; error isolation.",
|
||
"`npm test` green and `npm run typecheck` 0 errors."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F03"
|
||
],
|
||
"notes": [
|
||
"Passing `{ key }` alone is a no-op — always pass the full preset object.",
|
||
"`acceptFormat:'avif'` in a direct call equals a browser `Accept: image/avif` (verified) — reproduces the admin cache file without HTTP."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F05",
|
||
"title": "Upload hook + new-preset trigger",
|
||
"acceptance_steps": [
|
||
"On `files.upload` where `payload.type` starts with `image/` and `onUpload` is enabled, the hook runs the warmer for configured presets off the response path (fire-and-forget; never blocks or fails the upload).",
|
||
"A non-image upload performs no warming; a warmer failure never fails or slows the upload.",
|
||
"After uploading a test JPEG through the admin, the expected `<basename>__<hash>.avif` (and `.webp`) files for each configured `format:'auto'` preset appear in the storage root.",
|
||
"When a storage asset preset is added/changed, existing images are warmed for it (`settings.update` hook and/or a manual \"sync presets\" action); re-running is idempotent.",
|
||
"Under PM2 cluster the hook fires once per upload (logged once, not per worker)."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F04"
|
||
],
|
||
"notes": [
|
||
"`files.upload` is the correct/only post-write event in v11.17.4.",
|
||
"New-preset trigger mechanics are goal.md open item #3.",
|
||
"Decision: manual sync chosen — POST /run is idempotent and warms existing images for all presets; no settings.update auto-trigger."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F06",
|
||
"title": "Diagnostics & manual generation API + observability",
|
||
"acceptance_steps": [
|
||
"`POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` generates (or dry-runs) previews for ONE file and returns per `(key,format)` `{ ok, cached, filename, bytes, ms, error }`.",
|
||
"`GET /image-previews/resolve-plan?fileId=&key=` returns the resolved transforms array, the **predicted cache filename**, and whether it already exists — WITHOUT generating.",
|
||
"`GET /image-previews/ops?limit=` returns a recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error), capped in size.",
|
||
"A `debug` setting toggles verbose structured logs; logs carry ids/metadata only, never file contents.",
|
||
"All diagnostics endpoints except `/generate` are read-only and safe to call on a live instance.",
|
||
"Unit tests cover resolve-plan prediction (given a preset, the predicted filename equals the warmer's produced filename) and ops-ring-buffer behaviour.",
|
||
"`npm test` green and `npm run typecheck` 0 errors."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F04"
|
||
],
|
||
"notes": [
|
||
"resolve-plan is the debugging tool that proves hash-match before anything is written.",
|
||
"Also the autonomous loop's primary feedback channel — keep responses machine-readable."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F07",
|
||
"title": "Admin module UI: progress bar, disk usage, config form, diagnostics panel",
|
||
"acceptance_steps": [
|
||
"The \"Image Previews\" module renders and shows a progress bar with `done / total`, remaining, throughput, ETA, plus last-run and next-window times.",
|
||
"During a run the page live-updates (polls `/status`); \"Run now\" / \"Cancel\" controls start/stop a run.",
|
||
"A disk-usage panel shows total bytes + count of preview files (list the storage root, sum `stat()` sizes of `__<hash>` files); a refresh control recomputes it.",
|
||
"A config form edits `enabled`, maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`, persisting via `PUT /config`; changes take effect without rebuilding.",
|
||
"A diagnostics panel lists recent operations (`/ops`) and lets an operator run generate/dry-run/resolve-plan for a pasted file id.",
|
||
"Invalid input is rejected client-side with a clear message; a saved change is reflected after reload.",
|
||
"Smoke-verified on the staging stack (notes/screenshots in progress.txt)."
|
||
],
|
||
"passes": false,
|
||
"blocked_by": [
|
||
"F06"
|
||
],
|
||
"notes": [
|
||
"Use Directus UI components / the private-view shell; keep it minimal.",
|
||
"Concurrency is the throttle knob here because in-process generation bypasses the HTTP rate limiter."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F08",
|
||
"title": "Resumable backfill engine (window-aware, throttled, progress-reporting)",
|
||
"acceptance_steps": [
|
||
"The job enumerates `directus_files` with `type` starting `image/` (paginated) and runs the warmer for each file × configured preset × format.",
|
||
"It only runs inside the configured maintenance window, or on an explicit manual run; starting outside the window when scheduled is a logged no-op.",
|
||
"It honours the configured concurrency cap and appends completed pairs to `state.jsonl`; restarting after a kill resumes from the last completed pair.",
|
||
"Progress is observable during a run via `GET /status` and periodic log lines.",
|
||
"A second full run over an already-warmed library reports zero new generations (idempotent).",
|
||
"A >6000px original or a concurrency error is recorded as `failed` for that pair; the job continues.",
|
||
"`POST /run` starts a run, `POST /run/cancel` stops it; a second concurrent run returns 409 (cluster-wide lock file)."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F07"
|
||
],
|
||
"notes": [
|
||
"Run state is file-backed; only one run at a time across the cluster.",
|
||
"Scheduling: external cron hitting `/run` (default) vs in-process timer — goal.md open item #4."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F09",
|
||
"title": "Verification harness: hash-match gate + cold/warm timing + disk counts",
|
||
"acceptance_steps": [
|
||
"For one real file, the on-disk filename produced by an admin browser request for `?key=system-medium-cover` is byte-identical to the warmer's non-dry-run output for the same key+format (the avif variant matches the browser's avif request).",
|
||
"The same is demonstrated for `system-large-cover` (image field) and `system-small-cover` (image display).",
|
||
"A timing check prints cold vs warm `GET /assets/<id>?key=system-medium-cover`; warm is materially faster.",
|
||
"Preview-file count and total bytes are reported before/after a backfill chunk and increase by the expected number of (file × preset × format) entries.",
|
||
"Findings (observed admin query string, resulting filenames, uuids, timings) are recorded in progress.txt."
|
||
],
|
||
"passes": true,
|
||
"blocked_by": [
|
||
"F08"
|
||
],
|
||
"notes": [
|
||
"Make-or-break gate: if the warmer's filename does not match the admin's, the approach is wrong — revisit before rollout.",
|
||
"Also validates the system-preset copy from F04."
|
||
]
|
||
},
|
||
{
|
||
"feature_id": "F10",
|
||
"title": "Ops: fail-open/disabled-by-default, prod rollout + rollback, scheduling, docs, remote-feedback runbook",
|
||
"acceptance_steps": [
|
||
"Every entrypoint body is wrapped so a thrown error is logged and swallowed (never propagates into Directus core), and ALL behavior is gated by `enabled` (default false) — a loaded-but-disabled extension is inert, proven by loading it on staging with `enabled=false`.",
|
||
"A documented prod rollout procedure exists: backup previous `dist/` → deploy → healthcheck → enable via config; and a rollback that restores `dist/` and disables.",
|
||
"A documented scheduling setup runs the backfill inside the maintenance window (cron hitting `POST /image-previews/run`, by default).",
|
||
"`README.md` documents purpose, install/build/publish, every setting, presets/formats warmed, run/resume/cancel, disk reporting, and the maintenance window; `AGENTS.md` captures the transform invariants and safety rules.",
|
||
"A **remote-feedback runbook** documents the scoped token and/or restricted SSH access for the autonomous loop (staging only, no prod Docker socket, no prod admin token, private network), and the machine-readable endpoints used as the feedback channel.",
|
||
"Final end-to-end pass on staging: upload a fresh image → previews appear with no admin-visible delay; a small backfill chunk completes with the progress bar advancing and disk usage updating; then a controlled prod enable with rollback verified."
|
||
],
|
||
"passes": false,
|
||
"blocked_by": [
|
||
"F09"
|
||
],
|
||
"notes": [
|
||
"Promotion to prod is a human action; the loop proposes, a human deploys.",
|
||
"Record the exact cron entry, the scoped grants, and the rollback commands verbatim."
|
||
]
|
||
}
|
||
]
|
||
}
|