docs: track the full .opencode workspace
Keep goal, feature list, progress log, and the project skill together in the repo. Internal hostnames/paths sanitized.
This commit is contained in:
parent
1758f2f9c5
commit
73e837c165
5 changed files with 605 additions and 3 deletions
1
.opencode/.gitignore
vendored
Normal file
1
.opencode/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
node_modules
|
||||
206
.opencode/feature_list.json
Normal file
206
.opencode/feature_list.json
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
{
|
||||
"$schema": [
|
||||
"/home/rawman/.config/opencode/schemas/feature_list.schema.json"
|
||||
],
|
||||
"features": [
|
||||
{
|
||||
"feature_id": "F01",
|
||||
"title": "Bundle scaffold (module + hook + endpoint), build pipeline, and local load",
|
||||
"acceptance_steps": [
|
||||
"Run `npm install` → resolves the pinned devDependencies including vue.",
|
||||
"Run `npm run build` → emits `dist/app.js` and `dist/api.js`; `npm run typecheck` reports 0 errors.",
|
||||
"`directus-extension link` (or the dev compose stack) loads the bundle into a local Directus 11.17.4 with no error.",
|
||||
"The `image-previews` module appears in the admin sidebar, `GET /image-previews/status` answers, and a test upload emits the hook debug line.",
|
||||
"`npm run publish` copies `dist/` + `package.json` to the configured deploy target (staging only at this stage)."
|
||||
],
|
||||
"passes": false,
|
||||
"notes": [
|
||||
"Manifest is a bundle: `path:{app,api}`, `entries[{type:module|hook|endpoint,name,source}]`, `host:^10.10.0 || ^11.0.0`.",
|
||||
"Module sidebar fields (id/name/icon/routes) live in the entrypoint object, not package.json."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F02",
|
||||
"title": "Staging environment harness + deploy/rollback scripts (safe testing)",
|
||||
"acceptance_steps": [
|
||||
"`dev/compose.dev.yml` brings up Directus 11.17.4 with its own postgres + redis on a dedicated port, with the extension mounted.",
|
||||
"The stack seeds from a **restored anonymized DB dump** and a **copy of uploads** (`dev/seed/`), so previews can be tested against realistic data.",
|
||||
"A documented script builds and deploys to the staging instance, and a rollback script restores the previous `dist/` and disables the extension.",
|
||||
"Confirmed the local/staging stack shares NOTHING with prod (separate DB, separate storage, separate ports).",
|
||||
"README documents how to start/stop/seed the stack and the exact anonymization rules used for the dump."
|
||||
],
|
||||
"passes": false,
|
||||
"blocked_by": [
|
||||
"F01"
|
||||
],
|
||||
"notes": [
|
||||
"This must land before any feature is tested on real-ish data; never test on the prod instance.",
|
||||
"Decide staging placement (second compose project on the production host vs a separate dev box) — goal.md open item #5.",
|
||||
"Dev harness (dev/compose.dev.yml + sync-extension.sh) verified; staging compose/deploy/rollback templates added but not executed remotely."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F03",
|
||||
"title": "Persisted settings + progress-state store and the config/status API endpoints",
|
||||
"acceptance_steps": [
|
||||
"`src/lib/settings.ts` reads/writes a JSON settings file under `IMAGE_PREVIEWS_STATE_DIR` (default `/directus/extensions/.image-previews`), merged over env defaults; keys: `enabled` (default false), `windowStart`, `windowEnd`, `concurrency`, `presets`, `formats` (`avif,webp`), `onUpload`, `debug`.",
|
||||
"`GET /image-previews/config` returns effective settings; `PUT /image-previews/config` validates and persists them; invalid payloads return 400 without corrupting the file.",
|
||||
"`src/lib/state.ts` appends completed `{file_id,preset_key,format}` records to `state.jsonl` and loads them back; a resume helper returns completed pairs + totals.",
|
||||
"`GET /image-previews/status` returns `{ running, total, done, remaining, generated, hits, failed, rate, eta, lastRun, nextWindow }`.",
|
||||
"Unit tests cover settings merge/defaults/validation, config round-trip, state append/resume, status aggregation.",
|
||||
"`npm test` green and `npm run typecheck` 0 errors."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F02"
|
||||
],
|
||||
"notes": [
|
||||
"No native Directus extension-settings store exists; a JSON file under a persisted writable dir is the pragmatic choice.",
|
||||
"All state is file-backed (PM2 cluster, 8 workers — never in-memory)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F04",
|
||||
"title": "Preset registry (system + settings, dynamic) and the in-process warmer core",
|
||||
"acceptance_steps": [
|
||||
"`src/lib/presets.ts` returns the merged preset list in controller order (`[...SYSTEM_ASSET_ALLOW_LIST, ...storage_asset_presets]`) with first-match-wins on duplicate keys.",
|
||||
"System preset objects come from an `@directus/api` constant import if available, else the version-pinned copy in `src/lib/system-presets.ts`; the mechanism used is recorded in progress.txt.",
|
||||
"`src/lib/warmer.ts` calls `new services.AssetsService({ schema }).getAsset(fileId, { transformationParams: <verbatim preset object>, acceptFormat })` then destroys/unpipes the returned stream.",
|
||||
"For a `format:'auto'` preset the warmer emits BOTH `avif` and `webp`; a fixed-`format` preset emits one; a resize-only preset emits none.",
|
||||
"Concurrency honours the configured limit; a per-preset/format failure (concurrency error, `IllegalAssetTransformation`, missing dims) is caught, logged, and doesn't abort the rest; the result summarises `{ requested, ok, failed }`.",
|
||||
"Unit tests assert: merged preset order + first-match-wins; variant selection per format; exact `getAsset` args; concurrency cap; error isolation.",
|
||||
"`npm test` green and `npm run typecheck` 0 errors."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F03"
|
||||
],
|
||||
"notes": [
|
||||
"Passing `{ key }` alone is a no-op — always pass the full preset object.",
|
||||
"`acceptFormat:'avif'` in a direct call equals a browser `Accept: image/avif` (verified) — reproduces the admin cache file without HTTP."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F05",
|
||||
"title": "Upload hook + new-preset trigger",
|
||||
"acceptance_steps": [
|
||||
"On `files.upload` where `payload.type` starts with `image/` and `onUpload` is enabled, the hook runs the warmer for configured presets off the response path (fire-and-forget; never blocks or fails the upload).",
|
||||
"A non-image upload performs no warming; a warmer failure never fails or slows the upload.",
|
||||
"After uploading a test JPEG through the admin, the expected `<basename>__<hash>.avif` (and `.webp`) files for each configured `format:'auto'` preset appear in the storage root.",
|
||||
"When a storage asset preset is added/changed, existing images are warmed for it (`settings.update` hook and/or a manual \"sync presets\" action); re-running is idempotent.",
|
||||
"Under PM2 cluster the hook fires once per upload (logged once, not per worker)."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F04"
|
||||
],
|
||||
"notes": [
|
||||
"`files.upload` is the correct/only post-write event in v11.17.4.",
|
||||
"New-preset trigger mechanics are goal.md open item #3.",
|
||||
"Decision: manual sync chosen — POST /run is idempotent and warms existing images for all presets; no settings.update auto-trigger."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F06",
|
||||
"title": "Diagnostics & manual generation API + observability",
|
||||
"acceptance_steps": [
|
||||
"`POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` generates (or dry-runs) previews for ONE file and returns per `(key,format)` `{ ok, cached, filename, bytes, ms, error }`.",
|
||||
"`GET /image-previews/resolve-plan?fileId=&key=` returns the resolved transforms array, the **predicted cache filename**, and whether it already exists — WITHOUT generating.",
|
||||
"`GET /image-previews/ops?limit=` returns a recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error), capped in size.",
|
||||
"A `debug` setting toggles verbose structured logs; logs carry ids/metadata only, never file contents.",
|
||||
"All diagnostics endpoints except `/generate` are read-only and safe to call on a live instance.",
|
||||
"Unit tests cover resolve-plan prediction (given a preset, the predicted filename equals the warmer's produced filename) and ops-ring-buffer behaviour.",
|
||||
"`npm test` green and `npm run typecheck` 0 errors."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F04"
|
||||
],
|
||||
"notes": [
|
||||
"resolve-plan is the debugging tool that proves hash-match before anything is written.",
|
||||
"Also the autonomous loop's primary feedback channel — keep responses machine-readable."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F07",
|
||||
"title": "Admin module UI: progress bar, disk usage, config form, diagnostics panel",
|
||||
"acceptance_steps": [
|
||||
"The \"Image Previews\" module renders and shows a progress bar with `done / total`, remaining, throughput, ETA, plus last-run and next-window times.",
|
||||
"During a run the page live-updates (polls `/status`); \"Run now\" / \"Cancel\" controls start/stop a run.",
|
||||
"A disk-usage panel shows total bytes + count of preview files (list the storage root, sum `stat()` sizes of `__<hash>` files); a refresh control recomputes it.",
|
||||
"A config form edits `enabled`, maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`, persisting via `PUT /config`; changes take effect without rebuilding.",
|
||||
"A diagnostics panel lists recent operations (`/ops`) and lets an operator run generate/dry-run/resolve-plan for a pasted file id.",
|
||||
"Invalid input is rejected client-side with a clear message; a saved change is reflected after reload.",
|
||||
"Smoke-verified on the staging stack (notes/screenshots in progress.txt)."
|
||||
],
|
||||
"passes": false,
|
||||
"blocked_by": [
|
||||
"F06"
|
||||
],
|
||||
"notes": [
|
||||
"Use Directus UI components / the private-view shell; keep it minimal.",
|
||||
"Concurrency is the throttle knob here because in-process generation bypasses the HTTP rate limiter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F08",
|
||||
"title": "Resumable backfill engine (window-aware, throttled, progress-reporting)",
|
||||
"acceptance_steps": [
|
||||
"The job enumerates `directus_files` with `type` starting `image/` (paginated) and runs the warmer for each file × configured preset × format.",
|
||||
"It only runs inside the configured maintenance window, or on an explicit manual run; starting outside the window when scheduled is a logged no-op.",
|
||||
"It honours the configured concurrency cap and appends completed pairs to `state.jsonl`; restarting after a kill resumes from the last completed pair.",
|
||||
"Progress is observable during a run via `GET /status` and periodic log lines.",
|
||||
"A second full run over an already-warmed library reports zero new generations (idempotent).",
|
||||
"A >6000px original or a concurrency error is recorded as `failed` for that pair; the job continues.",
|
||||
"`POST /run` starts a run, `POST /run/cancel` stops it; a second concurrent run returns 409 (cluster-wide lock file)."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F07"
|
||||
],
|
||||
"notes": [
|
||||
"Run state is file-backed; only one run at a time across the cluster.",
|
||||
"Scheduling: external cron hitting `/run` (default) vs in-process timer — goal.md open item #4."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F09",
|
||||
"title": "Verification harness: hash-match gate + cold/warm timing + disk counts",
|
||||
"acceptance_steps": [
|
||||
"For one real file, the on-disk filename produced by an admin browser request for `?key=system-medium-cover` is byte-identical to the warmer's non-dry-run output for the same key+format (the avif variant matches the browser's avif request).",
|
||||
"The same is demonstrated for `system-large-cover` (image field) and `system-small-cover` (image display).",
|
||||
"A timing check prints cold vs warm `GET /assets/<id>?key=system-medium-cover`; warm is materially faster.",
|
||||
"Preview-file count and total bytes are reported before/after a backfill chunk and increase by the expected number of (file × preset × format) entries.",
|
||||
"Findings (observed admin query string, resulting filenames, uuids, timings) are recorded in progress.txt."
|
||||
],
|
||||
"passes": true,
|
||||
"blocked_by": [
|
||||
"F08"
|
||||
],
|
||||
"notes": [
|
||||
"Make-or-break gate: if the warmer's filename does not match the admin's, the approach is wrong — revisit before rollout.",
|
||||
"Also validates the system-preset copy from F04."
|
||||
]
|
||||
},
|
||||
{
|
||||
"feature_id": "F10",
|
||||
"title": "Ops: fail-open/disabled-by-default, prod rollout + rollback, scheduling, docs, remote-feedback runbook",
|
||||
"acceptance_steps": [
|
||||
"Every entrypoint body is wrapped so a thrown error is logged and swallowed (never propagates into Directus core), and ALL behavior is gated by `enabled` (default false) — a loaded-but-disabled extension is inert, proven by loading it on staging with `enabled=false`.",
|
||||
"A documented prod rollout procedure exists: backup previous `dist/` → deploy → healthcheck → enable via config; and a rollback that restores `dist/` and disables.",
|
||||
"A documented scheduling setup runs the backfill inside the maintenance window (cron hitting `POST /image-previews/run`, by default).",
|
||||
"`README.md` documents purpose, install/build/publish, every setting, presets/formats warmed, run/resume/cancel, disk reporting, and the maintenance window; `AGENTS.md` captures the transform invariants and safety rules.",
|
||||
"A **remote-feedback runbook** documents the scoped token and/or restricted SSH access for the autonomous loop (staging only, no prod Docker socket, no prod admin token, private network), and the machine-readable endpoints used as the feedback channel.",
|
||||
"Final end-to-end pass on staging: upload a fresh image → previews appear with no admin-visible delay; a small backfill chunk completes with the progress bar advancing and disk usage updating; then a controlled prod enable with rollback verified."
|
||||
],
|
||||
"passes": false,
|
||||
"blocked_by": [
|
||||
"F09"
|
||||
],
|
||||
"notes": [
|
||||
"Promotion to prod is a human action; the loop proposes, a human deploys.",
|
||||
"Record the exact cron entry, the scoped grants, and the rollback commands verbatim."
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
250
.opencode/goal.md
Normal file
250
.opencode/goal.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Goal: Directus image preview pre-generation
|
||||
|
||||
Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (independent of any host
|
||||
application).
|
||||
Location: `/home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews`
|
||||
Target: self-hosted Directus **11.17.4** (PM2 cluster, local disk storage).
|
||||
Project type: Directus **bundle** extension (app module + api hook + api endpoint).
|
||||
|
||||
## Summary
|
||||
|
||||
Pre-generate Directus image transform ("preview") cache files so the admin UI never pays the
|
||||
first-request sharp cost. One warmer, several triggers:
|
||||
|
||||
- **upload hook** — after an image upload, warm its previews.
|
||||
- **new-preset trigger** — when a storage asset preset is added/changed, warm it for existing images.
|
||||
- **backfill job** — a resumable, scheduled sweep over the existing `directus_files` library, inside a
|
||||
configurable maintenance window.
|
||||
- **manual / diagnostics** — generate (or dry-run) previews for a single file on demand, to debug.
|
||||
|
||||
Plus an **admin module** ("Image Previews") with a **progress bar**, **disk usage**, a **config form**
|
||||
(persisted, no rebuild), and a **diagnostics panel**.
|
||||
|
||||
## Verified mechanism (source-checked against tag v11.17.4)
|
||||
|
||||
The on-disk transform cache file is:
|
||||
|
||||
```
|
||||
<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>
|
||||
```
|
||||
|
||||
written **beside the original** in the storage root (`STORAGE_LOCAL_ROOT=./uploads`).
|
||||
|
||||
- `hash` = `object-hash@3` defaults: sha1/hex, `unorderedArrays:false` (**array order matters**),
|
||||
`unorderedObjects:true`. Hashed over the **RESOLVED** transform array from `resolvePreset`.
|
||||
- `resolvePreset` order is fixed: `(1) copy transformationParams.transforms[]`, `(2) push toFormat`
|
||||
(if `format`/`quality`), `(3) push resize` (+`extract` on focal cover).
|
||||
- `maybeExtractFormat` picks the last `toFormat` arg as the file extension.
|
||||
- `AssetsService.getAsset` checks `exists(assetFilename)` first (cache hit → no sharp); on miss runs
|
||||
sharp and **awaits** `storage.write(...)` before returning. Not draining the returned stream only
|
||||
leaks an FD (never truncates).
|
||||
- Called as sudo (`new AssetsService({ schema })`) it needs **no** HTTP request context.
|
||||
|
||||
## Why the naive / community approach FAILS (must not repeat)
|
||||
|
||||
1. **`{ key }` alone is a no-op.** `resolvePreset` never reads `key`. Pass the **full preset object**.
|
||||
2. **`format:'auto'` mismatch.** Every `system-*` preset is `format:'auto'`. `getFormat` with no
|
||||
`acceptFormat` → JPEG→`jpg`/PNG→`png`; a browser `Accept: image/avif` → `avif`. Different
|
||||
`toFormat` arg → different hash → **guaranteed miss**.
|
||||
3. Disproved myths: the write is awaited; the admin does NOT use `width`/`height`.
|
||||
4. Raw params resolve to `[toFormat, resize]`; a preset resolves to `[resize, toFormat]` → different hash.
|
||||
|
||||
## Chosen strategy — IN-PROCESS generation (no token, no HTTP, no rate limiter)
|
||||
|
||||
Because this is an internal extension, generate by calling Directus's own service in-process:
|
||||
|
||||
```ts
|
||||
const assets = new services.AssetsService({ schema }); // sudo
|
||||
const { stream } = await assets.getAsset(fileId, {
|
||||
transformationParams: presetObject, // FULL object, not { key }
|
||||
acceptFormat: 'avif', // or 'webp'
|
||||
});
|
||||
(stream as Readable).destroy?.();
|
||||
```
|
||||
|
||||
`acceptFormat` reproduces exactly what a browser's `Accept` header does, so the cache filename is
|
||||
byte-identical to the admin/browser read — **without any HTTP request, token, proxy, or rate limiter**.
|
||||
Generate **two variants** for `format:'auto'` presets (`avif`, `webp`); fixed-`format` presets yield
|
||||
one; resize-only presets (no `format`/`quality`) yield none.
|
||||
|
||||
HTTP-replay (`GET /assets/{id}?key=…` with a browser `Accept` + auth) is the documented **fallback**
|
||||
only — it needs a token and rate-limiter pacing, which is why it is not the default.
|
||||
|
||||
## Preset registry (system + settings, dynamic)
|
||||
|
||||
The controller builds its preset list inline as:
|
||||
|
||||
```
|
||||
res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]
|
||||
```
|
||||
|
||||
and resolves `?key=` with `.find(t => t.key === key)` → **first match wins** (a system key shadows a
|
||||
custom preset of the same key). There is **no** reusable helper and `GET /settings` exposes **only**
|
||||
custom presets.
|
||||
|
||||
Our registry (`src/lib/presets.ts`) mirrors this exact list:
|
||||
|
||||
- **System presets**: 6 objects in `SYSTEM_ASSET_ALLOW_LIST` (`system-small|medium|large` ×
|
||||
`cover|contain`), each `format:'auto'`, resize inside `transforms[]`; sourced at runtime from
|
||||
`@directus/api` **if** importable, else the version-pinned copy `src/lib/system-presets.ts` (v11.17.4).
|
||||
- **Custom presets**: read `directus_settings.storage_asset_presets` each run; pass the verbatim object.
|
||||
- **Dynamic**: newly created presets are picked up per run; the new-preset trigger warms them for
|
||||
existing images.
|
||||
|
||||
## Admin UI (module "Image Previews")
|
||||
|
||||
- **Progress**: bar + `done / total / remaining`, throughput, ETA, last-run, next-window; live during a run.
|
||||
- **Storage breakdown**: bytes + file counts separately for **preview assets** (`__<hash>`), **original
|
||||
assets**, and the total — raw and human-readable. Computed by listing the storage root and `stat()`-ing
|
||||
each file (listing yields paths only, no sizes); the state dir is excluded.
|
||||
- **Config form** (persisted, no rebuild; defaults `enabled:true`, window `01:00–04:00`): `enabled`,
|
||||
maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`; each field carries a
|
||||
plain-language explanation. **Run now** ignores the window; **Cancel** is shown only while a run is active.
|
||||
- **Diagnostics panel**: recent operations, per-file generate/dry-run, resolve-plan viewer.
|
||||
|
||||
## Diagnostics & manual generation (observability)
|
||||
|
||||
- `POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` → generate (or dry-run) for ONE
|
||||
file; returns per `(key,format)` `{ ok, failed, cached, filename, bytes, ms, error }`. Primary dev tool.
|
||||
- `GET /image-previews/resolve-plan?fileId=&key=` → resolved transforms + **predicted cache filename** +
|
||||
whether it already exists, WITHOUT generating. The hash-match debugging tool.
|
||||
- `GET /image-previews/ops?limit=` → recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error).
|
||||
- `debug` setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
|
||||
- Everything except `/generate` is read-only.
|
||||
|
||||
## Safe deployment & staging (must not crash prod)
|
||||
|
||||
- **Fail-open, inert by default**: every entrypoint is tiny, bodies wrapped in try/catch, hooks never
|
||||
throw into Directus core, and ALL behavior is gated by `enabled` (default `false`). A loaded-but-disabled
|
||||
extension does nothing.
|
||||
- **Local dev**: `directus-extension link` (or the dev compose stack) loads `src/` live for iteration.
|
||||
- **Staging stack**: a second compose project (`dev/compose.dev.yml`) running the SAME Directus 11.17.4
|
||||
image with its own pg+redis, a **restored anonymized dump**, and a **copy of uploads**, on a separate
|
||||
port — realistic data, zero prod risk.
|
||||
- **Rollout discipline**: build → verify on staging → back up the previous `dist/` → copy in → healthcheck
|
||||
→ enable via config → rollback = restore previous `dist/` + disable.
|
||||
- `EXTENSIONS_AUTO_RELOAD` in prod reloads on file change; deploy with it controlled and accept the brief
|
||||
reload, or restart deliberately.
|
||||
|
||||
## Remote feedback & autonomous operation (security)
|
||||
|
||||
- Feedback channel = the extension's own **machine-readable endpoints** (`/status`, `/ops`,
|
||||
`/disk-usage`, `/resolve-plan`). Automation reads those, not CMS content.
|
||||
- **Least privilege**:
|
||||
- A Directus **admin token = full CMS control**. Do NOT give it to an autonomous loop. Use a scoped
|
||||
token reaching only `/image-previews/*` (+ `directus_files` metadata read).
|
||||
- A **Docker socket = root-equivalent**. Never expose the prod socket to automation; keep the loop on staging.
|
||||
- Deployment access = a dedicated **non-sudo** SSH user with a restricted key (`restrict`, `from=`,
|
||||
`command=`), or run opencode on the staging host via `handoff`.
|
||||
- **Network**: staging reachable only over a private network / VPN / tunnel — never public.
|
||||
- **Prompt-injection**: treat logs, filenames, and image metadata as untrusted input; keep tooling scoped.
|
||||
- **Promotion to prod is a human action**: the loop proposes, a human merges/deploys.
|
||||
- Guard long autonomous runs with a cost/time budget.
|
||||
|
||||
## Design
|
||||
|
||||
```
|
||||
src/
|
||||
hook/index.ts (api) files.upload -> warmer; settings/presets change -> warm existing
|
||||
endpoint/index.ts (api) routes: /config (GET/PUT), /status, /run, /run/cancel,
|
||||
/disk-usage, /generate, /resolve-plan, /ops
|
||||
module/
|
||||
index.ts (app) defineModule({ id:'image-previews', name, icon, routes })
|
||||
module.vue progress + disk usage + config form + diagnostics panel
|
||||
lib/
|
||||
presets.ts merged registry (system + settings), mirrors controller semantics
|
||||
system-presets.ts version-pinned SYSTEM_ASSET_ALLOW_LIST copy (v11.17.4)
|
||||
warmer.ts in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
|
||||
backfill.ts enumerate directus_files (image/*), window-aware, resumable
|
||||
state.ts state.jsonl read/append/resume
|
||||
settings.ts config file read/write + env defaults
|
||||
disk.ts sum sizes of __<hash> files (driver.list + stat)
|
||||
window.ts maintenance-window logic (is-now-in-window, next-window)
|
||||
diagnostics.ts resolve-plan/dry-run (resolved transforms + predicted filename), ops ring buffer
|
||||
dev/
|
||||
compose.dev.yml local Directus 11.17.4 + postgres + redis, extension linked
|
||||
seed/ anonymized DB dump + sample images for local/staging runs
|
||||
```
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No Directus upgrade (server stays on 11.17.4; license constraint).
|
||||
- No change to admin preset configuration or to how the admin builds URLs.
|
||||
- Never hand-write `__<hash>.<ext>` filenames.
|
||||
- No CDN/nginx caching work (separate ops concern). No image `quality` policy change.
|
||||
- Automation never gets prod Docker socket or a full prod admin token.
|
||||
|
||||
## Constraints / risks
|
||||
|
||||
- `ASSETS_TRANSFORM_MAX_CONCURRENT=45` per worker; over it errors. Catch per file/preset and continue
|
||||
(never fail the upload).
|
||||
- `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000 is a **pixel dimension** guard in Directus, not a
|
||||
file-size limit. The extension imposes no limit of its own; raise the Directus env var to process larger
|
||||
images (watch memory). Upload-size limits (`FILES_MAX_UPLOAD_SIZE`, `MAX_PAYLOAD_SIZE`) are unrelated.
|
||||
- Local storage write is **non-atomic** → warm only AFTER an upload is committed; serialize so the
|
||||
upload hook and the backfill never generate the same file concurrently.
|
||||
- PM2 cluster (8 workers): the upload hook fires on the handling worker; the backfill runs in the worker
|
||||
that served the run request. Shared state MUST be file/DB-backed (never in-memory).
|
||||
- In-process backfill competes with live API serving → the maintenance window + concurrency cap govern
|
||||
that. (HTTP rate limiter is irrelevant in-process.)
|
||||
- Observed server bottleneck (2026-10-23): `storage:local:responseTime 1206ms`, `pg:responseTime`
|
||||
150–738ms; no 503s, no sharp errors.
|
||||
|
||||
## Verification policy
|
||||
|
||||
- `npm run typecheck` → 0 errors; `npm test` (node:test via tsx) green.
|
||||
- **The hash-match gate (F09)**: for a real file, the on-disk filename produced by a browser/admin
|
||||
request for `?key=system-medium-cover` is byte-identical to the warmer's output for the same key+format
|
||||
(+ the resolved extension matches, e.g. `.avif`).
|
||||
- Cold-vs-warm timing; preview-file count + total bytes before/after a run.
|
||||
- All feature testing happens on the **staging** stack; prod deploy only at goal completion, via the
|
||||
documented backup/rollback procedure.
|
||||
|
||||
## Expected files
|
||||
|
||||
- `package.json`, `tsconfig.json`, `.gitignore`
|
||||
- `src/hook/`, `src/endpoint/`, `src/module/`, `src/lib/`, plus `src/**/*.test.ts`
|
||||
- `dev/compose.dev.yml`, `dev/seed/`
|
||||
- `.opencode/goal.md`, `.opencode/feature_list.json`, `.opencode/progress.txt`
|
||||
- `README.md`, `AGENTS.md`
|
||||
|
||||
## Resolved decisions (2026-10-25)
|
||||
|
||||
1. State/config: JSON files under `<STORAGE_LOCAL_ROOT>/.image-previews` (default
|
||||
`/directus/uploads/.image-previews`), override with `IMAGE_PREVIEWS_STATE_DIR`. The former
|
||||
`/directus/extensions/.image-previews` default was rejected — root-owned volume → `EACCES`.
|
||||
2. System presets: version-pinned copy (`src/lib/system-presets.ts`, verbatim v11.17.4); no
|
||||
`@directus/api` import needed for presets.
|
||||
3. New-preset trigger: manual `POST /run` (idempotent) + the module's Run button; no
|
||||
`settings.update` auto-trigger.
|
||||
4. Backfill scheduling: **in-process** via `schedule('*/15 * * * *')` in the hook — no external cron or
|
||||
token. It runs only inside the maintenance window and pauses when the window closes.
|
||||
5. Dev/staging: isolated local compose stack (`dev/compose.dev.yml`). A staging project is
|
||||
still to be created for realistic-data runs.
|
||||
6. Autonomous-loop grants: still to be defined (scoped token / restricted SSH, staging only).
|
||||
|
||||
## Implementation status (2026-10-25)
|
||||
|
||||
Implemented and verified against an isolated Directus 11.17.4: settings/state, presets + warmer,
|
||||
upload hook, diagnostics, backfill, endpoints. `npm test` 32/32, typecheck clean, build emits both
|
||||
bundles. **Hash-match gate passed** (a real browser-style asset request produced exactly the
|
||||
predicted filename). Features F03–F06, F08, F09 pass.
|
||||
|
||||
Outstanding: admin-module browser verification (F07), staging deploy/rollback + prod rollout (F02/F10),
|
||||
and the F01 publish-to-staging step.
|
||||
|
||||
## Reference: source anchors (v11.17.4)
|
||||
|
||||
- `api/src/services/assets.ts` — `getAsset`, `getAssetSuffix`, filename build, awaited `write`.
|
||||
- `api/src/utils/transformations.ts` — `resolvePreset`, `getFormat`.
|
||||
- `api/src/controllers/assets.ts` — inline `shortcuts = [...SYSTEM_ASSET_ALLOW_LIST, ...settings]`,
|
||||
`acceptFormat` selection from `req.headers.accept`.
|
||||
- `api/src/constants.ts` — `SYSTEM_ASSET_ALLOW_LIST` (6 presets, `format:'auto'`);
|
||||
`ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000.
|
||||
- `app/src/utils/get-asset-url.ts`, `app/src/layouts/cards/components/card.vue`,
|
||||
`app/src/interfaces/file-image/file-image.vue`, `app/src/displays/image/image.vue` — admin URLs:
|
||||
Files grid `?key=system-medium-cover&v=<modified_on>`; image field `system-large-cover|contain`;
|
||||
image display `system-small-cover`.
|
||||
- `packages/storage-driver-local/src/index.ts` — non-atomic `write`; `list()` returns paths only.
|
||||
- Prior art that does NOT work as-is: `directus-extension-auto-generate-file-transformations`
|
||||
(utomic-media) — still falls foul of the `auto`→format mismatch unless it forces the format.
|
||||
148
.opencode/progress.txt
Normal file
148
.opencode/progress.txt
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
# progress.txt — directus-extension-generate-image-previews
|
||||
|
||||
Append-only log of decisions, evidence, and learnings. Newest at the bottom.
|
||||
|
||||
## 2026-10-25 — project scaffolded
|
||||
|
||||
- Opened this standalone extension project to pre-generate Directus image previews.
|
||||
- Source-verified the strategy against Directus **v11.17.4** (see `.opencode/goal.md`).
|
||||
- Root causes why existing/community extensions do not work:
|
||||
1. `resolvePreset` never reads `key` → `{transformationParams:{key}}` yields `transforms=[]` → no-op.
|
||||
2. `system-*` presets are `format:'auto'` → `jpg`/`png` server-side vs `avif` in-browser → different hash.
|
||||
3. "undrained stream truncates file" — false (write is awaited).
|
||||
4. admin uses `?key=system-*`, not raw width/height.
|
||||
- Admin URLs (v11.17.4 app source): files grid `?key=system-medium-cover&v=<modified_on>`;
|
||||
image field `system-large-cover|contain`; image display `system-small-cover`.
|
||||
- Server bottleneck evidence (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
|
||||
`pg:responseTime` 150–738ms; no 503s, no sharp errors.
|
||||
|
||||
## 2026-10-25 — scope expanded after user direction (v2)
|
||||
|
||||
- **No token needed.** Dropped HTTP-replay as primary. Strategy: **in-process**
|
||||
`AssetsService.getAsset(id, { transformationParams: <full preset object>, acceptFormat: 'avif'|'webp' })`.
|
||||
No HTTP → no token, no proxy, no rate limiter. (HTTP-replay kept as documented fallback.)
|
||||
- **Bundle extension**: api `hook` + api `endpoint` + app `module`.
|
||||
- **Presets: all, dynamically** — system `SYSTEM_ASSET_ALLOW_LIST` (6) + `directus_settings.storage_asset_presets`,
|
||||
controller order `[...SYSTEM, ...custom]`, first-match-wins; new presets picked up per run.
|
||||
- **Admin module**: progress bar, disk usage (`list`+`stat`), config form, diagnostics panel.
|
||||
- **Config**: maintenance window + concurrency (throttle knob; in-process bypasses HTTP rate limiter),
|
||||
persisted in JSON.
|
||||
- Confirmed `acceptFormat:'avif'` in a direct call equals browser `Accept: image/avif`.
|
||||
|
||||
## 2026-10-25 — diagnostics + safety + autonomy (v3)
|
||||
|
||||
- **Diagnostics / manual generation** (F06): `POST /generate` (one file, dry-run supported),
|
||||
`GET /resolve-plan` (resolved transforms + predicted filename, no write), `GET /ops` (ring buffer).
|
||||
- **Safe testing** (F02): staging compose stack + deploy/rollback scripts; fail-open, `enabled` default false.
|
||||
- **Autonomous remote feedback**: staging-only, least privilege (scoped token, no prod Docker socket,
|
||||
restricted SSH, private network, human-only prod promotion).
|
||||
|
||||
## 2026-10-25 — implementation + local verification (autonomous session)
|
||||
|
||||
Built the full extension locally and smoke-tested against an **isolated** Directus 11.17.4 stack
|
||||
(`dev/compose.dev.yml`, port 8057, own pg + volumes). No contact with the prod instance.
|
||||
|
||||
### Implemented
|
||||
- `src/lib/`: settings (env + JSON), state (`state.jsonl`, run state, lock, cancel flag),
|
||||
presets (pinned system copy + custom merge, controller parity), warmer (in-process `getAsset` with
|
||||
`acceptFormat` variants, concurrency, per-task error isolation, stream release), window, storage
|
||||
(dynamic `getStorage` + local-FS fallback), diagnostics (resolve-plan prediction + ops ring buffer),
|
||||
backfill (paginated, resumable, window-aware, idempotent).
|
||||
- `src/endpoint/index.ts`: `/status`, `/config` (GET/PUT), `/presets`, `/run`, `/run/cancel`,
|
||||
`/disk-usage`, `/generate`, `/resolve-plan`, `/ops`; admin-only guard.
|
||||
- `src/hook/index.ts`: `files.upload` → fire-and-forget warm.
|
||||
- `src/module/`: admin module (progress bar, disk usage, config form, diagnostics panel, ops table).
|
||||
- `dev/`: compose stack, `sync-extension.sh`, seed images.
|
||||
|
||||
### Bugs found & fixed during smoke test
|
||||
1. **Directus coerces env values** (`"true"` → boolean `true`, `"1"` → number). My string-only parsers
|
||||
threw `e.toLowerCase is not a function`, which silently broke the upload hook. Fixed `settings.ts`
|
||||
to accept `unknown` and stringify. Regression test added.
|
||||
2. **State dir EACCES.** `/directus/extensions/.image-previews` (named volume on a non-existent path)
|
||||
is root-owned; the API runs as `node`. This broke config writes and the backfill lock. Fixed: default
|
||||
state dir is now `<STORAGE_LOCAL_ROOT>/.image-previews` (i.e. `/directus/uploads/.image-previews`),
|
||||
which is writable; `IMAGE_PREVIEWS_STATE_DIR` still overrides.
|
||||
3. **Bind-mount reload gotcha.** `rm -rf` of the bind-mounted extension dir breaks the Docker mount
|
||||
(Directus removes the extension and never re-adds it). `sync-extension.sh` now updates in place;
|
||||
restart the dev container after a sync.
|
||||
|
||||
### Verified evidence (local stack)
|
||||
- `npm run typecheck` → 0 errors. `npm test` → **32/32 pass**. `npm run build` → `dist/app.js` + `dist/api.js`.
|
||||
- Directus logs: `Loaded extensions: directus-extension-generate-image-previews` (no errors).
|
||||
- `/status` → `{enabled:true, inWindow:true, locked:false}`; `/presets` → 6 system presets.
|
||||
- Upload a JPEG → hook warmed **12 files** (6 presets × avif/webp) in `/directus/uploads`.
|
||||
- **HASH-MATCH GATE PASSED:** after deleting the cached files, a real
|
||||
`GET /assets/<id>?key=system-medium-cover` with `Accept: image/avif` produced exactly
|
||||
`<id>__f15aa4b815722867859d2a3fa46ab50834dfc8dc.avif` — byte-identical to `resolve-plan`'s prediction.
|
||||
- `/generate` dry-run reports `exists:true/false` correctly; `/ops` shows per-task timing.
|
||||
- Backfill: `total=36` (3 images × 6 presets × 2 formats), `generated=36`, state `state.jsonl` = 36 lines.
|
||||
Second run: `generated=0, hits=36`, state unchanged, file count unchanged → **idempotent/resumable**.
|
||||
- Window gating: `/run` outside `03:00–05:00` → `409 outside-window` with `nextWindow`.
|
||||
- `/disk-usage` → `157633 bytes / 36 files (source: storage)`.
|
||||
- Cold vs warm (`system-medium-cover`): 114 ms → 16–30 ms. (`system-large-cover`): 304 ms → 36 ms.
|
||||
|
||||
### Decisions recorded
|
||||
- System presets: version-pinned copy in `src/lib/system-presets.ts` (verbatim from `api/src/constants.ts`
|
||||
v11.17.4). No `@directus/api` import needed for presets.
|
||||
- Storage/disk usage: `@directus/api/storage/index` loaded via dynamic import (hidden from the bundler),
|
||||
with a local-FS fallback; shares the host `getStorage()` singleton.
|
||||
- New-preset trigger: **manual** via `POST /run` (idempotent) + the module's Run button; no
|
||||
`settings.update` auto-trigger (avoids heavy background runs on unrelated settings changes).
|
||||
- Scheduling: external cron → `POST /image-previews/run` (default).
|
||||
- State/config dir: `<STORAGE_LOCAL_ROOT>/.image-previews` (writable), override with `IMAGE_PREVIEWS_STATE_DIR`.
|
||||
|
||||
### Additional verification (same session)
|
||||
- `GET /extensions` → bundle registered with entries `module`, `hook`, `endpoint`, all enabled.
|
||||
- Inert test: with `enabled:false`, uploading an image produced **zero** new preview files (25 → 25).
|
||||
- Added `dev/compose.staging.yml`, `dev/deploy-staging.sh`, `dev/rollback-staging.sh` (templates;
|
||||
NOT executed against any remote host).
|
||||
- Added `storage.test.ts`; total tests now **34/34**.
|
||||
- Restarted the dev stack after the final build; `/status`, `/disk-usage`, and bundle entries all good.
|
||||
|
||||
### Still open
|
||||
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
|
||||
- Cancel + concurrent-run 409 not exercised end-to-end (unit-tested only; local runs finish too fast).
|
||||
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
|
||||
- F01/F02/F07/F10 remain `passes:false` (F01 publish step, F02 staging run, F07 browser, F10 rollout).
|
||||
|
||||
## 2026-10-25 — feedback round (UI, defaults, custom formats, docs)
|
||||
|
||||
- **Defaults changed**: `enabled` now `true`; maintenance window defaults to `01:00–04:00`.
|
||||
- **Custom fixed-format presets verified**: `custom-png` (format `png`) generated `.png` and
|
||||
`resolve-plan` predicted the exact filename (`exists:true`). `custom-auto` generated avif+webp.
|
||||
An unsupported preset (`format: bmp`) failed with Directus's clear error
|
||||
("Expected one of: heic, heif, avif, jpeg, … received bmp") and is surfaced in Recent operations / `/ops`.
|
||||
- **Effective format fix**: ops/presets now report the preset's real output format (`png`) instead of
|
||||
`null`; backfill resume keys use the effective format (fixed a re-warm mismatch).
|
||||
- **Progress accounting fix**: already-cached tasks now count toward `done`/`hits`. Runs report
|
||||
`128/128` (120 hits + 8 expected failures) instead of `8/128`. After removing the unsupported preset:
|
||||
`120/120 hits, 0 failed, 0 generated`.
|
||||
- **Storage reporting**: `/disk-usage` + panel now split **previews** vs **originals** vs **total**
|
||||
(raw + human): e.g. previews 774.7 KB / 115 files, originals 24.9 MB / 9 files, total 25.6 MB / 124.
|
||||
- **Human-readable fields** added to `/status` (ISO + human dates, `etaHuman`, `rateHuman`),
|
||||
`/disk-usage` (`bytesHuman`), `/ops` (`tsHuman`, `msHuman`), `/generate` (`msHuman`).
|
||||
- **Admin module rewritten with Directus UI kit** (`v-button`, `v-input`, `v-checkbox`, `v-select`,
|
||||
`v-notice`, `v-progress-linear`), field explanations, preset table with effective formats + shadow
|
||||
warnings, storage table, error column in Recent operations, and a full collapsible **API reference**.
|
||||
- **Run now** now ignores the maintenance window; **Cancel** only renders while a run is active.
|
||||
- **Dimension vs file size clarified** in README/goal/AGENTS: the 6000 limit is a pixel-dimension guard in
|
||||
Directus, not a file-size limit; the extension imposes none.
|
||||
- Tests: **40/40** pass; typecheck clean; build OK.
|
||||
|
||||
### Still open
|
||||
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
|
||||
- Cancel + concurrent-run 409 not exercised end-to-end.
|
||||
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
|
||||
- F01/F02/F07/F10 remain `passes:false`.
|
||||
|
||||
## 2026-10-25 — production deploy + accuracy fixes + in-process scheduler
|
||||
|
||||
- Deployed to prod via local build + SCP archive (prod has no Node). A source clone is kept outside the
|
||||
watched extensions directory.
|
||||
- Verified: bundle loaded on all 4 PM2 workers, endpoints mounted (403 admin guard), no errors.
|
||||
- **Backfill counting fix**: pre-scan storage filenames; each task's predicted cache filename is checked
|
||||
before generating. Existing previews count as `already cached` and are skipped. (Commit 18db52a.)
|
||||
- **In-process nightly scheduler**: hook registers `schedule('*/15 * * * *')`; runs only inside the
|
||||
maintenance window, no external cron/token, pauses when the window closes and resumes next night.
|
||||
- UI now shows last-run errors and `paused (window closed)`.
|
||||
- Tests 43/43, typecheck clean.
|
||||
Loading…
Add table
Add a link
Reference in a new issue