Rawman Grey
c8f8bbc5ac
Bundle extension (upload hook + API endpoint + admin module) that warms Directus image transform cache in-process with an explicit acceptFormat, so the generated file matches the browser's cache key byte-for-byte. - resumable, window-aware backfill with progress + cancel - system + custom presets, effective-format reporting, shadow warnings - storage breakdown (previews vs originals), human-readable API fields - diagnostics: resolve-plan prediction, per-file generate, ops ring buffer - isolated dev/staging compose stack + deploy/rollback scripts Verified on Directus 11.17.4: 40/40 tests, hash-match gate passed, idempotent backfill, custom fixed-format presets, error surfacing.
129 lines
7.1 KiB
Markdown
129 lines
7.1 KiB
Markdown
# Agent notes — directus-extension-generate-image-previews
|
||
|
||
Common confusion points for agents working here. Read before touching the warmer.
|
||
|
||
## What this is
|
||
|
||
A Directus **bundle** extension that pre-generates image transform ("preview") cache files so the
|
||
admin UI never pays the first-request sharp cost. Target server: **Directus 11.17.4** on `land-nest`
|
||
(self-hosted, PM2 cluster, `STORAGE_LOCAL_ROOT=./uploads`).
|
||
|
||
Bundle entries: an api **hook** (`files.upload`), an api **endpoint** (`/image-previews/*`), and an
|
||
app **module** (`image-previews` admin page: progress, disk usage, config).
|
||
|
||
## Directus transform invariants (source-verified, tag v11.17.4) — DO NOT REGRESS
|
||
|
||
1. **Cached transform filename** = `<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>`,
|
||
written **beside the original** in the storage root (`./uploads`, no `.cache` subdir).
|
||
|
||
2. **Hash = `object-hash@3` defaults**: sha1/hex, `unorderedArrays:false` (**array order matters**),
|
||
`unorderedObjects:true`. Hashes the **RESOLVED** transform array from `resolvePreset`, NOT the raw
|
||
query/preset object.
|
||
|
||
3. **`resolvePreset` order is fixed**: copy `transformationParams.transforms[]` → push `toFormat`
|
||
(if `format`/`quality`) → push `resize` (+`extract` on focal cover). Raw params `?width=&height=&fit=&format=`
|
||
resolve to `[toFormat, resize]`; a preset resolves to `[resize, toFormat]` → **different hash for a
|
||
visually identical result**.
|
||
|
||
4. **`{ key }` alone is a NO-OP.** `resolvePreset` never reads `key`. Always pass the **full preset object**.
|
||
|
||
5. **`format:'auto'` is the killer trap.** All six `system-*` presets are `format:'auto'`. `getFormat`
|
||
with no `acceptFormat` → JPEG→`jpg`, PNG→`png`; a browser `Accept: image/avif` → `avif`. Different
|
||
`toFormat` arg → **different hash → guaranteed miss**.
|
||
|
||
6. **The write IS awaited.** Not draining the returned stream leaks an FD but never truncates.
|
||
Still: destroy/unpipe the returned stream to avoid FD leaks.
|
||
|
||
7. `AssetsService.getAsset` runs fine as sudo (`new AssetsService({ schema })`) with **no HTTP request
|
||
context** — this is the basis of our in-process strategy.
|
||
|
||
## Preset registry semantics (controller parity)
|
||
|
||
The controller builds, inline in `api/src/controllers/assets.ts`:
|
||
|
||
```
|
||
res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]
|
||
```
|
||
|
||
and resolves `?key=` with `.find(t => t.key === key)` → **first match wins** (a system key shadows a
|
||
custom preset of the same key). There is **no** reusable helper, and `GET /settings` exposes **only**
|
||
custom presets. Our registry (`src/lib/presets.ts`) must mirror this order exactly.
|
||
|
||
## What the admin actually requests (verbatim from app source)
|
||
|
||
| Surface | URL |
|
||
| ------------- | -------------------------------------------------------- |
|
||
| Files grid | `?key=system-medium-cover&v=<modified_on>` (cards, crop) |
|
||
| Image field | `?key=system-large-cover` / `system-large-contain` |
|
||
| Image display | `?key=system-small-cover` |
|
||
|
||
No explicit `width`/`height`/`format`. `format:auto` is implicit. `v=` is ignored for hashing.
|
||
|
||
## Chosen strategy — IN-PROCESS (no token, no HTTP, no rate limiter)
|
||
|
||
```ts
|
||
const assets = new services.AssetsService({ schema }); // sudo
|
||
const { stream } = await assets.getAsset(fileId, {
|
||
transformationParams: presetObject, // FULL object, not { key }
|
||
acceptFormat: 'avif', // or 'webp'
|
||
});
|
||
(stream as Readable).destroy?.();
|
||
```
|
||
|
||
`acceptFormat` reproduces what a browser's `Accept` header does → byte-identical cache filename with
|
||
no HTTP/token. Warm **both** `avif` and `webp` for `format:'auto'` presets.
|
||
|
||
HTTP-replay (`GET /assets/{id}?key=…`) is the documented **fallback** only (needs a token + rate-limit
|
||
pacing).
|
||
|
||
## Config + state
|
||
|
||
- Persisted settings + progress state are small JSON files under `<STORAGE_LOCAL_ROOT>/.image-previews`
|
||
(default `/directus/uploads/.image-previews`; override `IMAGE_PREVIEWS_STATE_DIR`). Do NOT default to
|
||
`/directus/extensions/...` — that path is a root-owned volume and the API runs as `node` → EACCES.
|
||
- `state.jsonl` = append-only completed `{file_id,preset_key,format}` → resume.
|
||
- All state is **file-backed** (PM2 cluster, 8 workers — never in-memory).
|
||
|
||
## Gotchas learned the hard way
|
||
|
||
- **Directus coerces env values**: `"true"` arrives as boolean `true`, `"1"` as number. Env parsing must
|
||
accept `unknown` and stringify (see `asString` in `settings.ts`). A string-only `.toLowerCase()` call
|
||
silently killed the upload hook.
|
||
- **Docker bind mounts**: never `rm -rf` the bind-mounted extension directory — it breaks the mount and
|
||
Directus removes the extension without re-adding it. Sync contents in place; restart the dev container.
|
||
- The dev stack (`dev/compose.dev.yml`) is the only place to test; never load this into prod while
|
||
iterating (prod has `EXTENSIONS_AUTO_RELOAD=true`).
|
||
|
||
|
||
## Server facts (land-nest)
|
||
|
||
- Directus image `directus/directus:11.17.4`; `PUBLIC_URL=https://data.landshaft.life`;
|
||
`PM2_INSTANCES=8`; `STORAGE_LOCAL_ROOT=./uploads`; logs at `~/apps/directus/logs`.
|
||
- `ASSETS_TRANSFORM_MAX_CONCURRENT=45` per worker → over it errors; catch per file/preset.
|
||
- `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000 is a **pixel dimension** guard, not a file-size
|
||
limit. The extension imposes no limit; raise the Directus env var to process larger images.
|
||
- Local storage writes are **non-atomic** — warm only after an upload is committed.
|
||
- `RATE_LIMITER_*` applies to HTTP only → irrelevant to in-process generation (this is why the admin
|
||
knob is *concurrency*, not a rate limit).
|
||
- Observed bottleneck (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
|
||
`pg:responseTime` 150–738ms; **no** 503s, **no** sharp errors. Pre-generation removes the
|
||
cold-transform cost; uploads-volume I/O latency is a separate ops concern.
|
||
|
||
## Safety (do not crash prod)
|
||
|
||
- Every entrypoint body is wrapped so thrown errors are logged and swallowed — never propagate into
|
||
Directus core. ALL behaviour is gated by `enabled` (default **false**): a loaded-but-disabled bundle
|
||
does nothing.
|
||
- Test only on the **staging** stack (`dev/compose.dev.yml`), never the prod instance.
|
||
- Rollout = backup previous `dist/` → deploy → healthcheck → enable; rollback = restore `dist/` + disable.
|
||
- Automation (autonomous loops) gets **staging only**: a scoped token limited to `/image-previews/*`
|
||
(+ `directus_files` metadata read), never a prod admin token; never the prod Docker socket; restricted
|
||
non-sudo SSH or opencode-on-host. Promotion to prod is a human action.
|
||
|
||
## Conventions
|
||
|
||
- Build via `@directus/extensions-sdk` (`npm run build` / `dev` / `link`); bundle emits `dist/app.js` + `dist/api.js`.
|
||
- Deploy with `npm run publish` (scp `dist` + `package.json` to land-nest extensions dir).
|
||
- Tests: `node:test` via `tsx` (`npm test`). Typecheck: `npm run typecheck`.
|
||
- Do not add comments unless asked. Follow the sibling extensions' style.
|
||
- Never commit without explicit request.
|