Rawman/directus-extension-generate-image-previews

Rawman Grey 871182d6d5 feat: persist failure journal and bounded retries (F11)

2026-10-30 10:22:29 +03:00

6.7 KiB

Raw Permalink Blame History

name	description
directus-image-previews	Work on the directus-extension-generate-image-previews bundle (Directus 11 image transform pre-generation). Use when changing the warmer, backfill, scheduler, admin module, endpoints, presets, deployment, or when debugging preview generation in this repo.

directus-image-previews — working rules

Read this before changing the extension. It encodes hard-won invariants; breaking them silently produces cache misses or production incidents. For general Directus-extension patterns, see the global directus-extensions skill.

Scope

A Directus bundle (tested on 11.17.4, manifest host: ^11.0.0) that pre-generates image transform cache files:

src/hook/ — files.upload warmer + in-process nightly schedule('*/15 * * * *').
src/endpoint/ — /image-previews/* admin API.
src/module/ — admin page (progress, storage, config, diagnostics, API reference).
src/lib/ — settings, state, presets, warmer, backfill, diagnostics, storage, window, format.
dev/ — isolated Directus 11.17.4 stack + deploy/rollback scripts.

Invariants — do not regress

Cache filename = <basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>, beside the original in the storage root.
Hash = object-hash@3 defaults (sha1/hex, unorderedArrays:false → array order matters, unorderedObjects:true), over the resolved transform array.
resolvePreset order: transforms[] → toFormat (if format/quality) → resize (+extract).
Never pass { key } alone — resolvePreset ignores key. Always pass the full preset object.
format:'auto' needs an explicit acceptFormat (avif/webp); otherwise server-side resolves to jpg/png and the hash won't match the browser.
Preset registry mirrors the controller: [...SYSTEM_PRESETS, ...storage_asset_presets], first match wins. GET /settings returns only custom presets.
AssetsService.getAsset awaits the write; destroy/unpipe the returned stream to free the FD.
Local storage writes are not atomic — never generate the same file from two workers concurrently.
State dir is <STORAGE_LOCAL_ROOT>/.image-previews (writable). Do not use /directus/extensions/... (root-owned → EACCES).
Env values from the Directus context are coerced (booleans/numbers). Stringify before parsing.
All authoritative state is file-backed (PM2 cluster). Never in-memory.
Hooks never throw into Directus core; all behavior is gated by enabled.

Presets / formats semantics

System presets are all format:'auto' → one file per configured formats (default avif,webp).
Custom preset with a fixed format → exactly that format, formats is ignored.
Custom preset with auto → follows formats.
Custom preset with no format/quality → resize-only (original extension, still hashed).
A custom key equal to a system key is shadowed (system wins) and flagged by /presets.

Backfill / scheduler semantics

Run now → POST /run with ignoreWindow:true; runs until cancelled.
Nightly → schedule('*/15 * * * *'): only inside the maintenance window, no active run, and not already completed today. It starts/resumes the backfill.
The backfill pauses when the window closes (run.paused=true) and resumes next night.
Pre-scans storage filenames and counts existing preview files as already cached (skips generation).
total = image files × tasksPerFile; done counts cached + generated + failed.
One run at a time (in-process guard + run.lock).

State files (on the Directus host/container)

<STORAGE_LOCAL_ROOT>/.image-previews/
  settings.json   effective config
  run.json        progress (running/done/total/generated/hits/failed/rate/eta/paused/cancelled/error)
  state.jsonl     completed {file_id,preset_key,format} → resume
  failures.jsonl  append-only failure events; replay = latest per task; compacted under run.lock
  run.lock        cluster-wide run lock
  cancel.flag     cancellation request

Debugging

GET /image-previews/status — progress, window, paused, last error.
GET /image-previews/presets — effective formats, shadowed keys.
GET /image-previews/resolve-plan?fileId=&key= — predicted cache filename, no write. Use this first.
POST /image-previews/generate {fileId,keys?,formats?,dryRun?} — one file, per-task results.
GET /image-previews/ops?limit= — recent operations with errors (e.g. unsupported preset format).
Directus log (file transport): Added extensions, Extensions reloaded, image-previews: ….
Common failure: custom preset format not supported by sharp (e.g. bmp) → error in /ops; fix the preset.

Verification gate

Before shipping a change that touches generation, prove:

resolve-plan filename for ?key=system-medium-cover is byte-identical to the file produced by a real browser request with Accept: image/avif.
npm test green, npm run typecheck clean, npm run build emits both bundles.
Isolated dev/ stack: upload → 12 files (6 presets × 2 formats), backfill idempotent, pause/cancel works.

Build / deploy

Never build on the production host (no Node; npm is unreliable there).
Build locally: npm ci && npm run build && npm test.
Package dist/ + package.json (+ README/LICENSE) into a tarball; SCP it.
Install: copy dist first, package.json last; back up the previous dist.
EXTENSIONS_AUTO_RELOAD=true in prod — never copy a half-built extension into the watched dir. If a reload looks ambiguous, restart the container.
Rollback: restore the backed-up dist or remove the extension directory.
Public repo: git@code-at-locus.im:Rawman/directus-extension-generate-image-previews.git, branch master. .opencode/ internal notes are untracked; only .opencode/skills/ is tracked.

Checklist for any change

Invariants above untouched (or explicitly justified).
Tests added/updated; 43+ green; typecheck clean.
resolve-plan hash-match still proven.
No new in-memory state; state dir still writable.
Endpoint stays admin-guarded; no secrets/file contents logged.
Deploy payload is dist + package.json; backup taken; reload verified.
README/AGENTS updated if behavior or settings changed.

6.7 KiB Raw Permalink Blame History Unescape Escape