directus-extension-generate-image-previews/README.md

directus-extension-generate-image-previews

Pre-generate Directus image transform ("preview") cache files so the admin UI and API never pay the first-request sharp cost.

A Directus bundle:

• upload hook — warm previews right after an image upload.
• new-preset trigger — Run now warms every preset (system + custom) for existing images.
• backfill job — resumable, window-aware sweep over the existing library.
• admin module — progress, storage breakdown, configuration, diagnostics, and an API reference.

AI-generated. This project was produced by an autonomous AI coding agent and reviewed by a human maintainer. Review and test it before relying on it in production.

Requirements

• Directus 11.x (tested on 11.17.4).
• Node 18+.
• A non-sandboxed local extension (it uses Directus's internal services at runtime — see Compatibility).
• Local disk storage is supported out of the box; other storage drivers work for disk reporting through the Directus storage manager.

Why this approach

The cache file is <filename_disk>__<object-hash(resolved transforms)>.<ext>. The hash depends on the RESOLVED transforms, and every system-* preset is format:'auto' — which resolves differently server-side (jpg/png) than in a browser (avif). This extension therefore generates in-process with an explicit acceptFormat (avif + webp), reproducing the browser's cache file byte-for-byte without any HTTP request. Naive warming (and several community extensions) silently misses because of this.

Install

git clone https://code.at.locus.im/Rawman/directus-extension-generate-image-previews.git
cd directus-extension-generate-image-previews
npm install
npm run build

Copy the build into your Directus extensions directory:

cp -r dist package.json /path/to/directus/extensions/directus-extension-generate-image-previews/

Restart Directus (or let EXTENSIONS_AUTO_RELOAD pick it up). The Image Previews module appears in the admin sidebar.

Development

npm run build        # emits dist/app.js + dist/api.js
npm run typecheck    # 0 errors
npm test             # unit tests
npm run dev          # watch build
npm run link         # link into a local Directus extensions dir

An isolated Directus 11.17.4 stack for smoke testing lives in dev/ — see dev/README.md.

Settings

Defaults are applied automatically; everything is editable at runtime from the admin module — no rebuild.

setting	default	meaning
enabled	true	master switch (off = hooks/backfill inert)
onUpload	true	generate previews immediately after an upload
windowStart / windowEnd	01:00 / 04:00	scheduled runs only start inside this window (local time)
concurrency	1	parallel generations
formats	avif,webp	output variants for format:'auto' presets only (all system presets)
presetKeys	null (all)	restrict to specific preset keys
debug	false	verbose generation logs in the Directus server log
IMAGE_PREVIEWS_STATE_DIR	<STORAGE_LOCAL_ROOT>/.image-previews	where settings + progress state live

Presets and formats

• System presets (system-small|medium|large × cover|contain) always use format:'auto', so they generate one file per entry in formats (default avif + webp), at their built-in sizes (64 / 300 / 800 px).
• Custom presets from Settings → File Storage are added automatically. A custom preset with a fixed format (e.g. png) ignores formats and generates exactly that format. A custom preset with format:'auto' follows formats.
• A custom preset whose key equals a system key is shadowed (system wins, same as Directus) and is flagged in the admin module.
• Unsupported format strings (e.g. bmp) fail at generation; the error is shown in Recent operations and GET /image-previews/ops.

Storage reporting

GET /image-previews/disk-usage (and the admin panel) reports bytes + file counts separately for preview assets and original assets, plus the total — in both raw bytes and human-readable form.

API endpoints

All require a Directus admin session or admin token. Human-readable fields accompany machine-readable ones.

• GET /image-previews/status — run progress, rate, ETA, window state, ISO + human dates
• GET /image-previews/config, PUT /image-previews/config
• GET /image-previews/presets — system + custom presets, effective formats, shadowed keys
• POST /image-previews/run — start backfill ({"ignoreWindow":true} for an immediate run)
• POST /image-previews/run/cancel
• GET /image-previews/disk-usage — previews vs originals vs total
• POST /image-previews/generate — one file ({fileId, keys?, formats?, dryRun?})
• GET /image-previews/resolve-plan?fileId=&key= — predicted cache filename, no write
• GET /image-previews/ops?limit= — recent operations with timing + error messages
• GET /image-previews/failures?status=&limit=&offset= — persisted failures with file metadata, attempts, timestamps, next retry and error (status = failed, recovered, or retry; limit max 200)

The admin module includes a full API reference with curl examples.

Scheduling

The extension schedules itself inside Directus — no external cron, no token:

• every 15 minutes it checks whether it is inside the maintenance window;
• if enabled, inside the window, and no run is active, it starts/resumes the backfill;
• the backfill pauses when the window closes and resumes the next night (it is resumable);
• once all tasks are done, it stays idle until the next day;
• before starting the backfill it runs any due failure retries (targeted pairs only, bounded batch).

Run now in the admin module ignores the window.

Failures and retries

Every failed (file, preset, format) task from the upload hook, POST /generate, and the backfill is appended to <state-dir>/failures.jsonl. Replay derives the latest state per task, so details survive restarts and PM2 workers. The journal is append-only; it is compacted under run.lock to keep unresolved tasks plus the last 1,000 recovered ones.

Retry policy: max 3 attempts (initial + 2 retries), backoff 5 minutes then 15 minutes. Due retries run inside the maintenance window only when no backfill is active, under the same run.lock, and process only the targeted file/preset/format pairs (default batch 50). A success appends the normal state.jsonl completion and marks the task recovered; an exhausted task stays failed with its last error and is not retried again. Failures are logged at warn with ids/metadata only; recoveries at info.

Safe testing & deployment

Never test against a production instance.

• Local / staging (dev/compose.dev.yml): Directus 11.17.4 + its own postgres + volumes, on separate ports. Shares nothing with production. See dev/README.md.
• Fail-open: entrypoints never throw into Directus core.
• Rollout: build → verify on staging → back up the previous dist/ → deploy → healthcheck → enable. Rollback: restore the previous dist/.
• Helpers: dev/deploy-staging.sh, dev/rollback-staging.sh, template dev/compose.staging.yml.

Caveats

• Transform dimension guard, not a file-size limit. Directus rejects transformations of images whose width or height exceeds ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION (default 6000 px). This is a Directus env var, not something this extension imposes. To process larger images, raise it on the Directus instance (and watch memory). Upload size limits (FILES_MAX_UPLOAD_SIZE, MAX_PAYLOAD_SIZE) are separate.
• Warm only after uploads are committed (local storage writes are non-atomic).
• In-process generation bypasses the HTTP rate limiter; bound it with concurrency + the window.
• PM2 cluster: shared state is file-backed, so all workers see the same run state.

Compatibility

Tested against Directus 11.17.4. The extension reads Directus's own transform pipeline and storage manager (@directus/api/utils/transformations, @directus/api/storage/index) via runtime imports; these are not a public API and may change between Directus versions. Re-verify after a Directus upgrade, especially the hash-match gate (resolve-plan vs a real browser asset request).

The extension is not sandboxed: Directus's sandbox forbids importing anything except directus:api, which this extension needs for storage/transform access.

License

MIT — see LICENSE.