# directus-extension-generate-image-previews

Pre-generate Directus image transform ("preview") cache files so the admin UI and API never pay the
first-request sharp cost.

A Directus **bundle**:

- **upload hook** — warm previews right after an image upload.
- **new-preset trigger** — `Run now` warms every preset (system + custom) for existing images.
- **backfill job** — resumable, window-aware sweep over the existing library.
- **admin module** — progress, storage breakdown, configuration, diagnostics, and an API reference.

> **AI-generated.** This project was produced by an autonomous AI coding agent and reviewed by a human
> maintainer. Review and test it before relying on it in production.

## Requirements

- Directus **11.x** (tested on **11.17.4**).
- Node 18+.
- A **non-sandboxed** local extension (it uses Directus's internal services at runtime — see
  Compatibility).
- Local disk storage is supported out of the box; other storage drivers work for disk reporting through
  the Directus storage manager.

## Why this approach

The cache file is `<filename_disk>__<object-hash(resolved transforms)>.<ext>`. The hash depends on the
RESOLVED transforms, and every `system-*` preset is `format:'auto'` — which resolves differently
server-side (jpg/png) than in a browser (avif). This extension therefore generates **in-process** with an
explicit `acceptFormat` (`avif` + `webp`), reproducing the browser's cache file byte-for-byte without any
HTTP request. Naive warming (and several community extensions) silently misses because of this.

## Install

```bash
git clone https://code.at.locus.im/Rawman/directus-extension-generate-image-previews.git
cd directus-extension-generate-image-previews
npm install
npm run build
```

Copy the build into your Directus extensions directory:

```bash
cp -r dist package.json /path/to/directus/extensions/directus-extension-generate-image-previews/
```

Restart Directus (or let `EXTENSIONS_AUTO_RELOAD` pick it up). The **Image Previews** module appears in
the admin sidebar.

## Development

```bash
npm run build        # emits dist/app.js + dist/api.js
npm run typecheck    # 0 errors
npm test             # unit tests
npm run dev          # watch build
npm run link         # link into a local Directus extensions dir
```

An isolated Directus 11.17.4 stack for smoke testing lives in `dev/` — see `dev/README.md`.

## Settings

Defaults are applied automatically; everything is editable at runtime from the admin module — no rebuild.

| setting                    | default                                | meaning                                                          |
| -------------------------- | -------------------------------------- | ---------------------------------------------------------------- |
| `enabled`                    | `true`                                   | master switch (off = hooks/backfill inert)                       |
| `onUpload`                   | `true`                                   | generate previews immediately after an upload                    |
| `windowStart` / `windowEnd`  | `01:00` / `04:00`                          | scheduled runs only start inside this window (local time)        |
| `concurrency`                | `1`                                      | parallel generations                                             |
| `formats`                    | `avif,webp`                              | output variants **for `format:'auto'` presets only** (all system presets) |
| `presetKeys`                 | `null` (all)                             | restrict to specific preset keys                                 |
| `debug`                      | `false`                                  | verbose generation logs in the Directus server log               |
| `IMAGE_PREVIEWS_STATE_DIR`   | `<STORAGE_LOCAL_ROOT>/.image-previews`   | where settings + progress state live                             |

### Presets and formats

- **System presets** (`system-small|medium|large` × `cover|contain`) always use `format:'auto'`, so they
  generate one file per entry in `formats` (default avif + webp), at their built-in sizes (64 / 300 / 800 px).
- **Custom presets** from Settings → File Storage are added automatically. A custom preset with a fixed
  `format` (e.g. `png`) ignores `formats` and generates exactly that format. A custom preset with
  `format:'auto'` follows `formats`.
- A custom preset whose `key` equals a system key is **shadowed** (system wins, same as Directus) and is
  flagged in the admin module.
- Unsupported format strings (e.g. `bmp`) fail at generation; the error is shown in Recent operations
  and `GET /image-previews/ops`.

## Storage reporting

`GET /image-previews/disk-usage` (and the admin panel) reports bytes + file counts separately for
**preview assets** and **original assets**, plus the total — in both raw bytes and human-readable form.

## API endpoints

All require a Directus admin session or admin token. Human-readable fields accompany machine-readable ones.

- `GET /image-previews/status` — run progress, rate, ETA, window state, ISO + human dates
- `GET /image-previews/config`, `PUT /image-previews/config`
- `GET /image-previews/presets` — system + custom presets, effective formats, shadowed keys
- `POST /image-previews/run` — start backfill (`{"ignoreWindow":true}` for an immediate run)
- `POST /image-previews/run/cancel`
- `GET /image-previews/disk-usage` — previews vs originals vs total
- `POST /image-previews/generate` — one file (`{fileId, keys?, formats?, dryRun?}`)
- `GET /image-previews/resolve-plan?fileId=&key=` — predicted cache filename, no write
- `GET /image-previews/ops?limit=` — recent operations with timing + error messages
- `GET /image-previews/failures?status=&limit=&offset=` — persisted failures with file metadata, attempts,
  timestamps, next retry and error (`status` = `failed`, `recovered`, or `retry`; limit max 200)

The admin module includes a full API reference with curl examples.

## Scheduling

The extension schedules itself **inside Directus** — no external cron, no token:

- every 15 minutes it checks whether it is inside the maintenance window;
- if enabled, inside the window, and no run is active, it starts/resumes the backfill;
- the backfill **pauses when the window closes** and resumes the next night (it is resumable);
- once all tasks are done, it stays idle until the next day;
- before starting the backfill it runs any due failure retries (targeted pairs only, bounded batch).

`Run now` in the admin module ignores the window.

## Failures and retries

Every failed `(file, preset, format)` task from the upload hook, `POST /generate`, and the backfill is
appended to `<state-dir>/failures.jsonl`. Replay derives the latest state per task, so details survive
restarts and PM2 workers. The journal is append-only; it is compacted under `run.lock` to keep unresolved
tasks plus the last 1,000 recovered ones.

Retry policy: max 3 attempts (initial + 2 retries), backoff 5 minutes then 15 minutes. Due retries run
inside the maintenance window only when no backfill is active, under the same `run.lock`, and process only
the targeted file/preset/format pairs (default batch 50). A success appends the normal `state.jsonl`
completion and marks the task `recovered`; an exhausted task stays `failed` with its last error and is not
retried again. Failures are logged at warn with ids/metadata only; recoveries at info.

## Safe testing & deployment

Never test against a production instance.

- **Local / staging** (`dev/compose.dev.yml`): Directus 11.17.4 + its own postgres + volumes, on separate
  ports. Shares nothing with production. See `dev/README.md`.
- **Fail-open**: entrypoints never throw into Directus core.
- **Rollout**: build → verify on staging → back up the previous `dist/` → deploy → healthcheck → enable.
  **Rollback**: restore the previous `dist/`.
- Helpers: `dev/deploy-staging.sh`, `dev/rollback-staging.sh`, template `dev/compose.staging.yml`.

## Caveats

- **Transform dimension guard, not a file-size limit.** Directus rejects transformations of images whose
  width or height exceeds `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` (default **6000 px**). This is a Directus
  env var, not something this extension imposes. To process larger images, raise it on the Directus
  instance (and watch memory). Upload size limits (`FILES_MAX_UPLOAD_SIZE`, `MAX_PAYLOAD_SIZE`) are separate.
- Warm only after uploads are committed (local storage writes are non-atomic).
- In-process generation bypasses the HTTP rate limiter; bound it with `concurrency` + the window.
- PM2 cluster: shared state is file-backed, so all workers see the same run state.

## Compatibility

Tested against Directus **11.17.4**. The extension reads Directus's own transform pipeline and storage
manager (`@directus/api/utils/transformations`, `@directus/api/storage/index`) via runtime imports; these
are **not a public API** and may change between Directus versions. Re-verify after a Directus upgrade,
especially the hash-match gate (`resolve-plan` vs a real browser asset request).

The extension is not sandboxed: Directus's sandbox forbids importing anything except `directus:api`, which
this extension needs for storage/transform access.

## License

MIT — see [LICENSE](LICENSE).