feat: pre-generate Directus image previews
Bundle extension (upload hook + API endpoint + admin module) that warms Directus image transform cache in-process with an explicit acceptFormat, so the generated file matches the browser's cache key byte-for-byte. - resumable, window-aware backfill with progress + cancel - system + custom presets, effective-format reporting, shadow warnings - storage breakdown (previews vs originals), human-readable API fields - diagnostics: resolve-plan prediction, per-file generate, ops ring buffer - isolated dev/staging compose stack + deploy/rollback scripts Verified on Directus 11.17.4: 40/40 tests, hash-match gate passed, idempotent backfill, custom fixed-format presets, error surfacing.
This commit is contained in:
commit
c8f8bbc5ac
42 changed files with 10005 additions and 0 deletions
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
.DS_Store
|
||||||
|
node_modules
|
||||||
|
dist
|
||||||
|
dev/extension
|
||||||
|
dev/.smoke.env
|
||||||
1
.opencode/.gitignore
vendored
Normal file
1
.opencode/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
node_modules
|
||||||
206
.opencode/feature_list.json
Normal file
206
.opencode/feature_list.json
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
{
|
||||||
|
"$schema": [
|
||||||
|
"/home/rawman/.config/opencode/schemas/feature_list.schema.json"
|
||||||
|
],
|
||||||
|
"features": [
|
||||||
|
{
|
||||||
|
"feature_id": "F01",
|
||||||
|
"title": "Bundle scaffold (module + hook + endpoint), build pipeline, and local load",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"Run `npm install` → resolves the pinned devDependencies including vue.",
|
||||||
|
"Run `npm run build` → emits `dist/app.js` and `dist/api.js`; `npm run typecheck` reports 0 errors.",
|
||||||
|
"`directus-extension link` (or the dev compose stack) loads the bundle into a local Directus 11.17.4 with no error.",
|
||||||
|
"The `image-previews` module appears in the admin sidebar, `GET /image-previews/status` answers, and a test upload emits the hook debug line.",
|
||||||
|
"`npm run publish` copies `dist/` + `package.json` to the configured deploy target (staging only at this stage)."
|
||||||
|
],
|
||||||
|
"passes": false,
|
||||||
|
"notes": [
|
||||||
|
"Manifest is a bundle: `path:{app,api}`, `entries[{type:module|hook|endpoint,name,source}]`, `host:^10.10.0 || ^11.0.0`.",
|
||||||
|
"Module sidebar fields (id/name/icon/routes) live in the entrypoint object, not package.json."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F02",
|
||||||
|
"title": "Staging environment harness + deploy/rollback scripts (safe testing)",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"`dev/compose.dev.yml` brings up Directus 11.17.4 with its own postgres + redis on a dedicated port, with the extension mounted.",
|
||||||
|
"The stack seeds from a **restored anonymized DB dump** and a **copy of uploads** (`dev/seed/`), so previews can be tested against realistic data.",
|
||||||
|
"A documented script builds and deploys to the staging instance, and a rollback script restores the previous `dist/` and disables the extension.",
|
||||||
|
"Confirmed the local/staging stack shares NOTHING with prod (separate DB, separate storage, separate ports).",
|
||||||
|
"README documents how to start/stop/seed the stack and the exact anonymization rules used for the dump."
|
||||||
|
],
|
||||||
|
"passes": false,
|
||||||
|
"blocked_by": [
|
||||||
|
"F01"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"This must land before any feature is tested on real-ish data; never test on the prod instance.",
|
||||||
|
"Decide staging placement (second compose project on land-nest vs a separate dev box) — goal.md open item #5.",
|
||||||
|
"Dev harness (dev/compose.dev.yml + sync-extension.sh) verified; staging compose/deploy/rollback templates added but not executed remotely."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F03",
|
||||||
|
"title": "Persisted settings + progress-state store and the config/status API endpoints",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"`src/lib/settings.ts` reads/writes a JSON settings file under `IMAGE_PREVIEWS_STATE_DIR` (default `/directus/extensions/.image-previews`), merged over env defaults; keys: `enabled` (default false), `windowStart`, `windowEnd`, `concurrency`, `presets`, `formats` (`avif,webp`), `onUpload`, `debug`.",
|
||||||
|
"`GET /image-previews/config` returns effective settings; `PUT /image-previews/config` validates and persists them; invalid payloads return 400 without corrupting the file.",
|
||||||
|
"`src/lib/state.ts` appends completed `{file_id,preset_key,format}` records to `state.jsonl` and loads them back; a resume helper returns completed pairs + totals.",
|
||||||
|
"`GET /image-previews/status` returns `{ running, total, done, remaining, generated, hits, failed, rate, eta, lastRun, nextWindow }`.",
|
||||||
|
"Unit tests cover settings merge/defaults/validation, config round-trip, state append/resume, status aggregation.",
|
||||||
|
"`npm test` green and `npm run typecheck` 0 errors."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F02"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"No native Directus extension-settings store exists; a JSON file under a persisted writable dir is the pragmatic choice.",
|
||||||
|
"All state is file-backed (PM2 cluster, 8 workers — never in-memory)."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F04",
|
||||||
|
"title": "Preset registry (system + settings, dynamic) and the in-process warmer core",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"`src/lib/presets.ts` returns the merged preset list in controller order (`[...SYSTEM_ASSET_ALLOW_LIST, ...storage_asset_presets]`) with first-match-wins on duplicate keys.",
|
||||||
|
"System preset objects come from an `@directus/api` constant import if available, else the version-pinned copy in `src/lib/system-presets.ts`; the mechanism used is recorded in progress.txt.",
|
||||||
|
"`src/lib/warmer.ts` calls `new services.AssetsService({ schema }).getAsset(fileId, { transformationParams: <verbatim preset object>, acceptFormat })` then destroys/unpipes the returned stream.",
|
||||||
|
"For a `format:'auto'` preset the warmer emits BOTH `avif` and `webp`; a fixed-`format` preset emits one; a resize-only preset emits none.",
|
||||||
|
"Concurrency honours the configured limit; a per-preset/format failure (concurrency error, `IllegalAssetTransformation`, missing dims) is caught, logged, and doesn't abort the rest; the result summarises `{ requested, ok, failed }`.",
|
||||||
|
"Unit tests assert: merged preset order + first-match-wins; variant selection per format; exact `getAsset` args; concurrency cap; error isolation.",
|
||||||
|
"`npm test` green and `npm run typecheck` 0 errors."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F03"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"Passing `{ key }` alone is a no-op — always pass the full preset object.",
|
||||||
|
"`acceptFormat:'avif'` in a direct call equals a browser `Accept: image/avif` (verified) — reproduces the admin cache file without HTTP."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F05",
|
||||||
|
"title": "Upload hook + new-preset trigger",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"On `files.upload` where `payload.type` starts with `image/` and `onUpload` is enabled, the hook runs the warmer for configured presets off the response path (fire-and-forget; never blocks or fails the upload).",
|
||||||
|
"A non-image upload performs no warming; a warmer failure never fails or slows the upload.",
|
||||||
|
"After uploading a test JPEG through the admin, the expected `<basename>__<hash>.avif` (and `.webp`) files for each configured `format:'auto'` preset appear in the storage root.",
|
||||||
|
"When a storage asset preset is added/changed, existing images are warmed for it (`settings.update` hook and/or a manual \"sync presets\" action); re-running is idempotent.",
|
||||||
|
"Under PM2 cluster the hook fires once per upload (logged once, not per worker)."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F04"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"`files.upload` is the correct/only post-write event in v11.17.4.",
|
||||||
|
"New-preset trigger mechanics are goal.md open item #3.",
|
||||||
|
"Decision: manual sync chosen — POST /run is idempotent and warms existing images for all presets; no settings.update auto-trigger."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F06",
|
||||||
|
"title": "Diagnostics & manual generation API + observability",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"`POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` generates (or dry-runs) previews for ONE file and returns per `(key,format)` `{ ok, cached, filename, bytes, ms, error }`.",
|
||||||
|
"`GET /image-previews/resolve-plan?fileId=&key=` returns the resolved transforms array, the **predicted cache filename**, and whether it already exists — WITHOUT generating.",
|
||||||
|
"`GET /image-previews/ops?limit=` returns a recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error), capped in size.",
|
||||||
|
"A `debug` setting toggles verbose structured logs; logs carry ids/metadata only, never file contents.",
|
||||||
|
"All diagnostics endpoints except `/generate` are read-only and safe to call on a live instance.",
|
||||||
|
"Unit tests cover resolve-plan prediction (given a preset, the predicted filename equals the warmer's produced filename) and ops-ring-buffer behaviour.",
|
||||||
|
"`npm test` green and `npm run typecheck` 0 errors."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F04"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"resolve-plan is the debugging tool that proves hash-match before anything is written.",
|
||||||
|
"Also the autonomous loop's primary feedback channel — keep responses machine-readable."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F07",
|
||||||
|
"title": "Admin module UI: progress bar, disk usage, config form, diagnostics panel",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"The \"Image Previews\" module renders and shows a progress bar with `done / total`, remaining, throughput, ETA, plus last-run and next-window times.",
|
||||||
|
"During a run the page live-updates (polls `/status`); \"Run now\" / \"Cancel\" controls start/stop a run.",
|
||||||
|
"A disk-usage panel shows total bytes + count of preview files (list the storage root, sum `stat()` sizes of `__<hash>` files); a refresh control recomputes it.",
|
||||||
|
"A config form edits `enabled`, maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`, persisting via `PUT /config`; changes take effect without rebuilding.",
|
||||||
|
"A diagnostics panel lists recent operations (`/ops`) and lets an operator run generate/dry-run/resolve-plan for a pasted file id.",
|
||||||
|
"Invalid input is rejected client-side with a clear message; a saved change is reflected after reload.",
|
||||||
|
"Smoke-verified on the staging stack (notes/screenshots in progress.txt)."
|
||||||
|
],
|
||||||
|
"passes": false,
|
||||||
|
"blocked_by": [
|
||||||
|
"F06"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"Use Directus UI components / the private-view shell; keep it minimal.",
|
||||||
|
"Concurrency is the throttle knob here because in-process generation bypasses the HTTP rate limiter."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F08",
|
||||||
|
"title": "Resumable backfill engine (window-aware, throttled, progress-reporting)",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"The job enumerates `directus_files` with `type` starting `image/` (paginated) and runs the warmer for each file × configured preset × format.",
|
||||||
|
"It only runs inside the configured maintenance window, or on an explicit manual run; starting outside the window when scheduled is a logged no-op.",
|
||||||
|
"It honours the configured concurrency cap and appends completed pairs to `state.jsonl`; restarting after a kill resumes from the last completed pair.",
|
||||||
|
"Progress is observable during a run via `GET /status` and periodic log lines.",
|
||||||
|
"A second full run over an already-warmed library reports zero new generations (idempotent).",
|
||||||
|
"A >6000px original or a concurrency error is recorded as `failed` for that pair; the job continues.",
|
||||||
|
"`POST /run` starts a run, `POST /run/cancel` stops it; a second concurrent run returns 409 (cluster-wide lock file)."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F07"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"Run state is file-backed; only one run at a time across the cluster.",
|
||||||
|
"Scheduling: external cron hitting `/run` (default) vs in-process timer — goal.md open item #4."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F09",
|
||||||
|
"title": "Verification harness: hash-match gate + cold/warm timing + disk counts",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"For one real file, the on-disk filename produced by an admin browser request for `?key=system-medium-cover` is byte-identical to the warmer's non-dry-run output for the same key+format (the avif variant matches the browser's avif request).",
|
||||||
|
"The same is demonstrated for `system-large-cover` (image field) and `system-small-cover` (image display).",
|
||||||
|
"A timing check prints cold vs warm `GET /assets/<id>?key=system-medium-cover`; warm is materially faster.",
|
||||||
|
"Preview-file count and total bytes are reported before/after a backfill chunk and increase by the expected number of (file × preset × format) entries.",
|
||||||
|
"Findings (observed admin query string, resulting filenames, uuids, timings) are recorded in progress.txt."
|
||||||
|
],
|
||||||
|
"passes": true,
|
||||||
|
"blocked_by": [
|
||||||
|
"F08"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"Make-or-break gate: if the warmer's filename does not match the admin's, the approach is wrong — revisit before rollout.",
|
||||||
|
"Also validates the system-preset copy from F04."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"feature_id": "F10",
|
||||||
|
"title": "Ops: fail-open/disabled-by-default, prod rollout + rollback, scheduling, docs, remote-feedback runbook",
|
||||||
|
"acceptance_steps": [
|
||||||
|
"Every entrypoint body is wrapped so a thrown error is logged and swallowed (never propagates into Directus core), and ALL behavior is gated by `enabled` (default false) — a loaded-but-disabled extension is inert, proven by loading it on staging with `enabled=false`.",
|
||||||
|
"A documented prod rollout procedure exists: backup previous `dist/` → deploy → healthcheck → enable via config; and a rollback that restores `dist/` and disables.",
|
||||||
|
"A documented scheduling setup runs the backfill inside the maintenance window (cron hitting `POST /image-previews/run`, by default).",
|
||||||
|
"`README.md` documents purpose, install/build/publish, every setting, presets/formats warmed, run/resume/cancel, disk reporting, and the maintenance window; `AGENTS.md` captures the transform invariants and safety rules.",
|
||||||
|
"A **remote-feedback runbook** documents the scoped token and/or restricted SSH access for the autonomous loop (staging only, no prod Docker socket, no prod admin token, private network), and the machine-readable endpoints used as the feedback channel.",
|
||||||
|
"Final end-to-end pass on staging: upload a fresh image → previews appear with no admin-visible delay; a small backfill chunk completes with the progress bar advancing and disk usage updating; then a controlled prod enable with rollback verified."
|
||||||
|
],
|
||||||
|
"passes": false,
|
||||||
|
"blocked_by": [
|
||||||
|
"F09"
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
"Promotion to prod is a human action; the loop proposes, a human deploys.",
|
||||||
|
"Record the exact cron entry, the scoped grants, and the rollback commands verbatim."
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
249
.opencode/goal.md
Normal file
249
.opencode/goal.md
Normal file
|
|
@ -0,0 +1,249 @@
|
||||||
|
# Goal: Directus image preview pre-generation
|
||||||
|
|
||||||
|
Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (not part of the
|
||||||
|
landshaft.life app repo).
|
||||||
|
Location: `/home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews`
|
||||||
|
Target server: `land-nest`, Directus **11.17.4** (self-hosted, PM2 cluster, local disk storage).
|
||||||
|
Project type: Directus **bundle** extension (app module + api hook + api endpoint).
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Pre-generate Directus image transform ("preview") cache files so the admin UI never pays the
|
||||||
|
first-request sharp cost. One warmer, several triggers:
|
||||||
|
|
||||||
|
- **upload hook** — after an image upload, warm its previews.
|
||||||
|
- **new-preset trigger** — when a storage asset preset is added/changed, warm it for existing images.
|
||||||
|
- **backfill job** — a resumable, scheduled sweep over the existing `directus_files` library, inside a
|
||||||
|
configurable maintenance window.
|
||||||
|
- **manual / diagnostics** — generate (or dry-run) previews for a single file on demand, to debug.
|
||||||
|
|
||||||
|
Plus an **admin module** ("Image Previews") with a **progress bar**, **disk usage**, a **config form**
|
||||||
|
(persisted, no rebuild), and a **diagnostics panel**.
|
||||||
|
|
||||||
|
## Verified mechanism (source-checked against tag v11.17.4)
|
||||||
|
|
||||||
|
The on-disk transform cache file is:
|
||||||
|
|
||||||
|
```
|
||||||
|
<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>
|
||||||
|
```
|
||||||
|
|
||||||
|
written **beside the original** in the storage root (`STORAGE_LOCAL_ROOT=./uploads`).
|
||||||
|
|
||||||
|
- `hash` = `object-hash@3` defaults: sha1/hex, `unorderedArrays:false` (**array order matters**),
|
||||||
|
`unorderedObjects:true`. Hashed over the **RESOLVED** transform array from `resolvePreset`.
|
||||||
|
- `resolvePreset` order is fixed: `(1) copy transformationParams.transforms[]`, `(2) push toFormat`
|
||||||
|
(if `format`/`quality`), `(3) push resize` (+`extract` on focal cover).
|
||||||
|
- `maybeExtractFormat` picks the last `toFormat` arg as the file extension.
|
||||||
|
- `AssetsService.getAsset` checks `exists(assetFilename)` first (cache hit → no sharp); on miss runs
|
||||||
|
sharp and **awaits** `storage.write(...)` before returning. Not draining the returned stream only
|
||||||
|
leaks an FD (never truncates).
|
||||||
|
- Called as sudo (`new AssetsService({ schema })`) it needs **no** HTTP request context.
|
||||||
|
|
||||||
|
## Why the naive / community approach FAILS (must not repeat)
|
||||||
|
|
||||||
|
1. **`{ key }` alone is a no-op.** `resolvePreset` never reads `key`. Pass the **full preset object**.
|
||||||
|
2. **`format:'auto'` mismatch.** Every `system-*` preset is `format:'auto'`. `getFormat` with no
|
||||||
|
`acceptFormat` → JPEG→`jpg`/PNG→`png`; a browser `Accept: image/avif` → `avif`. Different
|
||||||
|
`toFormat` arg → different hash → **guaranteed miss**.
|
||||||
|
3. Disproved myths: the write is awaited; the admin does NOT use `width`/`height`.
|
||||||
|
4. Raw params resolve to `[toFormat, resize]`; a preset resolves to `[resize, toFormat]` → different hash.
|
||||||
|
|
||||||
|
## Chosen strategy — IN-PROCESS generation (no token, no HTTP, no rate limiter)
|
||||||
|
|
||||||
|
Because this is an internal extension, generate by calling Directus's own service in-process:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const assets = new services.AssetsService({ schema }); // sudo
|
||||||
|
const { stream } = await assets.getAsset(fileId, {
|
||||||
|
transformationParams: presetObject, // FULL object, not { key }
|
||||||
|
acceptFormat: 'avif', // or 'webp'
|
||||||
|
});
|
||||||
|
(stream as Readable).destroy?.();
|
||||||
|
```
|
||||||
|
|
||||||
|
`acceptFormat` reproduces exactly what a browser's `Accept` header does, so the cache filename is
|
||||||
|
byte-identical to the admin/browser read — **without any HTTP request, token, proxy, or rate limiter**.
|
||||||
|
Generate **two variants** for `format:'auto'` presets (`avif`, `webp`); fixed-`format` presets yield
|
||||||
|
one; resize-only presets (no `format`/`quality`) yield none.
|
||||||
|
|
||||||
|
HTTP-replay (`GET /assets/{id}?key=…` with a browser `Accept` + auth) is the documented **fallback**
|
||||||
|
only — it needs a token and rate-limiter pacing, which is why it is not the default.
|
||||||
|
|
||||||
|
## Preset registry (system + settings, dynamic)
|
||||||
|
|
||||||
|
The controller builds its preset list inline as:
|
||||||
|
|
||||||
|
```
|
||||||
|
res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]
|
||||||
|
```
|
||||||
|
|
||||||
|
and resolves `?key=` with `.find(t => t.key === key)` → **first match wins** (a system key shadows a
|
||||||
|
custom preset of the same key). There is **no** reusable helper and `GET /settings` exposes **only**
|
||||||
|
custom presets.
|
||||||
|
|
||||||
|
Our registry (`src/lib/presets.ts`) mirrors this exact list:
|
||||||
|
|
||||||
|
- **System presets**: 6 objects in `SYSTEM_ASSET_ALLOW_LIST` (`system-small|medium|large` ×
|
||||||
|
`cover|contain`), each `format:'auto'`, resize inside `transforms[]`; sourced at runtime from
|
||||||
|
`@directus/api` **if** importable, else the version-pinned copy `src/lib/system-presets.ts` (v11.17.4).
|
||||||
|
- **Custom presets**: read `directus_settings.storage_asset_presets` each run; pass the verbatim object.
|
||||||
|
- **Dynamic**: newly created presets are picked up per run; the new-preset trigger warms them for
|
||||||
|
existing images.
|
||||||
|
|
||||||
|
## Admin UI (module "Image Previews")
|
||||||
|
|
||||||
|
- **Progress**: bar + `done / total / remaining`, throughput, ETA, last-run, next-window; live during a run.
|
||||||
|
- **Storage breakdown**: bytes + file counts separately for **preview assets** (`__<hash>`), **original
|
||||||
|
assets**, and the total — raw and human-readable. Computed by listing the storage root and `stat()`-ing
|
||||||
|
each file (listing yields paths only, no sizes); the state dir is excluded.
|
||||||
|
- **Config form** (persisted, no rebuild; defaults `enabled:true`, window `01:00–04:00`): `enabled`,
|
||||||
|
maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`; each field carries a
|
||||||
|
plain-language explanation. **Run now** ignores the window; **Cancel** is shown only while a run is active.
|
||||||
|
- **Diagnostics panel**: recent operations, per-file generate/dry-run, resolve-plan viewer.
|
||||||
|
|
||||||
|
## Diagnostics & manual generation (observability)
|
||||||
|
|
||||||
|
- `POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` → generate (or dry-run) for ONE
|
||||||
|
file; returns per `(key,format)` `{ ok, failed, cached, filename, bytes, ms, error }`. Primary dev tool.
|
||||||
|
- `GET /image-previews/resolve-plan?fileId=&key=` → resolved transforms + **predicted cache filename** +
|
||||||
|
whether it already exists, WITHOUT generating. The hash-match debugging tool.
|
||||||
|
- `GET /image-previews/ops?limit=` → recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error).
|
||||||
|
- `debug` setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
|
||||||
|
- Everything except `/generate` is read-only.
|
||||||
|
|
||||||
|
## Safe deployment & staging (must not crash prod)
|
||||||
|
|
||||||
|
- **Fail-open, inert by default**: every entrypoint is tiny, bodies wrapped in try/catch, hooks never
|
||||||
|
throw into Directus core, and ALL behavior is gated by `enabled` (default `false`). A loaded-but-disabled
|
||||||
|
extension does nothing.
|
||||||
|
- **Local dev**: `directus-extension link` (or the dev compose stack) loads `src/` live for iteration.
|
||||||
|
- **Staging stack**: a second compose project (`dev/compose.dev.yml`) running the SAME Directus 11.17.4
|
||||||
|
image with its own pg+redis, a **restored anonymized dump**, and a **copy of uploads**, on a separate
|
||||||
|
port — realistic data, zero prod risk.
|
||||||
|
- **Rollout discipline**: build → verify on staging → back up the previous `dist/` → copy in → healthcheck
|
||||||
|
→ enable via config → rollback = restore previous `dist/` + disable.
|
||||||
|
- `EXTENSIONS_AUTO_RELOAD` in prod reloads on file change; deploy with it controlled and accept the brief
|
||||||
|
reload, or restart deliberately.
|
||||||
|
|
||||||
|
## Remote feedback & autonomous operation (security)
|
||||||
|
|
||||||
|
- Feedback channel = the extension's own **machine-readable endpoints** (`/status`, `/ops`,
|
||||||
|
`/disk-usage`, `/resolve-plan`). Automation reads those, not CMS content.
|
||||||
|
- **Least privilege**:
|
||||||
|
- A Directus **admin token = full CMS control**. Do NOT give it to an autonomous loop. Use a scoped
|
||||||
|
token reaching only `/image-previews/*` (+ `directus_files` metadata read).
|
||||||
|
- A **Docker socket = root-equivalent**. Never expose the prod socket to automation; keep the loop on staging.
|
||||||
|
- Deployment access = a dedicated **non-sudo** SSH user with a restricted key (`restrict`, `from=`,
|
||||||
|
`command=`), or run opencode on the staging host via `handoff`.
|
||||||
|
- **Network**: staging reachable only over a private network / VPN / tunnel — never public.
|
||||||
|
- **Prompt-injection**: treat logs, filenames, and image metadata as untrusted input; keep tooling scoped.
|
||||||
|
- **Promotion to prod is a human action**: the loop proposes, a human merges/deploys.
|
||||||
|
- Guard long autonomous runs with a cost/time budget.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
hook/index.ts (api) files.upload -> warmer; settings/presets change -> warm existing
|
||||||
|
endpoint/index.ts (api) routes: /config (GET/PUT), /status, /run, /run/cancel,
|
||||||
|
/disk-usage, /generate, /resolve-plan, /ops
|
||||||
|
module/
|
||||||
|
index.ts (app) defineModule({ id:'image-previews', name, icon, routes })
|
||||||
|
module.vue progress + disk usage + config form + diagnostics panel
|
||||||
|
lib/
|
||||||
|
presets.ts merged registry (system + settings), mirrors controller semantics
|
||||||
|
system-presets.ts version-pinned SYSTEM_ASSET_ALLOW_LIST copy (v11.17.4)
|
||||||
|
warmer.ts in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
|
||||||
|
backfill.ts enumerate directus_files (image/*), window-aware, resumable
|
||||||
|
state.ts state.jsonl read/append/resume
|
||||||
|
settings.ts config file read/write + env defaults
|
||||||
|
disk.ts sum sizes of __<hash> files (driver.list + stat)
|
||||||
|
window.ts maintenance-window logic (is-now-in-window, next-window)
|
||||||
|
diagnostics.ts resolve-plan/dry-run (resolved transforms + predicted filename), ops ring buffer
|
||||||
|
dev/
|
||||||
|
compose.dev.yml local Directus 11.17.4 + postgres + redis, extension linked
|
||||||
|
seed/ anonymized DB dump + sample images for local/staging runs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- No Directus upgrade (server stays on 11.17.4; license constraint).
|
||||||
|
- No change to admin preset configuration or to how the admin builds URLs.
|
||||||
|
- Never hand-write `__<hash>.<ext>` filenames.
|
||||||
|
- No CDN/nginx caching work (separate ops concern). No image `quality` policy change.
|
||||||
|
- Automation never gets prod Docker socket or a full prod admin token.
|
||||||
|
|
||||||
|
## Constraints / risks
|
||||||
|
|
||||||
|
- `ASSETS_TRANSFORM_MAX_CONCURRENT=45` per worker; over it errors. Catch per file/preset and continue
|
||||||
|
(never fail the upload).
|
||||||
|
- `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000 is a **pixel dimension** guard in Directus, not a
|
||||||
|
file-size limit. The extension imposes no limit of its own; raise the Directus env var to process larger
|
||||||
|
images (watch memory). Upload-size limits (`FILES_MAX_UPLOAD_SIZE`, `MAX_PAYLOAD_SIZE`) are unrelated.
|
||||||
|
- Local storage write is **non-atomic** → warm only AFTER an upload is committed; serialize so the
|
||||||
|
upload hook and the backfill never generate the same file concurrently.
|
||||||
|
- PM2 cluster (8 workers): the upload hook fires on the handling worker; the backfill runs in the worker
|
||||||
|
that served the run request. Shared state MUST be file/DB-backed (never in-memory).
|
||||||
|
- In-process backfill competes with live API serving → the maintenance window + concurrency cap govern
|
||||||
|
that. (HTTP rate limiter is irrelevant in-process.)
|
||||||
|
- Observed server bottleneck (2026-10-23): `storage:local:responseTime 1206ms`, `pg:responseTime`
|
||||||
|
150–738ms; no 503s, no sharp errors.
|
||||||
|
|
||||||
|
## Verification policy
|
||||||
|
|
||||||
|
- `npm run typecheck` → 0 errors; `npm test` (node:test via tsx) green.
|
||||||
|
- **The hash-match gate (F09)**: for a real file, the on-disk filename produced by a browser/admin
|
||||||
|
request for `?key=system-medium-cover` is byte-identical to the warmer's output for the same key+format
|
||||||
|
(+ the resolved extension matches, e.g. `.avif`).
|
||||||
|
- Cold-vs-warm timing; preview-file count + total bytes before/after a run.
|
||||||
|
- All feature testing happens on the **staging** stack; prod deploy only at goal completion, via the
|
||||||
|
documented backup/rollback procedure.
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
- `package.json`, `tsconfig.json`, `.gitignore`
|
||||||
|
- `src/hook/`, `src/endpoint/`, `src/module/`, `src/lib/`, plus `src/**/*.test.ts`
|
||||||
|
- `dev/compose.dev.yml`, `dev/seed/`
|
||||||
|
- `.opencode/goal.md`, `.opencode/feature_list.json`, `.opencode/progress.txt`
|
||||||
|
- `README.md`, `AGENTS.md`
|
||||||
|
|
||||||
|
## Resolved decisions (2026-10-25)
|
||||||
|
|
||||||
|
1. State/config: JSON files under `<STORAGE_LOCAL_ROOT>/.image-previews` (default
|
||||||
|
`/directus/uploads/.image-previews`), override with `IMAGE_PREVIEWS_STATE_DIR`. The former
|
||||||
|
`/directus/extensions/.image-previews` default was rejected — root-owned volume → `EACCES`.
|
||||||
|
2. System presets: version-pinned copy (`src/lib/system-presets.ts`, verbatim v11.17.4); no
|
||||||
|
`@directus/api` import needed for presets.
|
||||||
|
3. New-preset trigger: manual `POST /run` (idempotent) + the module's Run button; no
|
||||||
|
`settings.update` auto-trigger.
|
||||||
|
4. Backfill scheduling: external cron → `POST /image-previews/run`.
|
||||||
|
5. Dev/staging: isolated local compose stack (`dev/compose.dev.yml`). A land-nest staging project is
|
||||||
|
still to be created for realistic-data runs.
|
||||||
|
6. Autonomous-loop grants: still to be defined (scoped token / restricted SSH, staging only).
|
||||||
|
|
||||||
|
## Implementation status (2026-10-25)
|
||||||
|
|
||||||
|
Implemented and verified against an isolated Directus 11.17.4: settings/state, presets + warmer,
|
||||||
|
upload hook, diagnostics, backfill, endpoints. `npm test` 32/32, typecheck clean, build emits both
|
||||||
|
bundles. **Hash-match gate passed** (a real browser-style asset request produced exactly the
|
||||||
|
predicted filename). Features F03–F06, F08, F09 pass.
|
||||||
|
|
||||||
|
Outstanding: admin-module browser verification (F07), staging deploy/rollback + prod rollout (F02/F10),
|
||||||
|
and the F01 publish-to-staging step.
|
||||||
|
|
||||||
|
## Reference: source anchors (v11.17.4)
|
||||||
|
|
||||||
|
- `api/src/services/assets.ts` — `getAsset`, `getAssetSuffix`, filename build, awaited `write`.
|
||||||
|
- `api/src/utils/transformations.ts` — `resolvePreset`, `getFormat`.
|
||||||
|
- `api/src/controllers/assets.ts` — inline `shortcuts = [...SYSTEM_ASSET_ALLOW_LIST, ...settings]`,
|
||||||
|
`acceptFormat` selection from `req.headers.accept`.
|
||||||
|
- `api/src/constants.ts` — `SYSTEM_ASSET_ALLOW_LIST` (6 presets, `format:'auto'`);
|
||||||
|
`ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000.
|
||||||
|
- `app/src/utils/get-asset-url.ts`, `app/src/layouts/cards/components/card.vue`,
|
||||||
|
`app/src/interfaces/file-image/file-image.vue`, `app/src/displays/image/image.vue` — admin URLs:
|
||||||
|
Files grid `?key=system-medium-cover&v=<modified_on>`; image field `system-large-cover|contain`;
|
||||||
|
image display `system-small-cover`.
|
||||||
|
- `packages/storage-driver-local/src/index.ts` — non-atomic `write`; `list()` returns paths only.
|
||||||
|
- Prior art that does NOT work as-is: `directus-extension-auto-generate-file-transformations`
|
||||||
|
(utomic-media) — still falls foul of the `auto`→format mismatch unless it forces the format.
|
||||||
136
.opencode/progress.txt
Normal file
136
.opencode/progress.txt
Normal file
|
|
@ -0,0 +1,136 @@
|
||||||
|
# progress.txt — directus-extension-generate-image-previews
|
||||||
|
|
||||||
|
Append-only log of decisions, evidence, and learnings. Newest at the bottom.
|
||||||
|
|
||||||
|
## 2026-10-25 — project scaffolded
|
||||||
|
|
||||||
|
- Opened this standalone extension project to pre-generate Directus image previews.
|
||||||
|
- Source-verified the strategy against Directus **v11.17.4** (see `.opencode/goal.md`).
|
||||||
|
- Root causes why existing/community extensions do not work:
|
||||||
|
1. `resolvePreset` never reads `key` → `{transformationParams:{key}}` yields `transforms=[]` → no-op.
|
||||||
|
2. `system-*` presets are `format:'auto'` → `jpg`/`png` server-side vs `avif` in-browser → different hash.
|
||||||
|
3. "undrained stream truncates file" — false (write is awaited).
|
||||||
|
4. admin uses `?key=system-*`, not raw width/height.
|
||||||
|
- Admin URLs (v11.17.4 app source): files grid `?key=system-medium-cover&v=<modified_on>`;
|
||||||
|
image field `system-large-cover|contain`; image display `system-small-cover`.
|
||||||
|
- Server bottleneck evidence (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
|
||||||
|
`pg:responseTime` 150–738ms; no 503s, no sharp errors.
|
||||||
|
|
||||||
|
## 2026-10-25 — scope expanded after user direction (v2)
|
||||||
|
|
||||||
|
- **No token needed.** Dropped HTTP-replay as primary. Strategy: **in-process**
|
||||||
|
`AssetsService.getAsset(id, { transformationParams: <full preset object>, acceptFormat: 'avif'|'webp' })`.
|
||||||
|
No HTTP → no token, no proxy, no rate limiter. (HTTP-replay kept as documented fallback.)
|
||||||
|
- **Bundle extension**: api `hook` + api `endpoint` + app `module`.
|
||||||
|
- **Presets: all, dynamically** — system `SYSTEM_ASSET_ALLOW_LIST` (6) + `directus_settings.storage_asset_presets`,
|
||||||
|
controller order `[...SYSTEM, ...custom]`, first-match-wins; new presets picked up per run.
|
||||||
|
- **Admin module**: progress bar, disk usage (`list`+`stat`), config form, diagnostics panel.
|
||||||
|
- **Config**: maintenance window + concurrency (throttle knob; in-process bypasses HTTP rate limiter),
|
||||||
|
persisted in JSON.
|
||||||
|
- Confirmed `acceptFormat:'avif'` in a direct call equals browser `Accept: image/avif`.
|
||||||
|
|
||||||
|
## 2026-10-25 — diagnostics + safety + autonomy (v3)
|
||||||
|
|
||||||
|
- **Diagnostics / manual generation** (F06): `POST /generate` (one file, dry-run supported),
|
||||||
|
`GET /resolve-plan` (resolved transforms + predicted filename, no write), `GET /ops` (ring buffer).
|
||||||
|
- **Safe testing** (F02): staging compose stack + deploy/rollback scripts; fail-open, `enabled` default false.
|
||||||
|
- **Autonomous remote feedback**: staging-only, least privilege (scoped token, no prod Docker socket,
|
||||||
|
restricted SSH, private network, human-only prod promotion).
|
||||||
|
|
||||||
|
## 2026-10-25 — implementation + local verification (autonomous session)
|
||||||
|
|
||||||
|
Built the full extension locally and smoke-tested against an **isolated** Directus 11.17.4 stack
|
||||||
|
(`dev/compose.dev.yml`, port 8057, own pg + volumes). No contact with the prod instance.
|
||||||
|
|
||||||
|
### Implemented
|
||||||
|
- `src/lib/`: settings (env + JSON), state (`state.jsonl`, run state, lock, cancel flag),
|
||||||
|
presets (pinned system copy + custom merge, controller parity), warmer (in-process `getAsset` with
|
||||||
|
`acceptFormat` variants, concurrency, per-task error isolation, stream release), window, storage
|
||||||
|
(dynamic `getStorage` + local-FS fallback), diagnostics (resolve-plan prediction + ops ring buffer),
|
||||||
|
backfill (paginated, resumable, window-aware, idempotent).
|
||||||
|
- `src/endpoint/index.ts`: `/status`, `/config` (GET/PUT), `/presets`, `/run`, `/run/cancel`,
|
||||||
|
`/disk-usage`, `/generate`, `/resolve-plan`, `/ops`; admin-only guard.
|
||||||
|
- `src/hook/index.ts`: `files.upload` → fire-and-forget warm.
|
||||||
|
- `src/module/`: admin module (progress bar, disk usage, config form, diagnostics panel, ops table).
|
||||||
|
- `dev/`: compose stack, `sync-extension.sh`, seed images.
|
||||||
|
|
||||||
|
### Bugs found & fixed during smoke test
|
||||||
|
1. **Directus coerces env values** (`"true"` → boolean `true`, `"1"` → number). My string-only parsers
|
||||||
|
threw `e.toLowerCase is not a function`, which silently broke the upload hook. Fixed `settings.ts`
|
||||||
|
to accept `unknown` and stringify. Regression test added.
|
||||||
|
2. **State dir EACCES.** `/directus/extensions/.image-previews` (named volume on a non-existent path)
|
||||||
|
is root-owned; the API runs as `node`. This broke config writes and the backfill lock. Fixed: default
|
||||||
|
state dir is now `<STORAGE_LOCAL_ROOT>/.image-previews` (i.e. `/directus/uploads/.image-previews`),
|
||||||
|
which is writable; `IMAGE_PREVIEWS_STATE_DIR` still overrides.
|
||||||
|
3. **Bind-mount reload gotcha.** `rm -rf` of the bind-mounted extension dir breaks the Docker mount
|
||||||
|
(Directus removes the extension and never re-adds it). `sync-extension.sh` now updates in place;
|
||||||
|
restart the dev container after a sync.
|
||||||
|
|
||||||
|
### Verified evidence (local stack)
|
||||||
|
- `npm run typecheck` → 0 errors. `npm test` → **32/32 pass**. `npm run build` → `dist/app.js` + `dist/api.js`.
|
||||||
|
- Directus logs: `Loaded extensions: directus-extension-generate-image-previews` (no errors).
|
||||||
|
- `/status` → `{enabled:true, inWindow:true, locked:false}`; `/presets` → 6 system presets.
|
||||||
|
- Upload a JPEG → hook warmed **12 files** (6 presets × avif/webp) in `/directus/uploads`.
|
||||||
|
- **HASH-MATCH GATE PASSED:** after deleting the cached files, a real
|
||||||
|
`GET /assets/<id>?key=system-medium-cover` with `Accept: image/avif` produced exactly
|
||||||
|
`<id>__f15aa4b815722867859d2a3fa46ab50834dfc8dc.avif` — byte-identical to `resolve-plan`'s prediction.
|
||||||
|
- `/generate` dry-run reports `exists:true/false` correctly; `/ops` shows per-task timing.
|
||||||
|
- Backfill: `total=36` (3 images × 6 presets × 2 formats), `generated=36`, state `state.jsonl` = 36 lines.
|
||||||
|
Second run: `generated=0, hits=36`, state unchanged, file count unchanged → **idempotent/resumable**.
|
||||||
|
- Window gating: `/run` outside `03:00–05:00` → `409 outside-window` with `nextWindow`.
|
||||||
|
- `/disk-usage` → `157633 bytes / 36 files (source: storage)`.
|
||||||
|
- Cold vs warm (`system-medium-cover`): 114 ms → 16–30 ms. (`system-large-cover`): 304 ms → 36 ms.
|
||||||
|
|
||||||
|
### Decisions recorded
|
||||||
|
- System presets: version-pinned copy in `src/lib/system-presets.ts` (verbatim from `api/src/constants.ts`
|
||||||
|
v11.17.4). No `@directus/api` import needed for presets.
|
||||||
|
- Storage/disk usage: `@directus/api/storage/index` loaded via dynamic import (hidden from the bundler),
|
||||||
|
with a local-FS fallback; shares the host `getStorage()` singleton.
|
||||||
|
- New-preset trigger: **manual** via `POST /run` (idempotent) + the module's Run button; no
|
||||||
|
`settings.update` auto-trigger (avoids heavy background runs on unrelated settings changes).
|
||||||
|
- Scheduling: external cron → `POST /image-previews/run` (default).
|
||||||
|
- State/config dir: `<STORAGE_LOCAL_ROOT>/.image-previews` (writable), override with `IMAGE_PREVIEWS_STATE_DIR`.
|
||||||
|
|
||||||
|
### Additional verification (same session)
|
||||||
|
- `GET /extensions` → bundle registered with entries `module`, `hook`, `endpoint`, all enabled.
|
||||||
|
- Inert test: with `enabled:false`, uploading an image produced **zero** new preview files (25 → 25).
|
||||||
|
- Added `dev/compose.staging.yml`, `dev/deploy-staging.sh`, `dev/rollback-staging.sh` (templates;
|
||||||
|
NOT executed against any remote host).
|
||||||
|
- Added `storage.test.ts`; total tests now **34/34**.
|
||||||
|
- Restarted the dev stack after the final build; `/status`, `/disk-usage`, and bundle entries all good.
|
||||||
|
|
||||||
|
### Still open
|
||||||
|
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
|
||||||
|
- Cancel + concurrent-run 409 not exercised end-to-end (unit-tested only; local runs finish too fast).
|
||||||
|
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
|
||||||
|
- F01/F02/F07/F10 remain `passes:false` (F01 publish step, F02 staging run, F07 browser, F10 rollout).
|
||||||
|
|
||||||
|
## 2026-10-25 — feedback round (UI, defaults, custom formats, docs)
|
||||||
|
|
||||||
|
- **Defaults changed**: `enabled` now `true`; maintenance window defaults to `01:00–04:00`.
|
||||||
|
- **Custom fixed-format presets verified**: `custom-png` (format `png`) generated `.png` and
|
||||||
|
`resolve-plan` predicted the exact filename (`exists:true`). `custom-auto` generated avif+webp.
|
||||||
|
An unsupported preset (`format: bmp`) failed with Directus's clear error
|
||||||
|
("Expected one of: heic, heif, avif, jpeg, … received bmp") and is surfaced in Recent operations / `/ops`.
|
||||||
|
- **Effective format fix**: ops/presets now report the preset's real output format (`png`) instead of
|
||||||
|
`null`; backfill resume keys use the effective format (fixed a re-warm mismatch).
|
||||||
|
- **Progress accounting fix**: already-cached tasks now count toward `done`/`hits`. Runs report
|
||||||
|
`128/128` (120 hits + 8 expected failures) instead of `8/128`. After removing the unsupported preset:
|
||||||
|
`120/120 hits, 0 failed, 0 generated`.
|
||||||
|
- **Storage reporting**: `/disk-usage` + panel now split **previews** vs **originals** vs **total**
|
||||||
|
(raw + human): e.g. previews 774.7 KB / 115 files, originals 24.9 MB / 9 files, total 25.6 MB / 124.
|
||||||
|
- **Human-readable fields** added to `/status` (ISO + human dates, `etaHuman`, `rateHuman`),
|
||||||
|
`/disk-usage` (`bytesHuman`), `/ops` (`tsHuman`, `msHuman`), `/generate` (`msHuman`).
|
||||||
|
- **Admin module rewritten with Directus UI kit** (`v-button`, `v-input`, `v-checkbox`, `v-select`,
|
||||||
|
`v-notice`, `v-progress-linear`), field explanations, preset table with effective formats + shadow
|
||||||
|
warnings, storage table, error column in Recent operations, and a full collapsible **API reference**.
|
||||||
|
- **Run now** now ignores the maintenance window; **Cancel** only renders while a run is active.
|
||||||
|
- **Dimension vs file size clarified** in README/goal/AGENTS: the 6000 limit is a pixel-dimension guard in
|
||||||
|
Directus, not a file-size limit; the extension imposes none.
|
||||||
|
- Tests: **40/40** pass; typecheck clean; build OK.
|
||||||
|
|
||||||
|
### Still open
|
||||||
|
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
|
||||||
|
- Cancel + concurrent-run 409 not exercised end-to-end.
|
||||||
|
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
|
||||||
|
- F01/F02/F07/F10 remain `passes:false`.
|
||||||
129
AGENTS.md
Normal file
129
AGENTS.md
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
# Agent notes — directus-extension-generate-image-previews
|
||||||
|
|
||||||
|
Common confusion points for agents working here. Read before touching the warmer.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
A Directus **bundle** extension that pre-generates image transform ("preview") cache files so the
|
||||||
|
admin UI never pays the first-request sharp cost. Target server: **Directus 11.17.4** on `land-nest`
|
||||||
|
(self-hosted, PM2 cluster, `STORAGE_LOCAL_ROOT=./uploads`).
|
||||||
|
|
||||||
|
Bundle entries: an api **hook** (`files.upload`), an api **endpoint** (`/image-previews/*`), and an
|
||||||
|
app **module** (`image-previews` admin page: progress, disk usage, config).
|
||||||
|
|
||||||
|
## Directus transform invariants (source-verified, tag v11.17.4) — DO NOT REGRESS
|
||||||
|
|
||||||
|
1. **Cached transform filename** = `<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>`,
|
||||||
|
written **beside the original** in the storage root (`./uploads`, no `.cache` subdir).
|
||||||
|
|
||||||
|
2. **Hash = `object-hash@3` defaults**: sha1/hex, `unorderedArrays:false` (**array order matters**),
|
||||||
|
`unorderedObjects:true`. Hashes the **RESOLVED** transform array from `resolvePreset`, NOT the raw
|
||||||
|
query/preset object.
|
||||||
|
|
||||||
|
3. **`resolvePreset` order is fixed**: copy `transformationParams.transforms[]` → push `toFormat`
|
||||||
|
(if `format`/`quality`) → push `resize` (+`extract` on focal cover). Raw params `?width=&height=&fit=&format=`
|
||||||
|
resolve to `[toFormat, resize]`; a preset resolves to `[resize, toFormat]` → **different hash for a
|
||||||
|
visually identical result**.
|
||||||
|
|
||||||
|
4. **`{ key }` alone is a NO-OP.** `resolvePreset` never reads `key`. Always pass the **full preset object**.
|
||||||
|
|
||||||
|
5. **`format:'auto'` is the killer trap.** All six `system-*` presets are `format:'auto'`. `getFormat`
|
||||||
|
with no `acceptFormat` → JPEG→`jpg`, PNG→`png`; a browser `Accept: image/avif` → `avif`. Different
|
||||||
|
`toFormat` arg → **different hash → guaranteed miss**.
|
||||||
|
|
||||||
|
6. **The write IS awaited.** Not draining the returned stream leaks an FD but never truncates.
|
||||||
|
Still: destroy/unpipe the returned stream to avoid FD leaks.
|
||||||
|
|
||||||
|
7. `AssetsService.getAsset` runs fine as sudo (`new AssetsService({ schema })`) with **no HTTP request
|
||||||
|
context** — this is the basis of our in-process strategy.
|
||||||
|
|
||||||
|
## Preset registry semantics (controller parity)
|
||||||
|
|
||||||
|
The controller builds, inline in `api/src/controllers/assets.ts`:
|
||||||
|
|
||||||
|
```
|
||||||
|
res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]
|
||||||
|
```
|
||||||
|
|
||||||
|
and resolves `?key=` with `.find(t => t.key === key)` → **first match wins** (a system key shadows a
|
||||||
|
custom preset of the same key). There is **no** reusable helper, and `GET /settings` exposes **only**
|
||||||
|
custom presets. Our registry (`src/lib/presets.ts`) must mirror this order exactly.
|
||||||
|
|
||||||
|
## What the admin actually requests (verbatim from app source)
|
||||||
|
|
||||||
|
| Surface | URL |
|
||||||
|
| ------------- | -------------------------------------------------------- |
|
||||||
|
| Files grid | `?key=system-medium-cover&v=<modified_on>` (cards, crop) |
|
||||||
|
| Image field | `?key=system-large-cover` / `system-large-contain` |
|
||||||
|
| Image display | `?key=system-small-cover` |
|
||||||
|
|
||||||
|
No explicit `width`/`height`/`format`. `format:auto` is implicit. `v=` is ignored for hashing.
|
||||||
|
|
||||||
|
## Chosen strategy — IN-PROCESS (no token, no HTTP, no rate limiter)
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const assets = new services.AssetsService({ schema }); // sudo
|
||||||
|
const { stream } = await assets.getAsset(fileId, {
|
||||||
|
transformationParams: presetObject, // FULL object, not { key }
|
||||||
|
acceptFormat: 'avif', // or 'webp'
|
||||||
|
});
|
||||||
|
(stream as Readable).destroy?.();
|
||||||
|
```
|
||||||
|
|
||||||
|
`acceptFormat` reproduces what a browser's `Accept` header does → byte-identical cache filename with
|
||||||
|
no HTTP/token. Warm **both** `avif` and `webp` for `format:'auto'` presets.
|
||||||
|
|
||||||
|
HTTP-replay (`GET /assets/{id}?key=…`) is the documented **fallback** only (needs a token + rate-limit
|
||||||
|
pacing).
|
||||||
|
|
||||||
|
## Config + state
|
||||||
|
|
||||||
|
- Persisted settings + progress state are small JSON files under `<STORAGE_LOCAL_ROOT>/.image-previews`
|
||||||
|
(default `/directus/uploads/.image-previews`; override `IMAGE_PREVIEWS_STATE_DIR`). Do NOT default to
|
||||||
|
`/directus/extensions/...` — that path is a root-owned volume and the API runs as `node` → EACCES.
|
||||||
|
- `state.jsonl` = append-only completed `{file_id,preset_key,format}` → resume.
|
||||||
|
- All state is **file-backed** (PM2 cluster, 8 workers — never in-memory).
|
||||||
|
|
||||||
|
## Gotchas learned the hard way
|
||||||
|
|
||||||
|
- **Directus coerces env values**: `"true"` arrives as boolean `true`, `"1"` as number. Env parsing must
|
||||||
|
accept `unknown` and stringify (see `asString` in `settings.ts`). A string-only `.toLowerCase()` call
|
||||||
|
silently killed the upload hook.
|
||||||
|
- **Docker bind mounts**: never `rm -rf` the bind-mounted extension directory — it breaks the mount and
|
||||||
|
Directus removes the extension without re-adding it. Sync contents in place; restart the dev container.
|
||||||
|
- The dev stack (`dev/compose.dev.yml`) is the only place to test; never load this into prod while
|
||||||
|
iterating (prod has `EXTENSIONS_AUTO_RELOAD=true`).
|
||||||
|
|
||||||
|
|
||||||
|
## Server facts (land-nest)
|
||||||
|
|
||||||
|
- Directus image `directus/directus:11.17.4`; `PUBLIC_URL=https://data.landshaft.life`;
|
||||||
|
`PM2_INSTANCES=8`; `STORAGE_LOCAL_ROOT=./uploads`; logs at `~/apps/directus/logs`.
|
||||||
|
- `ASSETS_TRANSFORM_MAX_CONCURRENT=45` per worker → over it errors; catch per file/preset.
|
||||||
|
- `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000 is a **pixel dimension** guard, not a file-size
|
||||||
|
limit. The extension imposes no limit; raise the Directus env var to process larger images.
|
||||||
|
- Local storage writes are **non-atomic** — warm only after an upload is committed.
|
||||||
|
- `RATE_LIMITER_*` applies to HTTP only → irrelevant to in-process generation (this is why the admin
|
||||||
|
knob is *concurrency*, not a rate limit).
|
||||||
|
- Observed bottleneck (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
|
||||||
|
`pg:responseTime` 150–738ms; **no** 503s, **no** sharp errors. Pre-generation removes the
|
||||||
|
cold-transform cost; uploads-volume I/O latency is a separate ops concern.
|
||||||
|
|
||||||
|
## Safety (do not crash prod)
|
||||||
|
|
||||||
|
- Every entrypoint body is wrapped so thrown errors are logged and swallowed — never propagate into
|
||||||
|
Directus core. ALL behaviour is gated by `enabled` (default **false**): a loaded-but-disabled bundle
|
||||||
|
does nothing.
|
||||||
|
- Test only on the **staging** stack (`dev/compose.dev.yml`), never the prod instance.
|
||||||
|
- Rollout = backup previous `dist/` → deploy → healthcheck → enable; rollback = restore `dist/` + disable.
|
||||||
|
- Automation (autonomous loops) gets **staging only**: a scoped token limited to `/image-previews/*`
|
||||||
|
(+ `directus_files` metadata read), never a prod admin token; never the prod Docker socket; restricted
|
||||||
|
non-sudo SSH or opencode-on-host. Promotion to prod is a human action.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Build via `@directus/extensions-sdk` (`npm run build` / `dev` / `link`); bundle emits `dist/app.js` + `dist/api.js`.
|
||||||
|
- Deploy with `npm run publish` (scp `dist` + `package.json` to land-nest extensions dir).
|
||||||
|
- Tests: `node:test` via `tsx` (`npm test`). Typecheck: `npm run typecheck`.
|
||||||
|
- Do not add comments unless asked. Follow the sibling extensions' style.
|
||||||
|
- Never commit without explicit request.
|
||||||
102
README.md
Normal file
102
README.md
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
# directus-extension-generate-image-previews
|
||||||
|
|
||||||
|
Pre-generate Directus image transform ("preview") cache files so the admin UI and API never pay the
|
||||||
|
first-request sharp cost.
|
||||||
|
|
||||||
|
A Directus **bundle**:
|
||||||
|
|
||||||
|
- **upload hook** — warm previews right after an image upload.
|
||||||
|
- **new-preset trigger** — `Run now` warms every preset (system + custom) for existing images.
|
||||||
|
- **backfill job** — resumable, window-aware sweep over the existing library.
|
||||||
|
- **admin module** — progress, storage breakdown, configuration, diagnostics, and API reference.
|
||||||
|
|
||||||
|
Target: self-hosted Directus **11.17.4** (PM2 cluster, local disk storage).
|
||||||
|
|
||||||
|
## Why this approach (read AGENTS.md too)
|
||||||
|
|
||||||
|
The cache file is `<filename_disk>__<object-hash(resolved transforms)>.<ext>`. The hash depends on the
|
||||||
|
RESOLVED transforms, and every `system-*` preset is `format:'auto'` — which resolves differently
|
||||||
|
server-side (jpg/png) than in a browser (avif). We therefore generate **in-process** with an explicit
|
||||||
|
`acceptFormat` (`avif` + `webp`), reproducing the browser's cache file byte-for-byte without any HTTP
|
||||||
|
request. Naive warming (or the popular community extension) silently misses.
|
||||||
|
|
||||||
|
## Install / build / publish
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
npm run build # emits dist/app.js + dist/api.js
|
||||||
|
npm run typecheck # 0 errors
|
||||||
|
npm test # unit tests
|
||||||
|
npm run dev # watch build
|
||||||
|
npm run publish # scp dist + package.json to the (staging) deploy target
|
||||||
|
```
|
||||||
|
|
||||||
|
## Settings
|
||||||
|
|
||||||
|
Defaults are applied automatically; everything is editable at runtime from the admin module — no rebuild.
|
||||||
|
|
||||||
|
| setting | default | meaning |
|
||||||
|
| -------------------------- | -------------------------------------- | ---------------------------------------------------------------- |
|
||||||
|
| `enabled` | `true` | master switch (off = hooks/backfill inert) |
|
||||||
|
| `onUpload` | `true` | generate previews immediately after an upload |
|
||||||
|
| `windowStart` / `windowEnd` | `01:00` / `04:00` | scheduled runs only start inside this window (local time) |
|
||||||
|
| `concurrency` | `1` | parallel generations |
|
||||||
|
| `formats` | `avif,webp` | output variants **for `format:'auto'` presets only** (all system presets) |
|
||||||
|
| `presetKeys` | `null` (all) | restrict to specific preset keys |
|
||||||
|
| `debug` | `false` | verbose generation logs in the Directus server log |
|
||||||
|
| `IMAGE_PREVIEWS_STATE_DIR` | `<STORAGE_LOCAL_ROOT>/.image-previews` | where settings + progress state live |
|
||||||
|
|
||||||
|
### Presets and formats
|
||||||
|
|
||||||
|
- **System presets** (`system-small|medium|large` × `cover|contain`) always use `format:'auto'`, so they
|
||||||
|
generate one file per entry in `formats` (default avif + webp), at their built-in sizes (64 / 300 / 800 px).
|
||||||
|
- **Custom presets** from Settings → File Storage are added automatically. A custom preset with a fixed
|
||||||
|
`format` (e.g. `png`) ignores `formats` and generates exactly that format. A custom preset with
|
||||||
|
`format:'auto'` follows `formats`.
|
||||||
|
- A custom preset whose `key` equals a system key is **shadowed** (system wins, same as Directus) and is
|
||||||
|
flagged in the admin module.
|
||||||
|
|
||||||
|
## Storage reporting
|
||||||
|
|
||||||
|
`GET /image-previews/disk-usage` (and the admin panel) reports bytes + file counts separately for
|
||||||
|
**preview assets** and **original assets**, plus the total — in both raw bytes and human-readable form.
|
||||||
|
|
||||||
|
## API endpoints
|
||||||
|
|
||||||
|
All require an admin session or admin token. Human-readable fields accompany machine-readable ones.
|
||||||
|
|
||||||
|
- `GET /image-previews/status` — run progress, rate, ETA, window state, ISO + human dates
|
||||||
|
- `GET /image-previews/config`, `PUT /image-previews/config`
|
||||||
|
- `GET /image-previews/presets` — system + custom presets, effective formats, shadowed keys
|
||||||
|
- `POST /image-previews/run` — start backfill (`{"ignoreWindow":true}` for an immediate run)
|
||||||
|
- `POST /image-previews/run/cancel`
|
||||||
|
- `GET /image-previews/disk-usage` — previews vs originals vs total
|
||||||
|
- `POST /image-previews/generate` — one file (`{fileId, keys?, formats?, dryRun?}`)
|
||||||
|
- `GET /image-previews/resolve-plan?fileId=&key=` — predicted cache filename, no write
|
||||||
|
- `GET /image-previews/ops?limit=` — recent operations with timing + error messages
|
||||||
|
|
||||||
|
The admin module includes a full API reference with curl examples.
|
||||||
|
|
||||||
|
## Safe testing & deployment
|
||||||
|
|
||||||
|
Never test against the prod instance.
|
||||||
|
|
||||||
|
- **Local / staging** (`dev/compose.dev.yml`): Directus 11.17.4 + its own postgres + volumes, on separate
|
||||||
|
ports. Shares nothing with prod. See `dev/README.md`.
|
||||||
|
- **Fail-open**: entrypoints never throw into Directus core.
|
||||||
|
- **Rollout**: build → verify on staging → back up previous `dist/` → deploy → healthcheck → enable.
|
||||||
|
**Rollback**: restore previous `dist/`.
|
||||||
|
- Scripts: `dev/deploy-staging.sh`, `dev/rollback-staging.sh`, template `dev/compose.staging.yml`.
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- **Transform dimension guard, not a file-size limit.** Directus rejects transformations of images whose
|
||||||
|
width or height exceeds `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` (default **6000 px**). This is a Directus
|
||||||
|
env var, not something this extension imposes. To process larger images, raise it on the Directus
|
||||||
|
instance (and watch memory). Upload size limits (`FILES_MAX_UPLOAD_SIZE`, `MAX_PAYLOAD_SIZE`) are separate.
|
||||||
|
- Warm only after uploads are committed (local writes are non-atomic).
|
||||||
|
- In-process generation bypasses the HTTP rate limiter; bound it with `concurrency` + the window.
|
||||||
|
- Unsupported custom preset formats (e.g. `bmp`) fail at generation time; the error appears in
|
||||||
|
Recent operations / `/ops`.
|
||||||
|
|
||||||
|
See `.opencode/goal.md` for the full contract and `.opencode/feature_list.json` for the task list.
|
||||||
48
dev/README.md
Normal file
48
dev/README.md
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
# Local dev / staging stack
|
||||||
|
|
||||||
|
Isolated Directus **11.17.4** with its own Postgres and volumes. Nothing is shared with the prod
|
||||||
|
instance (separate containers, volumes, and port **8057**).
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from the project root
|
||||||
|
./dev/sync-extension.sh # build + copy dist/package.json into dev/extension
|
||||||
|
docker compose -f dev/compose.dev.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Admin: `http://localhost:8057` — `admin@example.com` / `admin12345`.
|
||||||
|
The extension is loaded from `dev/extension` (bind mount). Check:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker logs image-previews-dev-directus-1 | grep -i "loaded extensions"
|
||||||
|
```
|
||||||
|
|
||||||
|
## After changing source
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./dev/sync-extension.sh
|
||||||
|
docker compose -f dev/compose.dev.yml restart directus
|
||||||
|
```
|
||||||
|
|
||||||
|
(`EXTENSIONS_AUTO_RELOAD` is on, but a restart is the reliable path. Never `rm -rf dev/extension` —
|
||||||
|
it breaks the bind mount.)
|
||||||
|
|
||||||
|
## Smoke test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
BASE=http://localhost:8057
|
||||||
|
TOKEN=$(curl -s -X POST "$BASE/auth/login" -H 'Content-Type: application/json' \
|
||||||
|
-d '{"email":"admin@example.com","password":"admin12345"}' | jq -r '.data.access_token')
|
||||||
|
|
||||||
|
curl -s "$BASE/image-previews/status" -H "Authorization: Bearer $TOKEN" | jq .
|
||||||
|
curl -s -X POST "$BASE/files" -H "Authorization: Bearer $TOKEN" -F "file=@dev/seed/test.jpg" | jq '.data | {id,filename_disk}'
|
||||||
|
docker exec image-previews-dev-directus-1 sh -c "ls -1 /directus/uploads | grep '__' | sort"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stop / reset
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f dev/compose.dev.yml down # stop, keep data
|
||||||
|
docker compose -f dev/compose.dev.yml down -v # stop and wipe DB + uploads
|
||||||
|
```
|
||||||
51
dev/compose.dev.yml
Normal file
51
dev/compose.dev.yml
Normal file
|
|
@ -0,0 +1,51 @@
|
||||||
|
name: image-previews-dev
|
||||||
|
|
||||||
|
services:
|
||||||
|
db:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: directus
|
||||||
|
POSTGRES_PASSWORD: directus
|
||||||
|
POSTGRES_DB: directus
|
||||||
|
volumes:
|
||||||
|
- db-data:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U directus"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
|
||||||
|
directus:
|
||||||
|
image: directus/directus:11.17.4
|
||||||
|
ports:
|
||||||
|
- "8057:8055"
|
||||||
|
depends_on:
|
||||||
|
db:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
KEY: dev-key
|
||||||
|
SECRET: dev-secret
|
||||||
|
ADMIN_EMAIL: admin@example.com
|
||||||
|
ADMIN_PASSWORD: admin12345
|
||||||
|
DB_CLIENT: pg
|
||||||
|
DB_HOST: db
|
||||||
|
DB_PORT: "5432"
|
||||||
|
DB_DATABASE: directus
|
||||||
|
DB_USER: directus
|
||||||
|
DB_PASSWORD: directus
|
||||||
|
STORAGE_LOCATIONS: local
|
||||||
|
STORAGE_LOCAL_ROOT: /directus/uploads
|
||||||
|
EXTENSIONS_AUTO_RELOAD: "true"
|
||||||
|
CACHE_ENABLED: "false"
|
||||||
|
RATE_LIMITER_ENABLED: "false"
|
||||||
|
ASSETS_TRANSFORM_MAX_CONCURRENT: "2"
|
||||||
|
IMAGE_PREVIEWS_ENABLED: "true"
|
||||||
|
IMAGE_PREVIEWS_CONCURRENCY: "1"
|
||||||
|
LOG_LEVEL: info
|
||||||
|
volumes:
|
||||||
|
- uploads:/directus/uploads
|
||||||
|
- ./extension:/directus/extensions/directus-extension-generate-image-previews
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
db-data:
|
||||||
|
uploads:
|
||||||
58
dev/compose.staging.yml
Normal file
58
dev/compose.staging.yml
Normal file
|
|
@ -0,0 +1,58 @@
|
||||||
|
# Staging stack template (run on the staging host only — never on prod).
|
||||||
|
#
|
||||||
|
# 1. Restore an anonymized DB dump into the `db` service.
|
||||||
|
# 2. Copy (or rsync) a subset of prod uploads into the `uploads` volume.
|
||||||
|
# 3. Deploy the extension with dev/deploy-staging.sh, then start this stack.
|
||||||
|
#
|
||||||
|
# Keep ports/volumes/network distinct from the prod stack.
|
||||||
|
|
||||||
|
name: land-nest-staging
|
||||||
|
|
||||||
|
services:
|
||||||
|
db:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: directus
|
||||||
|
POSTGRES_PASSWORD: change-me
|
||||||
|
POSTGRES_DB: directus
|
||||||
|
volumes:
|
||||||
|
- db-data:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U directus"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
|
||||||
|
directus:
|
||||||
|
image: directus/directus:11.17.4
|
||||||
|
ports:
|
||||||
|
- "8056:8055"
|
||||||
|
depends_on:
|
||||||
|
db:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
KEY: change-me
|
||||||
|
SECRET: change-me-to-at-least-32-bytes-long
|
||||||
|
ADMIN_EMAIL: staging@example.com
|
||||||
|
ADMIN_PASSWORD: change-me
|
||||||
|
DB_CLIENT: pg
|
||||||
|
DB_HOST: db
|
||||||
|
DB_PORT: "5432"
|
||||||
|
DB_DATABASE: directus
|
||||||
|
DB_USER: directus
|
||||||
|
DB_PASSWORD: change-me
|
||||||
|
STORAGE_LOCATIONS: local
|
||||||
|
STORAGE_LOCAL_ROOT: /directus/uploads
|
||||||
|
EXTENSIONS_AUTO_RELOAD: "true"
|
||||||
|
CACHE_ENABLED: "false"
|
||||||
|
RATE_LIMITER_ENABLED: "false"
|
||||||
|
IMAGE_PREVIEWS_ENABLED: "true"
|
||||||
|
IMAGE_PREVIEWS_CONCURRENCY: "1"
|
||||||
|
LOG_LEVEL: info
|
||||||
|
volumes:
|
||||||
|
- uploads:/directus/uploads
|
||||||
|
- ./extensions:/directus/extensions
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
db-data:
|
||||||
|
uploads:
|
||||||
21
dev/deploy-staging.sh
Executable file
21
dev/deploy-staging.sh
Executable file
|
|
@ -0,0 +1,21 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Deploy the built extension to a STAGING Directus extensions dir. Never target prod.
|
||||||
|
# Usage: ./dev/deploy-staging.sh <ssh-host> <staging-extensions-root>
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HOST="${1:?usage: deploy-staging.sh <ssh-host> <staging-extensions-root>}"
|
||||||
|
EXT_ROOT="${2:?usage: deploy-staging.sh <ssh-host> <staging-extensions-root>}"
|
||||||
|
NAME="directus-extension-generate-image-previews"
|
||||||
|
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||||
|
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
ssh "$HOST" "mkdir -p '$EXT_ROOT/$NAME' && if [ -d '$EXT_ROOT/$NAME/dist' ]; then cp -a '$EXT_ROOT/$NAME/dist' '$EXT_ROOT/$NAME/dist.bak-$STAMP'; fi"
|
||||||
|
|
||||||
|
rsync -az --delete dist/ "$HOST:$EXT_ROOT/$NAME/dist/"
|
||||||
|
scp package.json "$HOST:$EXT_ROOT/$NAME/package.json"
|
||||||
|
|
||||||
|
echo "deployed to $HOST:$EXT_ROOT/$NAME (backup stamp: $STAMP)"
|
||||||
|
echo "restart staging Directus, then check: docker logs <staging-directus> | grep -i 'loaded extensions'"
|
||||||
13
dev/rollback-staging.sh
Executable file
13
dev/rollback-staging.sh
Executable file
|
|
@ -0,0 +1,13 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Roll back a staging deploy to a previous dist backup.
|
||||||
|
# Usage: ./dev/rollback-staging.sh <ssh-host> <staging-extensions-root> <backup-stamp>
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HOST="${1:?usage: rollback-staging.sh <ssh-host> <staging-extensions-root> <backup-stamp>}"
|
||||||
|
EXT_ROOT="${2:?usage: rollback-staging.sh <ssh-host> <staging-extensions-root> <backup-stamp>}"
|
||||||
|
STAMP="${3:?usage: rollback-staging.sh <ssh-host> <staging-extensions-root> <backup-stamp>}"
|
||||||
|
NAME="directus-extension-generate-image-previews"
|
||||||
|
|
||||||
|
ssh "$HOST" "test -d '$EXT_ROOT/$NAME/dist.bak-$STAMP' && rm -rf '$EXT_ROOT/$NAME/dist' && cp -a '$EXT_ROOT/$NAME/dist.bak-$STAMP' '$EXT_ROOT/$NAME/dist'"
|
||||||
|
|
||||||
|
echo "rolled back $HOST:$EXT_ROOT/$NAME to dist.bak-$STAMP"
|
||||||
BIN
dev/seed/test-large.jpg
Normal file
BIN
dev/seed/test-large.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 498 KiB |
BIN
dev/seed/test.jpg
Normal file
BIN
dev/seed/test.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
13
dev/sync-extension.sh
Executable file
13
dev/sync-extension.sh
Executable file
|
|
@ -0,0 +1,13 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
mkdir -p dev/extension
|
||||||
|
rm -rf dev/extension/dist
|
||||||
|
cp -r dist dev/extension/dist
|
||||||
|
cp package.json dev/extension/package.json
|
||||||
|
|
||||||
|
echo "synced dev/extension (in place)"
|
||||||
6346
package-lock.json
generated
Normal file
6346
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load diff
63
package.json
Normal file
63
package.json
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
{
|
||||||
|
"name": "directus-extension-generate-image-previews",
|
||||||
|
"description": "Pre-generate Directus image transforms: on upload, on demand, and via a scheduled resumable backfill, with an admin UI (progress, disk usage, maintenance window)",
|
||||||
|
"author": {
|
||||||
|
"name": "Rawman Grey",
|
||||||
|
"email": "rawman@locus.im",
|
||||||
|
"url": "https://locus.im"
|
||||||
|
},
|
||||||
|
"icon": "extension",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"keywords": [
|
||||||
|
"directus",
|
||||||
|
"directus-extension",
|
||||||
|
"directus-extension-bundle"
|
||||||
|
],
|
||||||
|
"type": "module",
|
||||||
|
"files": [
|
||||||
|
"dist"
|
||||||
|
],
|
||||||
|
"directus:extension": {
|
||||||
|
"type": "bundle",
|
||||||
|
"path": {
|
||||||
|
"app": "dist/app.js",
|
||||||
|
"api": "dist/api.js"
|
||||||
|
},
|
||||||
|
"entries": [
|
||||||
|
{
|
||||||
|
"type": "module",
|
||||||
|
"name": "image-previews",
|
||||||
|
"source": "src/module/index.ts"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "hook",
|
||||||
|
"name": "image-previews-hook",
|
||||||
|
"source": "src/hook/index.ts"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "endpoint",
|
||||||
|
"name": "image-previews-endpoint",
|
||||||
|
"source": "src/endpoint/index.ts"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"host": "^10.10.0 || ^11.0.0"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"build": "directus-extension build",
|
||||||
|
"dev": "directus-extension build -w --no-minify",
|
||||||
|
"link": "directus-extension link",
|
||||||
|
"typecheck": "tsc --noEmit",
|
||||||
|
"test": "node --import tsx --test src/lib/*.test.ts",
|
||||||
|
"publish": "scp -r ./dist ./package.json land-nest:/home/oak/apps/directus/extensions/directus-extension-generate-image-previews/"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@directus/extensions-sdk": "13.0.1",
|
||||||
|
"@types/node": "^22.13.5",
|
||||||
|
"tsx": "^4.19.2",
|
||||||
|
"typescript": "^5.7.3",
|
||||||
|
"vue": "^3.5.13"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"object-hash": "^3.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
312
src/endpoint/index.ts
Normal file
312
src/endpoint/index.ts
Normal file
|
|
@ -0,0 +1,312 @@
|
||||||
|
import { defineEndpoint } from '@directus/extensions-sdk';
|
||||||
|
import type { Request, Response, NextFunction } from 'express';
|
||||||
|
import { runBackfill, requestCancel } from '../lib/backfill';
|
||||||
|
import { computePlan, getOps, recordOp } from '../lib/diagnostics';
|
||||||
|
import { humanBytes, humanDuration, humanMs, toIso } from '../lib/format';
|
||||||
|
import { buildRegistry, effectiveFormatsForPreset, formatsForPreset, loadCustomPresets } from '../lib/presets';
|
||||||
|
import { mergeSettings, readSettings, writeSettings } from '../lib/settings';
|
||||||
|
import { isLocked, readRun } from '../lib/state';
|
||||||
|
import { previewExists, storageUsage } from '../lib/storage';
|
||||||
|
import { SYSTEM_PRESETS } from '../lib/system-presets';
|
||||||
|
import { isInWindow, nextWindow } from '../lib/window';
|
||||||
|
import { warmFile } from '../lib/warmer';
|
||||||
|
|
||||||
|
const FILE_FIELDS = [
|
||||||
|
'id',
|
||||||
|
'filename_disk',
|
||||||
|
'type',
|
||||||
|
'width',
|
||||||
|
'height',
|
||||||
|
'focal_point_x',
|
||||||
|
'focal_point_y',
|
||||||
|
'storage',
|
||||||
|
];
|
||||||
|
|
||||||
|
function asyncHandler(handler: (req: Request, res: Response) => Promise<void>) {
|
||||||
|
return (req: Request, res: Response, next: NextFunction) => {
|
||||||
|
handler(req, res).catch(next);
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function bucket(bytes: number, count: number) {
|
||||||
|
return { bytes, count, bytesHuman: humanBytes(bytes) };
|
||||||
|
}
|
||||||
|
|
||||||
|
export default defineEndpoint({
|
||||||
|
id: 'image-previews',
|
||||||
|
handler: (router, context) => {
|
||||||
|
const { services, database, getSchema, env, logger } = context;
|
||||||
|
|
||||||
|
router.use((req: Request, res: Response, next: NextFunction) => {
|
||||||
|
const accountability = (req as Request & { accountability?: { admin?: boolean } }).accountability;
|
||||||
|
|
||||||
|
if (accountability?.admin === true) return next();
|
||||||
|
|
||||||
|
res.status(403).json({ error: 'admin access required' });
|
||||||
|
});
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/status',
|
||||||
|
asyncHandler(async (_req, res) => {
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
const run = await readRun(env);
|
||||||
|
const next = nextWindow(settings.windowStart, settings.windowEnd);
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
enabled: settings.enabled,
|
||||||
|
window: { start: settings.windowStart, end: settings.windowEnd },
|
||||||
|
inWindow: isInWindow(settings.windowStart, settings.windowEnd),
|
||||||
|
nextWindow: next?.toISOString() ?? null,
|
||||||
|
nextWindowHuman: next?.toLocaleString() ?? null,
|
||||||
|
locked: await isLocked(env),
|
||||||
|
run: run
|
||||||
|
? {
|
||||||
|
...run,
|
||||||
|
startedAtIso: toIso(run.startedAt),
|
||||||
|
finishedAtIso: toIso(run.finishedAt),
|
||||||
|
startedAtHuman: run.startedAt ? new Date(run.startedAt).toLocaleString() : null,
|
||||||
|
finishedAtHuman: run.finishedAt ? new Date(run.finishedAt).toLocaleString() : null,
|
||||||
|
etaHuman: humanDuration(run.eta),
|
||||||
|
rateHuman: `${Math.round(run.rate)} items/min`,
|
||||||
|
}
|
||||||
|
: null,
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/config',
|
||||||
|
asyncHandler(async (_req, res) => {
|
||||||
|
res.json(await readSettings(env));
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.put(
|
||||||
|
'/config',
|
||||||
|
asyncHandler(async (req, res) => {
|
||||||
|
try {
|
||||||
|
const current = await readSettings(env);
|
||||||
|
const merged = mergeSettings(current, req.body ?? {});
|
||||||
|
const saved = await writeSettings(merged, env);
|
||||||
|
res.json(saved);
|
||||||
|
} catch (error) {
|
||||||
|
res.status(400).json({ error: error instanceof Error ? error.message : String(error) });
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/presets',
|
||||||
|
asyncHandler(async (_req, res) => {
|
||||||
|
const schema = await getSchema();
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
const custom = await loadCustomPresets(new services['SettingsService']({ schema }));
|
||||||
|
const presets = buildRegistry(custom, null);
|
||||||
|
const systemKeys = new Set(SYSTEM_PRESETS.map((preset) => preset.key));
|
||||||
|
const shadowedKeys = custom
|
||||||
|
.filter((preset) => systemKeys.has(preset.key))
|
||||||
|
.map((preset) => preset.key);
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
shadowedKeys,
|
||||||
|
presets: presets.map((preset) => ({
|
||||||
|
key: preset.key,
|
||||||
|
format: preset.format ?? null,
|
||||||
|
effectiveFormats: effectiveFormatsForPreset(preset, settings.formats),
|
||||||
|
system: systemKeys.has(preset.key),
|
||||||
|
enabled: settings.presetKeys === null || settings.presetKeys.includes(preset.key),
|
||||||
|
})),
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.post(
|
||||||
|
'/run',
|
||||||
|
asyncHandler(async (req, res) => {
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
const ignoreWindow = req.body?.ignoreWindow === true;
|
||||||
|
|
||||||
|
if (!settings.enabled) {
|
||||||
|
res.status(409).json({ error: 'disabled' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!ignoreWindow && !isInWindow(settings.windowStart, settings.windowEnd)) {
|
||||||
|
const next = nextWindow(settings.windowStart, settings.windowEnd);
|
||||||
|
res.status(409).json({
|
||||||
|
error: 'outside-window',
|
||||||
|
nextWindow: next?.toISOString() ?? null,
|
||||||
|
nextWindowHuman: next?.toLocaleString() ?? null,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (await isLocked(env)) {
|
||||||
|
res.status(409).json({ error: 'already-running' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const presetKeys = Array.isArray(req.body?.presets) ? (req.body.presets as string[]) : undefined;
|
||||||
|
|
||||||
|
void runBackfill(
|
||||||
|
{ services, database, getSchema, env, logger },
|
||||||
|
{ presetKeys, ignoreWindow },
|
||||||
|
).catch((error) => logger.error(error, 'image-previews: run failed'));
|
||||||
|
|
||||||
|
res.status(202).json({ started: true });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.post(
|
||||||
|
'/run/cancel',
|
||||||
|
asyncHandler(async (_req, res) => {
|
||||||
|
await requestCancel(env);
|
||||||
|
res.json({ cancelling: true });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/disk-usage',
|
||||||
|
asyncHandler(async (_req, res) => {
|
||||||
|
const usage = await storageUsage(env);
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
previews: bucket(usage.previews.bytes, usage.previews.count),
|
||||||
|
originals: bucket(usage.originals.bytes, usage.originals.count),
|
||||||
|
total: bucket(usage.total.bytes, usage.total.count),
|
||||||
|
source: usage.source,
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/ops',
|
||||||
|
asyncHandler(async (req, res) => {
|
||||||
|
const limit = Number(req.query.limit) || 50;
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
ops: getOps(limit).map((op) => ({
|
||||||
|
...op,
|
||||||
|
tsIso: toIso(op.ts),
|
||||||
|
tsHuman: new Date(op.ts).toLocaleString(),
|
||||||
|
msHuman: humanMs(op.ms),
|
||||||
|
})),
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.post(
|
||||||
|
'/generate',
|
||||||
|
asyncHandler(async (req, res) => {
|
||||||
|
const fileId = req.body?.fileId;
|
||||||
|
|
||||||
|
if (typeof fileId !== 'string' || fileId.length === 0) {
|
||||||
|
res.status(400).json({ error: 'fileId is required' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
const schema = await getSchema();
|
||||||
|
const custom = await loadCustomPresets(new services['SettingsService']({ schema }));
|
||||||
|
const presetKeys = Array.isArray(req.body?.keys) ? (req.body.keys as string[]) : settings.presetKeys;
|
||||||
|
const formats = Array.isArray(req.body?.formats) ? (req.body.formats as string[]) : settings.formats;
|
||||||
|
const presets = buildRegistry(custom, presetKeys);
|
||||||
|
|
||||||
|
const items = new services['ItemsService']('directus_files', { schema });
|
||||||
|
|
||||||
|
if (req.body?.dryRun === true) {
|
||||||
|
const file = await items.readOne(fileId, { fields: FILE_FIELDS });
|
||||||
|
const plans = [];
|
||||||
|
|
||||||
|
for (const preset of presets) {
|
||||||
|
for (const format of formatsForPreset(preset, formats)) {
|
||||||
|
const plan = await computePlan(file, preset, format);
|
||||||
|
plans.push({
|
||||||
|
key: preset.key,
|
||||||
|
format,
|
||||||
|
effectiveFormat: plan.format,
|
||||||
|
filename: plan.filename,
|
||||||
|
exists: await previewExists(env, file.storage ?? null, plan.filename),
|
||||||
|
transforms: plan.transforms,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
res.json({ dryRun: true, fileId, plans });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const assets = new services['AssetsService']({ schema });
|
||||||
|
const started = Date.now();
|
||||||
|
const summary = await warmFile(assets, fileId, presets, {
|
||||||
|
concurrency: settings.concurrency,
|
||||||
|
formats,
|
||||||
|
onResult: (result) =>
|
||||||
|
recordOp({
|
||||||
|
ts: Date.now(),
|
||||||
|
fileId,
|
||||||
|
key: result.key,
|
||||||
|
format: result.format,
|
||||||
|
ok: result.ok,
|
||||||
|
ms: result.ms,
|
||||||
|
error: result.error,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
fileId,
|
||||||
|
ms: Date.now() - started,
|
||||||
|
msHuman: humanMs(Date.now() - started),
|
||||||
|
...summary,
|
||||||
|
results: summary.results.map((result) => ({ ...result, msHuman: humanMs(result.ms) })),
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
router.get(
|
||||||
|
'/resolve-plan',
|
||||||
|
asyncHandler(async (req, res) => {
|
||||||
|
const fileId = typeof req.query.fileId === 'string' ? req.query.fileId : '';
|
||||||
|
const key = typeof req.query.key === 'string' ? req.query.key : '';
|
||||||
|
|
||||||
|
if (!fileId || !key) {
|
||||||
|
res.status(400).json({ error: 'fileId and key are required' });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
const schema = await getSchema();
|
||||||
|
const custom = await loadCustomPresets(new services['SettingsService']({ schema }));
|
||||||
|
const presets = buildRegistry(custom, null);
|
||||||
|
const preset = presets.find((candidate) => candidate.key === key);
|
||||||
|
|
||||||
|
if (!preset) {
|
||||||
|
const systemKeys = SYSTEM_PRESETS.map((candidate) => candidate.key);
|
||||||
|
const shadowed = custom.some((candidate) => candidate.key === key);
|
||||||
|
res.status(404).json({
|
||||||
|
error: `unknown preset: ${key}`,
|
||||||
|
reason: shadowed && systemKeys.includes(key) ? 'shadowed-by-system' : 'not-found',
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const items = new services['ItemsService']('directus_files', { schema });
|
||||||
|
const file = await items.readOne(fileId, { fields: FILE_FIELDS });
|
||||||
|
const plans = [];
|
||||||
|
|
||||||
|
for (const format of formatsForPreset(preset, settings.formats)) {
|
||||||
|
const plan = await computePlan(file, preset, format);
|
||||||
|
plans.push({
|
||||||
|
format,
|
||||||
|
effectiveFormat: plan.format,
|
||||||
|
filename: plan.filename,
|
||||||
|
exists: await previewExists(env, file.storage ?? null, plan.filename),
|
||||||
|
transforms: plan.transforms,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
res.json({ fileId, key, plans });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
},
|
||||||
|
});
|
||||||
56
src/hook/index.ts
Normal file
56
src/hook/index.ts
Normal file
|
|
@ -0,0 +1,56 @@
|
||||||
|
import { defineHook } from '@directus/extensions-sdk';
|
||||||
|
import { recordOp } from '../lib/diagnostics';
|
||||||
|
import { buildRegistry, isImageType, loadCustomPresets } from '../lib/presets';
|
||||||
|
import { readSettings } from '../lib/settings';
|
||||||
|
import { warmFile } from '../lib/warmer';
|
||||||
|
|
||||||
|
interface UploadMeta {
|
||||||
|
key?: string;
|
||||||
|
collection?: string;
|
||||||
|
payload?: { type?: string | null; collection?: string };
|
||||||
|
}
|
||||||
|
|
||||||
|
export default defineHook(({ action }, { services, getSchema, env, logger }) => {
|
||||||
|
action('files.upload', (meta) => {
|
||||||
|
void warmUpload(meta as UploadMeta).catch((error) =>
|
||||||
|
logger.error(error, 'image-previews: upload warm failed'),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
async function warmUpload(meta: UploadMeta): Promise<void> {
|
||||||
|
const settings = await readSettings(env);
|
||||||
|
|
||||||
|
if (!settings.enabled || !settings.onUpload) return;
|
||||||
|
|
||||||
|
const fileId = meta?.key;
|
||||||
|
const payload = meta?.payload;
|
||||||
|
|
||||||
|
if (!fileId || !payload) return;
|
||||||
|
if (payload.collection && payload.collection !== 'directus_files') return;
|
||||||
|
if (!isImageType(payload.type)) return;
|
||||||
|
|
||||||
|
const schema = await getSchema();
|
||||||
|
const custom = await loadCustomPresets(new services['SettingsService']({ schema }));
|
||||||
|
const presets = buildRegistry(custom, settings.presetKeys);
|
||||||
|
|
||||||
|
if (presets.length === 0) return;
|
||||||
|
|
||||||
|
const assets = new services['AssetsService']({ schema });
|
||||||
|
const summary = await warmFile(assets, fileId, presets, {
|
||||||
|
concurrency: settings.concurrency,
|
||||||
|
formats: settings.formats,
|
||||||
|
onResult: (result) =>
|
||||||
|
recordOp({
|
||||||
|
ts: Date.now(),
|
||||||
|
fileId,
|
||||||
|
key: result.key,
|
||||||
|
format: result.format,
|
||||||
|
ok: result.ok,
|
||||||
|
ms: result.ms,
|
||||||
|
error: result.error,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
if (settings.debug) logger.info({ fileId, summary }, 'image-previews: warmed upload');
|
||||||
|
}
|
||||||
|
});
|
||||||
206
src/lib/backfill.ts
Normal file
206
src/lib/backfill.ts
Normal file
|
|
@ -0,0 +1,206 @@
|
||||||
|
import { recordOp } from './diagnostics';
|
||||||
|
import { buildRegistry, effectiveFormatsForPreset, isImageType, loadCustomPresets, tasksPerFile } from './presets';
|
||||||
|
import { readSettings } from './settings';
|
||||||
|
import {
|
||||||
|
acquireLock,
|
||||||
|
appendCompleted,
|
||||||
|
clearCancel,
|
||||||
|
completedKeyOf,
|
||||||
|
isCancelled,
|
||||||
|
loadCompleted,
|
||||||
|
readRun,
|
||||||
|
releaseLock,
|
||||||
|
requestCancel,
|
||||||
|
writeRun,
|
||||||
|
} from './state';
|
||||||
|
import type { RunState } from './types';
|
||||||
|
import { errorMessage, warmFile } from './warmer';
|
||||||
|
import { isInWindow } from './window';
|
||||||
|
|
||||||
|
export interface BackfillDeps {
|
||||||
|
services: Record<string, any>;
|
||||||
|
database: any;
|
||||||
|
getSchema: () => Promise<any>;
|
||||||
|
env: Record<string, any>;
|
||||||
|
logger: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BackfillOptions {
|
||||||
|
presetKeys?: string[] | null;
|
||||||
|
ignoreWindow?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BackfillOutcome {
|
||||||
|
status: 'completed' | 'skipped' | 'failed';
|
||||||
|
reason?: string;
|
||||||
|
run?: RunState;
|
||||||
|
}
|
||||||
|
|
||||||
|
const PAGE_SIZE = 100;
|
||||||
|
|
||||||
|
let inProcessRunning = false;
|
||||||
|
|
||||||
|
async function countImageFiles(database: any): Promise<number> {
|
||||||
|
try {
|
||||||
|
const rows = await database('directus_files').where('type', 'like', 'image/%').count({ count: '*' });
|
||||||
|
const row = Array.isArray(rows) ? rows[0] : rows;
|
||||||
|
return Number(row?.count) || 0;
|
||||||
|
} catch {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runBackfill(deps: BackfillDeps, options: BackfillOptions = {}): Promise<BackfillOutcome> {
|
||||||
|
if (inProcessRunning) return { status: 'skipped', reason: 'already-running' };
|
||||||
|
|
||||||
|
const settings = await readSettings(deps.env);
|
||||||
|
|
||||||
|
if (!settings.enabled) return { status: 'skipped', reason: 'disabled' };
|
||||||
|
|
||||||
|
if (!options.ignoreWindow && !isInWindow(settings.windowStart, settings.windowEnd)) {
|
||||||
|
return { status: 'skipped', reason: 'outside-window' };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!(await acquireLock(deps.env))) return { status: 'skipped', reason: 'already-running' };
|
||||||
|
|
||||||
|
inProcessRunning = true;
|
||||||
|
await clearCancel(deps.env);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const schema = await deps.getSchema();
|
||||||
|
const settingsService = new deps.services['SettingsService']({ schema });
|
||||||
|
const custom = await loadCustomPresets(settingsService);
|
||||||
|
const presetKeys = options.presetKeys !== undefined ? options.presetKeys : settings.presetKeys;
|
||||||
|
const presets = buildRegistry(custom, presetKeys);
|
||||||
|
const formats = settings.formats;
|
||||||
|
const perFile = tasksPerFile(presets, formats);
|
||||||
|
const totalFiles = await countImageFiles(deps.database);
|
||||||
|
|
||||||
|
const run: RunState = {
|
||||||
|
running: true,
|
||||||
|
startedAt: Date.now(),
|
||||||
|
finishedAt: null,
|
||||||
|
total: totalFiles * perFile,
|
||||||
|
done: 0,
|
||||||
|
generated: 0,
|
||||||
|
hits: 0,
|
||||||
|
failed: 0,
|
||||||
|
rate: 0,
|
||||||
|
eta: null,
|
||||||
|
cancelled: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
await writeRun(run, deps.env);
|
||||||
|
|
||||||
|
const completed = await loadCompleted(deps.env);
|
||||||
|
const items = new deps.services['ItemsService']('directus_files', { schema });
|
||||||
|
const assets = new deps.services['AssetsService']({ schema });
|
||||||
|
const started = Date.now();
|
||||||
|
let page = 1;
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
const files = await items.readByQuery({
|
||||||
|
filter: { type: { _starts_with: 'image/' } },
|
||||||
|
fields: [ 'id', 'filename_disk', 'type', 'width', 'height', 'focal_point_x', 'focal_point_y', 'storage' ],
|
||||||
|
limit: PAGE_SIZE,
|
||||||
|
page,
|
||||||
|
sort: [ 'id' ],
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!Array.isArray(files) || files.length === 0) break;
|
||||||
|
|
||||||
|
for (const file of files) {
|
||||||
|
if (await isCancelled(deps.env)) {
|
||||||
|
run.cancelled = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!isImageType(file.type)) continue;
|
||||||
|
|
||||||
|
const pending = presets.filter((preset) =>
|
||||||
|
effectiveFormatsForPreset(preset, formats).some(
|
||||||
|
(format) => !completed.has(completedKeyOf(file.id, preset.key, format)),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
const pendingTasks = pending.reduce(
|
||||||
|
(total, preset) => total + effectiveFormatsForPreset(preset, formats).length,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const alreadyDone = perFile - pendingTasks;
|
||||||
|
|
||||||
|
if (alreadyDone > 0) {
|
||||||
|
run.hits += alreadyDone;
|
||||||
|
run.done += alreadyDone;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (pending.length === 0) {
|
||||||
|
await writeRun(run, deps.env);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const summary = await warmFile(assets, file.id, pending, {
|
||||||
|
concurrency: settings.concurrency,
|
||||||
|
formats,
|
||||||
|
onResult: (result) =>
|
||||||
|
recordOp({
|
||||||
|
ts: Date.now(),
|
||||||
|
fileId: file.id,
|
||||||
|
key: result.key,
|
||||||
|
format: result.format,
|
||||||
|
ok: result.ok,
|
||||||
|
ms: result.ms,
|
||||||
|
error: result.error,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
for (const result of summary.results) {
|
||||||
|
if (result.ok) {
|
||||||
|
completed.add(completedKeyOf(file.id, result.key, result.format));
|
||||||
|
await appendCompleted(
|
||||||
|
{ file_id: file.id, preset_key: result.key, format: result.format, ts: Date.now() },
|
||||||
|
deps.env,
|
||||||
|
);
|
||||||
|
run.generated += 1;
|
||||||
|
} else {
|
||||||
|
run.failed += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
run.done += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const elapsedMinutes = (Date.now() - started) / 60000;
|
||||||
|
run.rate = elapsedMinutes > 0 ? run.done / elapsedMinutes : 0;
|
||||||
|
run.eta = run.rate > 0 ? Math.round(((run.total - run.done) / run.rate) * 60) : null;
|
||||||
|
await writeRun(run, deps.env);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (run.cancelled || files.length < PAGE_SIZE) break;
|
||||||
|
page += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
run.running = false;
|
||||||
|
run.finishedAt = Date.now();
|
||||||
|
await writeRun(run, deps.env);
|
||||||
|
|
||||||
|
return { status: 'completed', run };
|
||||||
|
} catch (error) {
|
||||||
|
deps.logger?.error?.(error, 'image-previews: backfill failed');
|
||||||
|
const existing = await readRun(deps.env);
|
||||||
|
|
||||||
|
if (existing) {
|
||||||
|
existing.running = false;
|
||||||
|
existing.finishedAt = Date.now();
|
||||||
|
existing.error = errorMessage(error);
|
||||||
|
await writeRun(existing, deps.env);
|
||||||
|
}
|
||||||
|
|
||||||
|
return { status: 'failed', reason: errorMessage(error) };
|
||||||
|
} finally {
|
||||||
|
inProcessRunning = false;
|
||||||
|
await releaseLock(deps.env);
|
||||||
|
await clearCancel(deps.env);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export { requestCancel };
|
||||||
77
src/lib/diagnostics.test.ts
Normal file
77
src/lib/diagnostics.test.ts
Normal file
|
|
@ -0,0 +1,77 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { computePlan, getOps, recordOp, type PlanFile } from './diagnostics';
|
||||||
|
import type { Preset } from './types';
|
||||||
|
|
||||||
|
const file: PlanFile = {
|
||||||
|
filename_disk: 'abc123.jpg',
|
||||||
|
type: 'image/jpeg',
|
||||||
|
width: 1200,
|
||||||
|
height: 800,
|
||||||
|
focal_point_x: null,
|
||||||
|
focal_point_y: null,
|
||||||
|
storage: 'local',
|
||||||
|
};
|
||||||
|
|
||||||
|
const preset: Preset = {
|
||||||
|
key: 'system-small-cover',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 64, height: 64, fit: 'cover' } ] ],
|
||||||
|
};
|
||||||
|
|
||||||
|
function utilsWith(format: string | undefined) {
|
||||||
|
return {
|
||||||
|
resolvePreset: () =>
|
||||||
|
format ? [ [ 'toFormat', format, { quality: undefined } ], [ 'resize', { width: 64 } ] ] : [ [ 'resize', { width: 64 } ] ],
|
||||||
|
maybeExtractFormat: () => format,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('computePlan predicts a hashed filename with the resolved extension', async () => {
|
||||||
|
const plan = await computePlan(file, preset, 'avif', {
|
||||||
|
utils: utilsWith('avif'),
|
||||||
|
hashFn: () => 'deadbeef',
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(plan.filename, 'abc123__deadbeef.avif');
|
||||||
|
assert.equal(plan.format, 'avif');
|
||||||
|
assert.equal(plan.transforms.length, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('computePlan without a toFormat keeps the original extension but still hashes the resize', async () => {
|
||||||
|
const plan = await computePlan(file, { key: 'resize-only', width: 100 }, null, {
|
||||||
|
utils: utilsWith(undefined),
|
||||||
|
hashFn: () => 'unused',
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(plan.filename, 'abc123__unused.jpg');
|
||||||
|
assert.equal(plan.format, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('computePlan passes acceptFormat into the transformation set', async () => {
|
||||||
|
let seen: Record<string, unknown> | null = null;
|
||||||
|
|
||||||
|
await computePlan(file, preset, 'webp', {
|
||||||
|
utils: {
|
||||||
|
resolvePreset: (transformation) => {
|
||||||
|
seen = transformation;
|
||||||
|
return [ [ 'toFormat', 'webp', {} ] ];
|
||||||
|
},
|
||||||
|
maybeExtractFormat: () => 'webp',
|
||||||
|
},
|
||||||
|
hashFn: () => 'x',
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(seen?.['acceptFormat'], 'webp');
|
||||||
|
assert.ok(seen?.['transformationParams']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('ops ring buffer is newest-first and bounded', () => {
|
||||||
|
recordOp({ ts: 1, fileId: 'a', key: 'k', format: 'avif', ok: true, ms: 1 });
|
||||||
|
recordOp({ ts: 2, fileId: 'b', key: 'k', format: 'avif', ok: false, ms: 2, error: 'x' });
|
||||||
|
|
||||||
|
const ops = getOps(1);
|
||||||
|
assert.equal(ops.length, 1);
|
||||||
|
assert.equal(ops[0]?.fileId, 'b');
|
||||||
|
assert.equal(getOps(0).length, 0);
|
||||||
|
});
|
||||||
97
src/lib/diagnostics.ts
Normal file
97
src/lib/diagnostics.ts
Normal file
|
|
@ -0,0 +1,97 @@
|
||||||
|
import { basename, extname } from 'node:path';
|
||||||
|
import objectHash from 'object-hash';
|
||||||
|
import type { Preset } from './types';
|
||||||
|
|
||||||
|
export interface PlanFile {
|
||||||
|
filename_disk: string;
|
||||||
|
type?: string | null;
|
||||||
|
width?: number | null;
|
||||||
|
height?: number | null;
|
||||||
|
focal_point_x?: number | null;
|
||||||
|
focal_point_y?: number | null;
|
||||||
|
storage?: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlanResult {
|
||||||
|
filename: string;
|
||||||
|
format: string | null;
|
||||||
|
transforms: unknown[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface TransformUtils {
|
||||||
|
resolvePreset(transformation: Record<string, unknown>, file: PlanFile): unknown[];
|
||||||
|
maybeExtractFormat(transforms: unknown[]): string | undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PlanDeps {
|
||||||
|
utils?: TransformUtils;
|
||||||
|
hashFn?: (value: unknown) => string;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadTransformUtils(): Promise<TransformUtils> {
|
||||||
|
const importer = new Function('return import("@directus/api/utils/transformations")') as () => Promise<
|
||||||
|
Record<string, unknown>
|
||||||
|
>;
|
||||||
|
const module = await importer();
|
||||||
|
const namespace = (module['TransformationUtils'] as Record<string, unknown> | undefined) ?? module;
|
||||||
|
|
||||||
|
const resolvePreset = (namespace['resolvePreset'] ?? module['resolvePreset']) as TransformUtils['resolvePreset'];
|
||||||
|
const maybeExtractFormat = (namespace['maybeExtractFormat'] ?? module['maybeExtractFormat']) as TransformUtils['maybeExtractFormat'];
|
||||||
|
|
||||||
|
if (typeof resolvePreset !== 'function' || typeof maybeExtractFormat !== 'function') {
|
||||||
|
throw new Error('Unable to load Directus transformation utils');
|
||||||
|
}
|
||||||
|
|
||||||
|
return { resolvePreset, maybeExtractFormat };
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function computePlan(
|
||||||
|
file: PlanFile,
|
||||||
|
preset: Preset,
|
||||||
|
acceptFormat: string | null,
|
||||||
|
deps: PlanDeps = {},
|
||||||
|
): Promise<PlanResult> {
|
||||||
|
const utils = deps.utils ?? (await loadTransformUtils());
|
||||||
|
const hashFn = deps.hashFn ?? ((value: unknown) => objectHash(value));
|
||||||
|
|
||||||
|
const transformationParams: Record<string, unknown> = { ...preset };
|
||||||
|
const transformation: Record<string, unknown> = { transformationParams };
|
||||||
|
|
||||||
|
if (acceptFormat) transformation['acceptFormat'] = acceptFormat;
|
||||||
|
|
||||||
|
const transforms = utils.resolvePreset(transformation, file);
|
||||||
|
const suffix = transforms.length > 0 ? `__${hashFn(transforms)}` : '';
|
||||||
|
const newFormat = utils.maybeExtractFormat(transforms);
|
||||||
|
const originalExtension = extname(file.filename_disk);
|
||||||
|
const base = basename(file.filename_disk, originalExtension);
|
||||||
|
const extension = newFormat ? `.${newFormat}` : originalExtension;
|
||||||
|
|
||||||
|
return {
|
||||||
|
filename: `${base}${suffix}${extension}`,
|
||||||
|
format: newFormat ?? null,
|
||||||
|
transforms,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OpRecord {
|
||||||
|
ts: number;
|
||||||
|
fileId: string;
|
||||||
|
key: string;
|
||||||
|
format: string | null;
|
||||||
|
ok: boolean;
|
||||||
|
ms: number;
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const MAX_OPS = 200;
|
||||||
|
const ops: OpRecord[] = [];
|
||||||
|
|
||||||
|
export function recordOp(op: OpRecord): void {
|
||||||
|
ops.unshift(op);
|
||||||
|
|
||||||
|
if (ops.length > MAX_OPS) ops.length = MAX_OPS;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getOps(limit = 50): OpRecord[] {
|
||||||
|
return ops.slice(0, Math.max(0, Math.min(limit, MAX_OPS)));
|
||||||
|
}
|
||||||
31
src/lib/format.test.ts
Normal file
31
src/lib/format.test.ts
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { humanBytes, humanDuration, humanMs, toIso } from './format';
|
||||||
|
|
||||||
|
test('humanBytes formats sizes', () => {
|
||||||
|
assert.equal(humanBytes(0), '0 B');
|
||||||
|
assert.equal(humanBytes(512), '512 B');
|
||||||
|
assert.equal(humanBytes(1024), '1.0 KB');
|
||||||
|
assert.equal(humanBytes(1536), '1.5 KB');
|
||||||
|
assert.equal(humanBytes(1024 * 1024), '1.0 MB');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('humanMs formats durations', () => {
|
||||||
|
assert.equal(humanMs(250), '250 ms');
|
||||||
|
assert.equal(humanMs(1500), '1.5 s');
|
||||||
|
assert.equal(humanMs(65000), '1 min 5 s');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('humanDuration formats seconds', () => {
|
||||||
|
assert.equal(humanDuration(null), '—');
|
||||||
|
assert.equal(humanDuration(30), '30 s');
|
||||||
|
assert.equal(humanDuration(90), '1 min 30 s');
|
||||||
|
assert.equal(humanDuration(3700), '1 h 1 min');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('toIso returns null for empty values', () => {
|
||||||
|
assert.equal(toIso(null), null);
|
||||||
|
assert.equal(toIso(undefined), null);
|
||||||
|
assert.equal(toIso(0), null);
|
||||||
|
assert.equal(toIso(1792880701827), new Date(1792880701827).toISOString());
|
||||||
|
});
|
||||||
34
src/lib/format.ts
Normal file
34
src/lib/format.ts
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
export function humanBytes(bytes: number): string {
|
||||||
|
if (!Number.isFinite(bytes) || bytes <= 0) return '0 B';
|
||||||
|
|
||||||
|
const units = [ 'B', 'KB', 'MB', 'GB', 'TB', 'PB' ];
|
||||||
|
const exponent = Math.min(units.length - 1, Math.floor(Math.log(bytes) / Math.log(1024)));
|
||||||
|
const value = bytes / Math.pow(1024, exponent);
|
||||||
|
|
||||||
|
return `${value.toFixed(exponent === 0 ? 0 : 1)} ${units[exponent]}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function humanMs(ms: number): string {
|
||||||
|
if (!Number.isFinite(ms) || ms < 0) return '—';
|
||||||
|
if (ms < 1000) return `${Math.round(ms)} ms`;
|
||||||
|
if (ms < 60000) return `${(ms / 1000).toFixed(1)} s`;
|
||||||
|
return `${Math.floor(ms / 60000)} min ${Math.round((ms % 60000) / 1000)} s`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function humanDuration(seconds: number | null | undefined): string {
|
||||||
|
if (seconds === null || seconds === undefined || !Number.isFinite(seconds) || seconds < 0) return '—';
|
||||||
|
if (seconds < 60) return `${Math.round(seconds)} s`;
|
||||||
|
|
||||||
|
const minutes = Math.floor(seconds / 60);
|
||||||
|
const rest = Math.round(seconds % 60);
|
||||||
|
|
||||||
|
if (minutes < 60) return rest > 0 ? `${minutes} min ${rest} s` : `${minutes} min`;
|
||||||
|
|
||||||
|
const hours = Math.floor(minutes / 60);
|
||||||
|
return `${hours} h ${minutes % 60} min`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function toIso(value: number | null | undefined): string | null {
|
||||||
|
if (!value || !Number.isFinite(value)) return null;
|
||||||
|
return new Date(value).toISOString();
|
||||||
|
}
|
||||||
63
src/lib/presets.test.ts
Normal file
63
src/lib/presets.test.ts
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { buildRegistry, effectiveFormatsForPreset, formatsForPreset, mergePresets, tasksPerFile, uniqueByKey } from './presets';
|
||||||
|
import { SYSTEM_PRESETS } from './system-presets';
|
||||||
|
import type { Preset } from './types';
|
||||||
|
|
||||||
|
test('system presets are the six expected keys', () => {
|
||||||
|
assert.deepEqual(
|
||||||
|
SYSTEM_PRESETS.map((preset) => preset.key),
|
||||||
|
[
|
||||||
|
'system-small-cover',
|
||||||
|
'system-small-contain',
|
||||||
|
'system-medium-cover',
|
||||||
|
'system-medium-contain',
|
||||||
|
'system-large-cover',
|
||||||
|
'system-large-contain',
|
||||||
|
],
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mergePresets appends custom after system', () => {
|
||||||
|
const custom: Preset[] = [ { key: 'custom', width: 100 } ];
|
||||||
|
const merged = mergePresets(SYSTEM_PRESETS, custom);
|
||||||
|
assert.equal(merged.length, SYSTEM_PRESETS.length + 1);
|
||||||
|
assert.equal(merged[merged.length - 1]?.key, 'custom');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('uniqueByKey keeps the first occurrence (controller parity)', () => {
|
||||||
|
const custom: Preset[] = [ { key: 'system-small-cover', width: 999 } ];
|
||||||
|
const merged = uniqueByKey(mergePresets(SYSTEM_PRESETS, custom));
|
||||||
|
const small = merged.find((preset) => preset.key === 'system-small-cover');
|
||||||
|
assert.equal(small?.width, undefined);
|
||||||
|
assert.equal(merged.filter((preset) => preset.key === 'system-small-cover').length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('buildRegistry selects by key or returns all', () => {
|
||||||
|
const all = buildRegistry([], null);
|
||||||
|
assert.equal(all.length, 6);
|
||||||
|
|
||||||
|
const selected = buildRegistry([], [ 'system-medium-cover' ]);
|
||||||
|
assert.deepEqual(selected.map((preset) => preset.key), [ 'system-medium-cover' ]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('formatsForPreset emits two variants for auto and one otherwise', () => {
|
||||||
|
const auto: Preset = { key: 'a', format: 'auto' };
|
||||||
|
const fixed: Preset = { key: 'b', format: 'webp' };
|
||||||
|
const none: Preset = { key: 'c', width: 100 };
|
||||||
|
|
||||||
|
assert.deepEqual(formatsForPreset(auto, [ 'avif', 'webp' ]), [ 'avif', 'webp' ]);
|
||||||
|
assert.deepEqual(formatsForPreset(fixed, [ 'avif', 'webp' ]), [ null ]);
|
||||||
|
assert.deepEqual(formatsForPreset(none, [ 'avif', 'webp' ]), [ null ]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('effectiveFormatsForPreset reports the real output format', () => {
|
||||||
|
assert.deepEqual(effectiveFormatsForPreset({ key: 'a', format: 'auto' }, [ 'avif', 'webp' ]), [ 'avif', 'webp' ]);
|
||||||
|
assert.deepEqual(effectiveFormatsForPreset({ key: 'b', format: 'png' }, [ 'avif', 'webp' ]), [ 'png' ]);
|
||||||
|
assert.deepEqual(effectiveFormatsForPreset({ key: 'c', width: 100 }, [ 'avif', 'webp' ]), [ null ]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('tasksPerFile counts variants across presets', () => {
|
||||||
|
assert.equal(tasksPerFile(SYSTEM_PRESETS, [ 'avif', 'webp' ]), 12);
|
||||||
|
assert.equal(tasksPerFile(SYSTEM_PRESETS, [ 'avif' ]), 6);
|
||||||
|
});
|
||||||
58
src/lib/presets.ts
Normal file
58
src/lib/presets.ts
Normal file
|
|
@ -0,0 +1,58 @@
|
||||||
|
import { SYSTEM_PRESETS } from './system-presets';
|
||||||
|
import type { Preset } from './types';
|
||||||
|
|
||||||
|
export function mergePresets(system: Preset[], custom: Preset[]): Preset[] {
|
||||||
|
return [ ...system, ...custom ];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function uniqueByKey(presets: Preset[]): Preset[] {
|
||||||
|
const seen = new Set<string>();
|
||||||
|
|
||||||
|
return presets.filter((preset) => {
|
||||||
|
if (seen.has(preset.key)) return false;
|
||||||
|
seen.add(preset.key);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function selectPresets(presets: Preset[], keys: string[] | null): Preset[] {
|
||||||
|
if (keys === null) return presets;
|
||||||
|
|
||||||
|
const wanted = new Set(keys);
|
||||||
|
return presets.filter((preset) => wanted.has(preset.key));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildRegistry(custom: Preset[], keys: string[] | null): Preset[] {
|
||||||
|
const merged = uniqueByKey(mergePresets(SYSTEM_PRESETS, custom));
|
||||||
|
return selectPresets(merged, keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatsForPreset(preset: Preset, formats: string[]): (string | null)[] {
|
||||||
|
if (preset.format === 'auto') return formats.length > 0 ? [ ...formats ] : [ 'avif', 'webp' ];
|
||||||
|
return [ null ];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function effectiveFormat(preset: Preset, variant: string | null): string | null {
|
||||||
|
return variant ?? preset.format ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function effectiveFormatsForPreset(preset: Preset, formats: string[]): (string | null)[] {
|
||||||
|
return formatsForPreset(preset, formats).map((variant) => effectiveFormat(preset, variant));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function tasksPerFile(presets: Preset[], formats: string[]): number {
|
||||||
|
return presets.reduce((total, preset) => total + formatsForPreset(preset, formats).length, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isImageType(type: unknown): boolean {
|
||||||
|
return typeof type === 'string' && type.startsWith('image/');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function loadCustomPresets(settingsService: {
|
||||||
|
readSingleton(query?: Record<string, unknown>): Promise<Record<string, unknown> | null>;
|
||||||
|
}): Promise<Preset[]> {
|
||||||
|
const settings = await settingsService.readSingleton({ fields: [ 'storage_asset_presets' ] });
|
||||||
|
const list = settings?.['storage_asset_presets'];
|
||||||
|
|
||||||
|
return Array.isArray(list) ? (list as Preset[]) : [];
|
||||||
|
}
|
||||||
69
src/lib/settings.test.ts
Normal file
69
src/lib/settings.test.ts
Normal file
|
|
@ -0,0 +1,69 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { envDefaults, mergeSettings, validateSettings } from './settings';
|
||||||
|
|
||||||
|
test('envDefaults applies safe defaults', () => {
|
||||||
|
const settings = envDefaults({});
|
||||||
|
assert.equal(settings.enabled, true);
|
||||||
|
assert.equal(settings.windowStart, '01:00');
|
||||||
|
assert.equal(settings.windowEnd, '04:00');
|
||||||
|
assert.equal(settings.concurrency, 1);
|
||||||
|
assert.deepEqual(settings.formats, [ 'avif', 'webp' ]);
|
||||||
|
assert.equal(settings.onUpload, true);
|
||||||
|
assert.equal(settings.presetKeys, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('envDefaults parses overrides', () => {
|
||||||
|
const settings = envDefaults({
|
||||||
|
IMAGE_PREVIEWS_ENABLED: 'true',
|
||||||
|
IMAGE_PREVIEWS_CONCURRENCY: '4',
|
||||||
|
IMAGE_PREVIEWS_FORMATS: 'webp',
|
||||||
|
IMAGE_PREVIEWS_PRESETS: 'system-small-cover, system-large-cover',
|
||||||
|
IMAGE_PREVIEWS_ON_UPLOAD: '0',
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(settings.enabled, true);
|
||||||
|
assert.equal(settings.concurrency, 4);
|
||||||
|
assert.deepEqual(settings.formats, [ 'webp' ]);
|
||||||
|
assert.deepEqual(settings.presetKeys, [ 'system-small-cover', 'system-large-cover' ]);
|
||||||
|
assert.equal(settings.onUpload, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('envDefaults tolerates Directus-coerced booleans and numbers', () => {
|
||||||
|
const settings = envDefaults({
|
||||||
|
IMAGE_PREVIEWS_ENABLED: true,
|
||||||
|
IMAGE_PREVIEWS_CONCURRENCY: 3,
|
||||||
|
IMAGE_PREVIEWS_ON_UPLOAD: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(settings.enabled, true);
|
||||||
|
assert.equal(settings.concurrency, 3);
|
||||||
|
assert.equal(settings.onUpload, false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mergeSettings keeps base values for absent overrides', () => {
|
||||||
|
const base = envDefaults({});
|
||||||
|
const merged = mergeSettings(base, { enabled: true });
|
||||||
|
assert.equal(merged.enabled, true);
|
||||||
|
assert.equal(merged.concurrency, base.concurrency);
|
||||||
|
assert.deepEqual(merged.formats, base.formats);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateSettings rejects invalid concurrency', () => {
|
||||||
|
const base = envDefaults({});
|
||||||
|
assert.throws(() => validateSettings({ ...base, concurrency: 0 }), /concurrency/);
|
||||||
|
assert.throws(() => validateSettings({ ...base, concurrency: 99 }), /concurrency/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateSettings rejects invalid window and format', () => {
|
||||||
|
const base = envDefaults({});
|
||||||
|
assert.throws(() => validateSettings({ ...base, windowStart: '25:00' }), /windowStart/);
|
||||||
|
assert.throws(() => validateSettings({ ...base, formats: [ 'bmp' ] }), /unsupported format/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateSettings normalises a valid payload', () => {
|
||||||
|
const base = envDefaults({});
|
||||||
|
const result = validateSettings({ ...base, windowStart: '03:00', windowEnd: '05:30' });
|
||||||
|
assert.equal(result.windowStart, '03:00');
|
||||||
|
assert.equal(result.windowEnd, '05:30');
|
||||||
|
});
|
||||||
156
src/lib/settings.ts
Normal file
156
src/lib/settings.ts
Normal file
|
|
@ -0,0 +1,156 @@
|
||||||
|
import { mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||||
|
import { dirname, isAbsolute, join } from 'node:path';
|
||||||
|
|
||||||
|
export interface Settings {
|
||||||
|
enabled: boolean;
|
||||||
|
windowStart: string | null;
|
||||||
|
windowEnd: string | null;
|
||||||
|
concurrency: number;
|
||||||
|
presetKeys: string[] | null;
|
||||||
|
formats: string[];
|
||||||
|
onUpload: boolean;
|
||||||
|
debug: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EnvLike {
|
||||||
|
[ key: string ]: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_STATE_SUBDIR = '.image-previews';
|
||||||
|
const DEFAULT_WINDOW_START = '01:00';
|
||||||
|
const DEFAULT_WINDOW_END = '04:00';
|
||||||
|
|
||||||
|
function asString(value: unknown): string | null {
|
||||||
|
if (value === undefined || value === null) return null;
|
||||||
|
if (typeof value === 'string') return value;
|
||||||
|
if (typeof value === 'number' || typeof value === 'boolean') return String(value);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function stateDir(env: EnvLike = process.env): string {
|
||||||
|
const configured = asString(env['IMAGE_PREVIEWS_STATE_DIR']);
|
||||||
|
|
||||||
|
if (configured && configured.length > 0) return configured;
|
||||||
|
|
||||||
|
const root = asString(env['STORAGE_LOCAL_ROOT']) || '/directus/uploads';
|
||||||
|
const absoluteRoot = isAbsolute(root) ? root : join(process.cwd(), root);
|
||||||
|
|
||||||
|
return join(absoluteRoot, DEFAULT_STATE_SUBDIR);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseBool(value: unknown, fallback: boolean): boolean {
|
||||||
|
const raw = asString(value);
|
||||||
|
if (raw === null || raw.trim() === '') return fallback;
|
||||||
|
return [ '1', 'true', 'yes', 'on' ].includes(raw.trim().toLowerCase());
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseList(value: unknown): string[] | null {
|
||||||
|
const raw = asString(value);
|
||||||
|
if (raw === null || raw.trim() === '') return null;
|
||||||
|
|
||||||
|
return raw
|
||||||
|
.split(',')
|
||||||
|
.map((part) => part.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseNumber(value: unknown, fallback: number): number {
|
||||||
|
const raw = asString(value);
|
||||||
|
if (raw === null || raw.trim() === '') return fallback;
|
||||||
|
const parsed = Number(raw);
|
||||||
|
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function envDefaults(env: EnvLike = process.env): Settings {
|
||||||
|
return {
|
||||||
|
enabled: parseBool(env['IMAGE_PREVIEWS_ENABLED'], true),
|
||||||
|
windowStart: asString(env['IMAGE_PREVIEWS_WINDOW_START']) ?? DEFAULT_WINDOW_START,
|
||||||
|
windowEnd: asString(env['IMAGE_PREVIEWS_WINDOW_END']) ?? DEFAULT_WINDOW_END,
|
||||||
|
concurrency: parseNumber(env['IMAGE_PREVIEWS_CONCURRENCY'], 1),
|
||||||
|
presetKeys: parseList(env['IMAGE_PREVIEWS_PRESETS']),
|
||||||
|
formats: parseList(env['IMAGE_PREVIEWS_FORMATS']) ?? [ 'avif', 'webp' ],
|
||||||
|
onUpload: parseBool(env['IMAGE_PREVIEWS_ON_UPLOAD'], true),
|
||||||
|
debug: parseBool(env['IMAGE_PREVIEWS_DEBUG'], false),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function settingsPath(env: EnvLike = process.env): string {
|
||||||
|
return join(stateDir(env), 'settings.json');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function mergeSettings(base: Settings, override: Partial<Settings>): Settings {
|
||||||
|
return {
|
||||||
|
...base,
|
||||||
|
...override,
|
||||||
|
formats: override.formats ?? base.formats,
|
||||||
|
presetKeys: override.presetKeys !== undefined ? override.presetKeys : base.presetKeys,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function readSettings(env: EnvLike = process.env): Promise<Settings> {
|
||||||
|
const defaults = envDefaults(env);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const raw = await readFile(settingsPath(env), 'utf8');
|
||||||
|
const parsed = JSON.parse(raw) as Partial<Settings>;
|
||||||
|
return validateSettings(mergeSettings(defaults, parsed));
|
||||||
|
} catch {
|
||||||
|
return defaults;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function writeSettings(settings: Settings, env: EnvLike = process.env): Promise<Settings> {
|
||||||
|
const validated = validateSettings(settings);
|
||||||
|
const target = settingsPath(env);
|
||||||
|
await mkdir(dirname(target), { recursive: true });
|
||||||
|
await writeFile(target, `${JSON.stringify(validated, null, 2)}\n`, 'utf8');
|
||||||
|
return validated;
|
||||||
|
}
|
||||||
|
|
||||||
|
const TIME_PATTERN = /^([01]\d|2[0-3]):[0-5]\d$/;
|
||||||
|
const ALLOWED_FORMATS = [ 'avif', 'webp', 'jpg', 'jpeg', 'png' ];
|
||||||
|
|
||||||
|
export function validateSettings(settings: Settings): Settings {
|
||||||
|
if (typeof settings.enabled !== 'boolean') throw new Error('enabled must be a boolean');
|
||||||
|
if (typeof settings.onUpload !== 'boolean') throw new Error('onUpload must be a boolean');
|
||||||
|
if (typeof settings.debug !== 'boolean') throw new Error('debug must be a boolean');
|
||||||
|
|
||||||
|
if (settings.windowStart !== null && !TIME_PATTERN.test(settings.windowStart)) {
|
||||||
|
throw new Error('windowStart must be HH:MM or null');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (settings.windowEnd !== null && !TIME_PATTERN.test(settings.windowEnd)) {
|
||||||
|
throw new Error('windowEnd must be HH:MM or null');
|
||||||
|
}
|
||||||
|
|
||||||
|
const concurrency = Number(settings.concurrency);
|
||||||
|
|
||||||
|
if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 32) {
|
||||||
|
throw new Error('concurrency must be an integer between 1 and 32');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Array.isArray(settings.formats) || settings.formats.length === 0) {
|
||||||
|
throw new Error('formats must be a non-empty array');
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const format of settings.formats) {
|
||||||
|
if (!ALLOWED_FORMATS.includes(format)) throw new Error(`unsupported format: ${format}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (settings.presetKeys !== null) {
|
||||||
|
if (!Array.isArray(settings.presetKeys) || settings.presetKeys.some((key) => typeof key !== 'string')) {
|
||||||
|
throw new Error('presetKeys must be an array of strings or null');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
enabled: settings.enabled,
|
||||||
|
windowStart: settings.windowStart,
|
||||||
|
windowEnd: settings.windowEnd,
|
||||||
|
concurrency,
|
||||||
|
presetKeys: settings.presetKeys,
|
||||||
|
formats: [ ...settings.formats ],
|
||||||
|
onUpload: settings.onUpload,
|
||||||
|
debug: settings.debug,
|
||||||
|
};
|
||||||
|
}
|
||||||
95
src/lib/state.test.ts
Normal file
95
src/lib/state.test.ts
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { appendFile, mkdtemp, rm } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import {
|
||||||
|
acquireLock,
|
||||||
|
appendCompleted,
|
||||||
|
completedKeyOf,
|
||||||
|
isCancelled,
|
||||||
|
loadCompleted,
|
||||||
|
readRun,
|
||||||
|
releaseLock,
|
||||||
|
requestCancel,
|
||||||
|
writeRun,
|
||||||
|
} from './state';
|
||||||
|
import type { RunState } from './types';
|
||||||
|
|
||||||
|
async function withTempEnv(run: (env: Record<string, string>) => Promise<void>): Promise<void> {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'image-previews-'));
|
||||||
|
|
||||||
|
try {
|
||||||
|
await run({ IMAGE_PREVIEWS_STATE_DIR: dir });
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test('appendCompleted + loadCompleted round-trips and resumes', async () => {
|
||||||
|
await withTempEnv(async (env) => {
|
||||||
|
const empty = await loadCompleted(env);
|
||||||
|
assert.equal(empty.size, 0);
|
||||||
|
|
||||||
|
await appendCompleted({ file_id: 'f1', preset_key: 'system-small-cover', format: 'avif', ts: 1 }, env);
|
||||||
|
await appendCompleted({ file_id: 'f1', preset_key: 'system-small-cover', format: 'webp', ts: 2 }, env);
|
||||||
|
|
||||||
|
const completed = await loadCompleted(env);
|
||||||
|
assert.equal(completed.size, 2);
|
||||||
|
assert.ok(completed.has(completedKeyOf('f1', 'system-small-cover', 'avif')));
|
||||||
|
assert.ok(completed.has(completedKeyOf('f1', 'system-small-cover', 'webp')));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('loadCompleted tolerates corrupt lines', async () => {
|
||||||
|
await withTempEnv(async (env) => {
|
||||||
|
await appendFile(join(env['IMAGE_PREVIEWS_STATE_DIR'] as string, 'state.jsonl'), 'not-json\n', 'utf8');
|
||||||
|
await appendCompleted({ file_id: 'f2', preset_key: 'k', format: null, ts: 1 }, env);
|
||||||
|
|
||||||
|
const completed = await loadCompleted(env);
|
||||||
|
assert.equal(completed.size, 1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('run state round-trips', async () => {
|
||||||
|
await withTempEnv(async (env) => {
|
||||||
|
assert.equal(await readRun(env), null);
|
||||||
|
|
||||||
|
const run: RunState = {
|
||||||
|
running: true,
|
||||||
|
startedAt: 1,
|
||||||
|
finishedAt: null,
|
||||||
|
total: 10,
|
||||||
|
done: 2,
|
||||||
|
generated: 2,
|
||||||
|
hits: 0,
|
||||||
|
failed: 0,
|
||||||
|
rate: 1,
|
||||||
|
eta: 8,
|
||||||
|
cancelled: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
await writeRun(run, env);
|
||||||
|
const loaded = await readRun(env);
|
||||||
|
assert.equal(loaded?.done, 2);
|
||||||
|
assert.equal(loaded?.total, 10);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('lock is exclusive and releasable', async () => {
|
||||||
|
await withTempEnv(async (env) => {
|
||||||
|
assert.equal(await acquireLock(env), true);
|
||||||
|
assert.equal(await acquireLock(env), false);
|
||||||
|
await releaseLock(env);
|
||||||
|
assert.equal(await acquireLock(env), true);
|
||||||
|
await releaseLock(env);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test('cancel flag toggles', async () => {
|
||||||
|
await withTempEnv(async (env) => {
|
||||||
|
assert.equal(await isCancelled(env), false);
|
||||||
|
await requestCancel(env);
|
||||||
|
assert.equal(await isCancelled(env), true);
|
||||||
|
});
|
||||||
|
});
|
||||||
107
src/lib/state.ts
Normal file
107
src/lib/state.ts
Normal file
|
|
@ -0,0 +1,107 @@
|
||||||
|
import { appendFile, mkdir, readFile, rm, stat, writeFile } from 'node:fs/promises';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { stateDir, type EnvLike } from './settings';
|
||||||
|
import type { CompletedRecord, RunState } from './types';
|
||||||
|
|
||||||
|
const STALE_LOCK_MS = 6 * 60 * 60 * 1000;
|
||||||
|
|
||||||
|
export function completedKey(record: CompletedRecord): string {
|
||||||
|
return completedKeyOf(record.file_id, record.preset_key, record.format);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function completedKeyOf(fileId: string, presetKey: string, format: string | null): string {
|
||||||
|
return `${fileId}::${presetKey}::${format ?? ''}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function stateFilePath(env: EnvLike, name: string): string {
|
||||||
|
return join(stateDir(env), name);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function ensureStateDir(env: EnvLike): Promise<void> {
|
||||||
|
await mkdir(stateDir(env), { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function loadCompleted(env: EnvLike): Promise<Set<string>> {
|
||||||
|
const completed = new Set<string>();
|
||||||
|
|
||||||
|
try {
|
||||||
|
const raw = await readFile(stateFilePath(env, 'state.jsonl'), 'utf8');
|
||||||
|
|
||||||
|
for (const line of raw.split('\n')) {
|
||||||
|
if (!line.trim()) continue;
|
||||||
|
|
||||||
|
try {
|
||||||
|
completed.add(completedKey(JSON.parse(line) as CompletedRecord));
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return completed;
|
||||||
|
}
|
||||||
|
|
||||||
|
return completed;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function appendCompleted(record: CompletedRecord, env: EnvLike): Promise<void> {
|
||||||
|
await ensureStateDir(env);
|
||||||
|
await appendFile(stateFilePath(env, 'state.jsonl'), `${JSON.stringify(record)}\n`, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function readRun(env: EnvLike): Promise<RunState | null> {
|
||||||
|
try {
|
||||||
|
return JSON.parse(await readFile(stateFilePath(env, 'run.json'), 'utf8')) as RunState;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function writeRun(run: RunState, env: EnvLike): Promise<void> {
|
||||||
|
await ensureStateDir(env);
|
||||||
|
await writeFile(stateFilePath(env, 'run.json'), `${JSON.stringify(run, null, 2)}\n`, 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function lockAge(env: EnvLike): Promise<number | null> {
|
||||||
|
try {
|
||||||
|
const info = JSON.parse(await readFile(stateFilePath(env, 'run.lock'), 'utf8')) as { ts?: number };
|
||||||
|
return Date.now() - (Number(info.ts) || 0);
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function isLocked(env: EnvLike): Promise<boolean> {
|
||||||
|
const age = await lockAge(env);
|
||||||
|
return age !== null && age < STALE_LOCK_MS;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function acquireLock(env: EnvLike): Promise<boolean> {
|
||||||
|
await ensureStateDir(env);
|
||||||
|
|
||||||
|
if (await isLocked(env)) return false;
|
||||||
|
|
||||||
|
await writeFile(stateFilePath(env, 'run.lock'), JSON.stringify({ ts: Date.now(), pid: process.pid }), 'utf8');
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function releaseLock(env: EnvLike): Promise<void> {
|
||||||
|
await rm(stateFilePath(env, 'run.lock'), { force: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function requestCancel(env: EnvLike): Promise<void> {
|
||||||
|
await ensureStateDir(env);
|
||||||
|
await writeFile(stateFilePath(env, 'cancel.flag'), String(Date.now()), 'utf8');
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function clearCancel(env: EnvLike): Promise<void> {
|
||||||
|
await rm(stateFilePath(env, 'cancel.flag'), { force: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function isCancelled(env: EnvLike): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
await stat(stateFilePath(env, 'cancel.flag'));
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
42
src/lib/storage.test.ts
Normal file
42
src/lib/storage.test.ts
Normal file
|
|
@ -0,0 +1,42 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { isPreviewPath, isStatePath, storageUsage } from './storage';
|
||||||
|
|
||||||
|
test('isPreviewPath matches transform cache files only', () => {
|
||||||
|
assert.equal(isPreviewPath('id__0123456789abcdef0123456789abcdef01234567.avif'), true);
|
||||||
|
assert.equal(isPreviewPath('id__0123456789abcdef0123456789abcdef01234567.webp'), true);
|
||||||
|
assert.equal(isPreviewPath('id.jpg'), false);
|
||||||
|
assert.equal(isPreviewPath('state.jsonl'), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('isStatePath detects the state directory', () => {
|
||||||
|
assert.equal(isStatePath('.image-previews/settings.json'), true);
|
||||||
|
assert.equal(isStatePath('.image-previews'), true);
|
||||||
|
assert.equal(isStatePath('abc.jpg'), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('storageUsage splits previews and originals, excluding state', async () => {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), 'previews-usage-'));
|
||||||
|
|
||||||
|
try {
|
||||||
|
await writeFile(join(dir, 'a__0123456789abcdef0123456789abcdef01234567.avif'), 'x'.repeat(10));
|
||||||
|
await writeFile(join(dir, 'a.jpg'), 'y'.repeat(99));
|
||||||
|
await mkdir(join(dir, '.image-previews'));
|
||||||
|
await writeFile(join(dir, '.image-previews', 'settings.json'), 'z'.repeat(500));
|
||||||
|
|
||||||
|
const usage = await storageUsage({ STORAGE_LOCAL_ROOT: dir, STORAGE_LOCATIONS: 'local' });
|
||||||
|
|
||||||
|
assert.equal(usage.source, 'local');
|
||||||
|
assert.equal(usage.previews.count, 1);
|
||||||
|
assert.equal(usage.previews.bytes, 10);
|
||||||
|
assert.equal(usage.originals.count, 1);
|
||||||
|
assert.equal(usage.originals.bytes, 99);
|
||||||
|
assert.equal(usage.total.count, 2);
|
||||||
|
assert.equal(usage.total.bytes, 109);
|
||||||
|
} finally {
|
||||||
|
await rm(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
162
src/lib/storage.ts
Normal file
162
src/lib/storage.ts
Normal file
|
|
@ -0,0 +1,162 @@
|
||||||
|
import { readdir, stat as fsStat } from 'node:fs/promises';
|
||||||
|
import { isAbsolute, join } from 'node:path';
|
||||||
|
import type { EnvLike } from './settings';
|
||||||
|
|
||||||
|
const PREVIEW_PATTERN = /__[0-9a-f]{8,}\.[a-z0-9]+$/i;
|
||||||
|
const STATE_DIR_NAME = '.image-previews';
|
||||||
|
|
||||||
|
function asString(value: unknown): string | null {
|
||||||
|
if (value === undefined || value === null) return null;
|
||||||
|
if (typeof value === 'string') return value;
|
||||||
|
if (typeof value === 'number' || typeof value === 'boolean') return String(value);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isPreviewPath(path: string): boolean {
|
||||||
|
return PREVIEW_PATTERN.test(path);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isStatePath(path: string): boolean {
|
||||||
|
return path === STATE_DIR_NAME || path.startsWith(`${STATE_DIR_NAME}/`) || path.includes(`/${STATE_DIR_NAME}/`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadStorageManager(): Promise<Record<string, unknown> | null> {
|
||||||
|
try {
|
||||||
|
const importer = new Function('return import("@directus/api/storage/index")') as () => Promise<Record<string, unknown>>;
|
||||||
|
const module = await importer();
|
||||||
|
const getStorage = module['getStorage'] as (() => Promise<Record<string, unknown>>) | undefined;
|
||||||
|
if (!getStorage) return null;
|
||||||
|
return await getStorage();
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function localStorageRoot(env: EnvLike): string {
|
||||||
|
const root = asString(env['STORAGE_LOCAL_ROOT']) || './uploads';
|
||||||
|
return isAbsolute(root) ? root : join(process.cwd(), root);
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Bucket {
|
||||||
|
bytes: number;
|
||||||
|
count: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StorageUsage {
|
||||||
|
previews: Bucket;
|
||||||
|
originals: Bucket;
|
||||||
|
total: Bucket;
|
||||||
|
source: 'storage' | 'local';
|
||||||
|
}
|
||||||
|
|
||||||
|
function emptyUsage(source: 'storage' | 'local'): StorageUsage {
|
||||||
|
return {
|
||||||
|
previews: { bytes: 0, count: 0 },
|
||||||
|
originals: { bytes: 0, count: 0 },
|
||||||
|
total: { bytes: 0, count: 0 },
|
||||||
|
source,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function add(usage: StorageUsage, path: string, size: number): void {
|
||||||
|
if (isPreviewPath(path)) {
|
||||||
|
usage.previews.bytes += size;
|
||||||
|
usage.previews.count += 1;
|
||||||
|
} else {
|
||||||
|
usage.originals.bytes += size;
|
||||||
|
usage.originals.count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
usage.total.bytes += size;
|
||||||
|
usage.total.count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function walkLocal(root: string, usage: StorageUsage): Promise<void> {
|
||||||
|
let entries;
|
||||||
|
|
||||||
|
try {
|
||||||
|
entries = await readdir(root, { withFileTypes: true });
|
||||||
|
} catch {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const entry of entries) {
|
||||||
|
const full = join(root, entry.name);
|
||||||
|
|
||||||
|
if (entry.isDirectory()) {
|
||||||
|
if (entry.name === STATE_DIR_NAME) continue;
|
||||||
|
await walkLocal(full, usage);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!entry.isFile()) continue;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const info = await fsStat(full);
|
||||||
|
add(usage, entry.name, info.size);
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function storageUsage(env: EnvLike): Promise<StorageUsage> {
|
||||||
|
const manager = await loadStorageManager();
|
||||||
|
|
||||||
|
if (manager) {
|
||||||
|
const usage = emptyUsage('storage');
|
||||||
|
|
||||||
|
const locations = (asString(env['STORAGE_LOCATIONS']) || 'local')
|
||||||
|
.split(',')
|
||||||
|
.map((part) => part.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
|
||||||
|
for (const location of locations) {
|
||||||
|
try {
|
||||||
|
const driver = (manager['location'] as (name: string) => Record<string, unknown>)(location);
|
||||||
|
const list = (driver['list'] as () => AsyncIterable<string>)();
|
||||||
|
|
||||||
|
for await (const path of list) {
|
||||||
|
if (isStatePath(path)) continue;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const info = (await (driver['stat'] as (p: string) => Promise<{ size: number }>)(path)) as {
|
||||||
|
size: number;
|
||||||
|
};
|
||||||
|
add(usage, path, Number(info?.size) || 0);
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return usage;
|
||||||
|
}
|
||||||
|
|
||||||
|
const usage = emptyUsage('local');
|
||||||
|
await walkLocal(localStorageRoot(env), usage);
|
||||||
|
return usage;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function previewExists(env: EnvLike, location: string | null, filename: string): Promise<boolean | null> {
|
||||||
|
const manager = await loadStorageManager();
|
||||||
|
|
||||||
|
if (manager && location) {
|
||||||
|
try {
|
||||||
|
const driver = (manager['location'] as (name: string) => Record<string, unknown>)(location);
|
||||||
|
return await (driver['exists'] as (p: string) => Promise<boolean>)(filename);
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await fsStat(join(localStorageRoot(env), filename));
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
36
src/lib/system-presets.ts
Normal file
36
src/lib/system-presets.ts
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
import type { Preset } from './types';
|
||||||
|
|
||||||
|
export const SYSTEM_PRESETS: Preset[] = [
|
||||||
|
{
|
||||||
|
key: 'system-small-cover',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 64, height: 64, fit: 'cover' } ] ],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'system-small-contain',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 64, fit: 'contain' } ] ],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'system-medium-cover',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 300, height: 300, fit: 'cover' } ] ],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'system-medium-contain',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 300, fit: 'contain' } ] ],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'system-large-cover',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 800, height: 800, fit: 'cover' } ] ],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'system-large-contain',
|
||||||
|
format: 'auto',
|
||||||
|
transforms: [ [ 'resize', { width: 800, fit: 'contain' } ] ],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
export const DEFAULT_FORMATS = [ 'avif', 'webp' ];
|
||||||
50
src/lib/types.ts
Normal file
50
src/lib/types.ts
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
export interface Preset {
|
||||||
|
key: string;
|
||||||
|
format?: string;
|
||||||
|
quality?: number | string;
|
||||||
|
width?: number | string;
|
||||||
|
height?: number | string;
|
||||||
|
fit?: string;
|
||||||
|
withoutEnlargement?: boolean;
|
||||||
|
focal_point_x?: number | string | null;
|
||||||
|
focal_point_y?: number | string | null;
|
||||||
|
transforms?: unknown[];
|
||||||
|
[key: string]: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WarmResultItem {
|
||||||
|
key: string;
|
||||||
|
format: string | null;
|
||||||
|
ok: boolean;
|
||||||
|
ms: number;
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WarmSummary {
|
||||||
|
requested: number;
|
||||||
|
ok: number;
|
||||||
|
failed: number;
|
||||||
|
results: WarmResultItem[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CompletedRecord {
|
||||||
|
file_id: string;
|
||||||
|
preset_key: string;
|
||||||
|
format: string | null;
|
||||||
|
ts: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RunState {
|
||||||
|
running: boolean;
|
||||||
|
startedAt: number | null;
|
||||||
|
finishedAt: number | null;
|
||||||
|
total: number;
|
||||||
|
done: number;
|
||||||
|
generated: number;
|
||||||
|
hits: number;
|
||||||
|
failed: number;
|
||||||
|
rate: number;
|
||||||
|
eta: number | null;
|
||||||
|
cancelled: boolean;
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
92
src/lib/warmer.test.ts
Normal file
92
src/lib/warmer.test.ts
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import type { Preset } from './types';
|
||||||
|
import { warmFile, type AssetsLike } from './warmer';
|
||||||
|
|
||||||
|
const presets: Preset[] = [
|
||||||
|
{ key: 'system-small-cover', format: 'auto', transforms: [ [ 'resize', { width: 64, height: 64, fit: 'cover' } ] ] },
|
||||||
|
{ key: 'custom-fixed', format: 'webp', width: 200 },
|
||||||
|
];
|
||||||
|
|
||||||
|
test('warmFile passes the full preset object and the accept variant', async () => {
|
||||||
|
const calls: Array<Record<string, unknown>> = [];
|
||||||
|
|
||||||
|
const assets: AssetsLike = {
|
||||||
|
async getAsset(_id, transformation) {
|
||||||
|
calls.push(transformation ?? {});
|
||||||
|
return { stream: { destroy() {} } };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const summary = await warmFile(assets, 'file-1', presets, { concurrency: 1, formats: [ 'avif', 'webp' ] });
|
||||||
|
|
||||||
|
assert.equal(summary.requested, 3);
|
||||||
|
assert.equal(summary.ok, 3);
|
||||||
|
assert.equal(calls.length, 3);
|
||||||
|
|
||||||
|
const autoCalls = calls.filter((call) => (call['transformationParams'] as Preset).key === 'system-small-cover');
|
||||||
|
assert.equal(autoCalls.length, 2);
|
||||||
|
assert.deepEqual(
|
||||||
|
autoCalls.map((call) => call['acceptFormat']).sort(),
|
||||||
|
[ 'avif', 'webp' ],
|
||||||
|
);
|
||||||
|
assert.ok((autoCalls[0]?.['transformationParams'] as Preset).transforms);
|
||||||
|
|
||||||
|
const fixed = calls.find((call) => (call['transformationParams'] as Preset).key === 'custom-fixed');
|
||||||
|
assert.equal(fixed?.['acceptFormat'], undefined);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('warmFile isolates per-task errors', async () => {
|
||||||
|
const assets: AssetsLike = {
|
||||||
|
async getAsset(_id, transformation) {
|
||||||
|
const preset = transformation?.['transformationParams'] as Preset;
|
||||||
|
if (preset.key === 'custom-fixed') throw new Error('boom');
|
||||||
|
return { stream: { destroy() {} } };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const summary = await warmFile(assets, 'file-1', presets, { concurrency: 1, formats: [ 'avif', 'webp' ] });
|
||||||
|
|
||||||
|
assert.equal(summary.ok, 2);
|
||||||
|
assert.equal(summary.failed, 1);
|
||||||
|
assert.equal(summary.results.find((result) => result.key === 'custom-fixed')?.error, 'boom');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('warmFile respects the concurrency cap', async () => {
|
||||||
|
let active = 0;
|
||||||
|
let maxActive = 0;
|
||||||
|
|
||||||
|
const assets: AssetsLike = {
|
||||||
|
async getAsset() {
|
||||||
|
active += 1;
|
||||||
|
maxActive = Math.max(maxActive, active);
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||||
|
active -= 1;
|
||||||
|
return { stream: { destroy() {} } };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
await warmFile(assets, 'file-1', presets, { concurrency: 2, formats: [ 'avif', 'webp' ] });
|
||||||
|
|
||||||
|
assert.ok(maxActive <= 2, `maxActive=${maxActive}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('warmFile releases the returned stream', async () => {
|
||||||
|
let destroyed = 0;
|
||||||
|
|
||||||
|
const assets: AssetsLike = {
|
||||||
|
async getAsset() {
|
||||||
|
return {
|
||||||
|
stream: {
|
||||||
|
destroy() {
|
||||||
|
destroyed += 1;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
await warmFile(assets, 'file-1', [ presets[0] as Preset ], { concurrency: 1, formats: [ 'avif' ] });
|
||||||
|
|
||||||
|
assert.equal(destroyed, 1);
|
||||||
|
});
|
||||||
110
src/lib/warmer.ts
Normal file
110
src/lib/warmer.ts
Normal file
|
|
@ -0,0 +1,110 @@
|
||||||
|
import { effectiveFormat, formatsForPreset } from './presets';
|
||||||
|
import type { Preset, WarmResultItem, WarmSummary } from './types';
|
||||||
|
|
||||||
|
export interface AssetsLike {
|
||||||
|
getAsset(
|
||||||
|
id: string,
|
||||||
|
transformation?: Record<string, unknown>,
|
||||||
|
range?: unknown,
|
||||||
|
deferStream?: boolean,
|
||||||
|
): Promise<{ stream?: unknown }>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WarmOptions {
|
||||||
|
concurrency: number;
|
||||||
|
formats: string[];
|
||||||
|
onResult?: (result: WarmResultItem) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface WarmTask {
|
||||||
|
preset: Preset;
|
||||||
|
format: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function errorMessage(error: unknown): string {
|
||||||
|
if (error instanceof Error) return error.message;
|
||||||
|
if (typeof error === 'string') return error;
|
||||||
|
|
||||||
|
try {
|
||||||
|
return JSON.stringify(error);
|
||||||
|
} catch {
|
||||||
|
return String(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function releaseStream(stream: unknown): void {
|
||||||
|
const candidate = stream as { destroy?: () => void; unpipe?: () => void } | undefined;
|
||||||
|
|
||||||
|
if (candidate && typeof candidate.destroy === 'function') {
|
||||||
|
candidate.destroy();
|
||||||
|
} else if (candidate && typeof candidate.unpipe === 'function') {
|
||||||
|
candidate.unpipe();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function warmFile(
|
||||||
|
assets: AssetsLike,
|
||||||
|
fileId: string,
|
||||||
|
presets: Preset[],
|
||||||
|
options: WarmOptions,
|
||||||
|
): Promise<WarmSummary> {
|
||||||
|
const tasks: WarmTask[] = [];
|
||||||
|
|
||||||
|
for (const preset of presets) {
|
||||||
|
for (const format of formatsForPreset(preset, options.formats)) {
|
||||||
|
tasks.push({ preset, format });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const results: WarmResultItem[] = new Array(tasks.length);
|
||||||
|
let cursor = 0;
|
||||||
|
|
||||||
|
const workerCount = Math.max(1, Math.min(options.concurrency, tasks.length || 1));
|
||||||
|
|
||||||
|
const workers = Array.from({ length: workerCount }, async () => {
|
||||||
|
while (cursor < tasks.length) {
|
||||||
|
const index = cursor;
|
||||||
|
cursor += 1;
|
||||||
|
const task = tasks[index];
|
||||||
|
|
||||||
|
if (!task) break;
|
||||||
|
|
||||||
|
const transformation: Record<string, unknown> = { transformationParams: task.preset };
|
||||||
|
|
||||||
|
if (task.format) transformation['acceptFormat'] = task.format;
|
||||||
|
|
||||||
|
const started = Date.now();
|
||||||
|
const outputFormat = effectiveFormat(task.preset, task.format);
|
||||||
|
let result: WarmResultItem;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const asset = await assets.getAsset(fileId, transformation);
|
||||||
|
releaseStream(asset?.stream);
|
||||||
|
result = { key: task.preset.key, format: outputFormat, ok: true, ms: Date.now() - started };
|
||||||
|
} catch (error) {
|
||||||
|
result = {
|
||||||
|
key: task.preset.key,
|
||||||
|
format: outputFormat,
|
||||||
|
ok: false,
|
||||||
|
ms: Date.now() - started,
|
||||||
|
error: errorMessage(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
results[index] = result;
|
||||||
|
options.onResult?.(result);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
await Promise.all(workers);
|
||||||
|
|
||||||
|
const settled = results.filter((result): result is WarmResultItem => Boolean(result));
|
||||||
|
const ok = settled.filter((result) => result.ok).length;
|
||||||
|
|
||||||
|
return {
|
||||||
|
requested: tasks.length,
|
||||||
|
ok,
|
||||||
|
failed: settled.length - ok,
|
||||||
|
results: settled,
|
||||||
|
};
|
||||||
|
}
|
||||||
47
src/lib/window.test.ts
Normal file
47
src/lib/window.test.ts
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { isInWindow, nextWindow, parseMinutes } from './window';
|
||||||
|
|
||||||
|
function at(hours: number, minutes: number): Date {
|
||||||
|
return new Date(2026, 9, 25, hours, minutes, 0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
test('parseMinutes parses HH:MM', () => {
|
||||||
|
assert.equal(parseMinutes('00:00'), 0);
|
||||||
|
assert.equal(parseMinutes('03:30'), 210);
|
||||||
|
assert.equal(parseMinutes('23:59'), 1439);
|
||||||
|
assert.equal(parseMinutes(null), null);
|
||||||
|
assert.equal(parseMinutes('bad'), null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('no window means always inside', () => {
|
||||||
|
assert.equal(isInWindow(null, null, at(12, 0)), true);
|
||||||
|
assert.equal(isInWindow('03:00', null, at(12, 0)), true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('daytime window', () => {
|
||||||
|
assert.equal(isInWindow('03:00', '05:00', at(2, 59)), false);
|
||||||
|
assert.equal(isInWindow('03:00', '05:00', at(3, 0)), true);
|
||||||
|
assert.equal(isInWindow('03:00', '05:00', at(4, 59)), true);
|
||||||
|
assert.equal(isInWindow('03:00', '05:00', at(5, 0)), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('overnight window', () => {
|
||||||
|
assert.equal(isInWindow('23:00', '02:00', at(23, 30)), true);
|
||||||
|
assert.equal(isInWindow('23:00', '02:00', at(1, 30)), true);
|
||||||
|
assert.equal(isInWindow('23:00', '02:00', at(12, 0)), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('nextWindow returns the upcoming start', () => {
|
||||||
|
const next = nextWindow('03:00', '05:00', at(12, 0));
|
||||||
|
assert.ok(next);
|
||||||
|
assert.equal(next?.getDate(), 26);
|
||||||
|
assert.equal(next?.getHours(), 3);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('nextWindow today when still before the start', () => {
|
||||||
|
const next = nextWindow('03:00', '05:00', at(1, 0));
|
||||||
|
assert.ok(next);
|
||||||
|
assert.equal(next?.getDate(), 25);
|
||||||
|
assert.equal(next?.getHours(), 3);
|
||||||
|
});
|
||||||
32
src/lib/window.ts
Normal file
32
src/lib/window.ts
Normal file
|
|
@ -0,0 +1,32 @@
|
||||||
|
export function parseMinutes(value: string | null): number | null {
|
||||||
|
if (value === null) return null;
|
||||||
|
const match = /^([01]\d|2[0-3]):([0-5]\d)$/.exec(value);
|
||||||
|
if (!match) return null;
|
||||||
|
return Number(match[1]) * 60 + Number(match[2]);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isInWindow(start: string | null, end: string | null, now: Date = new Date()): boolean {
|
||||||
|
const from = parseMinutes(start);
|
||||||
|
const to = parseMinutes(end);
|
||||||
|
|
||||||
|
if (from === null || to === null) return true;
|
||||||
|
if (from === to) return true;
|
||||||
|
|
||||||
|
const current = now.getHours() * 60 + now.getMinutes();
|
||||||
|
|
||||||
|
if (from < to) return current >= from && current < to;
|
||||||
|
|
||||||
|
return current >= from || current < to;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function nextWindow(start: string | null, end: string | null, now: Date = new Date()): Date | null {
|
||||||
|
const from = parseMinutes(start);
|
||||||
|
if (from === null || parseMinutes(end) === null) return null;
|
||||||
|
|
||||||
|
const next = new Date(now);
|
||||||
|
next.setHours(Math.floor(from / 60), from % 60, 0, 0);
|
||||||
|
|
||||||
|
if (next.getTime() <= now.getTime()) next.setDate(next.getDate() + 1);
|
||||||
|
|
||||||
|
return next;
|
||||||
|
}
|
||||||
14
src/module/index.ts
Normal file
14
src/module/index.ts
Normal file
|
|
@ -0,0 +1,14 @@
|
||||||
|
import { defineModule } from '@directus/extensions-sdk';
|
||||||
|
import ModuleComponent from './module.vue';
|
||||||
|
|
||||||
|
export default defineModule({
|
||||||
|
id: 'image-previews',
|
||||||
|
name: 'Image Previews',
|
||||||
|
icon: 'image',
|
||||||
|
routes: [
|
||||||
|
{
|
||||||
|
path: '',
|
||||||
|
component: ModuleComponent,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
579
src/module/module.vue
Normal file
579
src/module/module.vue
Normal file
|
|
@ -0,0 +1,579 @@
|
||||||
|
<template>
|
||||||
|
<private-view title="Image Previews">
|
||||||
|
<main class="ip">
|
||||||
|
<v-notice v-if="status && !status.enabled" type="warning">
|
||||||
|
Preview generation is disabled. Enable it in Configuration below.
|
||||||
|
</v-notice>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head">
|
||||||
|
<h2>Backfill progress</h2>
|
||||||
|
<div class="ip__actions">
|
||||||
|
<v-button kind="primary" :loading="starting" @click="startRun">Run now</v-button>
|
||||||
|
<v-button v-if="status?.run?.running" kind="danger" @click="cancelRun">Cancel</v-button>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<p class="ip__hint">
|
||||||
|
“Run now” processes the whole library for all enabled presets and ignores the maintenance
|
||||||
|
window. Progress refreshes every 3 seconds.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<v-progress-linear :value="percent" />
|
||||||
|
|
||||||
|
<div class="ip__grid">
|
||||||
|
<div><span class="ip__label">done</span> {{ status?.run?.done ?? 0 }} / {{ status?.run?.total ?? 0 }}</div>
|
||||||
|
<div><span class="ip__label">remaining</span> {{ remaining }}</div>
|
||||||
|
<div><span class="ip__label">generated</span> {{ status?.run?.generated ?? 0 }}</div>
|
||||||
|
<div><span class="ip__label">already cached</span> {{ status?.run?.hits ?? 0 }}</div>
|
||||||
|
<div><span class="ip__label">failed</span> {{ status?.run?.failed ?? 0 }}</div>
|
||||||
|
<div><span class="ip__label">rate</span> {{ status?.run?.rateHuman ?? '—' }}</div>
|
||||||
|
<div><span class="ip__label">eta</span> {{ status?.run?.etaHuman ?? '—' }}</div>
|
||||||
|
<div><span class="ip__label">last started</span> {{ status?.run?.startedAtHuman ?? '—' }}</div>
|
||||||
|
<div><span class="ip__label">last finished</span> {{ status?.run?.finishedAtHuman ?? '—' }}</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p class="ip__hint">
|
||||||
|
{{ status?.enabled ? 'enabled' : 'disabled' }} ·
|
||||||
|
{{ status?.inWindow ? 'inside maintenance window' : 'outside maintenance window' }} ·
|
||||||
|
next window {{ status?.nextWindowHuman ?? '—' }}
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head">
|
||||||
|
<h2>Storage</h2>
|
||||||
|
<v-button @click="loadDisk">Refresh</v-button>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<table class="ip__table" v-if="disk">
|
||||||
|
<thead>
|
||||||
|
<tr><th></th><th>size</th><th>files</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Preview assets</td><td>{{ disk.previews.bytesHuman }}</td><td>{{ disk.previews.count }}</td></tr>
|
||||||
|
<tr><td>Original assets</td><td>{{ disk.originals.bytesHuman }}</td><td>{{ disk.originals.count }}</td></tr>
|
||||||
|
<tr><td><strong>Total</strong></td><td><strong>{{ disk.total.bytesHuman }}</strong></td><td><strong>{{ disk.total.count }}</strong></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p v-else class="ip__hint">Loading…</p>
|
||||||
|
<p class="ip__hint">Preview assets are the generated transforms (<code><id>__<hash>.avif|webp</code>). Originals are the uploaded files.</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head"><h2>Configuration</h2></header>
|
||||||
|
|
||||||
|
<form v-if="config" class="ip__form" @submit.prevent="saveConfig">
|
||||||
|
<v-checkbox v-model="config.enabled" label="Enabled" />
|
||||||
|
<p class="ip__hint">Master switch. When off, no previews are generated (upload hook and backfill are inert).</p>
|
||||||
|
|
||||||
|
<v-checkbox v-model="config.onUpload" label="Warm on upload" />
|
||||||
|
<p class="ip__hint">
|
||||||
|
Generate previews immediately after an image is uploaded, so admin thumbnails load
|
||||||
|
instantly. When off, previews are generated on first view instead.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<v-checkbox v-model="config.debug" label="Debug logging" />
|
||||||
|
<p class="ip__hint">
|
||||||
|
Write verbose generation logs to the Directus server log (container logs /
|
||||||
|
<code>logs/directus.log</code>).
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="ip__row">
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>Maintenance window start</span>
|
||||||
|
<v-input v-model="config.windowStart" placeholder="01:00" />
|
||||||
|
</label>
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>Maintenance window end</span>
|
||||||
|
<v-input v-model="config.windowEnd" placeholder="04:00" />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<p class="ip__hint">
|
||||||
|
Scheduled runs only start inside this window (local time). “Run now” ignores it.
|
||||||
|
Default: 01:00–04:00.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>Concurrency</span>
|
||||||
|
<v-input v-model="config.concurrency" type="number" />
|
||||||
|
</label>
|
||||||
|
<p class="ip__hint">How many previews are generated in parallel. Keep low on CPU-limited hosts.</p>
|
||||||
|
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>Formats for “auto” presets</span>
|
||||||
|
<v-input v-model="formatsInput" placeholder="avif,webp" />
|
||||||
|
</label>
|
||||||
|
<p class="ip__hint">
|
||||||
|
Used only by presets whose format is <code>auto</code> (all system presets). Fixed-format
|
||||||
|
custom presets ignore this list and use their own format. One file is generated per
|
||||||
|
format listed here.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="ip__actions">
|
||||||
|
<v-button kind="primary" @click="saveConfig">Save</v-button>
|
||||||
|
<span class="ip__hint">{{ message }}</span>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head"><h2>Presets</h2></header>
|
||||||
|
<p class="ip__hint">
|
||||||
|
System presets are always generated with their built-in sizes (64 / 300 / 800 px,
|
||||||
|
cover/contain). Custom presets from Settings → File Storage are added automatically;
|
||||||
|
“Run now” also generates them for existing images.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<v-notice v-if="shadowedKeys.length" type="warning">
|
||||||
|
These custom preset keys are shadowed by a system preset and will not be used:
|
||||||
|
{{ shadowedKeys.join(', ') }}
|
||||||
|
</v-notice>
|
||||||
|
|
||||||
|
<table class="ip__table">
|
||||||
|
<thead>
|
||||||
|
<tr><th>key</th><th>format</th><th>generated formats</th><th>source</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr v-for="preset in presets" :key="preset.key">
|
||||||
|
<td class="ip__mono">{{ preset.key }}</td>
|
||||||
|
<td>{{ preset.format ?? 'original' }}</td>
|
||||||
|
<td>{{ preset.effectiveFormats.map((f) => f ?? 'original').join(', ') }}</td>
|
||||||
|
<td>{{ preset.system ? 'system' : 'custom' }}</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head"><h2>Diagnostics</h2></header>
|
||||||
|
<div class="ip__row">
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>File id</span>
|
||||||
|
<v-input v-model="fileId" placeholder="file uuid" />
|
||||||
|
</label>
|
||||||
|
<label class="ip__field">
|
||||||
|
<span>Preset</span>
|
||||||
|
<v-select v-model="selectedKey" :items="presetItems" placeholder="Select a preset" />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div class="ip__actions">
|
||||||
|
<v-button :disabled="!fileId || !selectedKey" @click="resolvePlan">Resolve plan</v-button>
|
||||||
|
<v-button :disabled="!fileId || !selectedKey" @click="generate(true)">Dry run</v-button>
|
||||||
|
<v-button kind="primary" :disabled="!fileId || !selectedKey" @click="generate(false)">Generate</v-button>
|
||||||
|
</div>
|
||||||
|
<p class="ip__hint">
|
||||||
|
“Resolve plan” predicts the exact cache filename without writing anything — use it to
|
||||||
|
verify a custom preset maps to the file the browser will request.
|
||||||
|
</p>
|
||||||
|
<pre v-if="plan" class="ip__pre">{{ JSON.stringify(plan, null, 2) }}</pre>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head">
|
||||||
|
<h2>Recent operations</h2>
|
||||||
|
<v-button @click="loadStatus">Refresh</v-button>
|
||||||
|
</header>
|
||||||
|
<table class="ip__table">
|
||||||
|
<thead>
|
||||||
|
<tr><th>time</th><th>file</th><th>key</th><th>format</th><th>ok</th><th>duration</th><th>error</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr v-for="(op, index) in ops" :key="index">
|
||||||
|
<td>{{ op.tsHuman }}</td>
|
||||||
|
<td class="ip__mono">{{ op.fileId.slice(0, 8) }}</td>
|
||||||
|
<td>{{ op.key }}</td>
|
||||||
|
<td>{{ op.format ?? 'original' }}</td>
|
||||||
|
<td>{{ op.ok ? '✓' : '✗' }}</td>
|
||||||
|
<td>{{ op.msHuman }}</td>
|
||||||
|
<td class="ip__error">{{ op.error ?? '' }}</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="ip__card">
|
||||||
|
<header class="ip__head"><h2>API reference</h2></header>
|
||||||
|
<p class="ip__hint">
|
||||||
|
Base path <code>/image-previews</code>. Every endpoint requires an admin session or an
|
||||||
|
admin token: <code>-H "Authorization: Bearer <token>"</code>.
|
||||||
|
</p>
|
||||||
|
<details v-for="endpoint in endpoints" :key="endpoint.method + endpoint.path" class="ip__details">
|
||||||
|
<summary><code>{{ endpoint.method }} {{ endpoint.path }}</code> — {{ endpoint.summary }}</summary>
|
||||||
|
<p class="ip__hint">{{ endpoint.description }}</p>
|
||||||
|
<pre class="ip__pre">{{ endpoint.example }}</pre>
|
||||||
|
</details>
|
||||||
|
</section>
|
||||||
|
</main>
|
||||||
|
</private-view>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup lang="ts">
|
||||||
|
import { computed, onMounted, onUnmounted, ref } from 'vue';
|
||||||
|
|
||||||
|
interface Bucket {
|
||||||
|
bytes: number;
|
||||||
|
count: number;
|
||||||
|
bytesHuman: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface RunState {
|
||||||
|
running: boolean;
|
||||||
|
done: number;
|
||||||
|
total: number;
|
||||||
|
generated: number;
|
||||||
|
hits: number;
|
||||||
|
failed: number;
|
||||||
|
rateHuman: string;
|
||||||
|
etaHuman: string;
|
||||||
|
startedAtHuman: string | null;
|
||||||
|
finishedAtHuman: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Status {
|
||||||
|
enabled: boolean;
|
||||||
|
inWindow: boolean;
|
||||||
|
nextWindowHuman: string | null;
|
||||||
|
run: RunState | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PresetInfo {
|
||||||
|
key: string;
|
||||||
|
format: string | null;
|
||||||
|
effectiveFormats: (string | null)[];
|
||||||
|
system: boolean;
|
||||||
|
enabled: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
const base = '/image-previews';
|
||||||
|
const status = ref<Status | null>(null);
|
||||||
|
const config = ref<Record<string, any> | null>(null);
|
||||||
|
const disk = ref<{ previews: Bucket; originals: Bucket; total: Bucket } | null>(null);
|
||||||
|
const ops = ref<any[]>([]);
|
||||||
|
const presets = ref<PresetInfo[]>([]);
|
||||||
|
const shadowedKeys = ref<string[]>([]);
|
||||||
|
const fileId = ref('');
|
||||||
|
const selectedKey = ref('');
|
||||||
|
const plan = ref<any>(null);
|
||||||
|
const message = ref('');
|
||||||
|
const formatsInput = ref('');
|
||||||
|
const starting = ref(false);
|
||||||
|
let timer: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
||||||
|
const presetItems = computed(() => presets.value.map((preset) => ({ text: preset.key, value: preset.key })));
|
||||||
|
|
||||||
|
const percent = computed(() => {
|
||||||
|
const run = status.value?.run;
|
||||||
|
if (!run || !run.total) return 0;
|
||||||
|
return Math.min(100, Math.round((run.done / run.total) * 100));
|
||||||
|
});
|
||||||
|
|
||||||
|
const remaining = computed(() => {
|
||||||
|
const run = status.value?.run;
|
||||||
|
if (!run) return 0;
|
||||||
|
return Math.max(0, run.total - run.done);
|
||||||
|
});
|
||||||
|
|
||||||
|
const endpoints = [
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/status',
|
||||||
|
summary: 'current run + window state',
|
||||||
|
description: 'Progress counters, rate, ETA, maintenance-window state, and human-readable dates.',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" $BASE/image-previews/status',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/config',
|
||||||
|
summary: 'read settings',
|
||||||
|
description: 'Returns the effective settings (env defaults merged with the persisted config).',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" $BASE/image-previews/config',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'PUT',
|
||||||
|
path: '/config',
|
||||||
|
summary: 'update settings',
|
||||||
|
description: 'Persists enabled, windowStart/windowEnd, concurrency, presetKeys, formats, onUpload, debug.',
|
||||||
|
example:
|
||||||
|
'curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \\\n -d \'{"enabled":true,"windowStart":"01:00","windowEnd":"04:00","concurrency":1,"formats":["avif","webp"],"onUpload":true,"debug":false,"presetKeys":null}\' \\\n $BASE/image-previews/config',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/presets',
|
||||||
|
summary: 'list presets',
|
||||||
|
description: 'System + custom presets with the formats each one generates; flags custom keys shadowed by system presets.',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" $BASE/image-previews/presets',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'POST',
|
||||||
|
path: '/run',
|
||||||
|
summary: 'start a backfill',
|
||||||
|
description: 'Body { "ignoreWindow": true } runs immediately; omit it to respect the maintenance window. Optional { "presets": ["key"] }.',
|
||||||
|
example: 'curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \\\n -d \'{"ignoreWindow":true}\' $BASE/image-previews/run',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'POST',
|
||||||
|
path: '/run/cancel',
|
||||||
|
summary: 'cancel a running backfill',
|
||||||
|
description: 'Sets a cancel flag; the backfill stops after the current file.',
|
||||||
|
example: 'curl -X POST -H "Authorization: Bearer $TOKEN" $BASE/image-previews/run/cancel',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/disk-usage',
|
||||||
|
summary: 'preview vs original storage',
|
||||||
|
description: 'Bytes and file counts for preview assets, original assets, and the total (machine + human-readable).',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" $BASE/image-previews/disk-usage',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'POST',
|
||||||
|
path: '/generate',
|
||||||
|
summary: 'generate one file',
|
||||||
|
description: 'Body { fileId, keys?, formats?, dryRun? }. Returns per preset/format results, filenames, sizes and durations.',
|
||||||
|
example: 'curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \\\n -d \'{"fileId":"<uuid>","keys":["system-medium-cover"],"dryRun":true}\' $BASE/image-previews/generate',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/resolve-plan',
|
||||||
|
summary: 'predict a cache filename',
|
||||||
|
description: 'Returns resolved transforms and the exact cache filename without writing anything.',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" "$BASE/image-previews/resolve-plan?fileId=<uuid>&key=system-medium-cover"',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
method: 'GET',
|
||||||
|
path: '/ops',
|
||||||
|
summary: 'recent operations',
|
||||||
|
description: 'In-memory ring buffer of the last 200 generation attempts with timing and error messages.',
|
||||||
|
example: 'curl -H "Authorization: Bearer $TOKEN" "$BASE/image-previews/ops?limit=50"',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
async function api(path: string, init?: RequestInit) {
|
||||||
|
const response = await fetch(`${base}${path}`, {
|
||||||
|
credentials: 'include',
|
||||||
|
headers: { 'Content-Type': 'application/json', ...(init?.headers ?? {}) },
|
||||||
|
...init,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) throw new Error(`${response.status} ${await response.text()}`);
|
||||||
|
|
||||||
|
return response.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadStatus() {
|
||||||
|
try {
|
||||||
|
status.value = await api('/status');
|
||||||
|
ops.value = (await api('/ops?limit=50')).ops;
|
||||||
|
} catch {
|
||||||
|
/* ignore polling errors */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadConfig() {
|
||||||
|
config.value = await api('/config');
|
||||||
|
formatsInput.value = (config.value?.formats ?? []).join(',');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadDisk() {
|
||||||
|
disk.value = await api('/disk-usage');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadPresets() {
|
||||||
|
const result = await api('/presets');
|
||||||
|
presets.value = result.presets;
|
||||||
|
shadowedKeys.value = result.shadowedKeys ?? [];
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveConfig() {
|
||||||
|
if (!config.value) return;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const payload = {
|
||||||
|
...config.value,
|
||||||
|
concurrency: Number(config.value.concurrency),
|
||||||
|
formats: formatsInput.value.split(',').map((part) => part.trim()).filter(Boolean),
|
||||||
|
};
|
||||||
|
|
||||||
|
config.value = await api('/config', { method: 'PUT', body: JSON.stringify(payload) });
|
||||||
|
formatsInput.value = (config.value?.formats ?? []).join(',');
|
||||||
|
message.value = 'saved';
|
||||||
|
await loadStatus();
|
||||||
|
} catch (error) {
|
||||||
|
message.value = String(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function startRun() {
|
||||||
|
starting.value = true;
|
||||||
|
message.value = '';
|
||||||
|
|
||||||
|
try {
|
||||||
|
await api('/run', { method: 'POST', body: JSON.stringify({ ignoreWindow: true }) });
|
||||||
|
message.value = 'backfill started';
|
||||||
|
await loadStatus();
|
||||||
|
} catch (error) {
|
||||||
|
message.value = String(error);
|
||||||
|
} finally {
|
||||||
|
starting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function cancelRun() {
|
||||||
|
try {
|
||||||
|
await api('/run/cancel', { method: 'POST', body: '{}' });
|
||||||
|
message.value = 'cancelling after the current file';
|
||||||
|
} catch (error) {
|
||||||
|
message.value = String(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function resolvePlan() {
|
||||||
|
try {
|
||||||
|
plan.value = await api(
|
||||||
|
`/resolve-plan?fileId=${encodeURIComponent(fileId.value)}&key=${encodeURIComponent(selectedKey.value)}`,
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
message.value = String(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function generate(dryRun: boolean) {
|
||||||
|
try {
|
||||||
|
plan.value = await api('/generate', {
|
||||||
|
method: 'POST',
|
||||||
|
body: JSON.stringify({ fileId: fileId.value, keys: [ selectedKey.value ], dryRun }),
|
||||||
|
});
|
||||||
|
await loadStatus();
|
||||||
|
await loadDisk();
|
||||||
|
} catch (error) {
|
||||||
|
message.value = String(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
onMounted(async () => {
|
||||||
|
await Promise.all([ loadStatus(), loadConfig(), loadDisk(), loadPresets() ]);
|
||||||
|
timer = setInterval(loadStatus, 3000);
|
||||||
|
});
|
||||||
|
|
||||||
|
onUnmounted(() => {
|
||||||
|
if (timer) clearInterval(timer);
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.ip {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 16px;
|
||||||
|
padding: 24px;
|
||||||
|
max-width: 1100px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__card {
|
||||||
|
border: 1px solid var(--border-normal);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__head {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
margin-bottom: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__head h2 {
|
||||||
|
font-size: 15px;
|
||||||
|
font-weight: 600;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__actions {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__row {
|
||||||
|
display: flex;
|
||||||
|
gap: 12px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__field {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 4px;
|
||||||
|
min-width: 180px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__form {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
|
||||||
|
gap: 8px;
|
||||||
|
margin: 12px 0;
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__label {
|
||||||
|
color: var(--foreground-subdued);
|
||||||
|
margin-right: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__hint {
|
||||||
|
color: var(--foreground-subdued);
|
||||||
|
font-size: 13px;
|
||||||
|
margin: 2px 0 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__pre {
|
||||||
|
background: var(--background-subdued);
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 12px;
|
||||||
|
overflow: auto;
|
||||||
|
max-height: 320px;
|
||||||
|
font-size: 12px;
|
||||||
|
white-space: pre-wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__table {
|
||||||
|
width: 100%;
|
||||||
|
border-collapse: collapse;
|
||||||
|
font-size: 13px;
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__table th,
|
||||||
|
.ip__table td {
|
||||||
|
text-align: left;
|
||||||
|
padding: 4px 8px;
|
||||||
|
border-bottom: 1px solid var(--border-subdued);
|
||||||
|
vertical-align: top;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__mono {
|
||||||
|
font-family: monospace;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__error {
|
||||||
|
color: var(--danger);
|
||||||
|
max-width: 320px;
|
||||||
|
word-break: break-word;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__details {
|
||||||
|
border-bottom: 1px solid var(--border-subdued);
|
||||||
|
padding: 6px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ip__details summary {
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
11
src/shims.d.ts
vendored
Normal file
11
src/shims.d.ts
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
declare module '*.vue' {
|
||||||
|
import type { DefineComponent } from 'vue';
|
||||||
|
|
||||||
|
const component: DefineComponent<{}, {}, any>;
|
||||||
|
export default component;
|
||||||
|
}
|
||||||
|
|
||||||
|
declare module 'object-hash' {
|
||||||
|
const hash: (value: unknown) => string;
|
||||||
|
export default hash;
|
||||||
|
}
|
||||||
28
tsconfig.json
Normal file
28
tsconfig.json
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2019",
|
||||||
|
"lib": ["ES2019", "DOM"],
|
||||||
|
"moduleResolution": "node",
|
||||||
|
"strict": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"noImplicitAny": true,
|
||||||
|
"noImplicitThis": true,
|
||||||
|
"noImplicitReturns": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUncheckedIndexedAccess": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"alwaysStrict": true,
|
||||||
|
"strictNullChecks": true,
|
||||||
|
"strictFunctionTypes": true,
|
||||||
|
"strictBindCallApply": true,
|
||||||
|
"strictPropertyInitialization": true,
|
||||||
|
"resolveJsonModule": false,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"allowSyntheticDefaultImports": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"rootDir": "./src"
|
||||||
|
},
|
||||||
|
"include": ["./src/**/*.ts"]
|
||||||
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue