chore: prepare for public release
- add repository/homepage/bugs metadata; target Directus ^11 - genericize internal deployment references - document install, compatibility, scheduling, security - stop tracking internal .opencode work artifacts
This commit is contained in:
parent
c8f8bbc5ac
commit
94f197bdec
10 changed files with 181 additions and 706 deletions
|
|
@ -1,249 +0,0 @@
|
|||
# Goal: Directus image preview pre-generation
|
||||
|
||||
Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (not part of the
|
||||
landshaft.life app repo).
|
||||
Location: `/home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews`
|
||||
Target server: `land-nest`, Directus **11.17.4** (self-hosted, PM2 cluster, local disk storage).
|
||||
Project type: Directus **bundle** extension (app module + api hook + api endpoint).
|
||||
|
||||
## Summary
|
||||
|
||||
Pre-generate Directus image transform ("preview") cache files so the admin UI never pays the
|
||||
first-request sharp cost. One warmer, several triggers:
|
||||
|
||||
- **upload hook** — after an image upload, warm its previews.
|
||||
- **new-preset trigger** — when a storage asset preset is added/changed, warm it for existing images.
|
||||
- **backfill job** — a resumable, scheduled sweep over the existing `directus_files` library, inside a
|
||||
configurable maintenance window.
|
||||
- **manual / diagnostics** — generate (or dry-run) previews for a single file on demand, to debug.
|
||||
|
||||
Plus an **admin module** ("Image Previews") with a **progress bar**, **disk usage**, a **config form**
|
||||
(persisted, no rebuild), and a **diagnostics panel**.
|
||||
|
||||
## Verified mechanism (source-checked against tag v11.17.4)
|
||||
|
||||
The on-disk transform cache file is:
|
||||
|
||||
```
|
||||
<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>
|
||||
```
|
||||
|
||||
written **beside the original** in the storage root (`STORAGE_LOCAL_ROOT=./uploads`).
|
||||
|
||||
- `hash` = `object-hash@3` defaults: sha1/hex, `unorderedArrays:false` (**array order matters**),
|
||||
`unorderedObjects:true`. Hashed over the **RESOLVED** transform array from `resolvePreset`.
|
||||
- `resolvePreset` order is fixed: `(1) copy transformationParams.transforms[]`, `(2) push toFormat`
|
||||
(if `format`/`quality`), `(3) push resize` (+`extract` on focal cover).
|
||||
- `maybeExtractFormat` picks the last `toFormat` arg as the file extension.
|
||||
- `AssetsService.getAsset` checks `exists(assetFilename)` first (cache hit → no sharp); on miss runs
|
||||
sharp and **awaits** `storage.write(...)` before returning. Not draining the returned stream only
|
||||
leaks an FD (never truncates).
|
||||
- Called as sudo (`new AssetsService({ schema })`) it needs **no** HTTP request context.
|
||||
|
||||
## Why the naive / community approach FAILS (must not repeat)
|
||||
|
||||
1. **`{ key }` alone is a no-op.** `resolvePreset` never reads `key`. Pass the **full preset object**.
|
||||
2. **`format:'auto'` mismatch.** Every `system-*` preset is `format:'auto'`. `getFormat` with no
|
||||
`acceptFormat` → JPEG→`jpg`/PNG→`png`; a browser `Accept: image/avif` → `avif`. Different
|
||||
`toFormat` arg → different hash → **guaranteed miss**.
|
||||
3. Disproved myths: the write is awaited; the admin does NOT use `width`/`height`.
|
||||
4. Raw params resolve to `[toFormat, resize]`; a preset resolves to `[resize, toFormat]` → different hash.
|
||||
|
||||
## Chosen strategy — IN-PROCESS generation (no token, no HTTP, no rate limiter)
|
||||
|
||||
Because this is an internal extension, generate by calling Directus's own service in-process:
|
||||
|
||||
```ts
|
||||
const assets = new services.AssetsService({ schema }); // sudo
|
||||
const { stream } = await assets.getAsset(fileId, {
|
||||
transformationParams: presetObject, // FULL object, not { key }
|
||||
acceptFormat: 'avif', // or 'webp'
|
||||
});
|
||||
(stream as Readable).destroy?.();
|
||||
```
|
||||
|
||||
`acceptFormat` reproduces exactly what a browser's `Accept` header does, so the cache filename is
|
||||
byte-identical to the admin/browser read — **without any HTTP request, token, proxy, or rate limiter**.
|
||||
Generate **two variants** for `format:'auto'` presets (`avif`, `webp`); fixed-`format` presets yield
|
||||
one; resize-only presets (no `format`/`quality`) yield none.
|
||||
|
||||
HTTP-replay (`GET /assets/{id}?key=…` with a browser `Accept` + auth) is the documented **fallback**
|
||||
only — it needs a token and rate-limiter pacing, which is why it is not the default.
|
||||
|
||||
## Preset registry (system + settings, dynamic)
|
||||
|
||||
The controller builds its preset list inline as:
|
||||
|
||||
```
|
||||
res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]
|
||||
```
|
||||
|
||||
and resolves `?key=` with `.find(t => t.key === key)` → **first match wins** (a system key shadows a
|
||||
custom preset of the same key). There is **no** reusable helper and `GET /settings` exposes **only**
|
||||
custom presets.
|
||||
|
||||
Our registry (`src/lib/presets.ts`) mirrors this exact list:
|
||||
|
||||
- **System presets**: 6 objects in `SYSTEM_ASSET_ALLOW_LIST` (`system-small|medium|large` ×
|
||||
`cover|contain`), each `format:'auto'`, resize inside `transforms[]`; sourced at runtime from
|
||||
`@directus/api` **if** importable, else the version-pinned copy `src/lib/system-presets.ts` (v11.17.4).
|
||||
- **Custom presets**: read `directus_settings.storage_asset_presets` each run; pass the verbatim object.
|
||||
- **Dynamic**: newly created presets are picked up per run; the new-preset trigger warms them for
|
||||
existing images.
|
||||
|
||||
## Admin UI (module "Image Previews")
|
||||
|
||||
- **Progress**: bar + `done / total / remaining`, throughput, ETA, last-run, next-window; live during a run.
|
||||
- **Storage breakdown**: bytes + file counts separately for **preview assets** (`__<hash>`), **original
|
||||
assets**, and the total — raw and human-readable. Computed by listing the storage root and `stat()`-ing
|
||||
each file (listing yields paths only, no sizes); the state dir is excluded.
|
||||
- **Config form** (persisted, no rebuild; defaults `enabled:true`, window `01:00–04:00`): `enabled`,
|
||||
maintenance window, concurrency, preset selection, formats, `onUpload`, `debug`; each field carries a
|
||||
plain-language explanation. **Run now** ignores the window; **Cancel** is shown only while a run is active.
|
||||
- **Diagnostics panel**: recent operations, per-file generate/dry-run, resolve-plan viewer.
|
||||
|
||||
## Diagnostics & manual generation (observability)
|
||||
|
||||
- `POST /image-previews/generate` `{ fileId, keys?, formats?, dryRun? }` → generate (or dry-run) for ONE
|
||||
file; returns per `(key,format)` `{ ok, failed, cached, filename, bytes, ms, error }`. Primary dev tool.
|
||||
- `GET /image-previews/resolve-plan?fileId=&key=` → resolved transforms + **predicted cache filename** +
|
||||
whether it already exists, WITHOUT generating. The hash-match debugging tool.
|
||||
- `GET /image-previews/ops?limit=` → recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error).
|
||||
- `debug` setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
|
||||
- Everything except `/generate` is read-only.
|
||||
|
||||
## Safe deployment & staging (must not crash prod)
|
||||
|
||||
- **Fail-open, inert by default**: every entrypoint is tiny, bodies wrapped in try/catch, hooks never
|
||||
throw into Directus core, and ALL behavior is gated by `enabled` (default `false`). A loaded-but-disabled
|
||||
extension does nothing.
|
||||
- **Local dev**: `directus-extension link` (or the dev compose stack) loads `src/` live for iteration.
|
||||
- **Staging stack**: a second compose project (`dev/compose.dev.yml`) running the SAME Directus 11.17.4
|
||||
image with its own pg+redis, a **restored anonymized dump**, and a **copy of uploads**, on a separate
|
||||
port — realistic data, zero prod risk.
|
||||
- **Rollout discipline**: build → verify on staging → back up the previous `dist/` → copy in → healthcheck
|
||||
→ enable via config → rollback = restore previous `dist/` + disable.
|
||||
- `EXTENSIONS_AUTO_RELOAD` in prod reloads on file change; deploy with it controlled and accept the brief
|
||||
reload, or restart deliberately.
|
||||
|
||||
## Remote feedback & autonomous operation (security)
|
||||
|
||||
- Feedback channel = the extension's own **machine-readable endpoints** (`/status`, `/ops`,
|
||||
`/disk-usage`, `/resolve-plan`). Automation reads those, not CMS content.
|
||||
- **Least privilege**:
|
||||
- A Directus **admin token = full CMS control**. Do NOT give it to an autonomous loop. Use a scoped
|
||||
token reaching only `/image-previews/*` (+ `directus_files` metadata read).
|
||||
- A **Docker socket = root-equivalent**. Never expose the prod socket to automation; keep the loop on staging.
|
||||
- Deployment access = a dedicated **non-sudo** SSH user with a restricted key (`restrict`, `from=`,
|
||||
`command=`), or run opencode on the staging host via `handoff`.
|
||||
- **Network**: staging reachable only over a private network / VPN / tunnel — never public.
|
||||
- **Prompt-injection**: treat logs, filenames, and image metadata as untrusted input; keep tooling scoped.
|
||||
- **Promotion to prod is a human action**: the loop proposes, a human merges/deploys.
|
||||
- Guard long autonomous runs with a cost/time budget.
|
||||
|
||||
## Design
|
||||
|
||||
```
|
||||
src/
|
||||
hook/index.ts (api) files.upload -> warmer; settings/presets change -> warm existing
|
||||
endpoint/index.ts (api) routes: /config (GET/PUT), /status, /run, /run/cancel,
|
||||
/disk-usage, /generate, /resolve-plan, /ops
|
||||
module/
|
||||
index.ts (app) defineModule({ id:'image-previews', name, icon, routes })
|
||||
module.vue progress + disk usage + config form + diagnostics panel
|
||||
lib/
|
||||
presets.ts merged registry (system + settings), mirrors controller semantics
|
||||
system-presets.ts version-pinned SYSTEM_ASSET_ALLOW_LIST copy (v11.17.4)
|
||||
warmer.ts in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
|
||||
backfill.ts enumerate directus_files (image/*), window-aware, resumable
|
||||
state.ts state.jsonl read/append/resume
|
||||
settings.ts config file read/write + env defaults
|
||||
disk.ts sum sizes of __<hash> files (driver.list + stat)
|
||||
window.ts maintenance-window logic (is-now-in-window, next-window)
|
||||
diagnostics.ts resolve-plan/dry-run (resolved transforms + predicted filename), ops ring buffer
|
||||
dev/
|
||||
compose.dev.yml local Directus 11.17.4 + postgres + redis, extension linked
|
||||
seed/ anonymized DB dump + sample images for local/staging runs
|
||||
```
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No Directus upgrade (server stays on 11.17.4; license constraint).
|
||||
- No change to admin preset configuration or to how the admin builds URLs.
|
||||
- Never hand-write `__<hash>.<ext>` filenames.
|
||||
- No CDN/nginx caching work (separate ops concern). No image `quality` policy change.
|
||||
- Automation never gets prod Docker socket or a full prod admin token.
|
||||
|
||||
## Constraints / risks
|
||||
|
||||
- `ASSETS_TRANSFORM_MAX_CONCURRENT=45` per worker; over it errors. Catch per file/preset and continue
|
||||
(never fail the upload).
|
||||
- `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000 is a **pixel dimension** guard in Directus, not a
|
||||
file-size limit. The extension imposes no limit of its own; raise the Directus env var to process larger
|
||||
images (watch memory). Upload-size limits (`FILES_MAX_UPLOAD_SIZE`, `MAX_PAYLOAD_SIZE`) are unrelated.
|
||||
- Local storage write is **non-atomic** → warm only AFTER an upload is committed; serialize so the
|
||||
upload hook and the backfill never generate the same file concurrently.
|
||||
- PM2 cluster (8 workers): the upload hook fires on the handling worker; the backfill runs in the worker
|
||||
that served the run request. Shared state MUST be file/DB-backed (never in-memory).
|
||||
- In-process backfill competes with live API serving → the maintenance window + concurrency cap govern
|
||||
that. (HTTP rate limiter is irrelevant in-process.)
|
||||
- Observed server bottleneck (2026-10-23): `storage:local:responseTime 1206ms`, `pg:responseTime`
|
||||
150–738ms; no 503s, no sharp errors.
|
||||
|
||||
## Verification policy
|
||||
|
||||
- `npm run typecheck` → 0 errors; `npm test` (node:test via tsx) green.
|
||||
- **The hash-match gate (F09)**: for a real file, the on-disk filename produced by a browser/admin
|
||||
request for `?key=system-medium-cover` is byte-identical to the warmer's output for the same key+format
|
||||
(+ the resolved extension matches, e.g. `.avif`).
|
||||
- Cold-vs-warm timing; preview-file count + total bytes before/after a run.
|
||||
- All feature testing happens on the **staging** stack; prod deploy only at goal completion, via the
|
||||
documented backup/rollback procedure.
|
||||
|
||||
## Expected files
|
||||
|
||||
- `package.json`, `tsconfig.json`, `.gitignore`
|
||||
- `src/hook/`, `src/endpoint/`, `src/module/`, `src/lib/`, plus `src/**/*.test.ts`
|
||||
- `dev/compose.dev.yml`, `dev/seed/`
|
||||
- `.opencode/goal.md`, `.opencode/feature_list.json`, `.opencode/progress.txt`
|
||||
- `README.md`, `AGENTS.md`
|
||||
|
||||
## Resolved decisions (2026-10-25)
|
||||
|
||||
1. State/config: JSON files under `<STORAGE_LOCAL_ROOT>/.image-previews` (default
|
||||
`/directus/uploads/.image-previews`), override with `IMAGE_PREVIEWS_STATE_DIR`. The former
|
||||
`/directus/extensions/.image-previews` default was rejected — root-owned volume → `EACCES`.
|
||||
2. System presets: version-pinned copy (`src/lib/system-presets.ts`, verbatim v11.17.4); no
|
||||
`@directus/api` import needed for presets.
|
||||
3. New-preset trigger: manual `POST /run` (idempotent) + the module's Run button; no
|
||||
`settings.update` auto-trigger.
|
||||
4. Backfill scheduling: external cron → `POST /image-previews/run`.
|
||||
5. Dev/staging: isolated local compose stack (`dev/compose.dev.yml`). A land-nest staging project is
|
||||
still to be created for realistic-data runs.
|
||||
6. Autonomous-loop grants: still to be defined (scoped token / restricted SSH, staging only).
|
||||
|
||||
## Implementation status (2026-10-25)
|
||||
|
||||
Implemented and verified against an isolated Directus 11.17.4: settings/state, presets + warmer,
|
||||
upload hook, diagnostics, backfill, endpoints. `npm test` 32/32, typecheck clean, build emits both
|
||||
bundles. **Hash-match gate passed** (a real browser-style asset request produced exactly the
|
||||
predicted filename). Features F03–F06, F08, F09 pass.
|
||||
|
||||
Outstanding: admin-module browser verification (F07), staging deploy/rollback + prod rollout (F02/F10),
|
||||
and the F01 publish-to-staging step.
|
||||
|
||||
## Reference: source anchors (v11.17.4)
|
||||
|
||||
- `api/src/services/assets.ts` — `getAsset`, `getAssetSuffix`, filename build, awaited `write`.
|
||||
- `api/src/utils/transformations.ts` — `resolvePreset`, `getFormat`.
|
||||
- `api/src/controllers/assets.ts` — inline `shortcuts = [...SYSTEM_ASSET_ALLOW_LIST, ...settings]`,
|
||||
`acceptFormat` selection from `req.headers.accept`.
|
||||
- `api/src/constants.ts` — `SYSTEM_ASSET_ALLOW_LIST` (6 presets, `format:'auto'`);
|
||||
`ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` default 6000.
|
||||
- `app/src/utils/get-asset-url.ts`, `app/src/layouts/cards/components/card.vue`,
|
||||
`app/src/interfaces/file-image/file-image.vue`, `app/src/displays/image/image.vue` — admin URLs:
|
||||
Files grid `?key=system-medium-cover&v=<modified_on>`; image field `system-large-cover|contain`;
|
||||
image display `system-small-cover`.
|
||||
- `packages/storage-driver-local/src/index.ts` — non-atomic `write`; `list()` returns paths only.
|
||||
- Prior art that does NOT work as-is: `directus-extension-auto-generate-file-transformations`
|
||||
(utomic-media) — still falls foul of the `auto`→format mismatch unless it forces the format.
|
||||
Loading…
Add table
Add a link
Reference in a new issue