directus-extension-generate-image-previews/.opencode/goal.md

Goal: Directus image preview pre-generation

Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (not part of the landshaft.life app repo). Location: /home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews Target server: land-nest, Directus 11.17.4 (self-hosted, PM2 cluster, local disk storage). Project type: Directus bundle extension (app module + api hook + api endpoint).

Summary

Pre-generate Directus image transform ("preview") cache files so the admin UI never pays the first-request sharp cost. One warmer, several triggers:

• upload hook — after an image upload, warm its previews.
• new-preset trigger — when a storage asset preset is added/changed, warm it for existing images.
• backfill job — a resumable, scheduled sweep over the existing directus_files library, inside a configurable maintenance window.
• manual / diagnostics — generate (or dry-run) previews for a single file on demand, to debug.

Plus an admin module ("Image Previews") with a progress bar, disk usage, a config form (persisted, no rebuild), and a diagnostics panel.

Verified mechanism (source-checked against tag v11.17.4)

The on-disk transform cache file is:

<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>

written beside the original in the storage root (STORAGE_LOCAL_ROOT=./uploads).

• hash = object-hash@3 defaults: sha1/hex, unorderedArrays:false (array order matters), unorderedObjects:true. Hashed over the RESOLVED transform array from resolvePreset.
• resolvePreset order is fixed: (1) copy transformationParams.transforms[], (2) push toFormat (if format/quality), (3) push resize (+extract on focal cover).
• maybeExtractFormat picks the last toFormat arg as the file extension.
• AssetsService.getAsset checks exists(assetFilename) first (cache hit → no sharp); on miss runs sharp and awaits storage.write(...) before returning. Not draining the returned stream only leaks an FD (never truncates).
• Called as sudo (new AssetsService({ schema })) it needs no HTTP request context.

Why the naive / community approach FAILS (must not repeat)

1. { key } alone is a no-op. resolvePreset never reads key. Pass the full preset object.
2. format:'auto' mismatch. Every system-* preset is format:'auto'. getFormat with no acceptFormat → JPEG→jpg/PNG→png; a browser Accept: image/avif → avif. Different toFormat arg → different hash → guaranteed miss.
3. Disproved myths: the write is awaited; the admin does NOT use width/height.
4. Raw params resolve to [toFormat, resize]; a preset resolves to [resize, toFormat] → different hash.

Chosen strategy — IN-PROCESS generation (no token, no HTTP, no rate limiter)

Because this is an internal extension, generate by calling Directus's own service in-process:

const assets = new services.AssetsService({ schema }); // sudo
const { stream } = await assets.getAsset(fileId, {
  transformationParams: presetObject,                  // FULL object, not { key }
  acceptFormat: 'avif',                                // or 'webp'
});
(stream as Readable).destroy?.();

acceptFormat reproduces exactly what a browser's Accept header does, so the cache filename is byte-identical to the admin/browser read — without any HTTP request, token, proxy, or rate limiter. Generate two variants for format:'auto' presets (avif, webp); fixed-format presets yield one; resize-only presets (no format/quality) yield none.

HTTP-replay (GET /assets/{id}?key=… with a browser Accept + auth) is the documented fallback only — it needs a token and rate-limiter pacing, which is why it is not the default.

Preset registry (system + settings, dynamic)

The controller builds its preset list inline as:

res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]

and resolves ?key= with .find(t => t.key === key) → first match wins (a system key shadows a custom preset of the same key). There is no reusable helper and GET /settings exposes only custom presets.

Our registry (src/lib/presets.ts) mirrors this exact list:

• System presets: 6 objects in SYSTEM_ASSET_ALLOW_LIST (system-small|medium|large × cover|contain), each format:'auto', resize inside transforms[]; sourced at runtime from @directus/api if importable, else the version-pinned copy src/lib/system-presets.ts (v11.17.4).
• Custom presets: read directus_settings.storage_asset_presets each run; pass the verbatim object.
• Dynamic: newly created presets are picked up per run; the new-preset trigger warms them for existing images.

Admin UI (module "Image Previews")

• Progress: bar + done / total / remaining, throughput, ETA, last-run, next-window; live during a run.
• Storage breakdown: bytes + file counts separately for preview assets (__<hash>), original assets, and the total — raw and human-readable. Computed by listing the storage root and stat()-ing each file (listing yields paths only, no sizes); the state dir is excluded.
• Config form (persisted, no rebuild; defaults enabled:true, window 01:00–04:00): enabled, maintenance window, concurrency, preset selection, formats, onUpload, debug; each field carries a plain-language explanation. Run now ignores the window; Cancel is shown only while a run is active.
• Diagnostics panel: recent operations, per-file generate/dry-run, resolve-plan viewer.

Diagnostics & manual generation (observability)

• POST /image-previews/generate { fileId, keys?, formats?, dryRun? } → generate (or dry-run) for ONE file; returns per (key,format) { ok, failed, cached, filename, bytes, ms, error }. Primary dev tool.
• GET /image-previews/resolve-plan?fileId=&key= → resolved transforms + predicted cache filename + whether it already exists, WITHOUT generating. The hash-match debugging tool.
• GET /image-previews/ops?limit= → recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error).
• debug setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
• Everything except /generate is read-only.

Safe deployment & staging (must not crash prod)

• Fail-open, inert by default: every entrypoint is tiny, bodies wrapped in try/catch, hooks never throw into Directus core, and ALL behavior is gated by enabled (default false). A loaded-but-disabled extension does nothing.
• Local dev: directus-extension link (or the dev compose stack) loads src/ live for iteration.
• Staging stack: a second compose project (dev/compose.dev.yml) running the SAME Directus 11.17.4 image with its own pg+redis, a restored anonymized dump, and a copy of uploads, on a separate port — realistic data, zero prod risk.
• Rollout discipline: build → verify on staging → back up the previous dist/ → copy in → healthcheck → enable via config → rollback = restore previous dist/ + disable.
• EXTENSIONS_AUTO_RELOAD in prod reloads on file change; deploy with it controlled and accept the brief reload, or restart deliberately.

Remote feedback & autonomous operation (security)

• Feedback channel = the extension's own machine-readable endpoints (/status, /ops, /disk-usage, /resolve-plan). Automation reads those, not CMS content.
• Least privilege:
  • A Directus admin token = full CMS control. Do NOT give it to an autonomous loop. Use a scoped token reaching only /image-previews/* (+ directus_files metadata read).
  • A Docker socket = root-equivalent. Never expose the prod socket to automation; keep the loop on staging.
  • Deployment access = a dedicated non-sudo SSH user with a restricted key (restrict, from=, command=), or run opencode on the staging host via handoff.
• Network: staging reachable only over a private network / VPN / tunnel — never public.
• Prompt-injection: treat logs, filenames, and image metadata as untrusted input; keep tooling scoped.
• Promotion to prod is a human action: the loop proposes, a human merges/deploys.
• Guard long autonomous runs with a cost/time budget.

Design

src/
  hook/index.ts          (api) files.upload -> warmer; settings/presets change -> warm existing
  endpoint/index.ts      (api) routes: /config (GET/PUT), /status, /run, /run/cancel,
                                /disk-usage, /generate, /resolve-plan, /ops
  module/
    index.ts             (app) defineModule({ id:'image-previews', name, icon, routes })
    module.vue           progress + disk usage + config form + diagnostics panel
  lib/
    presets.ts           merged registry (system + settings), mirrors controller semantics
    system-presets.ts    version-pinned SYSTEM_ASSET_ALLOW_LIST copy (v11.17.4)
    warmer.ts            in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
    backfill.ts          enumerate directus_files (image/*), window-aware, resumable
    state.ts             state.jsonl read/append/resume
    settings.ts          config file read/write + env defaults
    disk.ts              sum sizes of __<hash> files (driver.list + stat)
    window.ts            maintenance-window logic (is-now-in-window, next-window)
    diagnostics.ts       resolve-plan/dry-run (resolved transforms + predicted filename), ops ring buffer
dev/
  compose.dev.yml        local Directus 11.17.4 + postgres + redis, extension linked
  seed/                  anonymized DB dump + sample images for local/staging runs

Non-goals

• No Directus upgrade (server stays on 11.17.4; license constraint).
• No change to admin preset configuration or to how the admin builds URLs.
• Never hand-write __<hash>.<ext> filenames.
• No CDN/nginx caching work (separate ops concern). No image quality policy change.
• Automation never gets prod Docker socket or a full prod admin token.

Constraints / risks

• ASSETS_TRANSFORM_MAX_CONCURRENT=45 per worker; over it errors. Catch per file/preset and continue (never fail the upload).
• ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION default 6000 is a pixel dimension guard in Directus, not a file-size limit. The extension imposes no limit of its own; raise the Directus env var to process larger images (watch memory). Upload-size limits (FILES_MAX_UPLOAD_SIZE, MAX_PAYLOAD_SIZE) are unrelated.
• Local storage write is non-atomic → warm only AFTER an upload is committed; serialize so the upload hook and the backfill never generate the same file concurrently.
• PM2 cluster (8 workers): the upload hook fires on the handling worker; the backfill runs in the worker that served the run request. Shared state MUST be file/DB-backed (never in-memory).
• In-process backfill competes with live API serving → the maintenance window + concurrency cap govern that. (HTTP rate limiter is irrelevant in-process.)
• Observed server bottleneck (2026-10-23): storage:local:responseTime 1206ms, pg:responseTime 150–738ms; no 503s, no sharp errors.

Verification policy

• npm run typecheck → 0 errors; npm test (node:test via tsx) green.
• The hash-match gate (F09): for a real file, the on-disk filename produced by a browser/admin request for ?key=system-medium-cover is byte-identical to the warmer's output for the same key+format (+ the resolved extension matches, e.g. .avif).
• Cold-vs-warm timing; preview-file count + total bytes before/after a run.
• All feature testing happens on the staging stack; prod deploy only at goal completion, via the documented backup/rollback procedure.

Expected files

• package.json, tsconfig.json, .gitignore
• src/hook/, src/endpoint/, src/module/, src/lib/, plus src/**/*.test.ts
• dev/compose.dev.yml, dev/seed/
• .opencode/goal.md, .opencode/feature_list.json, .opencode/progress.txt
• README.md, AGENTS.md

Resolved decisions (2026-10-25)

1. State/config: JSON files under <STORAGE_LOCAL_ROOT>/.image-previews (default /directus/uploads/.image-previews), override with IMAGE_PREVIEWS_STATE_DIR. The former /directus/extensions/.image-previews default was rejected — root-owned volume → EACCES.
2. System presets: version-pinned copy (src/lib/system-presets.ts, verbatim v11.17.4); no @directus/api import needed for presets.
3. New-preset trigger: manual POST /run (idempotent) + the module's Run button; no settings.update auto-trigger.
4. Backfill scheduling: external cron → POST /image-previews/run.
5. Dev/staging: isolated local compose stack (dev/compose.dev.yml). A land-nest staging project is still to be created for realistic-data runs.
6. Autonomous-loop grants: still to be defined (scoped token / restricted SSH, staging only).

Implementation status (2026-10-25)

Implemented and verified against an isolated Directus 11.17.4: settings/state, presets + warmer, upload hook, diagnostics, backfill, endpoints. npm test 32/32, typecheck clean, build emits both bundles. Hash-match gate passed (a real browser-style asset request produced exactly the predicted filename). Features F03–F06, F08, F09 pass.

Outstanding: admin-module browser verification (F07), staging deploy/rollback + prod rollout (F02/F10), and the F01 publish-to-staging step.

Reference: source anchors (v11.17.4)

• api/src/services/assets.ts — getAsset, getAssetSuffix, filename build, awaited write.
• api/src/utils/transformations.ts — resolvePreset, getFormat.
• api/src/controllers/assets.ts — inline shortcuts = [...SYSTEM_ASSET_ALLOW_LIST, ...settings], acceptFormat selection from req.headers.accept.
• api/src/constants.ts — SYSTEM_ASSET_ALLOW_LIST (6 presets, format:'auto'); ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION default 6000.
• app/src/utils/get-asset-url.ts, app/src/layouts/cards/components/card.vue, app/src/interfaces/file-image/file-image.vue, app/src/displays/image/image.vue — admin URLs: Files grid ?key=system-medium-cover&v=<modified_on>; image field system-large-cover|contain; image display system-small-cover.
• packages/storage-driver-local/src/index.ts — non-atomic write; list() returns paths only.
• Prior art that does NOT work as-is: directus-extension-auto-generate-file-transformations (utomic-media) — still falls foul of the auto→format mismatch unless it forces the format.