Rawman/directus-extension-generate-image-previews
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
directus-extension-generate.../.opencode/goal.md

18 KiB
Raw Blame History

Goal: Directus image preview pre-generation

Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (independent of any host application). Location: /home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews Branch: goal/image-previews (created from master 2026-10-30). Target: self-hosted Directus 11.17.4 (PM2 cluster, local disk storage). Project type: Directus bundle extension (app module + api hook + api endpoint).

Summary

Pre-generate Directus image transform ("preview") cache files so the admin UI never pays the first-request sharp cost. One warmer, several triggers:

  • upload hook — after an image upload, warm its previews.
  • new-preset trigger — when a storage asset preset is added/changed, warm it for existing images.
  • backfill job — a resumable, scheduled sweep over the existing directus_files library, inside a configurable maintenance window.
  • manual / diagnostics — generate (or dry-run) previews for a single file on demand, to debug.

Plus an admin module ("Image Previews") with a progress bar, disk usage, a config form (persisted, no rebuild), and a diagnostics panel.

Verified mechanism (source-checked against tag v11.17.4)

The on-disk transform cache file is:

<basename(filename_disk)>__<object-hash(resolved transforms)>.<ext>

written beside the original in the storage root (STORAGE_LOCAL_ROOT=./uploads).

  • hash = object-hash@3 defaults: sha1/hex, unorderedArrays:false (array order matters), unorderedObjects:true. Hashed over the RESOLVED transform array from resolvePreset.
  • resolvePreset order is fixed: (1) copy transformationParams.transforms[], (2) push toFormat (if format/quality), (3) push resize (+extract on focal cover).
  • maybeExtractFormat picks the last toFormat arg as the file extension.
  • AssetsService.getAsset checks exists(assetFilename) first (cache hit → no sharp); on miss runs sharp and awaits storage.write(...) before returning. Not draining the returned stream only leaks an FD (never truncates).
  • Called as sudo (new AssetsService({ schema })) it needs no HTTP request context.

Why the naive / community approach FAILS (must not repeat)

  1. { key } alone is a no-op. resolvePreset never reads key. Pass the full preset object.
  2. format:'auto' mismatch. Every system-* preset is format:'auto'. getFormat with no acceptFormat → JPEG→jpg/PNG→png; a browser Accept: image/avif → avif. Different toFormat arg → different hash → guaranteed miss.
  3. Disproved myths: the write is awaited; the admin does NOT use width/height.
  4. Raw params resolve to [toFormat, resize]; a preset resolves to [resize, toFormat] → different hash.

Chosen strategy — IN-PROCESS generation (no token, no HTTP, no rate limiter)

Because this is an internal extension, generate by calling Directus's own service in-process:

const assets = new services.AssetsService({ schema }); // sudo
const { stream } = await assets.getAsset(fileId, {
  transformationParams: presetObject,                  // FULL object, not { key }
  acceptFormat: 'avif',                                // or 'webp'
});
(stream as Readable).destroy?.();

acceptFormat reproduces exactly what a browser's Accept header does, so the cache filename is byte-identical to the admin/browser read — without any HTTP request, token, proxy, or rate limiter. Generate two variants for format:'auto' presets (avif, webp); fixed-format presets yield one; resize-only presets (no format/quality) yield none.

HTTP-replay (GET /assets/{id}?key=… with a browser Accept + auth) is the documented fallback only — it needs a token and rate-limiter pacing, which is why it is not the default.

Preset registry (system + settings, dynamic)

The controller builds its preset list inline as:

res.locals['shortcuts'] = [...SYSTEM_ASSET_ALLOW_LIST, ...(directus_settings.storage_asset_presets ?? [])]

and resolves ?key= with .find(t => t.key === key) → first match wins (a system key shadows a custom preset of the same key). There is no reusable helper and GET /settings exposes only custom presets.

Our registry (src/lib/presets.ts) mirrors this exact list:

  • System presets: 6 objects in SYSTEM_ASSET_ALLOW_LIST (system-small|medium|large × cover|contain), each format:'auto', resize inside transforms[]; sourced at runtime from @directus/api if importable, else the version-pinned copy src/lib/system-presets.ts (v11.17.4).
  • Custom presets: read directus_settings.storage_asset_presets each run; pass the verbatim object.
  • Dynamic: newly created presets are picked up per run; the new-preset trigger warms them for existing images.

Admin UI (module "Image Previews")

  • Progress: bar + done / total / remaining, throughput, ETA, last-run, next-window; live during a run.
  • Storage breakdown: bytes + file counts separately for preview assets (__<hash>), original assets, and the total — raw and human-readable. Computed by listing the storage root and stat()-ing each file (listing yields paths only, no sizes); the state dir is excluded.
  • Config form (persisted, no rebuild; defaults enabled:true, window 01:00–04:00): enabled, maintenance window, concurrency, preset selection, formats, onUpload, debug; each field carries a plain-language explanation. Run now ignores the window; Cancel is shown only while a run is active.
  • Diagnostics panel: recent operations, per-file generate/dry-run, resolve-plan viewer.

Diagnostics & manual generation (observability)

  • POST /image-previews/generate { fileId, keys?, formats?, dryRun? } → generate (or dry-run) for ONE file; returns per (key,format) { ok, failed, cached, filename, bytes, ms, error }. Primary dev tool.
  • GET /image-previews/resolve-plan?fileId=&key= → resolved transforms + predicted cache filename + whether it already exists, WITHOUT generating. The hash-match debugging tool.
  • GET /image-previews/ops?limit= → recent-operations ring buffer (timestamp, fileId, key, format, ok, ms, error).
  • debug setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
  • Everything except /generate is read-only.

Failure journal & bounded retries (2026-10-30)

Backfill failures are currently only counted in run.json and held in the in-memory /ops ring buffer (last 200 entries, per PM2 worker). After a long run the details are lost. Add a durable failure journal and a bounded retry path.

  • failures.jsonl under the state dir = append-only events {type:'failed'|'retry'|'recovered', ts, file_id, preset_key, format, attempt, next_retry_at?, error?, ms?}. Replay derives the latest state per (file_id, preset_key, format); append-only so PM2 workers can write safely.
  • Every failure (upload hook, manual /generate, backfill) is journaled. Default retry policy: max 3 attempts (initial + 2 retries), backoff 5 min then 15 min. Due retries run only while enabled, inside the maintenance window (manual Run now may ignore it), with no active backfill, under the same run.lock; targeted file/preset/format retries — never a full-library sweep.
  • Success after retry appends a normal state.jsonl completion and marks the failure recovered. Exhausted retries stay failed with the last error and remain readable.
  • GET /image-previews/failures?status=&limit=&offset= returns persisted records + file metadata, attempts, timestamps, next retry and error text. Admin-only, read-only.
  • Module: "Failed previews" table (file, preset, format, attempts, last error, next retry, status). No toasts or user-facing notifications.
  • Log every failure at warn with ids/metadata + error; log recoveries at info. No file contents.
  • Journal is compacted under the run lock: keep unresolved rows + last 1,000 recovered; never grow unbounded. /ops remains the live in-memory view.

Host data fix: uploaded_on for TUS uploads (2026-10-30)

Scope split (user decision): DB access and execution stay with the user; this repo only ships the SQL script + runbook. No extension code change for this item.

Directus 11.17.4 TUS uploads omit uploaded_on (TusDataStore.create + onUploadFinish); the regular POST /files path sets it. Upstream issue #28042 is fixed only in 12.4.0; this host stays on 11.17.4.

  • Fix existing rows directly in Postgres: update directus_files set uploaded_on = coalesce(created_on, now()) where uploaded_on is null; in a transaction, after a table backup. Raw SQL touches only uploaded_on; it does not bump modified_on/modified_by, emit events, or create revisions.
  • Add a UTC column default so future TUS inserts get a date: alter table directus_files alter column uploaded_on set default (now() at time zone 'utc'); Confirm one new TUS upload has non-null uploaded_on and tus_id = null.
  • Keep TUS_ENABLED=true; no Directus upgrade and no extension code required. Latent TUS defects (client-declared type/filesize, stale tus_id placeholders) are documented, not changed.

Safe deployment & staging (must not crash prod)

  • Fail-open, inert by default: every entrypoint is tiny, bodies wrapped in try/catch, hooks never throw into Directus core, and ALL behavior is gated by enabled (default false). A loaded-but-disabled extension does nothing.
  • Local dev: directus-extension link (or the dev compose stack) loads src/ live for iteration.
  • Staging stack: a second compose project (dev/compose.dev.yml) running the SAME Directus 11.17.4 image with its own pg+redis, a restored anonymized dump, and a copy of uploads, on a separate port — realistic data, zero prod risk.
  • Rollout discipline: build → verify on staging → back up the previous dist/ → copy in → healthcheck → enable via config → rollback = restore previous dist/ + disable.
  • EXTENSIONS_AUTO_RELOAD in prod reloads on file change; deploy with it controlled and accept the brief reload, or restart deliberately.

Remote feedback & autonomous operation (security)

  • Feedback channel = the extension's own machine-readable endpoints (/status, /ops, /disk-usage, /resolve-plan). Automation reads those, not CMS content.
  • Least privilege:
    • A Directus admin token = full CMS control. Do NOT give it to an autonomous loop. Use a scoped token reaching only /image-previews/* (+ directus_files metadata read).
    • A Docker socket = root-equivalent. Never expose the prod socket to automation; keep the loop on staging.
    • Deployment access = a dedicated non-sudo SSH user with a restricted key (restrict, from=, command=), or run opencode on the staging host via handoff.
  • Network: staging reachable only over a private network / VPN / tunnel — never public.
  • Prompt-injection: treat logs, filenames, and image metadata as untrusted input; keep tooling scoped.
  • Promotion to prod is a human action: the loop proposes, a human merges/deploys.
  • Guard long autonomous runs with a cost/time budget.

Design

src/
  hook/index.ts          (api) files.upload -> warmer; settings/presets change -> warm existing
  endpoint/index.ts      (api) routes: /config (GET/PUT), /status, /run, /run/cancel,
                                /disk-usage, /generate, /resolve-plan, /ops
  module/
    index.ts             (app) defineModule({ id:'image-previews', name, icon, routes })
    module.vue           progress + disk usage + config form + diagnostics panel
  lib/
    presets.ts           merged registry (system + settings), mirrors controller semantics
    system-presets.ts    version-pinned SYSTEM_ASSET_ALLOW_LIST copy (v11.17.4)
    warmer.ts            in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
    backfill.ts          enumerate directus_files (image/*), window-aware, resumable
    state.ts             state.jsonl read/append/resume
    failures.ts          failures.jsonl append/replay/compact
    retry.ts             due-retry selection + backoff/max-attempt policy
    settings.ts          config file read/write + env defaults
    disk.ts              sum sizes of __<hash> files (driver.list + stat)
    window.ts            maintenance-window logic (is-now-in-window, next-window)
    diagnostics.ts       resolve-plan/dry-run (resolved transforms + predicted filename), ops ring buffer
dev/
  compose.dev.yml        local Directus 11.17.4 + postgres + redis, extension linked
  seed/                  anonymized DB dump + sample images for local/staging runs

Non-goals

  • No Directus upgrade (server stays on 11.17.4; license constraint).
  • No change to admin preset configuration or to how the admin builds URLs.
  • Never hand-write __<hash>.<ext> filenames.
  • No CDN/nginx caching work (separate ops concern). No image quality policy change.
  • Automation never gets prod Docker socket or a full prod admin token.

Constraints / risks

  • ASSETS_TRANSFORM_MAX_CONCURRENT=45 per worker; over it errors. Catch per file/preset and continue (never fail the upload).
  • ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION default 6000 is a pixel dimension guard in Directus, not a file-size limit. The extension imposes no limit of its own; raise the Directus env var to process larger images (watch memory). Upload-size limits (FILES_MAX_UPLOAD_SIZE, MAX_PAYLOAD_SIZE) are unrelated.
  • Local storage write is non-atomic → warm only AFTER an upload is committed; serialize so the upload hook and the backfill never generate the same file concurrently.
  • PM2 cluster (8 workers): the upload hook fires on the handling worker; the backfill runs in the worker that served the run request. Shared state MUST be file/DB-backed (never in-memory).
  • In-process backfill competes with live API serving → the maintenance window + concurrency cap govern that. (HTTP rate limiter is irrelevant in-process.)
  • Observed server bottleneck (2026-10-23): storage:local:responseTime 1206ms, pg:responseTime 150–738ms; no 503s, no sharp errors.

Verification policy

  • npm run typecheck → 0 errors; npm test (node:test via tsx) green.
  • The hash-match gate (F09): for a real file, the on-disk filename produced by a browser/admin request for ?key=system-medium-cover is byte-identical to the warmer's output for the same key+format (+ the resolved extension matches, e.g. .avif).
  • Cold-vs-warm timing; preview-file count + total bytes before/after a run.
  • All feature testing happens on the staging stack; prod deploy only at goal completion, via the documented backup/rollback procedure.

Expected files

  • package.json, tsconfig.json, .gitignore
  • src/hook/, src/endpoint/, src/module/, src/lib/, plus src/**/*.test.ts
  • dev/compose.dev.yml, dev/seed/
  • .opencode/goal.md, .opencode/feature_list.json, .opencode/progress.txt
  • README.md, AGENTS.md

Resolved decisions (2026-10-25)

  1. State/config: JSON files under <STORAGE_LOCAL_ROOT>/.image-previews (default /directus/uploads/.image-previews), override with IMAGE_PREVIEWS_STATE_DIR. The former /directus/extensions/.image-previews default was rejected — root-owned volume → EACCES.
  2. System presets: version-pinned copy (src/lib/system-presets.ts, verbatim v11.17.4); no @directus/api import needed for presets.
  3. New-preset trigger: manual POST /run (idempotent) + the module's Run button; no settings.update auto-trigger.
  4. Backfill scheduling: in-process via schedule('*/15 * * * *') in the hook — no external cron or token. It runs only inside the maintenance window and pauses when the window closes.
  5. Dev/staging: isolated local compose stack (dev/compose.dev.yml). A staging project is still to be created for realistic-data runs.
  6. Autonomous-loop grants: still to be defined (scoped token / restricted SSH, staging only).

Implementation status (2026-10-25)

Implemented and verified against an isolated Directus 11.17.4: settings/state, presets + warmer, upload hook, diagnostics, backfill, endpoints. npm test 32/32, typecheck clean, build emits both bundles. Hash-match gate passed (a real browser-style asset request produced exactly the predicted filename). Features F03–F06, F08, F09 pass.

Outstanding: admin-module browser verification (F07), staging deploy/rollback + prod rollout (F02/F10), and the F01 publish-to-staging step.

Reference: source anchors (v11.17.4)

  • api/src/services/assets.ts — getAsset, getAssetSuffix, filename build, awaited write.
  • api/src/utils/transformations.ts — resolvePreset, getFormat.
  • api/src/controllers/assets.ts — inline shortcuts = [...SYSTEM_ASSET_ALLOW_LIST, ...settings], acceptFormat selection from req.headers.accept.
  • api/src/constants.ts — SYSTEM_ASSET_ALLOW_LIST (6 presets, format:'auto'); ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION default 6000.
  • app/src/utils/get-asset-url.ts, app/src/layouts/cards/components/card.vue, app/src/interfaces/file-image/file-image.vue, app/src/displays/image/image.vue — admin URLs: Files grid ?key=system-medium-cover&v=<modified_on>; image field system-large-cover|contain; image display system-small-cover.
  • packages/storage-driver-local/src/index.ts — non-atomic write; list() returns paths only.
  • Prior art that does NOT work as-is: directus-extension-auto-generate-file-transformations (utomic-media) — still falls foul of the auto→format mismatch unless it forces the format.