directus-extension-generate-image-previews/.opencode/progress.txt

# progress.txt — directus-extension-generate-image-previews

Append-only log of decisions, evidence, and learnings. Newest at the bottom.

## 2026-10-25 — project scaffolded

- Opened this standalone extension project to pre-generate Directus image previews.
- Source-verified the strategy against Directus **v11.17.4** (see `.opencode/goal.md`).
- Root causes why existing/community extensions do not work:
  1. `resolvePreset` never reads `key` → `{transformationParams:{key}}` yields `transforms=[]` → no-op.
  2. `system-*` presets are `format:'auto'` → `jpg`/`png` server-side vs `avif` in-browser → different hash.
  3. "undrained stream truncates file" — false (write is awaited).
  4. admin uses `?key=system-*`, not raw width/height.
- Admin URLs (v11.17.4 app source): files grid `?key=system-medium-cover&v=<modified_on>`;
  image field `system-large-cover|contain`; image display `system-small-cover`.
- Server bottleneck evidence (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
  `pg:responseTime` 150–738ms; no 503s, no sharp errors.

## 2026-10-25 — scope expanded after user direction (v2)

- **No token needed.** Dropped HTTP-replay as primary. Strategy: **in-process**
  `AssetsService.getAsset(id, { transformationParams: <full preset object>, acceptFormat: 'avif'|'webp' })`.
  No HTTP → no token, no proxy, no rate limiter. (HTTP-replay kept as documented fallback.)
- **Bundle extension**: api `hook` + api `endpoint` + app `module`.
- **Presets: all, dynamically** — system `SYSTEM_ASSET_ALLOW_LIST` (6) + `directus_settings.storage_asset_presets`,
  controller order `[...SYSTEM, ...custom]`, first-match-wins; new presets picked up per run.
- **Admin module**: progress bar, disk usage (`list`+`stat`), config form, diagnostics panel.
- **Config**: maintenance window + concurrency (throttle knob; in-process bypasses HTTP rate limiter),
  persisted in JSON.
- Confirmed `acceptFormat:'avif'` in a direct call equals browser `Accept: image/avif`.

## 2026-10-25 — diagnostics + safety + autonomy (v3)

- **Diagnostics / manual generation** (F06): `POST /generate` (one file, dry-run supported),
  `GET /resolve-plan` (resolved transforms + predicted filename, no write), `GET /ops` (ring buffer).
- **Safe testing** (F02): staging compose stack + deploy/rollback scripts; fail-open, `enabled` default false.
- **Autonomous remote feedback**: staging-only, least privilege (scoped token, no prod Docker socket,
  restricted SSH, private network, human-only prod promotion).

## 2026-10-25 — implementation + local verification (autonomous session)

Built the full extension locally and smoke-tested against an **isolated** Directus 11.17.4 stack
(`dev/compose.dev.yml`, port 8057, own pg + volumes). No contact with the prod instance.

### Implemented
- `src/lib/`: settings (env + JSON), state (`state.jsonl`, run state, lock, cancel flag),
  presets (pinned system copy + custom merge, controller parity), warmer (in-process `getAsset` with
  `acceptFormat` variants, concurrency, per-task error isolation, stream release), window, storage
  (dynamic `getStorage` + local-FS fallback), diagnostics (resolve-plan prediction + ops ring buffer),
  backfill (paginated, resumable, window-aware, idempotent).
- `src/endpoint/index.ts`: `/status`, `/config` (GET/PUT), `/presets`, `/run`, `/run/cancel`,
  `/disk-usage`, `/generate`, `/resolve-plan`, `/ops`; admin-only guard.
- `src/hook/index.ts`: `files.upload` → fire-and-forget warm.
- `src/module/`: admin module (progress bar, disk usage, config form, diagnostics panel, ops table).
- `dev/`: compose stack, `sync-extension.sh`, seed images.

### Bugs found & fixed during smoke test
1. **Directus coerces env values** (`"true"` → boolean `true`, `"1"` → number). My string-only parsers
   threw `e.toLowerCase is not a function`, which silently broke the upload hook. Fixed `settings.ts`
   to accept `unknown` and stringify. Regression test added.
2. **State dir EACCES.** `/directus/extensions/.image-previews` (named volume on a non-existent path)
   is root-owned; the API runs as `node`. This broke config writes and the backfill lock. Fixed: default
   state dir is now `<STORAGE_LOCAL_ROOT>/.image-previews` (i.e. `/directus/uploads/.image-previews`),
   which is writable; `IMAGE_PREVIEWS_STATE_DIR` still overrides.
3. **Bind-mount reload gotcha.** `rm -rf` of the bind-mounted extension dir breaks the Docker mount
   (Directus removes the extension and never re-adds it). `sync-extension.sh` now updates in place;
   restart the dev container after a sync.

### Verified evidence (local stack)
- `npm run typecheck` → 0 errors. `npm test` → **32/32 pass**. `npm run build` → `dist/app.js` + `dist/api.js`.
- Directus logs: `Loaded extensions: directus-extension-generate-image-previews` (no errors).
- `/status` → `{enabled:true, inWindow:true, locked:false}`; `/presets` → 6 system presets.
- Upload a JPEG → hook warmed **12 files** (6 presets × avif/webp) in `/directus/uploads`.
- **HASH-MATCH GATE PASSED:** after deleting the cached files, a real
  `GET /assets/<id>?key=system-medium-cover` with `Accept: image/avif` produced exactly
  `<id>__f15aa4b815722867859d2a3fa46ab50834dfc8dc.avif` — byte-identical to `resolve-plan`'s prediction.
- `/generate` dry-run reports `exists:true/false` correctly; `/ops` shows per-task timing.
- Backfill: `total=36` (3 images × 6 presets × 2 formats), `generated=36`, state `state.jsonl` = 36 lines.
  Second run: `generated=0, hits=36`, state unchanged, file count unchanged → **idempotent/resumable**.
- Window gating: `/run` outside `03:00–05:00` → `409 outside-window` with `nextWindow`.
- `/disk-usage` → `157633 bytes / 36 files (source: storage)`.
- Cold vs warm (`system-medium-cover`): 114 ms → 16–30 ms. (`system-large-cover`): 304 ms → 36 ms.

### Decisions recorded
- System presets: version-pinned copy in `src/lib/system-presets.ts` (verbatim from `api/src/constants.ts`
  v11.17.4). No `@directus/api` import needed for presets.
- Storage/disk usage: `@directus/api/storage/index` loaded via dynamic import (hidden from the bundler),
  with a local-FS fallback; shares the host `getStorage()` singleton.
- New-preset trigger: **manual** via `POST /run` (idempotent) + the module's Run button; no
  `settings.update` auto-trigger (avoids heavy background runs on unrelated settings changes).
- Scheduling: external cron → `POST /image-previews/run` (default).
- State/config dir: `<STORAGE_LOCAL_ROOT>/.image-previews` (writable), override with `IMAGE_PREVIEWS_STATE_DIR`.

### Additional verification (same session)
- `GET /extensions` → bundle registered with entries `module`, `hook`, `endpoint`, all enabled.
- Inert test: with `enabled:false`, uploading an image produced **zero** new preview files (25 → 25).
- Added `dev/compose.staging.yml`, `dev/deploy-staging.sh`, `dev/rollback-staging.sh` (templates;
  NOT executed against any remote host).
- Added `storage.test.ts`; total tests now **34/34**.
- Restarted the dev stack after the final build; `/status`, `/disk-usage`, and bundle entries all good.

### Still open
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
- Cancel + concurrent-run 409 not exercised end-to-end (unit-tested only; local runs finish too fast).
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
- F01/F02/F07/F10 remain `passes:false` (F01 publish step, F02 staging run, F07 browser, F10 rollout).

## 2026-10-25 — feedback round (UI, defaults, custom formats, docs)

- **Defaults changed**: `enabled` now `true`; maintenance window defaults to `01:00–04:00`.
- **Custom fixed-format presets verified**: `custom-png` (format `png`) generated `.png` and
  `resolve-plan` predicted the exact filename (`exists:true`). `custom-auto` generated avif+webp.
  An unsupported preset (`format: bmp`) failed with Directus's clear error
  ("Expected one of: heic, heif, avif, jpeg, … received bmp") and is surfaced in Recent operations / `/ops`.
- **Effective format fix**: ops/presets now report the preset's real output format (`png`) instead of
  `null`; backfill resume keys use the effective format (fixed a re-warm mismatch).
- **Progress accounting fix**: already-cached tasks now count toward `done`/`hits`. Runs report
  `128/128` (120 hits + 8 expected failures) instead of `8/128`. After removing the unsupported preset:
  `120/120 hits, 0 failed, 0 generated`.
- **Storage reporting**: `/disk-usage` + panel now split **previews** vs **originals** vs **total**
  (raw + human): e.g. previews 774.7 KB / 115 files, originals 24.9 MB / 9 files, total 25.6 MB / 124.
- **Human-readable fields** added to `/status` (ISO + human dates, `etaHuman`, `rateHuman`),
  `/disk-usage` (`bytesHuman`), `/ops` (`tsHuman`, `msHuman`), `/generate` (`msHuman`).
- **Admin module rewritten with Directus UI kit** (`v-button`, `v-input`, `v-checkbox`, `v-select`,
  `v-notice`, `v-progress-linear`), field explanations, preset table with effective formats + shadow
  warnings, storage table, error column in Recent operations, and a full collapsible **API reference**.
- **Run now** now ignores the maintenance window; **Cancel** only renders while a run is active.
- **Dimension vs file size clarified** in README/goal/AGENTS: the 6000 limit is a pixel-dimension guard in
  Directus, not a file-size limit; the extension imposes none.
- Tests: **40/40** pass; typecheck clean; build OK.

### Still open
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
- Cancel + concurrent-run 409 not exercised end-to-end.
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
- F01/F02/F07/F10 remain `passes:false`.

## 2026-10-25 — production deploy + accuracy fixes + in-process scheduler

- Deployed to prod via local build + SCP archive (prod has no Node). A source clone is kept outside the
  watched extensions directory.
- Verified: bundle loaded on all 4 PM2 workers, endpoints mounted (403 admin guard), no errors.
- **Backfill counting fix**: pre-scan storage filenames; each task's predicted cache filename is checked
  before generating. Existing previews count as `already cached` and are skipped. (Commit 18db52a.)
- **In-process nightly scheduler**: hook registers `schedule('*/15 * * * *')`; runs only inside the
  maintenance window, no external cron/token, pauses when the window closes and resumes next night.
- UI now shows last-run errors and `paused (window closed)`.
- Tests 43/43, typecheck clean.