# progress.txt — directus-extension-generate-image-previews

Append-only log of decisions, evidence, and learnings. Newest at the bottom.

## 2026-10-25 — project scaffolded

- Opened this standalone extension project to pre-generate Directus image previews.
- Source-verified the strategy against Directus **v11.17.4** (see `.opencode/goal.md`).
- Root causes why existing/community extensions do not work:
  1. `resolvePreset` never reads `key` → `{transformationParams:{key}}` yields `transforms=[]` → no-op.
  2. `system-*` presets are `format:'auto'` → `jpg`/`png` server-side vs `avif` in-browser → different hash.
  3. "undrained stream truncates file" — false (write is awaited).
  4. admin uses `?key=system-*`, not raw width/height.
- Admin URLs (v11.17.4 app source): files grid `?key=system-medium-cover&v=<modified_on>`;
  image field `system-large-cover|contain`; image display `system-small-cover`.
- Server bottleneck evidence (2026-10-23 log triage): `storage:local:responseTime 1206ms`,
  `pg:responseTime` 150–738ms; no 503s, no sharp errors.

## 2026-10-25 — scope expanded after user direction (v2)

- **No token needed.** Dropped HTTP-replay as primary. Strategy: **in-process**
  `AssetsService.getAsset(id, { transformationParams: <full preset object>, acceptFormat: 'avif'|'webp' })`.
  No HTTP → no token, no proxy, no rate limiter. (HTTP-replay kept as documented fallback.)
- **Bundle extension**: api `hook` + api `endpoint` + app `module`.
- **Presets: all, dynamically** — system `SYSTEM_ASSET_ALLOW_LIST` (6) + `directus_settings.storage_asset_presets`,
  controller order `[...SYSTEM, ...custom]`, first-match-wins; new presets picked up per run.
- **Admin module**: progress bar, disk usage (`list`+`stat`), config form, diagnostics panel.
- **Config**: maintenance window + concurrency (throttle knob; in-process bypasses HTTP rate limiter),
  persisted in JSON.
- Confirmed `acceptFormat:'avif'` in a direct call equals browser `Accept: image/avif`.

## 2026-10-25 — diagnostics + safety + autonomy (v3)

- **Diagnostics / manual generation** (F06): `POST /generate` (one file, dry-run supported),
  `GET /resolve-plan` (resolved transforms + predicted filename, no write), `GET /ops` (ring buffer).
- **Safe testing** (F02): staging compose stack + deploy/rollback scripts; fail-open, `enabled` default false.
- **Autonomous remote feedback**: staging-only, least privilege (scoped token, no prod Docker socket,
  restricted SSH, private network, human-only prod promotion).

## 2026-10-25 — implementation + local verification (autonomous session)

Built the full extension locally and smoke-tested against an **isolated** Directus 11.17.4 stack
(`dev/compose.dev.yml`, port 8057, own pg + volumes). No contact with the prod instance.

### Implemented
- `src/lib/`: settings (env + JSON), state (`state.jsonl`, run state, lock, cancel flag),
  presets (pinned system copy + custom merge, controller parity), warmer (in-process `getAsset` with
  `acceptFormat` variants, concurrency, per-task error isolation, stream release), window, storage
  (dynamic `getStorage` + local-FS fallback), diagnostics (resolve-plan prediction + ops ring buffer),
  backfill (paginated, resumable, window-aware, idempotent).
- `src/endpoint/index.ts`: `/status`, `/config` (GET/PUT), `/presets`, `/run`, `/run/cancel`,
  `/disk-usage`, `/generate`, `/resolve-plan`, `/ops`; admin-only guard.
- `src/hook/index.ts`: `files.upload` → fire-and-forget warm.
- `src/module/`: admin module (progress bar, disk usage, config form, diagnostics panel, ops table).
- `dev/`: compose stack, `sync-extension.sh`, seed images.

### Bugs found & fixed during smoke test
1. **Directus coerces env values** (`"true"` → boolean `true`, `"1"` → number). My string-only parsers
   threw `e.toLowerCase is not a function`, which silently broke the upload hook. Fixed `settings.ts`
   to accept `unknown` and stringify. Regression test added.
2. **State dir EACCES.** `/directus/extensions/.image-previews` (named volume on a non-existent path)
   is root-owned; the API runs as `node`. This broke config writes and the backfill lock. Fixed: default
   state dir is now `<STORAGE_LOCAL_ROOT>/.image-previews` (i.e. `/directus/uploads/.image-previews`),
   which is writable; `IMAGE_PREVIEWS_STATE_DIR` still overrides.
3. **Bind-mount reload gotcha.** `rm -rf` of the bind-mounted extension dir breaks the Docker mount
   (Directus removes the extension and never re-adds it). `sync-extension.sh` now updates in place;
   restart the dev container after a sync.

### Verified evidence (local stack)
- `npm run typecheck` → 0 errors. `npm test` → **32/32 pass**. `npm run build` → `dist/app.js` + `dist/api.js`.
- Directus logs: `Loaded extensions: directus-extension-generate-image-previews` (no errors).
- `/status` → `{enabled:true, inWindow:true, locked:false}`; `/presets` → 6 system presets.
- Upload a JPEG → hook warmed **12 files** (6 presets × avif/webp) in `/directus/uploads`.
- **HASH-MATCH GATE PASSED:** after deleting the cached files, a real
  `GET /assets/<id>?key=system-medium-cover` with `Accept: image/avif` produced exactly
  `<id>__f15aa4b815722867859d2a3fa46ab50834dfc8dc.avif` — byte-identical to `resolve-plan`'s prediction.
- `/generate` dry-run reports `exists:true/false` correctly; `/ops` shows per-task timing.
- Backfill: `total=36` (3 images × 6 presets × 2 formats), `generated=36`, state `state.jsonl` = 36 lines.
  Second run: `generated=0, hits=36`, state unchanged, file count unchanged → **idempotent/resumable**.
- Window gating: `/run` outside `03:00–05:00` → `409 outside-window` with `nextWindow`.
- `/disk-usage` → `157633 bytes / 36 files (source: storage)`.
- Cold vs warm (`system-medium-cover`): 114 ms → 16–30 ms. (`system-large-cover`): 304 ms → 36 ms.

### Decisions recorded
- System presets: version-pinned copy in `src/lib/system-presets.ts` (verbatim from `api/src/constants.ts`
  v11.17.4). No `@directus/api` import needed for presets.
- Storage/disk usage: `@directus/api/storage/index` loaded via dynamic import (hidden from the bundler),
  with a local-FS fallback; shares the host `getStorage()` singleton.
- New-preset trigger: **manual** via `POST /run` (idempotent) + the module's Run button; no
  `settings.update` auto-trigger (avoids heavy background runs on unrelated settings changes).
- Scheduling: external cron → `POST /image-previews/run` (default).
- State/config dir: `<STORAGE_LOCAL_ROOT>/.image-previews` (writable), override with `IMAGE_PREVIEWS_STATE_DIR`.

### Additional verification (same session)
- `GET /extensions` → bundle registered with entries `module`, `hook`, `endpoint`, all enabled.
- Inert test: with `enabled:false`, uploading an image produced **zero** new preview files (25 → 25).
- Added `dev/compose.staging.yml`, `dev/deploy-staging.sh`, `dev/rollback-staging.sh` (templates;
  NOT executed against any remote host).
- Added `storage.test.ts`; total tests now **34/34**.
- Restarted the dev stack after the final build; `/status`, `/disk-usage`, and bundle entries all good.

### Still open
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
- Cancel + concurrent-run 409 not exercised end-to-end (unit-tested only; local runs finish too fast).
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
- F01/F02/F07/F10 remain `passes:false` (F01 publish step, F02 staging run, F07 browser, F10 rollout).

## 2026-10-25 — feedback round (UI, defaults, custom formats, docs)

- **Defaults changed**: `enabled` now `true`; maintenance window defaults to `01:00–04:00`.
- **Custom fixed-format presets verified**: `custom-png` (format `png`) generated `.png` and
  `resolve-plan` predicted the exact filename (`exists:true`). `custom-auto` generated avif+webp.
  An unsupported preset (`format: bmp`) failed with Directus's clear error
  ("Expected one of: heic, heif, avif, jpeg, … received bmp") and is surfaced in Recent operations / `/ops`.
- **Effective format fix**: ops/presets now report the preset's real output format (`png`) instead of
  `null`; backfill resume keys use the effective format (fixed a re-warm mismatch).
- **Progress accounting fix**: already-cached tasks now count toward `done`/`hits`. Runs report
  `128/128` (120 hits + 8 expected failures) instead of `8/128`. After removing the unsupported preset:
  `120/120 hits, 0 failed, 0 generated`.
- **Storage reporting**: `/disk-usage` + panel now split **previews** vs **originals** vs **total**
  (raw + human): e.g. previews 774.7 KB / 115 files, originals 24.9 MB / 9 files, total 25.6 MB / 124.
- **Human-readable fields** added to `/status` (ISO + human dates, `etaHuman`, `rateHuman`),
  `/disk-usage` (`bytesHuman`), `/ops` (`tsHuman`, `msHuman`), `/generate` (`msHuman`).
- **Admin module rewritten with Directus UI kit** (`v-button`, `v-input`, `v-checkbox`, `v-select`,
  `v-notice`, `v-progress-linear`), field explanations, preset table with effective formats + shadow
  warnings, storage table, error column in Recent operations, and a full collapsible **API reference**.
- **Run now** now ignores the maintenance window; **Cancel** only renders while a run is active.
- **Dimension vs file size clarified** in README/goal/AGENTS: the 6000 limit is a pixel-dimension guard in
  Directus, not a file-size limit; the extension imposes none.
- Tests: **40/40** pass; typecheck clean; build OK.

### Still open
- Admin module UI not yet verified in a real browser (bundle + entries registered; rendering/polling untested).
- Cancel + concurrent-run 409 not exercised end-to-end.
- Staging deploy not executed (scripts/templates provided); prod rollout not done.
- F01/F02/F07/F10 remain `passes:false`.

## 2026-10-25 — production deploy + accuracy fixes + in-process scheduler

- Deployed to prod via local build + SCP archive (prod has no Node). A source clone is kept outside the
  watched extensions directory.
- Verified: bundle loaded on all 4 PM2 workers, endpoints mounted (403 admin guard), no errors.
- **Backfill counting fix**: pre-scan storage filenames; each task's predicted cache filename is checked
  before generating. Existing previews count as `already cached` and are skipped. (Commit 18db52a.)
- **In-process nightly scheduler**: hook registers `schedule('*/15 * * * *')`; runs only inside the
  maintenance window, no external cron/token, pauses when the window closes and resumes next night.
- UI now shows last-run errors and `paused (window closed)`.
- Tests 43/43, typecheck clean.

## 2026-10-30 — file-library ordering / TUS `uploaded_on` / failure observability

- User report: fresh files absent from folder list unless filtered by `uploaded_on`; 4,649-item Documents
  folder; page size raised to 1000 remained slow.
- Directus 11.17.4 source check: filtered result is an `_and` subset of unfiltered; no app item cache.
  Postgres `ORDER BY uploaded_on DESC` = NULLS FIRST, so a large null-date block fills every page before
  fresh rows. TUS (`TusDataStore.create` + `onUploadFinish`) never sets `uploaded_on`; regular
  `POST /files` does. Upstream #28042 fixed only in 12.4.0; host stays 11.17.4.
- Host checks (pgAdmin): `created_on IS NULL` = **0**; `directus_files` total **39,350**,
  `uploaded_on IS NULL` = **2,836**, dated = **36,514**.
- `uploaded_on` default added successfully: `DEFAULT (now() AT TIME ZONE 'utc')`.
- **Backfill UPDATE was not applied**: `uploaded_on IS NULL` still 2,836. Defaults do not backfill
  history. Re-run `UPDATE directus_files SET uploaded_on = created_on WHERE uploaded_on IS NULL;`
  (transaction, backup) and verify null count = 0.
- Slow thumbnails: backfill only ~95,744 / 661,232 tasks done (~14%); loading 1,000 cards issues 1,000
  transform requests, most cold → on-demand sharp. Use 25–100 page size; let backfill finish or run
  outside-window with low concurrency.
- Added feature specs F11 (durable `failures.jsonl` + bounded retry + `/failures` + module table) and
  F12 (host `uploaded_on` remediation). F12 remains `passes:false` until the backfill UPDATE is
  actually applied and one new TUS upload is verified.

## 2026-10-30 — production outage: uncommitted pgAdmin DDL held lock (resolved)

- Symptom: Directus UI/API slow, 500s, health timeouts, container unhealthy. `directus.log` showed
  repeated `KnexTimeoutError: Timeout acquiring a connection`, `Can't connect to the database`, and
  60s request aborts.
- Root cause: pgAdmin backend **PID 674331** (`pgAdmin 4 - CONN:4204762`) was `idle in transaction`
  holding a granted `AccessExclusiveLock` on `public.directus_files`. Its last query was the
  `ALTER TABLE ... ALTER COLUMN uploaded_on SET DEFAULT ...`; it was never committed. All 80 Directus
  pool connections (4 workers × 20) were active waiting on `Lock/relation` for `directus_files`, so the
  API pool was exhausted.
- A `commit;` from a different pgAdmin connection (PID 674244) does not commit another session's
  transaction. DDL must be run with auto-commit on or `BEGIN; ...; COMMIT;` in the same session.
- Recovery: terminated PID 674331 (`pg_terminate_backend`). Its uncommitted ALTER rolled back; lock
  released; Directus pool drained; `/server/health` returned `{"status":"ok"}` and container health
  returned `healthy`.
- Re-applied default safely with `lock_timeout=3s`:
  `DEFAULT (now() AT TIME ZONE 'utc')`. Verified in `information_schema.columns`.
- Applied backfill in one transaction with `lock_timeout=5s`, `statement_timeout=30s`:
  `UPDATE directus_files SET uploaded_on = created_on WHERE uploaded_on IS NULL;` → **UPDATE 2836**,
  COMMIT. Verified `null_dates = 0`, `dated = 39,350`, zero `Lock` waiters afterwards.
- No code/config deploy involved. Raw SQL did not touch `modified_on`/`modified_by`.
- Remaining verification: upload one TUS file and confirm non-null `uploaded_on` + `tus_id = null`;
  then confirm the folder list shows it at top with `-uploaded_on`.
- Verified on 2026-10-30 after recovery:
  - latest 5 TUS uploads all have non-null `uploaded_on`, `tus_id = null`, correct `folder`
  - `uploaded_on - created_on` ≈ 0–0.24s (default is applied at INSERT, finish is immediate)
  - `show timezone` = `Etc/UTC`, so `(now() AT TIME ZONE 'utc')` cast to `timestamptz` is correct
  - zero rows `state = 'idle in transaction'`; no lock waiters
- DB/TUS acceptance for F12 is done. Only remaining check: browser Files view shows a fresh TUS file
  at the top of the unfiltered folder with `sort=-uploaded_on` and `page=1`.
- F12 passes. Browser shows fresh files at top when the cards sort control sends `-uploaded_on`.
- Apparent "inverted sorting" is a Directus cards UI icon bug, not data. v11.17.4
  `app/src/layouts/cards/components/header.vue` flips the `sort` glyph for descending, while tabular
  flips it for ascending. Query semantics are correct: `sort[]=uploaded_on` = ascending (oldest first),
  `sort[]=-uploaded_on` = descending (newest first). Selecting a field resets direction to ascending.
  Workaround: toggle until DevTools shows `-uploaded_on`, ignore glyph; or use tabular.

## 2026-10-30 — F11 persistent failure journal + bounded retry + /failures + module table

Applicant: coder
Feature: F11 — Persistent failure journal, bounded retry, failure API and module view
Branch: feat/F11 @ de25e50 (base goal/image-previews @ 7951f82)

RUN: 2026-10-30T06:26:28Z | coder | /home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews | feat/F11 @ de25e50 | npx tsc --noEmit; npm test; npm run build | specs=0 | exit=0 | pass=55 | log=.opencode/tmp/runs/f11-test.log

Summary:
- Added `src/lib/failures.ts`: append-only `<state-dir>/failures.jsonl`, replay of the latest record per
  `(file_id,preset_key,format)` (key = `completedKeyOf`), corrupt-line tolerance, atomic compaction
  (unresolved + last 1,000 recovered), and `listFailures` with status filter + limit/offset.
- Added `src/lib/retry.ts`: MAX_ATTEMPTS=3, 5m/15m backoff, `nextRetryAt`, `dueRetries`,
  `journalFailure`/`markRecovered`, and `runDueRetries` (enabled/window/lock gated, targeted bounded
  batch default 50, grouped by file, full preset + acceptFormat via `warmVariant`, completion + recovered
  on success, attempt+1 + next retry on failure, compaction under `run.lock`, never throws).
- Hooked failure journaling into the upload warm hook, `POST /generate`, and the backfill; successes mark
  prior failures recovered. Journal writes never alter responses or hook behavior.
- Hook scheduler tick now runs due retries first, then the existing backfill.
- Added admin-only `GET /image-previews/failures?status=&limit=&offset=` with one batched
  `directus_files` metadata query (`filename_download`, `type`, `folder`) and ISO/human timestamps.
- Module: “Failed previews” table + Refresh (no toasts); API reference includes `/failures`; polling
  interval unchanged.
- Types added; README/AGENTS/project SKILL state-file docs updated.
- Tests: `failures.test.ts` (append/replay/latest-state/corrupt/idempotent/compaction/list) and
  `retry.test.ts` (backoff/exhaustion/due selection/bounded batch/grouping/missing preset/recovered/
  failure/disabled).

Files changed:
- src/lib/failures.ts (new), src/lib/failures.test.ts (new)
- src/lib/retry.ts (new), src/lib/retry.test.ts (new)
- src/lib/types.ts, src/lib/warmer.ts, src/lib/backfill.ts
- src/hook/index.ts, src/endpoint/index.ts, src/module/module.vue
- README.md, AGENTS.md, .opencode/skills/directus-image-previews/SKILL.md, .gitignore
- .opencode/progress.txt

Commands:
- `npx tsc --noEmit` → exit 0, 0 errors (`.opencode/tmp/runs/f11-tsc.log`)
- `npm test` → exit 0, 55/55 pass (`.opencode/tmp/runs/f11-test.log`)
- `npm run build` → exit 0, `dist/app.js` + `dist/api.js` emitted (`.opencode/tmp/runs/f11-build.log`)

Manual verification steps:
1. Cause a failure (unsupported preset format or missing file) from upload, `POST /generate`, and backfill;
   confirm one line per failure in `<state-dir>/failures.jsonl` and a warn log with `fileId`,
   `preset_key`, `format`, `attempt`, `next_retry_at`, `error`.
2. Inside the maintenance window with no backfill active, wait for the next 15-minute tick; confirm a due
   pair is retried, success writes `state.jsonl` + a `recovered` event and logs info, failure writes
   `attempt+1` with 15m backoff (or exhausted at 3).
3. `GET /image-previews/failures?status=failed&limit=50&offset=0` as admin: records include file metadata,
   attempts, first/last failure, next retry raw+ISO/human, status, error, ms; non-admin gets 403.
4. Module “Failed previews” table renders and Refresh reloads; no toast/notification appears.

Not verified locally:
- Browser rendering of the new module table (no live Directus/admin session in this run).
- Live PM2 `schedule('*/15 * * * *')` tick and maintenance-window retry execution.
- Concurrent PM2 append behavior and compaction race with upload-hook appends.

## 2026-10-30 — F11 review fixes

- Head: `feat/F11` @ `e10819c`.
- Replay now derives attempts from `event.attempt`; fail→recover→fail starts a fresh cycle (attempt 1) and duplicate failed events are idempotent.
- Journal writer lock (`failures.lock`, 20×10 ms, 30 s stale): appends serialize, compaction returns `false` if it cannot lock, and append still falls back to writing.
- Backfill loads the journal once, skips unresolved failures per pair, and caps journal attempts at `MAX_ATTEMPTS`; recovered pairs still append `recovered`.
- Added `/failures` shape/validation tests, retry lock/window tests, fail→recover→fail, attempt cap, and 50-append compaction race tests.

## 2026-10-30 — F11 second review fixes

- Implementation head: `feat/F11` @ `7f46493` (progress commit follows).
- Journal writes now require the `failures.lock` directory (`owner.json` with random token + pid) and never append unlocked. Blocking acquisition defaults to 10 s / 10 ms poll; timeout drops the event with `image-previews: could not journal failure` and `fileId`/`preset_key`/`format`.
- Stale locks are broken only when the owner pid is dead (`process.kill(pid, 0)` → `ESRCH`); mtime staleness removed. Release only removes the lock when `owner.json.token` still matches.
- `compactFailures` uses the same lock across read→write temp→rename, returns `false` when it cannot acquire, and uses a pid+token-unique temp filename.
- Tests now cover live-owner append refusal, dead-pid recovery, foreign-token release, live-lock compaction refusal, temporarily-held-lock 50-append race, and `planFileTasks` total/resume/skip counting.
- `npx tsc --noEmit` 0 errors; `npm test` 69/69 pass; `npm run build` emits both bundles. Logs: `.opencode/tmp/runs/f11-fix2-{tsc,test,build}.log`.
- Remaining manual checks: live PM2 scheduler retry tick, browser module rendering, and a real multi-worker compaction/append run.

## 2026-10-30 — F11 final lock hardening

- Implementation head: `feat/F11` @ `bc20c6a` (progress commit follows).
- Lock is now an atomic `failures.lock` file written with `flag:'wx'`; owner carries `token`, `pid`, `bootId`, `startTime`, `ts`. Identity readers use `/proc/sys/kernel/random/boot_id` and `/proc/<pid>/stat` field 22, with injectable readers for tests.
- Dead-lock reclaim runs only under `failures.lock.reclaim` (mkdir guard). Inside the guard it re-stats/re-reads the current owner and unlinks only a dead owner or an unreadable owner older than the 2 s grace. Live owners are never broken; stale guards older than 30 s are removed.
- Release unlinks only when the current `owner.token` matches; no unlink on mismatch or read error.
- Tests added: concurrent reclaim of a dead lock, missing/corrupt owner grace, PID reuse by bootId and startTime, guard-held reclaim and stale-guard recovery. Existing live-owner timeout, foreign-token release, 50-append race, and compaction-blocked tests stay green.
- `npx tsc --noEmit` 0 errors; `npm test` 76/76 pass; `npm run build` emits both bundles. Logs: `.opencode/tmp/runs/f11-lock-{test,build}.log` and `f11-lock-final-*`.
- Remaining manual checks: live PM2 multi-worker append/compaction, live scheduler retry tick, browser module rendering.

## 2026-10-30 — F11 reclaim-guard lifecycle fix

- Implementation head: `feat/F11` @ `cd973ca` (progress commit follows).
- Stale guard removal is now atomic: `rename(guardPath, guardPath.stale.<staleKey>)` with a marker inside the quarantined dir; losers retry instead of deleting a successor guard. Guard writes `owner.json` `{token,pid,ts}` and its `finally` only removes the guard when the token matches.
- Added injectable `now` clock. Tests: two waiters on one stale guard (both events survive), foreign-token guard release keeps the successor, plus existing lock/race/compaction tests.
- `npx tsc --noEmit` 0; `npm test` 78/78 pass; `npm run build` emits both bundles. Logs: `.opencode/tmp/runs/f11-guard2-{test,build}.log` and `f11-guard-final-*`.
- Remaining manual checks: live PM2 multi-worker append/compaction, live scheduler retry tick, browser module rendering.

## 2026-10-30 — F11 stale-guard snapshot verification

- Implementation head: `feat/F11` @ `4bd69dd` (progress commit follows).
- Stale-guard branch now snapshots `observedMtime` + `observedOwner` before any re-read, quarantines with `guardPath.stale.<pid>.<uuid>`, then verifies the moved directory's owner token (or both null) and mtime against the snapshot. Mismatch restores the successor guard and retries; restore failure returns false without running the action.
- Added deterministic successor-replacement test via guard hooks; both appends survive and the successor guard is restored.
- `npx tsc --noEmit` 0; `npm test` 79/79 pass; `npm run build` emits both bundles. Logs: `.opencode/tmp/runs/f11-guard3-{test,build}.log` and `f11-guard3-final-*`.
- Remaining manual checks: live PM2 multi-worker append/compaction, live scheduler retry tick, browser module rendering.