docs: add failure journal + uploaded_on features to goal artifacts

.opencode/feature_list.json

@@ -201,6 +201,51 @@
         "Promotion to prod is a human action; the loop proposes, a human deploys.",
         "Record the exact cron entry, the scoped grants, and the rollback commands verbatim."
       ]
+    },
+    {
+      "feature_id": "F11",
+      "title": "Persistent failure journal, bounded retry, failure API and module view",
+      "acceptance_steps": [
+        "Every failed (file, preset, format) task from the upload hook, manual `/generate`, and backfill appends an event to `<state-dir>/failures.jsonl`; replay derives the latest state per key across restarts and PM2 workers.",
+        "Default retry policy is max 3 attempts (initial + 2 retries) with 5 min then 15 min backoff; a failed task gets at least one automatic retry when the scheduler next runs with no active backfill inside the maintenance window.",
+        "Due retries run under the existing `run.lock`, process only the targeted file/preset/format pairs (bounded batch, default 50), and never trigger a full-library sweep; a manual `Run now` may ignore the window.",
+        "A recovered task appends a normal `state.jsonl` completion and is marked `recovered`; an exhausted task stays `failed` with attempts, last error, and timestamps, and is not retried again.",
+        "`GET /image-previews/failures?status=&limit=&offset=` returns persisted records plus file metadata (`filename_download`, `type`, `folder` when available), attempts, first/last failure, next retry, status, and error text; admin-only and read-only.",
+        "The admin module shows a \"Failed previews\" table with file, preset, format, attempts, last error, next retry, and status; no toasts or user-facing notifications are added.",
+        "Every failure logs a structured warn with `fileId`, `preset_key`, `format`, `attempt`, `next_retry_at`, and `error`; recovery logs at info. No file contents are logged.",
+        "Journal compaction under the run lock retains unresolved rows plus the last 1,000 recovered rows, so the file cannot grow without bound.",
+        "Targeted unit tests cover journal append/replay/idempotency, backoff schedule, max-attempt exhaustion, due-retry selection, and the failure endpoint response shape."
+      ],
+      "passes": false,
+      "blocked_by": [
+        "F08"
+      ],
+      "notes": [
+        "Root cause: `/ops` is an in-memory 200-entry ring buffer per PM2 worker; after a long run the 17 failure details were unrecoverable.",
+        "Do not reuse `/ops` for persistence; keep it as the live view. Journal is append-only for cluster safety; compact only under `run.lock`.",
+        "Retry policy starts with constants; expose settings only if tuning is needed after the first real run.",
+        "No user-facing notifications; logs and the read-only API/module table are the observability surface."
+      ]
+    },
+    {
+      "feature_id": "F12",
+      "title": "Host fix: `uploaded_on` integrity for TUS and legacy files (no Directus upgrade)",
+      "acceptance_steps": [
+        "Before change, record `count(*)`, `count(*) filter (where uploaded_on is null)`, and `count(*) filter (where created_on is null)` for `directus_files`; confirm the legacy null block explains the file-library ordering.",
+        "In a transaction after a table backup, run `update directus_files set uploaded_on = coalesce(created_on, now()) where uploaded_on is null;` and verify only `uploaded_on` changed (`modified_on`/`modified_by` unchanged, no new activity/revisions).",
+        "Set the column default to `(now() at time zone 'utc')` when absent, so future TUS inserts that omit `uploaded_on` receive a date.",
+        "Upload one test file with `TUS_ENABLED=true`; confirm `uploaded_on` is non-null, `tus_id`/`tus_data` are null after finish, and the file appears at the top of the unfiltered folder with `-uploaded_on`.",
+        "Keep `TUS_ENABLED=true`; no Directus upgrade and no extension code change required for this fix.",
+        "Document latent 11.17.4 TUS defects not changed here: client-declared `type`/`filesize`, and stale `tus_id` placeholders hidden from `/files` if finish/cleanup fails."
+      ],
+      "passes": false,
+      "blocked_by": [],
+      "notes": [
+        "Source: v11.17.4 `api/src/services/tus/{data-store,server}.ts` omits `uploaded_on`; only `/files` (`FilesService.uploadOne`) sets it. Upstream #28042 fixed in 12.4.0, not backported to 11.x.",
+        "Raw SQL is the only way to avoid bumping `modified_on`/`modified_by`; `ItemsService.updateOne` always runs payload specials even with `emitEvents:false`.",
+        "TUS row insert omits `uploaded_on` entirely, so a DB default applies to future inserts. `now() at time zone 'utc'` matches Directus's UTC writes for the tz-less timestamp column.",
+        "This is host data remediation, not extension code; keep it separate from the preview warmer."
+      ]
     }
   ]
 }

.opencode/goal.md

@@ -3,6 +3,7 @@
 Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (independent of any host
 application).
 Location: `/home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews`
+Branch: `goal/image-previews` (created from master 2026-10-30).
 Target: self-hosted Directus **11.17.4** (PM2 cluster, local disk storage).
 Project type: Directus **bundle** extension (app module + api hook + api endpoint).
 
@@ -112,6 +113,48 @@ Our registry (`src/lib/presets.ts`) mirrors this exact list:
 - `debug` setting enables verbose structured logs; logs carry ids/metadata only, never file contents.
 - Everything except `/generate` is read-only.
 
+## Failure journal & bounded retries (2026-10-30)
+
+Backfill failures are currently only counted in `run.json` and held in the in-memory `/ops` ring
+buffer (last 200 entries, per PM2 worker). After a long run the details are lost. Add a durable
+failure journal and a bounded retry path.
+
+- `failures.jsonl` under the state dir = append-only events `{type:'failed'|'retry'|'recovered',
+  ts, file_id, preset_key, format, attempt, next_retry_at?, error?, ms?}`. Replay derives the latest
+  state per `(file_id, preset_key, format)`; append-only so PM2 workers can write safely.
+- Every failure (upload hook, manual `/generate`, backfill) is journaled. Default retry policy:
+  max 3 attempts (initial + 2 retries), backoff 5 min then 15 min. Due retries run only while enabled,
+  inside the maintenance window (manual `Run now` may ignore it), with no active backfill, under the
+  same `run.lock`; targeted file/preset/format retries — never a full-library sweep.
+- Success after retry appends a normal `state.jsonl` completion and marks the failure `recovered`.
+  Exhausted retries stay `failed` with the last error and remain readable.
+- `GET /image-previews/failures?status=&limit=&offset=` returns persisted records + file metadata,
+  attempts, timestamps, next retry and error text. Admin-only, read-only.
+- Module: "Failed previews" table (file, preset, format, attempts, last error, next retry, status).
+  No toasts or user-facing notifications.
+- Log every failure at warn with ids/metadata + error; log recoveries at info. No file contents.
+- Journal is compacted under the run lock: keep unresolved rows + last 1,000 recovered; never grow
+  unbounded. `/ops` remains the live in-memory view.
+
+## Host data fix: `uploaded_on` for TUS uploads (2026-10-30)
+
+Scope split (user decision): DB access and execution stay with the user; this repo only ships the SQL
+script + runbook. No extension code change for this item.
+
+Directus 11.17.4 TUS uploads omit `uploaded_on` (`TusDataStore.create` + `onUploadFinish`); the
+regular `POST /files` path sets it. Upstream issue #28042 is fixed only in 12.4.0; this host stays on
+11.17.4.
+
+- Fix existing rows directly in Postgres: `update directus_files set uploaded_on =
+  coalesce(created_on, now()) where uploaded_on is null;` in a transaction, after a table backup.
+  Raw SQL touches only `uploaded_on`; it does not bump `modified_on`/`modified_by`, emit events, or
+  create revisions.
+- Add a UTC column default so future TUS inserts get a date:
+  `alter table directus_files alter column uploaded_on set default (now() at time zone 'utc');`
+  Confirm one new TUS upload has non-null `uploaded_on` and `tus_id = null`.
+- Keep `TUS_ENABLED=true`; no Directus upgrade and no extension code required. Latent TUS defects
+  (client-declared `type`/`filesize`, stale `tus_id` placeholders) are documented, not changed.
+
 ## Safe deployment & staging (must not crash prod)
 
 - **Fail-open, inert by default**: every entrypoint is tiny, bodies wrapped in try/catch, hooks never
@@ -157,6 +200,8 @@ src/
     warmer.ts            in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency
     backfill.ts          enumerate directus_files (image/*), window-aware, resumable
     state.ts             state.jsonl read/append/resume
+    failures.ts          failures.jsonl append/replay/compact
+    retry.ts             due-retry selection + backoff/max-attempt policy
     settings.ts          config file read/write + env defaults
     disk.ts              sum sizes of __<hash> files (driver.list + stat)
     window.ts            maintenance-window logic (is-now-in-window, next-window)