From 7951f82dd5cbcee7e04ed50267535a4040e51b5c Mon Sep 17 00:00:00 2001 From: Rawman Grey Date: Fri, 30 Oct 2026 09:18:59 +0300 Subject: [PATCH] docs: add failure journal + uploaded_on features to goal artifacts --- .opencode/feature_list.json | 45 +++++++++++++++++++++++++++++++++++++ .opencode/goal.md | 45 +++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/.opencode/feature_list.json b/.opencode/feature_list.json index 541f494..5c3b508 100644 --- a/.opencode/feature_list.json +++ b/.opencode/feature_list.json @@ -201,6 +201,51 @@ "Promotion to prod is a human action; the loop proposes, a human deploys.", "Record the exact cron entry, the scoped grants, and the rollback commands verbatim." ] + }, + { + "feature_id": "F11", + "title": "Persistent failure journal, bounded retry, failure API and module view", + "acceptance_steps": [ + "Every failed (file, preset, format) task from the upload hook, manual `/generate`, and backfill appends an event to `/failures.jsonl`; replay derives the latest state per key across restarts and PM2 workers.", + "Default retry policy is max 3 attempts (initial + 2 retries) with 5 min then 15 min backoff; a failed task gets at least one automatic retry when the scheduler next runs with no active backfill inside the maintenance window.", + "Due retries run under the existing `run.lock`, process only the targeted file/preset/format pairs (bounded batch, default 50), and never trigger a full-library sweep; a manual `Run now` may ignore the window.", + "A recovered task appends a normal `state.jsonl` completion and is marked `recovered`; an exhausted task stays `failed` with attempts, last error, and timestamps, and is not retried again.", + "`GET /image-previews/failures?status=&limit=&offset=` returns persisted records plus file metadata (`filename_download`, `type`, `folder` when available), attempts, first/last failure, next retry, status, and error text; admin-only and read-only.", + "The admin module shows a \"Failed previews\" table with file, preset, format, attempts, last error, next retry, and status; no toasts or user-facing notifications are added.", + "Every failure logs a structured warn with `fileId`, `preset_key`, `format`, `attempt`, `next_retry_at`, and `error`; recovery logs at info. No file contents are logged.", + "Journal compaction under the run lock retains unresolved rows plus the last 1,000 recovered rows, so the file cannot grow without bound.", + "Targeted unit tests cover journal append/replay/idempotency, backoff schedule, max-attempt exhaustion, due-retry selection, and the failure endpoint response shape." + ], + "passes": false, + "blocked_by": [ + "F08" + ], + "notes": [ + "Root cause: `/ops` is an in-memory 200-entry ring buffer per PM2 worker; after a long run the 17 failure details were unrecoverable.", + "Do not reuse `/ops` for persistence; keep it as the live view. Journal is append-only for cluster safety; compact only under `run.lock`.", + "Retry policy starts with constants; expose settings only if tuning is needed after the first real run.", + "No user-facing notifications; logs and the read-only API/module table are the observability surface." + ] + }, + { + "feature_id": "F12", + "title": "Host fix: `uploaded_on` integrity for TUS and legacy files (no Directus upgrade)", + "acceptance_steps": [ + "Before change, record `count(*)`, `count(*) filter (where uploaded_on is null)`, and `count(*) filter (where created_on is null)` for `directus_files`; confirm the legacy null block explains the file-library ordering.", + "In a transaction after a table backup, run `update directus_files set uploaded_on = coalesce(created_on, now()) where uploaded_on is null;` and verify only `uploaded_on` changed (`modified_on`/`modified_by` unchanged, no new activity/revisions).", + "Set the column default to `(now() at time zone 'utc')` when absent, so future TUS inserts that omit `uploaded_on` receive a date.", + "Upload one test file with `TUS_ENABLED=true`; confirm `uploaded_on` is non-null, `tus_id`/`tus_data` are null after finish, and the file appears at the top of the unfiltered folder with `-uploaded_on`.", + "Keep `TUS_ENABLED=true`; no Directus upgrade and no extension code change required for this fix.", + "Document latent 11.17.4 TUS defects not changed here: client-declared `type`/`filesize`, and stale `tus_id` placeholders hidden from `/files` if finish/cleanup fails." + ], + "passes": false, + "blocked_by": [], + "notes": [ + "Source: v11.17.4 `api/src/services/tus/{data-store,server}.ts` omits `uploaded_on`; only `/files` (`FilesService.uploadOne`) sets it. Upstream #28042 fixed in 12.4.0, not backported to 11.x.", + "Raw SQL is the only way to avoid bumping `modified_on`/`modified_by`; `ItemsService.updateOne` always runs payload specials even with `emitEvents:false`.", + "TUS row insert omits `uploaded_on` entirely, so a DB default applies to future inserts. `now() at time zone 'utc'` matches Directus's UTC writes for the tz-less timestamp column.", + "This is host data remediation, not extension code; keep it separate from the preview warmer." + ] } ] } diff --git a/.opencode/goal.md b/.opencode/goal.md index bcc22dc..3ecf86a 100644 --- a/.opencode/goal.md +++ b/.opencode/goal.md @@ -3,6 +3,7 @@ Status: SCAFFOLDED — opened 2026-10-25. Standalone extension project (independent of any host application). Location: `/home/rawman/Projects/DirectusExtensions/directus-extension-generate-image-previews` +Branch: `goal/image-previews` (created from master 2026-10-30). Target: self-hosted Directus **11.17.4** (PM2 cluster, local disk storage). Project type: Directus **bundle** extension (app module + api hook + api endpoint). @@ -112,6 +113,48 @@ Our registry (`src/lib/presets.ts`) mirrors this exact list: - `debug` setting enables verbose structured logs; logs carry ids/metadata only, never file contents. - Everything except `/generate` is read-only. +## Failure journal & bounded retries (2026-10-30) + +Backfill failures are currently only counted in `run.json` and held in the in-memory `/ops` ring +buffer (last 200 entries, per PM2 worker). After a long run the details are lost. Add a durable +failure journal and a bounded retry path. + +- `failures.jsonl` under the state dir = append-only events `{type:'failed'|'retry'|'recovered', + ts, file_id, preset_key, format, attempt, next_retry_at?, error?, ms?}`. Replay derives the latest + state per `(file_id, preset_key, format)`; append-only so PM2 workers can write safely. +- Every failure (upload hook, manual `/generate`, backfill) is journaled. Default retry policy: + max 3 attempts (initial + 2 retries), backoff 5 min then 15 min. Due retries run only while enabled, + inside the maintenance window (manual `Run now` may ignore it), with no active backfill, under the + same `run.lock`; targeted file/preset/format retries — never a full-library sweep. +- Success after retry appends a normal `state.jsonl` completion and marks the failure `recovered`. + Exhausted retries stay `failed` with the last error and remain readable. +- `GET /image-previews/failures?status=&limit=&offset=` returns persisted records + file metadata, + attempts, timestamps, next retry and error text. Admin-only, read-only. +- Module: "Failed previews" table (file, preset, format, attempts, last error, next retry, status). + No toasts or user-facing notifications. +- Log every failure at warn with ids/metadata + error; log recoveries at info. No file contents. +- Journal is compacted under the run lock: keep unresolved rows + last 1,000 recovered; never grow + unbounded. `/ops` remains the live in-memory view. + +## Host data fix: `uploaded_on` for TUS uploads (2026-10-30) + +Scope split (user decision): DB access and execution stay with the user; this repo only ships the SQL +script + runbook. No extension code change for this item. + +Directus 11.17.4 TUS uploads omit `uploaded_on` (`TusDataStore.create` + `onUploadFinish`); the +regular `POST /files` path sets it. Upstream issue #28042 is fixed only in 12.4.0; this host stays on +11.17.4. + +- Fix existing rows directly in Postgres: `update directus_files set uploaded_on = + coalesce(created_on, now()) where uploaded_on is null;` in a transaction, after a table backup. + Raw SQL touches only `uploaded_on`; it does not bump `modified_on`/`modified_by`, emit events, or + create revisions. +- Add a UTC column default so future TUS inserts get a date: + `alter table directus_files alter column uploaded_on set default (now() at time zone 'utc');` + Confirm one new TUS upload has non-null `uploaded_on` and `tus_id = null`. +- Keep `TUS_ENABLED=true`; no Directus upgrade and no extension code required. Latent TUS defects + (client-declared `type`/`filesize`, stale `tus_id` placeholders) are documented, not changed. + ## Safe deployment & staging (must not crash prod) - **Fail-open, inert by default**: every entrypoint is tiny, bodies wrapped in try/catch, hooks never @@ -157,6 +200,8 @@ src/ warmer.ts in-process getAsset(x, { transformationParams, acceptFormat }) + concurrency backfill.ts enumerate directus_files (image/*), window-aware, resumable state.ts state.jsonl read/append/resume + failures.ts failures.jsonl append/replay/compact + retry.ts due-retry selection + backoff/max-attempt policy settings.ts config file read/write + env defaults disk.ts sum sizes of __ files (driver.list + stat) window.ts maintenance-window logic (is-now-in-window, next-window)