feat: persist failure journal and bounded retries (F11)

This commit is contained in:
Rawman Grey 2026-10-30 10:22:29 +03:00
parent ec3e0c16e0
commit 871182d6d5
15 changed files with 2769 additions and 23 deletions

View file

@ -105,6 +105,8 @@ All require a Directus admin session or admin token. Human-readable fields accom
- `POST /image-previews/generate` — one file (`{fileId, keys?, formats?, dryRun?}`)
- `GET /image-previews/resolve-plan?fileId=&key=` — predicted cache filename, no write
- `GET /image-previews/ops?limit=` — recent operations with timing + error messages
- `GET /image-previews/failures?status=&limit=&offset=` — persisted failures with file metadata, attempts,
timestamps, next retry and error (`status` = `failed`, `recovered`, or `retry`; limit max 200)
The admin module includes a full API reference with curl examples.
@ -115,10 +117,24 @@ The extension schedules itself **inside Directus** — no external cron, no toke
- every 15 minutes it checks whether it is inside the maintenance window;
- if enabled, inside the window, and no run is active, it starts/resumes the backfill;
- the backfill **pauses when the window closes** and resumes the next night (it is resumable);
- once all tasks are done, it stays idle until the next day.
- once all tasks are done, it stays idle until the next day;
- before starting the backfill it runs any due failure retries (targeted pairs only, bounded batch).
`Run now` in the admin module ignores the window.
## Failures and retries
Every failed `(file, preset, format)` task from the upload hook, `POST /generate`, and the backfill is
appended to `<state-dir>/failures.jsonl`. Replay derives the latest state per task, so details survive
restarts and PM2 workers. The journal is append-only; it is compacted under `run.lock` to keep unresolved
tasks plus the last 1,000 recovered ones.
Retry policy: max 3 attempts (initial + 2 retries), backoff 5 minutes then 15 minutes. Due retries run
inside the maintenance window only when no backfill is active, under the same `run.lock`, and process only
the targeted file/preset/format pairs (default batch 50). A success appends the normal `state.jsonl`
completion and marks the task `recovered`; an exhausted task stays `failed` with its last error and is not
retried again. Failures are logged at warn with ids/metadata only; recoveries at info.
## Safe testing & deployment
Never test against a production instance.