feat: persist failure journal and bounded retries (F11)
This commit is contained in:
parent
ec3e0c16e0
commit
871182d6d5
15 changed files with 2769 additions and 23 deletions
18
README.md
18
README.md
|
|
@ -105,6 +105,8 @@ All require a Directus admin session or admin token. Human-readable fields accom
|
|||
- `POST /image-previews/generate` — one file (`{fileId, keys?, formats?, dryRun?}`)
|
||||
- `GET /image-previews/resolve-plan?fileId=&key=` — predicted cache filename, no write
|
||||
- `GET /image-previews/ops?limit=` — recent operations with timing + error messages
|
||||
- `GET /image-previews/failures?status=&limit=&offset=` — persisted failures with file metadata, attempts,
|
||||
timestamps, next retry and error (`status` = `failed`, `recovered`, or `retry`; limit max 200)
|
||||
|
||||
The admin module includes a full API reference with curl examples.
|
||||
|
||||
|
|
@ -115,10 +117,24 @@ The extension schedules itself **inside Directus** — no external cron, no toke
|
|||
- every 15 minutes it checks whether it is inside the maintenance window;
|
||||
- if enabled, inside the window, and no run is active, it starts/resumes the backfill;
|
||||
- the backfill **pauses when the window closes** and resumes the next night (it is resumable);
|
||||
- once all tasks are done, it stays idle until the next day.
|
||||
- once all tasks are done, it stays idle until the next day;
|
||||
- before starting the backfill it runs any due failure retries (targeted pairs only, bounded batch).
|
||||
|
||||
`Run now` in the admin module ignores the window.
|
||||
|
||||
## Failures and retries
|
||||
|
||||
Every failed `(file, preset, format)` task from the upload hook, `POST /generate`, and the backfill is
|
||||
appended to `<state-dir>/failures.jsonl`. Replay derives the latest state per task, so details survive
|
||||
restarts and PM2 workers. The journal is append-only; it is compacted under `run.lock` to keep unresolved
|
||||
tasks plus the last 1,000 recovered ones.
|
||||
|
||||
Retry policy: max 3 attempts (initial + 2 retries), backoff 5 minutes then 15 minutes. Due retries run
|
||||
inside the maintenance window only when no backfill is active, under the same `run.lock`, and process only
|
||||
the targeted file/preset/format pairs (default batch 50). A success appends the normal `state.jsonl`
|
||||
completion and marks the task `recovered`; an exhausted task stays `failed` with its last error and is not
|
||||
retried again. Failures are logged at warn with ids/metadata only; recoveries at info.
|
||||
|
||||
## Safe testing & deployment
|
||||
|
||||
Never test against a production instance.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue