Reproduction steps:
- Build a saga store whose Nth persist_event() raises RuntimeError("database is
locked") — the exact WAL-contention case the saga code cites — and let all
step functions succeed.
- Run two steps with compensators, arranging the persist failure to land on the
SECOND step's success-path persist:
with Saga(name="wf_demo", db=flaky_store) as saga:
saga.step("charge_card", lambda: {"charge_id":"ch_1"},
compensate=lambda r: fired.append("REFUND "+r["charge_id"]))
saga.step("send_email", lambda: {"msg_id":"m_1"},
compensate=lambda r: fired.append("UNSEND "+r["msg_id"]))
- Control: same two steps persist fine, then a third step raises a normal
business exception.
Expected:
Both cases roll back the two committed effects — a step whose durable record
cannot be written is no more "done" than a step whose function raised, and the
saga's _fail() is explicitly hardened so "a failure to PERSIST ... must NEVER
abort the compensation obligation."
Actual:
A. persist "database is locked" on a successful step:
effects happened: ['$50.00 charge','email'] compensators fired: NONE
CONTROL. business exception in a later step:
effects happened: ['$50.00 charge','email'] compensators fired: BOTH
Saga.step() calls self._persist() OUTSIDE the try/except that fires _fail(), so a
persist failure on a successful step propagates without any compensator running —
the charge and the email stand, un-rolled-back. Downstream,
workflow_engine.run_workflow() catches the propagated exception, sets outcome =
"ROLLED_BACK" and stamps the workflow receipt compensated: True, so the sealed
receipt reports a clean rollback that never happened. (Aggravator: the engine's
per-node retry loop catches this exception and re-runs the node's function first,
so a step whose effect succeeded but failed to persist can be re-executed —
double charge — before the run is abandoned; the "a failed saga.step is not
committed, so retrying is safe" assumption does not hold when the function
succeeded and only the persist failed.) LocalSagaStore uses one shared
sagalog.db across all workflow runs, so concurrent runs make "database is locked"
a real, recurring trigger, not an exotic one.
Suggested fix:
Route a success-path persist failure through the same compensation obligation as
a step failure. In Saga.step(), after appending to _completed:
step_record["finished_at"] = time.time()
step_record["status"] = "ok"
self._steps.append(step_record)
self._completed.append((step_record, result))
try:
self._persist()
except Exception as pe:
# A committed step whose durable record was lost must be rolled back,
# not left as an un-logged, un-compensated external effect. _fail() is
# already hardened against persist errors during compensation.
if critical:
self._fail(pe)
raise
return result
This makes compensated: True truthful — compensators fire for the committed
steps — and closes the retry-recharge window, because the step is now driven into
the failure/compensation path instead of propagating a bare persist error the
node-retry loop re-executes.
Station version (railcall version): station-v0.99
Module slug + version: N/A (platform — workbench/primitives/sagalog.py + workbench/workflow_engine.py)