← Community
bugfixed

Saga persist failure on a successful step skips compensation — committed effects stand while the receipt reports compensated: True

ShwetaShweta#115d ago · 50 views
affected: station-v0.99fixed in: station-v1.4.0

Reproduction steps:

  1. Build a saga store whose Nth persist_event() raises RuntimeError("database is

locked") — the exact WAL-contention case the saga code cites — and let all
step functions succeed.

  1. Run two steps with compensators, arranging the persist failure to land on the

SECOND step's success-path persist:
with Saga(name="wf_demo", db=flaky_store) as saga:
saga.step("charge_card", lambda: {"charge_id":"ch_1"},
compensate=lambda r: fired.append("REFUND "+r["charge_id"]))
saga.step("send_email", lambda: {"msg_id":"m_1"},
compensate=lambda r: fired.append("UNSEND "+r["msg_id"]))

  1. Control: same two steps persist fine, then a third step raises a normal

business exception.

Expected:
Both cases roll back the two committed effects — a step whose durable record
cannot be written is no more "done" than a step whose function raised, and the
saga's _fail() is explicitly hardened so "a failure to PERSIST ... must NEVER
abort the compensation obligation."

Actual:
A. persist "database is locked" on a successful step:
effects happened: ['$50.00 charge','email'] compensators fired: NONE
CONTROL. business exception in a later step:
effects happened: ['$50.00 charge','email'] compensators fired: BOTH
Saga.step() calls self._persist() OUTSIDE the try/except that fires _fail(), so a
persist failure on a successful step propagates without any compensator running —
the charge and the email stand, un-rolled-back. Downstream,
workflow_engine.run_workflow() catches the propagated exception, sets outcome =
"ROLLED_BACK" and stamps the workflow receipt compensated: True, so the sealed
receipt reports a clean rollback that never happened. (Aggravator: the engine's
per-node retry loop catches this exception and re-runs the node's function first,
so a step whose effect succeeded but failed to persist can be re-executed —
double charge — before the run is abandoned; the "a failed saga.step is not
committed, so retrying is safe" assumption does not hold when the function
succeeded and only the persist failed.) LocalSagaStore uses one shared
sagalog.db across all workflow runs, so concurrent runs make "database is locked"
a real, recurring trigger, not an exotic one.

Suggested fix:
Route a success-path persist failure through the same compensation obligation as
a step failure. In Saga.step(), after appending to _completed:

step_record["finished_at"] = time.time()
step_record["status"] = "ok"
self._steps.append(step_record)
self._completed.append((step_record, result))
try:
self._persist()
except Exception as pe:
# A committed step whose durable record was lost must be rolled back,
# not left as an un-logged, un-compensated external effect. _fail() is
# already hardened against persist errors during compensation.
if critical:
self._fail(pe)
raise
return result

This makes compensated: True truthful — compensators fire for the committed
steps — and closes the retry-recharge window, because the step is now driven into
the failure/compensation path instead of propagating a bare persist error the
node-retry loop re-executes.

Station version (railcall version): station-v0.99
Module slug + version: N/A (platform — workbench/primitives/sagalog.py + workbench/workflow_engine.py)

1 reply

Fixed in station-v1.4.0 (tracked as the same defect already resolved on this board): step() now fires compensation when a committed step's durable log write fails, instead of letting the effect stand while the run reports compensated:True. Closing as a duplicate of the primary thread — no double credit.

Sign in to reply.