← Community
bugfixed

Concurrent /api/commands/execute double-fires one single-use approval: the consume is an unlocked read-modify-write on a threaded server

ShwetaShweta#111d ago · 29 views
affected: station-v1.0.0fixed in: station-v1.3.0

The airlock command path enforces "one approval, one execution" as an unlocked
read-check-write over WS/pending_approvals.json. On the Studio server — which is
threaded — two concurrent execute requests for the same payload both pass the
"already consumed?" check before either records the consume, and both fire the
effect. One human approval produces two live effects.

routes/commands.execute_command:

pending_map = _pending() # jload from disk
pend = pending_map.get(idem)
...
if pend.get("status") == "executed": # (A) the single-use check
return blocked_by_policy("approval already consumed ...")
...
pending_map[idem]["status"] = "executed" # (B) consume, in memory
_pending_save(pending_map) # (C) jsave to disk
_rate_limit_consume(cmd_id)
output, artifact = LOCAL_HANDLERS[cmd_id](...) # (D) the real effect

Nothing serializes (A) through (C). The in-line comment claims "Loopback
single-thread makes this atomic enough for the current threat model", but the
server is socketserver.ThreadingMixIn (studio_server.py:7230), so requests run
on separate threads and two _pending() reads can both observe status !=
"executed" before either save lands.

The maintainers already treat this concurrency as real elsewhere: the REQUESTS
queue path serializes the identical read-check-fire-delete under a dedicated
lock — studio_server.py:4091, _APPROVE_LOCK = threading.Lock(), commented
"two concurrent approves on a ThreadingMixIn server can't both pass the pending
guard and fire the same held payload twice (the 'one-shot' guarantee,
enforced)". That lock is taken in routes/requests.py:78 for the queue path and
is NOT taken on the direct command endpoints (/api/commands/approve,
/api/commands/execute in the cap-off wave routes, the MCP module-command path
_station_execute_command, and the Studio Approve+execute button), so the command
store's consume runs unprotected.

The single-use guarantee is documented and was regression-tested against exactly
this effect. execute_command's own comment: "SINGLE-USE APPROVAL: an approval is
consumed on FIRST execute attempt (success OR failure) and cannot be replayed.
Without this, a captured approval to a mutating command could be re-fired
indefinitely against the same payload — proven live 2026-07-22 by the
local.csv_append E2E test writing two identical rows from one approval." The race
reintroduces that exact double-write.

Same-root amplification: _rate_limit_consume (studio_server.py:5223) is the same
jload -> increment -> jsave with no lock ("Atomic-ish increment"), so the
per-command daily rate cap — the DoS control that also fronts a Stripe refund —
is overshootable by the same concurrency, and a lost increment lets more than
per_day effects through.

For a real mutating command (stripe.create_refund, twilio.send_sms,
pagerduty.trigger_incident, email.send_followup) a single human authorization
becomes N duplicated live effects, where N is how many execute requests race.

Reproduction steps:

  1. Extract the station-v1.0.0 tarball to a clean directory. Put workbench/ on

sys.path, import studio_server, routes.commands, routes.handlers; warm each
module's _LATE names and point their WS at a scratch workspace.

  1. Write ONE approval into pending_approvals.json bound to a payload's

payload_hash (status "pending_approval") — the state after a person clicks
Approve once. Use local.csv_append with {"file":"out.csv","row":{...}} — the
command the docstring names.

  1. SEQUENTIAL control: call execute_command twice in a row; confirm the second

returns blocked_by_policy ("approval already consumed") and exactly one CSV
row is written.

  1. CONCURRENT: rebind one approval, then call execute_command from two threads.

To realize the interleaving the threaded server permits, wrap _pending() with
a threading.Barrier(2) so both threads finish their jload before either
saves (the only instrumentation — the logic under test is unchanged).

  1. Count rows in the CSV the real handler wrote.

Expected: at most one execution per approval regardless of concurrency —
the second concurrent call blocks exactly as the sequential one does.

Actual:
sequential -> ["executed", "blocked_by_policy"], 1 row
concurrent -> ["executed", "executed"], 2 rows, 2 distinct receipts
(one approval, two identical live writes)

Root cause: a one-shot guarantee is implemented as check-then-set across a file
read and a file write with no mutual exclusion, on a server that runs requests
concurrently. The lock that fixes the sibling queue path was never extended to
the command store.

Suggested fix: take _APPROVE_LOCK (or a dedicated command-approval lock) across
the whole read-check-consume-fire sequence in execute_command, the same way
routes/requests.py wraps the queue path, so the consume and the effect are one
critical section per idempotency_key. Do the same for _rate_limit_consume, or
fold the counter into the same locked region. A hardened version would move
approvals to SQLite with a UNIQUE constraint on (idempotency_key, consumed) and
consume via a conditional UPDATE, as the code's own comment already contemplates.
Delete the stale "Loopback single-thread makes this atomic" comment — the server
is threaded and a second lock in this tree proves it.

3 pts

1 reply

Fixed in station-v1.3.0. The single-use approval consume is now a _CONSUME_LOCK-guarded atomic compare-and-set, so two concurrent /api/commands/execute calls can't both win the read-modify-write.

Thanks for the report — credited.

Sign in to reply.