Reproduction steps:
- Two threads run team_mesh.verify_envelope's replay step for the SAME
envelope_id concurrently, arranged (barrier) so both read the seen-set before
either writes — exactly what happens when the relay auto-poll thread and a
session-gated POST /api/relay/poll process the same delivery at once:
seen = team_mesh._load_seen(ws) # read whole seen_envelopes.json
replayed = eid in seen # check
# (barrier: both threads reach here before either _remember writes)
if not replayed:
team_mesh._remember(ws, eid, exp, seen) # commit; handler then runs
- Control: run the same two verifies sequentially.
Expected:
The replay cache rejects the second occurrence of an envelope_id, so a signed
envelope is consumed once. verify_envelope's own docstring: "verify is
consume-once by design."
Actual:
Concurrent : BOTH verifies see the id as fresh and pass -> both would run the
handler (double execution).
Sequential : the second is correctly rejected (executes once).
_load_seen (read whole JSON) -> if eid in seen -> _remember (rewrite whole
JSON) takes no lock, so two overlapping verifies both read a seen-set lacking the
id and both proceed. Because team_mesh.receive() runs the per-kind handler after
verify_envelope returns ok, the same signed envelope executes twice: a
capability_use envelope charges the holder's shared credential twice; a team-job
envelope fires its effect twice (team_jobs has no (team_id, job_id) single-flight
either). Second, related defect surfaced by the same repro: _remember writes to a
FIXED temp name (seen_envelopes.json.tmp), so two concurrent commits collide and
one raises FileNotFoundError out of verify_envelope (the same fixed-.tmp
concurrency class already reported for schedule_store/team_approval).
Suggested fix:
Make the replay claim atomic instead of a read-modify-write. Replace the
JSON-dict cache with a per-envelope marker created via an exclusive open, so
exactly one concurrent verify wins and the loser is treated as a replay:
import errno
def _claim_envelope(ws, eid, expires_at):
d = os.path.join(ws, "team", "seen_envelopes")
os.makedirs(d, mode=0o700, exist_ok=True)
path = os.path.join(d, re.sub(r"[^A-Za-z0-9_.-]", "_", eid))
try:
fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except OSError as e:
if e.errno == errno.EEXIST:
return False # already claimed -> replay
raise
with os.fdopen(fd, "w") as f:
f.write(expires_at)
return True # we won the claim; safe to run the handler
# verify_envelope: if not _claim_envelope(ws, eid, env["expires_at"]): reject
O_CREAT|O_EXCL is atomic across threads and processes, eliminating the TOCTOU and
the fixed-.tmp collision at once (prune stale marker files by their stored expiry
on a periodic pass). Alternatively, hold a per-workspace file lock around the
check-and-commit, and give _remember a unique temp name (tempfile.mkstemp in the
same dir) as schedule_store's fix did.
Station version (railcall version): station-v0.99
Module slug + version: N/A (platform — workbench/primitives/team_mesh.py)