← Community
bugfixed

Team worker executes the same job_id twice when a retry uses a new envelope_id

DaveDave#313d ago · 86 views
fixed in: station-v1.4.0

Reproduction steps:

  1. Run a clean station-v0.97 source/workbench with a temporary workspace containing a valid signed Team manifest and a current authorized member.
  2. Construct two valid signed Team job-offer envelopes with the same team_id, job_id: "job_same", workflow payload, and sender, but different envelope_id values.
  3. Deliver both envelopes through the real team_mesh.receive() path to team_jobs.handle_job_offer().
  4. Inject a harmless runner stub that records each invocation and returns a synthetic result. No provider, HTTP, Stripe, or financial write is performed.
  5. Deliver the first byte-for-byte identical envelope again as a replay control.

Expected:

A retry of one logical Team job must be single-flight/idempotent by (team_id, job_id). A second envelope carrying the same job must return the existing/in-flight result or a duplicate response; it must not invoke the workflow runner a second time.

Actual:

Two different valid envelope IDs with the same job_id both invoke the runner:

different_envelope_ids: True True
runner_calls: 2
run_ids: ['jobrun_job_same', 'jobrun_job_same']
same_envelope_replay: True
runner_calls_after_control: 2

The exact same envelope replay is rejected by the envelope replay cache and does not add a third runner call. The failure therefore depends on a new envelope ID for the same logical job, not on forged signatures or an invalid member.

Root cause / affected path:

workbench/primitives/team_jobs.py, handle_job_offer() (approximately lines 199–278) validates the offer and immediately calls the injected runner(...). There is no persisted worker-side job state or single-flight guard keyed by (team_id, job_id). The derived run_id (jobrun_ + job_id) is used for approval/receipt binding, but workflow_engine.run_workflow() does not enforce caller-supplied run-ID idempotency before starting the runner.

Impact:

One logical Team job can execute its runner twice when delivery retries with a new envelope. If the workflow contains an external or financial effect, this can cause duplicate execution unless a separate downstream provider idempotency key happens to protect it. The attached reproduction proves duplicate runner invocation only; it intentionally performs no external write.

Station version (railcall version):
station-v0.97

Module slug + version:
N/A — Station Team-job runtime primitive; tested with synthetic workflow data.

Suggested fix:

Persist a Team-job record before invoking the runner, keyed by (team_id, job_id), with states such as RECEIVED, RUNNING, COMPLETED, and FAILED. Return the stored/in-flight result for duplicate offers, and enforce a single-flight/idempotency check in the DAG runner for the derived run ID as a defense in depth. The record must be written atomically and bound to the signed job payload so a changed payload cannot reuse the old result.

5 pts

1 reply

Fixed in station-v1.4.0. The worker now enforces at-most-once per job_id: a live job claims its id under a lock and persists its terminal result to WS/team/jobs_in/<job_id>.json; a redelivery under any fresh envelope_id replays that result instead of re-running. Concurrent offers serialise on the claim, and a failed attempt is terminal (not silently retried). Dry runs stay un-deduped. Great catch on the envelope_id vs job_id distinction.

Thanks for the report — credited.

Sign in to reply.