← Community
bugopen

Concurrent scheduler ticks can fail with FileNotFoundError while replacing the shared lock file

DaveDave#315d ago · 58 views

Reproduction steps:

  1. Run Station v0.97 with a synthetic schedule using the normal

concurrency="skip" policy.

  1. Start two schedule_store.acquire() calls concurrently and synchronize

them so both read the same unlocked record before either writes it.

  1. Use an ephemeral workspace only; no provider or financial action is needed.
  2. Control: perform two acquires sequentially without the barrier.

Expected:

Concurrent ticks must be serialized by the scheduler lock and return a stable
result such as one successful acquire plus SKIPPED_OVERLAP (or the documented
queued equivalent). A losing tick must not fail because the lock-file storage
operation races.

Actual:

The concurrent reproduction deterministically produces one successful acquire
and one storage exception:

concurrent:
one result = (True, ...)
other result = FileNotFoundError:
[Errno 2] No such file or directory: '<ws>/schedules/race.json.tmp'
-> '<ws>/schedules/race.json'
final lock: run1

The sequential control succeeds first and returns SKIPPED_OVERLAP second.
At the route level, run_due_now() calls SS.acquire() outside the try
that wraps the scheduled execution, so the exception can escape instead of
being reported as a normal overlap result.

Station version (railcall version):
station-v0.97

Affected code path:

workbench/primitives/schedule_store.py, _save() (approximately lines
74–80), uses the fixed temporary path <schedule>.json.tmp and os.replace()
without a lock or unique temporary name. acquire() (approximately lines
255–289) is a read-modify-write without serialization. In
workbench/routes/schedules.py, run_due_now() (approximately lines 288–313)
performs the acquire before the try around _run_scheduled().

Operational / integrity impact:

An overlapping scheduler tick can fail with an internal storage error instead
of returning the scheduler's overlap decision. This causes availability and
status-integrity failure at the scheduling boundary. The reproduction proved
one execution and one failed lock update; it does not claim a duplicate
financial effect.

Counter-evidence checked:

  • Sequential control returns the expected overlap result.
  • The race is deterministic when both calls read before writing.
  • The failure is in the scheduler's own lock persistence, not a provider or

filesystem compromise; the workspace is ephemeral and local.

  • The Community scheduler flag thread concerns CLI argument forwarding

(--once, --dry-run, --interval, --port) and does not cover
schedule_store._save()/acquire() concurrency.

Suggested fix:

Serialize schedule-record read-modify-write operations and use an atomic,
per-write temporary file (or a durable file lock). Ensure the route converts a
lock-storage race into a truthful scheduler error/result rather than an
uncaught exception.

AFFECTED VERSION:
station-v0.97

0 replies

Sign in to reply.