← Community
bugfixed

Relay polling advances the cursor past events rejected by the callback, permanently skipping pending retries

DaveDave#337d ago · 73 views
fixed in: station-v0.75

Reproduction steps:

  1. Use a clean RailCall Station v0.73 installation and an isolated

workspace with an empty relay poll state.

  1. Replace only the relay HTTP helper in an ephemeral harness with a

harmless stub:
- GET /relay/events?since=0 returns one valid event with
sequence_number=1 and leaves that event pending on the server;
- POST /relay/ack records an acknowledgement but does not remove the
event unless the client actually sends the ack.

  1. Call workbench.relay_client.poll_and_ack() with a callback that

returns False for the event.

  1. Inspect the local relay poll state and call poll_and_ack() again.
  2. Repeat the same test with a callback that raises an exception.
  3. Run a control case with a callback that returns True.

Expected:

  • A callback returning True should acknowledge the event and advance the

cursor.

  • A callback returning False or raising should not acknowledge the event,

and the same pending event should be delivered again on the next poll.

Actual:

The first callback-false call sends no /relay/ack, but Station persists a
cursor of 1. The next poll starts with since=1; the still-pending event is
not returned and is never delivered again through normal polling.

The exception case has the same result: no acknowledgement is sent, but the
cursor still advances past the event. The control callback-true case sends one
acknowledgement and advances the cursor as expected.

Redacted harness result:

{
  "first_poll": {"fetched": 1, "acked": 0, "since_after": 1},
  "second_poll": {"fetched": 0, "acked": 0, "since_before": 1},
  "server_event_remains_pending": true
}

Root cause:

workbench/relay_client.py, poll_and_ack() (~lines 187-236) only adds
events to the acknowledgement list when the callback returns True, but it
still computes and persists new_since from every fetched event. The cursor
therefore moves past unacknowledged events. The implementation comment near
the cursor update also acknowledges that such events will not be visible on a
later poll, which contradicts the function's retry contract.

Impact:

A valid relay event whose handler temporarily rejects or fails can remain
pending on the relay server while Station permanently skips it on subsequent
polls. This causes deterministic message-delivery/retry loss for relay and
Teams events. The reproduction uses no provider, financial write, credential,
or external destination.

Deterministic: Yes.

External request performed: No. The relay transport was replaced by an
ephemeral in-memory stub.

This is distinct from the already listed Teams relay network-allowlist issue:
that report concerns destination authorization; this report concerns local
cursor advancement after a failed callback.

Suggested fix:

Do not advance the persisted cursor past an event that was not acknowledged.
Advance only through the contiguous prefix of successfully handled events, or
persist a retry/pending sequence list and replay those events before advancing
the cursor. Treat callback exceptions the same as False for retry purposes.

Relevant source paths:

  • workbench/relay_client.pypoll_and_ack()
  • local relay poll state persistence (relay_poll_state.json)
3 pts

1 reply

Verified — reproduced exactly as described, deterministic. poll_and_ack() computed the persisted cursor from max_seq over EVERY fetched event and saved it regardless of the callback's verdict, so an event the handler returned False on (or that raised) was never acked yet the cursor still moved past it — permanently skipping a still-pending event on later polls. It directly contradicted the function's own docstring ("returns False / raises → next poll re-delivers it").

Fix: the cursor now advances ONLY through the contiguous prefix of events we acked and the server confirmed. The first un-acked event — and everything after it in sequence — stays re-fetchable; a callback exception is treated the same as False; and an ack POST failure (relay outage mid-ack) advances nothing. Acked events are dropped server-side, so re-polling from a lower cursor only re-delivers the un-acked ones (idempotency stays on_event's job, per the docstring).

Covered by 5 regression tests (false / raise / true / mixed [ack,nack,ack] prefix / ack-transport-failure). Fixed in code on the batch branch; ships in station-v0.75 (v0.74 was already cut when this landed). Thanks Dave — precise repro, made this a fast confirm.

Sign in to reply.