← Community
bugfixed

seen_window_seconds is persisted but never enforced by incremental dedupe

DaveDave#339d ago · 143 views
affected: station-v0.65fixed in: station-v0.66

Reproduction steps:

  1. Configure an incremental contract with seen_window_seconds: 60.
  2. Persist a cursor in the seen set after processing.
  3. Re-deliver the same cursor after the configured window has elapsed.
  4. Pass it through fresh_items().

Expected:
Once the configured seen window has elapsed, the cursor should no longer be suppressed by the seen-set.

Actual:
The cursor is still treated as a duplicate after the window has elapsed.

Runtime reproduction with a 60-second window and 7200-second elapsed interval:

window exceeded : True
fresh items : 0
duplicates : 1
cursor suppressed : True

Result: WINDOW NOT ENFORCED

Root cause:
seen_window_seconds is parsed from the incremental contract and persisted in the watermark record, but is never used by the dedupe path.

The persisted seen state contains only cursor strings with no timestamp/age metadata, and fresh_items() performs plain membership suppression:

if str(cur) in seen:

The only eviction is _compact_seen(seen, max_seen), which is count-based (default 5000 entries), not time-based.

As a result, seen_window_seconds currently has no effect on cursor expiry. A cursor remains suppressed until it is pushed out by the count limit or the watermark state is reset.

Station version (railcall version):
station-v0.65

Module slug + version:
Synthetic incremental-contract reproduction

3 pts

1 reply

Confirmed — seen_window_seconds is parsed and persisted into the watermark record (watermark_store.py:142) but never read by the dedupe path; fresh_items() does plain str(cur) in seen membership (182) and the only eviction is count-based _compact_seen. So the declared window has no effect on cursor expiry. Credited 3 pts (moderate). Fixing to make the seen-set time-aware so cursors age out per the configured window.

Sign in to reply.