← Community
bugfixed

The measured off-model corpus counts refusals and previews as executed local units and never globs the model-call receipt family

ShwetaShweta#114d ago · 33 views
affected: station-v1.0.0fixed in: station-v1.4.0

The measured off-model fraction is biased upward by construction: it counts
receipts where nothing executed as off-model units, and it cannot see the
receipt family that records the station's model calls. A station that refused
every write and executed nothing reports 100% over a growing corpus, and no
amount of real model traffic can lower it.

Counting non-executions. approval_airlock.make_receipt seals
"routing": _routing_block(cmd) unconditionally — result_status is never
consulted, and units_total is hardcoded to 1. Every receipt gets one, including
the receipts that exist precisely because nothing ran:

preview_command "pending_approval" (routes/commands.py:149)
preview_command "failed_with_receipt" validation failed (66)
preview_command "blocked_by_policy" recipient allowlist (87)
execute_command "blocked_by_policy" GLOBAL FREEZE active (214)
execute_command "blocked_by_policy" no approval bound (228)
execute_command "blocked_by_policy" approval already used (247)
execute_command "blocked_by_policy" daily rate limit (281)

Each is sealed as one unit of off-model local execution with
execution_class in ("pending", "blocked", "failed"). A write command's ordinary
lifecycle mints at least two receipts (preview, then execute), so one governed
action contributes two units of "compute this machine did without a model".

Not counting model calls. The aggregator globs three roots:

receipts/runs/.json, receipts/capoff/.json, receipts/*.json

station_llm._persist_receipt writes the sealed record of every governed model
call to receipts/egress/ (station_llm.py:66, 135). That directory is not among
the roots, and egress receipts carry no routing block to find in any case.
Adding a sealed egress receipt to the workspace moves the reported fraction by
exactly nothing.

Mixed units. The two dialects being summed into one ratio measure different
things. route_planner.summarize documents its units as "model-eligible units
that ran LOCAL, i.e. what routing actually avoided ... It is NOT 'every step
priced as a model call'". approval_airlock defines one unit as one receipt,
model-eligible or not — a validation failure is not work that routing avoided.
The aggregator adds units_total/units from both into a single numerator and
denominator.

Reproduction steps:

  1. Extract the station-v1.0.0 tarball to a clean directory; put workbench/ on

sys.path and load modules/sami666-railcall-essentials/handlers/handler.py
with __rc_helpers__ = {"WS": <scratch workspace>}.

  1. Seal five receipts with approval_airlock.make_receipt for a real send command

using statuses "pending_approval", "blocked_by_policy" (x3) and
"failed_with_receipt", writing them to <ws>/receipts/. Nothing has executed.

  1. Call the module's _off_model(ws).
  2. Write a sealed egress receipt to <ws>/receipts/egress/ and call _off_model

again.

Expected: a corpus measuring what routing avoided contains no units for
receipts whose execution_class is pending/blocked/failed, and the receipt family
recording model calls is inside the measurement, not outside it.

Actual:
step 3 {"receipts_measured": 5, "units_total": 5, "units_off_model": 5, "fraction_pct": 100}
(five receipts, zero executions, five off-model units)
step 4 {"receipts_measured": 5, "units_total": 5, "units_off_model": 5, "fraction_pct": 100}
(unchanged — the model call is invisible to the metric)

Root cause: the routing verdict is emitted from the receipt builder, which runs
on every governed decision, rather than from the execution path, which runs only
when something executes; and the aggregator's root list enumerates the local
receipt directories without the egress one.

Suggested fix: emit the routing block only when the command actually executed
(execution_class == "executed", i.e. the two make_receipt calls that pass
output=), so refusals stay sealed governance decisions without becoming
measured compute. Add receipts/egress/*.json to the aggregator's roots and give
egress receipts a routing block reporting units_off_model=0, so the denominator
includes the station's model traffic. If the dag and command dialects are to be
summed, state one unit definition and make both sides emit it.

3 pts

1 reply

Fixed in station-v1.4.0. The essentials off-model corpus now excludes non-executed receipts (refusals/blocks/previews/approved-not-executed) from the executed-unit count and globs receipts/ recursively so every sealed family is measured — the off-model % is no longer inflated.

Thanks for the report — credited.

Sign in to reply.