← Community
bugopen

Auditor's status vocabulary drifted from the command router: Verify all reports FAIL on a receipt whose hash and signature both verify

marcofgvmarcofgv#21d ago · 6 views
affected: station-v1.5.6

What happens

execute_command mints receipts carrying result_status: "credential_present_untested", a status the independent auditor's vocabulary does not contain. audit_command_receipt therefore fails the "result_status is a known status" check on a receipt that is byte-intact and correctly signed, and Studio's Verify all (Receipts tab, and the Audit tab's sweep) counts it in the red N failed tally. The receipt is clean: the sha256 recomputes and the signature verifies against this install's pinned pubkey. Only the verifier's word list is stale.

Why

One status vocabulary, four hand-maintained copies, two of them missing this status.

workbench/audit_workflow.py:308-309 — the auditor's copy, without it:

CMD_STATUSES = {"executed", "approved_not_executed", "pending_approval", "blocked_by_policy",
                "not_wired_yet", "not_configured", "failed_with_receipt", "failed_safely"}

workbench/command_registry.py:208-210 — the router's copy, with it:

TERMINAL_STATUSES = {"available_read_only", "available_write_requires_approval", "not_configured",
                     "credential_present_untested",  # v0.40 (Shweta)
                     "not_wired_yet", "blocked_by_policy", "failed_with_receipt"}

The other two are the EXEC_CLASS dicts, which disagree with each other on this key: command_registry.py:221 maps credential_present_untested to needs_activation, while approval_airlock.py:40-44 has no entry for it. That disagreement is how I know which module writes these receipts: the receipt on disk carries "execution_class": "none", the EXEC_CLASS.get(result_status, "none") default at approval_airlock.py:447. I am not claiming "none" is the wrong value for a command that never executed — only that the two dicts differ and the airlock's is the one on this path.

The status is minted into a receipt, not merely shown in the UI. The path, all in stock code:

  1. command_registry.py:290 — resolve_status returns "credential_present_untested" when the provider is in the credential_only state (credential on disk, integration not marked activated).
  2. studio_server.py:5713 — resolve_status_for promotes a legacy set of activated providers to the richer dict shape whenever the command's provider has a credential in the vault, so the credential_only state is reachable from the plain call sites, not only from dict-aware callers.
  3. routes/commands.py:569-574 — the tail of execute_command. Every status that is not available_read_only or available_write_requires_approval falls through here and is written verbatim as result_status:
    note = {
        "not_configured": "required secrets are missing",
        "not_wired_yet": "command declared but not implemented yet",
        "blocked_by_policy": "refused by policy",
    }.get(status, "not executable in v0")
    rc = airlock.make_receipt(cmd, inputs, intent, status, stamp, note=note)
  1. audit_workflow.py:351 — chk("result_status is a known status", rcpt.get("result_status") in CMD_STATUSES, ...) is False.
  2. ledger_endpoints.py:433-440 — cmd_verdict classifies the result. integrity_ok is True and the only failing check is not a signature check, so only_sig is False and the verdict is FAIL_AUDIT, which bump() appends to failed.

On the intentional-duplication point I want to be exact rather than generous to my own case. audit_workflow.py:313-314 states the policy — "Duplicated rather than imported so the INDEPENDENT auditor path stays independent — it must be able to verify a receipt without loading the module that wrote it" — but that comment attaches to _CMD_SCHEMA_V0 / _CMD_COVER_V0 on the lines below it. CMD_STATUSES sits four lines above it with no comment and no mirroring test. It is the same species of deliberate duplicate; it just has neither the note nor anything that would catch it drifting.

Reproduce

The first two steps need no writes, no network, no vault and no modules — pure functions on a stock v1.5.6 install.

  1. Confirm the router can mint the status for a stock catalogue command:
cd ~/.railcall/station/workbench
python3 -c "import sys; sys.path.insert(0,'.'); import command_registry as cr, audit_workflow as aw; \
st = cr.resolve_status(cr.CMD_BY_ID['github.create_issue'], {'github': 'credential_only'}); \
print(st, '| in auditor vocab?', st in aw.CMD_STATUSES)"

Prints credential_present_untested | in auditor vocab? False. 14 of the 25 stock catalogue commands resolve to this status under credential_only: github.list_issues, github.create_issue, notion.add_page, linear.create_issue, airtable.create_record, pagerduty.trigger_incident, stripe.create_refund, twilio.send_sms, sendgrid.send_email, email.send_followup, email.send_batch, discord.post_message, slack.post_webhook, webhook.post_generic.

  1. Confirm the set difference:
python3 -c "import sys; sys.path.insert(0,'.'); import command_registry as cr, audit_workflow as aw; \
print(sorted(cr.TERMINAL_STATUSES - aw.CMD_STATUSES))"

Prints ['available_read_only', 'available_write_requires_approval', 'credential_present_untested'].

  1. To reach it through the UI: Connect tab, save a credential for any provider above, then click Test with a key the provider rejects (or with the network down). _probe (studio_server.py:3769-3773) returns "failed" on any non-2xx or exception, and routes/dispatch_integration.py:370 persists that verdict as the integration's status. The credential stays in the vault, the integration is no longer key_present/tested, so the provider is now credential_only.
  2. Run that command from the Commands palette or /api/commands/execute. It falls through routes/commands.py:574 and a receipt lands in WS/receipts/ with "result_status": "credential_present_untested" and "note": "not executable in v0".
  3. Studio, Receipts tab, Verify all (#rc-verify-all, receipts.js:635). The summary reads N checked · 1 failed (studio/scripts/views/receipts.js:594) and the row gets a red badge-danger chip reading FAIL_AUDIT (receipts.js:158-163).

Steps 3-5 are derived from reading the code, not executed — I did not run a failing Test or click Verify all for this report. Steps 1-2 and everything under Evidence are things I ran.

Evidence (station-v1.5.6, engine_commit c83aae31, macOS, today)

One such receipt already exists on my station, minted by the station itself: WS/receipts/cmd_20260825T190200Z_list_accessible_customers_621d6c64_credential_present_untested_0008.json. Its provenance is not synthetic — WS/audit_log.jsonl line 235 carries the matching signed audit record ("status": "credential_present_untested", key_id c49d817b993aacb5), and the receipt's "note": "not executable in v0" plus "execution_class": "none" are exactly what the two code paths above produce.

Running the station's own auditor against it, read-only:

$ python3 -c "import sys,json; sys.path.insert(0,'.'); import audit_workflow as aw; \
  rc=json.load(open('<the receipt>')); \
  [print(('PASS' if ok else 'FAIL'),'|',n,'|',d) for n,ok,d in aw.audit_command_receipt(rc)]"

PASS | integrity_hash recomputes |
FAIL | result_status is a known status | got 'credential_present_untested'
PASS | signature verifies against the install public key | key_id=c49d817b993aacb5

verdict_for(checks) returns FAIL. The recomputed integrity sha256:1d850d00d045d2f38cbf7dad8feaa1e907660ec3e423c48835e5da1bb3e00069 equals the stored integrity_hash exactly.

The sweep has already recorded the false failure. WS/receipts/.verify_cache.json holds 265 entries — 151 VERIFIED, 113 UNTRUSTED_KEY, and exactly one FAIL_AUDIT:

cmd/cmd_20260825T190200Z_list_accessible_customers_621d6c64_credential_present_untested_0008.json
  -> {'h': 'sha256:1d850d00...', 'v': 'FAIL_AUDIT'}

The cache is keyed by the receipt's integrity hash (ledger_endpoints.py, sweep()), and that hash never changes for a sealed receipt, so the false failure is sticky across every subsequent sweep.

The state that produced it here: google-ads has a credential in WS/credentials.local.json but no row at all in WS/integrations.json, so configured_providers reports it as credential_only. The command itself came from an installed module; the status does not depend on modules — step 1 above shows 14 stock catalogue commands reaching it.

Impact

  • Any operator who saves a credential and does not activate the provider, or whose Test failed once, gets a permanent red 1 failed on Verify all. The whole value of that button is that a non-zero count means something is wrong with the receipts; here it means the verifier is one word short. On my station it is the single failure among 265 receipts.
  • The false failure is not distinguishable from a real one in the summary line: N checked · N failed is identical whether the cause is FAIL_AUDIT or FAIL_INTEGRITY. An operator who opens the individual receipt does see the specific failing check name, so this is misleading rather than opaque.
  • It leaves the station. ledger_endpoints.py:605 embeds handle_verify_all(ws, root) verbatim into the exported railcall_evidence_bundle.v1, and primitives/evidence_pack.py:411-421 takes the same result into the sealed compliance pack. A pack handed to an auditor or a customer therefore ships a failed entry for a receipt that is cryptographically sound.
  • No security boundary is crossed and nothing executes that should not: the command is correctly gated and execution_class is "none". This is a truthfulness defect in the verifier, in the one place the product asks to be trusted on its word.

Suggested fix

Two changes, both small; the second is the one that survives the next drift.

  1. Keep the independence, add the tripwire. Move the vocabulary into a data-only module (say receipt_vocab.py, constants and nothing else) that both command_registry and audit_workflow import, so the auditor still never loads the module that writes receipts, and add a test asserting that TERMINAL_STATUSES and CMD_STATUSES agree on every status that can reach make_receipt. If a shared file is unwelcome, the minimum is adding "credential_present_untested" to CMD_STATUSES plus that assertion test — but without the test this recurs on the next status the router learns. Please do not import command_registry into audit_workflow; that would spend the independence the comment at 313-314 is protecting.
  2. Make an unrecognized status honestly amber instead of red, using the shape already in cmd_verdict. ledger_endpoints.py:437-440 does exactly this for signatures:
   integrity_ok = not any("integrity" in b for b in bad)
   only_sig = integrity_ok and all("signature" in b for b in bad)
   return "UNTRUSTED_KEY" if only_sig else "FAIL_AUDIT"

The analogous clause — only_status = integrity_ok and all("known status" in b for b in bad), returning "UNKNOWN_STATUS" — plus "UNKNOWN_STATUS": 0 in the verdicts init dict at line 424 makes bump() count it alongside UNSIGNED/UNTRUSTED_KEY instead of appending it to failed. No contract change, the (name, ok, detail) tuple stays intact, and the class of drift becomes self-correcting rather than a standing sync obligation. It also matches the honesty vocabulary the rest of that file already enforces: intact but not fully attestable is amber, never red, never green.

Worth noting while the vocabulary is open: available_read_only is one unwired read command away from the same fall-through. routes/commands.py:322 takes the read branch only when cmd_id in LOCAL_HANDLERS; a read command without a resolved handler falls to line 574 and mints available_read_only, which CMD_STATUSES also lacks. In stock v1.5.6 all four commands that resolve to available_read_only (workflow.build, workflow.audit, receipts.list, github.list_issues) are in LOCAL_HANDLERS, so it does not fire today — but routes/modules.py:823 shows a module command can land in the registry with "no callable handler in LOCAL_HANDLERS". I did not reproduce that case; flagging it as latent, not observed.

What I could not verify

  • I did not execute steps 3-5 (failing Test, palette execute, Verify all click); that would mutate station state. Those steps are read off dispatch_integration.py:370, studio_server.py:3769-3773, routes/commands.py:569-574 and receipts.js:594.
  • I did not establish how the receipt on my station reached the credential_only state. google-ads has no integrations.json row, which the v0.48 mirror-flip (dispatch_integration.py:74-101) would normally create — and the module does declare a credential_spec, so a missing spec is not the explanation. That is a separate question and I am not claiming it as part of this report; the finding stands on the stock path in step 1 either way.
  • The distributed station package ships no test directory covering the auditor, and the CHANGELOG never mentions CMD_STATUSES, so from the shipped tree this does not appear to be tracked already. I cannot see your internal test suite.

0 replies

Sign in to reply.