External Benchmark Replay

SMERC against ILION-Bench v2

ILION-Bench v2 tests whether proposed agentic AI actions should be allowed or blocked. SMERC replays those scenarios through a recoverability checkpoint and returns richer runtime postures: allow, throttle, freeze, deny, or escalate.

ALLOW THROTTLE FREEZE DENY ESCALATE

Result Summary

Binary safety labels become recoverability postures

The replay used 400 ILION-Bench v2 scenarios: 200 expected `ALLOW` and 200 expected `BLOCK`. SMERC did not simply mirror binary labels. It preserved hard denials for most blocked actions and used middle states where recoverability called for restraint rather than a flat allow/block response.

400External benchmark scenarios replayed locally.
71%Exact binary match rate: 94 expected allows and 190 expected blocks matched directly.
0ILION `BLOCK` scenarios were converted into SMERC `ALLOW`.
29%Middle-state rate: `THROTTLE` or `FREEZE` instead of simple allow/block.

Why It Matters

SMERC is testing the space between allow and block

Most action-safety benchmarks classify whether an action should be allowed or blocked. SMERC asks a second question: if the action is technically permissible or ambiguous, should it still be constrained, paused, denied, or escalated because recovery would be weak?

This replay supports the current product direction: SMERC is not a content filter or generic policy engine. It is a recoverability checkpoint before consequential automated actions execute.

Next Question

What this changes

The project now has an external benchmark path. The next useful test is customer or design-partner shadow-mode data where reviewer agreement, false release risk, false constraint rate, latency, and rollback outcomes can be measured.