Safety monitor failures can hide evidence from llm decisions

Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries

Artificial IntelligenceCryptography and SecuritySoftware Engineering

Summary

Large language models (LLMs) use safety monitors to check and control risky outputs during operation. This paper shows that sometimes the monitors detect a problem, but the warning does not reach the part of the system that decides what to do, meaning unsafe outputs can slip through. The authors found many examples of this issue in real LLM safety frameworks by testing with special test cases that highlight these hidden evidence paths. Their work reveals that safety checks need regular testing and updates as systems evolve to ensure all warnings lead to the right actions.

What this means in practice

  • For llm safety engineers: Identify and fix hidden evidence path failures that prevent safety monitors from enforcing rules in LLM frameworks.
  • For ai platform developers: Design regression tests that track evidence flow through evolving schemas and adapters to maintain safety monitor effectiveness.

Authors

Junchi Liao

Abstract

Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision. This differs from jailbreaks and detector errors because the evidence is already recognized yet disconnected from enforcement. We study this repair-boundary signature with canary contracts: parent/fixed confirmation units that place a known payload only on the changed evidence surface and record monitor visibility together with the downstream decision for action-boundary cases. Applying the protocol to public framework repairs, we confirm 18 evidence-path failures across 11 LLM or agent safety project groups and four implementation ecosystems. Nine are established-boundary repairs, eight are new-surface policy-coverage gaps, and one is a boundary-only evidence case. Fourteen reach E1 enforcement boundaries: seven native action outcomes, six configured enforcement boundaries, and one synthetic-consumer consequence. The remaining cases affect verifier, safety-prompt, or observability channels. The cases form carrier, adapter, authority, and surface families and show how ordinary application behavior can remain intact while a safety-specific evidence path misses its decision. Exact revisions, deterministic outcomes, negative controls, and a frozen prospective audit document the confirmation rule. Monitor assumptions therefore need regression tests at the schemas, streams, tools, and adapters that frameworks evolve.