Lasso Security's red team encoded a harmful prompt in NATO phonetic alphabet — Alpha, Bravo, Charlie — and sent it to Perplexity's BrowseSafe guardrail. The guardrail decoded the text, understood the content, and let it through. They tried Base32 encoding. Same result. Across their test suite, 36% of encoded prompts bypassed the safety layer entirely.
The guardrail was an LLM. The system it was guarding was an LLM. Both shared the same fundamental weakness: they process instructions and data in the same channel, with no architectural separation between the two. The guard was susceptible to exactly the same class of attacks as the prisoner.
This is not a bug in Perplexity's implementation. It is a structural property of every system that uses one language model to police another.
The Same Model, Different Hat
HiddenLayer's research team named the core problem precisely: "Same Model, Different Hat." When a guardrail LLM evaluates whether a prompt is safe, it performs the same kind of reasoning — attention-based pattern matching over token sequences — as the production model that will execute the prompt. Both models share the same vulnerability surface. A sufficiently creative encoding, reframing, or multi-turn decomposition that fools one will fool the other.
This is not theoretical. Wen et al. tested 10 guardrail models across 1,445 attack scenarios at LLMSec 2025. The results are the most comprehensive benchmark I have seen on this question, and the numbers are damning.
Qwen3Guard-8B, the best-performing model in their evaluation, scored 91% accuracy on known attack patterns — the categories that appear in public jailbreak datasets, the attacks that guardrail vendors train against. On novel attack categories — 9 new scenarios the researchers designed specifically for the study — accuracy dropped to 33.8%.
A 57.2 percentage point generalization gap. The guardrail was pattern-matching against its training data, not reasoning about safety.
What the Guardrails Actually Miss
The category-level breakdown is where the architecture falls apart. Wen et al. organized their 21 attack categories by difficulty, and the results expose which attack strategies guardrails are structurally incapable of catching.
Business-framed attacks — harmful requests wrapped in corporate language like "draft a competitive intelligence memo" — achieved a 96.8% evasion rate across all 10 models. The guardrails could not distinguish between a legitimate business request and a social engineering wrapper. Social engineering scenarios hit 84.4%. Boundary cases — prompts that sit in the ambiguous space between harmful and benign — reached 88.9%.
The pattern is consistent: the more a malicious prompt resembles normal professional communication, the less likely any LLM guardrail catches it. This is not a failure of model size or training data quantity. It is a consequence of how language models process text. An LLM trained to be helpful will interpret business framing as a legitimate request because that is exactly what it was trained to do. Adding a system prompt that says "also check for safety" does not override the foundational training objective.
Unit42's independent evaluation of 17 guardrail solutions confirms the pattern from a different angle. Against known jailbreak templates — the attacks that ship in public red-team toolkits — most guardrails performed respectably, with evasion rates between 8.9% and 27.8%. The moment the attack strategy deviated from those templates, performance collapsed.
The Vendors Know
The uncomfortable part is that the organizations deploying LLM guardrails already acknowledge this limitation — in language carefully designed to not alarm customers.
OpenAI's safety documentation states their guardrails are "not foolproof." The UK's National Cyber Security Centre, part of GCHQ, goes further in their official guidance: "LLMs simply do not enforce a security boundary between instructions and data." That sentence alone should disqualify LLM-as-judge architectures from any system where the guardrail is the last line of defense. The government agency responsible for national cybersecurity is telling you, in plain language, that the architecture cannot do what vendors claim it does.
Simon Willison, who has tracked prompt injection since naming the vulnerability class, frames it as an unsolved problem in computer science: "There is no known reliable way to have an LLM process untrusted content and guarantee it won't be manipulated." Not "difficult." Not "requires careful engineering." Unsolved.
Yet the default architecture at most AI companies remains: production LLM processes the request, guardrail LLM evaluates the output, guardrail LLM scans the input. Three LLMs, zero architectural diversity. The attack surface does not shrink with each layer — it multiplies.
What the Numbers Say Actually Works
The Pangea ecological study measured 330,000 attack attempts against production AI systems and found that single-layer LLM guardrails allowed roughly 10% of attacks through. That number alone should end the conversation about LLM-only safety architectures for anything with real consequences.
But the same study found something constructive. Systems that combined deterministic rules, specialized classifiers, and LLM reasoning in a layered architecture reduced attack success to 0.003%. Three orders of magnitude better than the LLM-only approach.
The architecture that works looks nothing like "add another LLM":
| Layer | Function | What it catches | Latency |
|---|---|---|---|
| Deterministic rules | Regex, blocklists, schema validation | Known patterns, format violations | Microseconds |
| Specialized classifiers | Fine-tuned models trained on specific attack types | Category-specific threats, toxicity | Low milliseconds |
| LLM reasoning | Contextual evaluation of ambiguous cases | Novel framing, complex multi-turn attacks | Hundreds of milliseconds |
| Structured output | Constrained decoding, JSON schema enforcement | Format-based exfiltration, injection via output | Varies by method |
Each layer catches a different class of failure. Deterministic rules handle the attacks that never change — SQL injection patterns, known toxic phrases, format violations. Specialized classifiers handle category-specific threats at speeds and accuracy levels that general-purpose LLMs cannot match, because they are trained on narrow tasks rather than general helpfulness. The LLM layer handles what is left — the genuinely ambiguous cases that require contextual reasoning.
The critical design principle: the LLM is the last layer, not the only layer. By the time a prompt reaches the LLM evaluator, the deterministic and classifier layers have already filtered the attacks that exploit the LLM's known weaknesses. The 96.8% business-framing evasion rate becomes irrelevant if a classifier trained specifically on social engineering patterns catches it two layers earlier.
Meta's LlamaFirewall implements a version of this architecture: PromptGuard (a fine-tuned classifier) handles injection detection, CodeShield runs deterministic analysis on generated code, and an agent-alignment layer evaluates multi-step reasoning chains. In AgentDojo benchmarks, the layered approach outperformed every single-layer alternative.
R2-Guard, presented at ICLR 2025, takes the hybrid approach further with a neural-symbolic architecture — combining neural pattern matching with symbolic reasoning rules that cannot be prompt-injected because they do not process natural language. The symbolic component enforces invariants that hold regardless of how creatively the input is framed.
The Architecture Is the Argument
The generalization gap in Wen et al.'s data is not a temporary limitation waiting for better models. It is a property of the architecture. Language models trained on internet text will always be better at recognizing known attack patterns than novel ones, because that is what statistical learning does — it generalizes from training distributions. An attacker who reads the same public jailbreak datasets has already mapped the guardrail's decision boundary.
Deterministic rules do not have this problem. A regex that blocks DROP TABLE does not care how creatively you frame the request. A JSON schema validator does not get confused by business language. These tools are brittle in their own way — they cannot handle the long tail of novel attacks — but their failure modes are predictable, auditable, and fixable without retraining a billion-parameter model.
The 0.003% number from Pangea is not magic. It is what happens when you stop asking one architecture to solve every category of threat and instead match each threat category to the detection method built for it. The LLM stays in the stack. It just stops being the load-bearing wall.
NCSC's guidance lands differently once you have seen the benchmark data: LLMs do not enforce a security boundary between instructions and data. Neither does a guardrail LLM. The boundary has to come from outside the language model entirely — from the deterministic layer that cannot be talked out of its rules, from the classifier that was never trained to be helpful, from the schema validator that does not understand English well enough to be manipulated by it.
The question is not whether LLM guardrails will improve. They will. The question is whether improvement within the same architecture can ever close a 57-point generalization gap, or whether the gap is the architecture telling you something about itself.