Large language model (LLM) agents deployed in safety-critical domains are increasingly governed by symbolic constraints compiled from domain knowledge bases. We identify and formalise a failure mode of this design, which we term WikiBreach: an agent output passes every symbolic predicate check while pragmatically conveying the content that the constraint was written to prohibit. We define WikiBreach as the conjunction of five predicates—symbolic pass, semantic relevance, user-specific application, operationalisation of prohibited management, and absence of adequate escalation—and organise eight evasion strategies against the symbolic blind spot each exploits. We release WikiBreach-v1, a benchmark of 1540 samples derived from the Prolog safety rules of a deployed General Practitioner agent, and evaluate a layered semantic audit pipeline against four baselines and three prevalence-aware trivial baselines. Two findings follow. First, F1 is unsuitable at the prevalence typical of red-teaming benchmarks: an all-positive classifier attains a higher F1 than any method evaluated, and measured by the Matthews correlation coefficient no baseline separates breaches from hard negatives to a useful degree (MCC ≤ +0.130). Second, the user-application predicate carries the discriminative burden: an ablation in which it is computed but not enforced attains MCC +0.107, whereas enforcing it as a gate attains +0.444 and reduces false positives on hard negatives from 0.835 to 0.271. On the multi-turn subset the proposed state tracker attains MCC +0.431 where the NLI baseline attains −0.184 at identical recall. We present the framework as a safety auditing instrument rather than a deployable filter.
- Open Access
- Article
WikiBreach: Detecting Semantic Evasion of Wiki-Compiled Constraints in LLM Agents
Author Information
Received: 07 June 2026 | Revised: 18 September 2026 | Accepted: 22 October 2026 | Published: 05 November 2026
Abstract
Keywords
symbolic constraints | LLM agents | semantic safety | NLI entailment | multi-turn dialogue | safety audit
References
- 1.
Karpathy, A. LLM Wiki. Available online: https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f (accessed on 15 April 2026).
- 2.
Krakovna, V.; Uesato, J.; Mikulik, V.; et al. Specification Gaming: The Flip Side of AI Ingenuity. Available online: https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ (accessed on 15 April 2026).
- 3.
Perez, E.; Huang, S.; Song, F.; et al. Red Teaming Language Models with Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 7–11 December 2022; pp. 3419–3448. https://doi.org/10.18653/v1/2022.emnlp-main.225.
- 4.
Singhal, K.; Azizi, S.; Tu, T.; et al. Large Language Models Encode Clinical Knowledge. Nature 2023, 620, 172–180. https://doi.org/10.1038/s41586-023-06291-2.
- 5.
Inan, H.; Upasani, K.; Chi, J.; et al. Llama Guard: LLM-Based Input-Output Safeguard for Human-AI Conversations. arXiv 2023, arXiv:2312.06674.
- 6.
Kang, M.; Chen, Z.; Xu, C.; et al. PolyGuard: Massive Multi-Domain Safety Policy-Grounded Guardrail Dataset. In Advances in Neural Information Processing Systems 38; Curran Associates, Inc.: Red Hook, NY, USA, 2025; pp. 13512–13632. https://doi.org/10.52202/085713-0406.
- 7.
Cao, H.; Jing, S.; Wang, Y.; et al. SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks. arXiv 2025, arXiv:2502.11090.
- 8.
Brown, T.B.; Mann, B.; Ryder, N.; et al. Language Models Are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Online, 6–12 December 2020; Volume 33, pp. 1877–1901.
- 9.
Ouyang, L.; Wu, J.; Jiang, X.; et al. Training Language Models to Follow Instructions with Human Feedback. In Proceedings of the Advances in Neural Information Processing Systems 35, New Orleans, LA, USA, 28 November–9 December 2022; Volume 35, pp. 27730–27744. https://doi.org/10.52202/068431-2011.
- 10.
Wielemaker, J.; Schrijvers, T.; Triska, M.; et al. SWI-Prolog. Theory Pract. Log. Program. 2012, 12, 67–96. https://doi.org/10.1017/s1471068411000494.
- 11.
Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; pp. 3980–3992. https://doi.org/10.18653/v1/d19-1410.
- 12.
He, P.; Liu, X.; Gao, J.; et al. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In Proceedings of the International Conference on Learning Representations (ICLR), Online, 3–7 May 2021.
- 13.
Devlin, J.; Chang, M.W.; Lee, K.; et al. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA, 2–7 June 2019; pp. 4171–4186. https://doi.org/10.18653/v1/n19-1423.
- 14.
Du, Z.; Qian, Y.; Liu, X.; et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 22–27 May 2022; Volume 1, pp. 320–335. https://doi.org/10.18653/v1/2022.acl-long.26.
- 15.
Pedregosa, F.; Varoquaux, G.; Gramfort, A.; et al. Scikit-Learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830.
This work is licensed under a Creative Commons Attribution 4.0 International License.
