Aussie AI

Jailbreak

Last Updated 22 October, 2025

by David Spuler, Ph.D.

Research on Jailbreak

Research papers include:

Andy Arditi, Oscar Obeso, Aaquib111, wesg, Neel Nanda, 27th Apr 2024, Refusal in LLMs is mediated by a single direction, LessWrong, https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
Adva Nakash Peleg, May 30, 2024, An LLM Journey: From POC to Production, https://medium.com/cyberark-engineering/an-llm-journey-from-poc-to-production-6c5ec6a172fb
Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, Chaowei Xiao, 14 Mar 2024, AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting, https://arxiv.org/abs/2403.09513 Code: https://github.com/rain305f/AdaShield
Jinhwa Kim, Ali Derakhshan, Ian G. Harris, 31 Oct 2023, Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield, https://arxiv.org/abs/2311.00172
Zixuan Ni, Longhui Wei, Jiacheng Li, Siliang Tang, Yueting Zhuang, Qi Tian, 8 Aug 2023 (v2), Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion, https://arxiv.org/abs/2308.02552
Xiao Peng, Tao Liu, Ying Wang, 3 Jun 2024 (v2), Genshin: General Shield for Natural Language Processing with Large Language Models, https://arxiv.org/abs/2405.18741
Ayushi Nirmal, Amrita Bhattacharjee, Paras Sheth, Huan Liu, 8 May 2024 ( v2), Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales, https://arxiv.org/abs/2403.12403 Code: https://github.com/AmritaBh/shield
Shweta Sharma, 27 Jun 2024, Microsoft warns of ‘Skeleton Key’ jailbreak affecting many generative AI models, https://www.csoonline.com/article/2507702/microsoft-warns-of-novel-jailbreak-affecting-many-generative-ai-models.html
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri, 26 Jun 2024, WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs, https://arxiv.org/abs/2406.18495
Maksym Andriushchenko, Nicolas Flammarion, 16 Jul 2024, Does Refusal Training in LLMs Generalize to the Past Tense? https://arxiv.org/abs/2407.11969 Code: https://github.com/tml-epfl/llm-past-tense
Kylie Robison, Jul 20, 2024, OpenAI’s latest model will block the ‘ignore all previous instructions’ loophole, https://www.theverge.com/2024/7/19/24201414/openai-chatgpt-gpt-4o-prompt-injection-instruction-hierarchy
Chip Huyen, Jul 25, 2024, Building A Generative AI Platform, https://huyenchip.com/2024/07/25/genai-platform.html
Jaymari Chua, Yun Li, Shiyi Yang, Chen Wang, Lina Yao, 6 Jul 2024, AI Safety in Generative AI Large Language Models: A Survey, https://arxiv.org/abs/2407.18369
Ayush RoyChowdhury, Mulong Luo,, Prateek Sahu,, Sarbartha Banerjee, Mohit Tiwari, Aug 2024, ConfusedPilot: Confused Deputy Risks in RAG-based LLMs, https://confusedpilot.info/confused_pilot_new.pdf
Dr. Ashish Bamania, Sep 2024, ‘MathPrompt’ Embarassingly Jailbreaks All LLMs Available On The Market Today. A deep dive into how a novel LLM Jailbreaking technique called ‘MathPrompt’ works, why it is so effective, and why it needs to be patched as soon as possible to prevent harmful LLM content generation, https://bamania-ashish.medium.com/mathprompt-embarassingly-jailbreaks-all-llms-available-on-the-market-today-d749da26c6e8
Y. Bai et al., "Backdoor Attack and Defense on Deep Learning: A Survey," in IEEE Transactions on Computational Social Systems, doi: 10.1109/TCSS.2024.3482723. https://ieeexplore.ieee.org/abstract/document/10744415
Steve Jones, Oct 3, 2024, LLM Prompt Injection: Never send the request to the model. Classify, rewrite and reject, https://blog.metamirror.io/llm-prompt-injection-never-send-the-request-to-the-model-e8017269b96a
Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, Peyman Najafirad, 5 Nov 2024 (v2), Jailbreaking Large Language Models with Symbolic Mathematics, https://arxiv.org/abs/2409.11445
Alwin Peng, Julian Michael, Henry Sleight, Ethan Perez, Mrinank Sharma, 12 Nov 2024, Rapid Response: Mitigating LLM Jailbreaks with a Few Examples, https://arxiv.org/abs/2411.07494
Kyle O'Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, Forough Poursabzi-Sangde, 18 Nov 2024, Steering Language Model Refusal with Sparse Autoencoders, https://arxiv.org/abs/2411.11296
Zachary Coalson, Jeonghyun Woo, Shiyang Chen, Yu Sun, Lishan Yang, Prashant Nair, Bo Fang, Sanghyun Hong, 10 Dec 2024, PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips, https://arxiv.org/abs/2412.07192
Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, Prasanna Sattigeri, 10 Dec 2024, Granite Guardian, https://arxiv.org/abs/2412.07724 https://github.com/ibm-granite/granite-guardian (Open-sourcing of safety models with many capabilities.)
Mohit Sewak, Dec 6, 2024, Prompt Injection Attacks on Large Language Models, https://pub.towardsai.net/prompt-injection-attacks-on-large-language-models-bd8062fa1bb7
Sicheng Zhu, Brandon Amos, Yuandong Tian, Chuan Guo, Ivan Evtimov, 13 Dec 2024, AdvPrefix: An Objective for Nuanced LLM Jailbreaks, https://arxiv.org/abs/2412.10321
Aditi Bodhankar, Jan 16, 2025, How to Safeguard AI Agents for Customer Service with NVIDIA NeMo Guardrails, https://developer.nvidia.com/blog/how-to-safeguard-ai-agents-for-customer-service-with-nvidia-nemo-guardrails/
Xin Yi, Yue Li, Linlin Wang, Xiaoling Wang, Liang He, 18 Jan 2025, Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks, https://arxiv.org/abs/2501.10639
Yue Liu, Hongcheng Gao, Shengfang Zhai, Jun Xia, Tianyi Wu, Zhiwei Xue, Yulin Chen, Kenji Kawaguchi, Jiaheng Zhang, Bryan Hooi, 30 Jan 2025, GuardReasoner: Towards Reasoning-based LLM Safeguards, https://arxiv.org/abs/2501.18492
Taryn Plumb, February 3, 2025, Anthropic claims new AI security method blocks 95% of jailbreaks, invites red teamers to try, https://venturebeat.com/security/anthropic-claims-new-ai-security-method-blocks-95-of-jailbreaks-invites-red-teamers-to-try/
Holistic AI Team, March 6, 2025, Anthropic’s Claude 3.7 Sonnet Jailbreaking & Red Teaming Audit: The Most Secure Model Yet? https://www.holisticai.com/blog/claude-3-7-sonnet-jailbreaking-audit
Jiacheng Liang, Tanqiu Jiang, Yuhui Wang, Rongyi Zhu, Fenglong Ma, Ting Wang, 16 May 2025, AutoRAN: Weak-to-Strong Jailbreaking of Large Reasoning Models, https://arxiv.org/abs/2505.10846
Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, Amelia Glaese, 31 Jan 2025, Trading Inference-Time Compute for Adversarial Robustness, https://arxiv.org/abs/2501.18841
Manuel Cossio, 3 Aug 2025, A comprehensive taxonomy of hallucinations in Large Language Models, https://arxiv.org/abs/2508.01781
Anthropic, 13 Aug 2025, Building Safeguards for Claude, https://www.anthropic.com/news/building-safeguards-for-claude
Wenpeng Xing, Mohan Li, Chunqiang Hu, Haitao XuNingyu Zhang, Bo Lin, Meng Han, 8 Aug 2025, Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs, https://arxiv.org/abs/2508.10029
Fan Yang, 9 Aug 2025, The Cost of Thinking: Increased Jailbreak Risk in Large Language Models, https://arxiv.org/abs/2508.10032
Xiaoxue Yang, Jaeha Lee, Anna-Katharina Dick, Jasper Timm, Fei Xie, Diogo Cruz, 11 Aug 2025, Multi-Turn Jailbreaks Are Simpler Than They Seem, https://arxiv.org/abs/2508.07646
Xianjun Yang, Liqiang Xiao, Shiyang Li, Faisal Ladhak, Hyokun Yun, Linda Ruth Petzold, Yi Xu, William Yang Wang, 9 Aug 2025, Many-Turn Jailbreaking, https://arxiv.org/abs/2508.06755
Xuancun Lu, Zhengxian Huang, Xinfeng Li, Chi Zhang, Xiaoyu ji, Wenyuan Xu, 11 Aug 2025, POEX: Towards Policy Executable Jailbreak Attacks Against the LLM-based Robots, https://arxiv.org/abs/2412.16633
Tatia Tsmindashvili, Ana Kolkhidashvili, Dachi Kurtskhalia, Nino Maghlakelidze, Elene Mekvabishvili, Guram Dentoshvili, Orkhan Shamilov, Zaal Gachechiladze, Steven Saporta, David Dachi Choladze, 11 Aug 2025, Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration, https://arxiv.org/abs/2505.17066
Jirui Yang, Zheyu Lin, Zhihui Lu, Yinggui Wang, Lei Wang, Tao Wei, Xin Du, Shuhan Yang, 31 Jul 2025, CEE: An Inference-Time Jailbreak Defense for Embodied Intelligence via Subspace Concept Rotation, https://arxiv.org/abs/2504.13201
Zheng Zhang, Peilin Zhao, Deheng Ye, Hao Wang, 28 Jul 2025, Enhancing Jailbreak Attacks on LLMs via Persona Prompts, https://arxiv.org/abs/2507.22171
Jiecong Wang, Haoran Li, Hao Peng, Ziqian Zeng, Zihao Wang, Haohua Du, Zhengtao Yu, 1 Aug 2025, Activation-Guided Local Editing for Jailbreaking Attacks, https://arxiv.org/abs/2508.00555
Yelim Ahn, Jaejin Lee, 2 Aug 2025, PUZZLED: Jailbreaking LLMs through Word-Based Puzzles, https://arxiv.org/abs/2508.01306
Yik Siu Chan, Narutatsu Ri, Yuxin Xiao, Marzyeh Ghassemi, 2 Aug 2025, Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions, https://arxiv.org/abs/2502.04322
Muyang Zheng, Yuanzhi Yao, Changting Lin, Rui Wang, Caihong Kai, 4 Aug 2025, MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning, https://arxiv.org/abs/2506.16792
Rui Pu, Chaozhuo Li, Rui Ha, Litian Zhang, Lirong Qiu, Xi Zhang, 5 Aug 2025, Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning, https://arxiv.org/abs/2508.03054
Bodam Kim, Hiskias Dingeto, Taeyoun Kwon, Dasol Choi, DongGeon Lee, Haon Park, JaeHoon Lee, Jongho Shin, 5 Aug 2025, When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs, https://arxiv.org/abs/2508.03365
Giovanni Cherubin, Andrew Paverd, 4 Aug 2025, Highlight & Summarize: RAG without the jailbreaks, https://arxiv.org/abs/2508.02872
Ruofan Wang, Juncheng Li, Yixu Wang, Bo Wang, Xiaosen Wang, Yan Teng, Yingchun Wang, Xingjun Ma, Yu-Gang Jiang, 5 Aug 2025, IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves, https://arxiv.org/abs/2411.00827
Junwoo Ha, Hyunjun Kim, Sangyoon Yu, Haon Park, Ashkan Yousefpour, Yuna Park, Suhyun Kim, 5 Aug 2025, M2S: Multi-turn to Single-turn jailbreak in Red Teaming for LLMs, https://arxiv.org/abs/2503.04856
Thilo Hagendorff, Erik Derner, Nuria Oliver, 4 Aug 2025, Large Reasoning Models Are Autonomous Jailbreak Agents, https://arxiv.org/abs/2508.04039
Xiaohu Li and Yunfeng Ning and Zepeng Bao and Mayi Xu and Jianhao Chen and Tieyun Qian, 6 Aug 2025, CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations, https://arxiv.org/abs/2507.06043
Renmiao Chen, Shiyao Cui, Xuancheng Huang, Chengwei Pan, Victor Shea-Jay Huang, QingLin Zhang, Xuan Ouyang, Zhexin Zhang, Hongning Wang, and Minlie Huang, 7 Aug 2025, JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering, https://arxiv.org/abs/2508.05087
Jesson Wang, Zhanhao Hu, David Wagner, 7 Aug 2025, JULI: Jailbreak Large Language Models by Self-Introspection, https://arxiv.org/abs/2505.11790
Shuang Liang, Zhihao Xu, Jialing Tao, Hui Xue, Xiting Wang, 8 Aug 2025, Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models: A Unified and Accurate Approach, https://arxiv.org/abs/2508.09201
Zuoou Li, Weitong Zhang, Jingyuan Wang, Shuyuan Zhang, Wenjia Bai, Bernhard Kainz, Mengyun Qiao, 11 Aug 2025, Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity, https://arxiv.org/abs/2508.09218
Boyuan Chen, Minghao Shao, Abdul Basit, Siddharth Garg, Muhammad Shafique, 13 Aug 2025, MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs, https://arxiv.org/abs/2506.22557
Ma Teng and Jia Xiaojun and Duan Ranjie and Li Xinfeng and Huang Yihao and Jia Xiaoshuang and Chu Zhixuan and Ren Wenqi, 18 Aug 2025, Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models, https://arxiv.org/abs/2412.05934
Zhipeng Wei, Yuqi Liu, N. Benjamin Erichson, 16 Aug 2025, Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection, https://arxiv.org/abs/2411.01077
Yangyang Guo and Yangyan Li and Mohan Kankanhalli, 18 Aug 2025, Involuntary Jailbreak, https://arxiv.org/abs/2508.13246
Jiaming Hu, Haoyu Wang, Debarghya Mukherjee, Ioannis Ch. Paschalidis, 19 Aug 2025, CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protection, https://arxiv.org/abs/2508.14128
Xiangman Li, Xiaodong Wu, Qi Li, Jianbing Ni, and Rongxing Lu, 21 Aug 2025, SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks, https://arxiv.org/abs/2508.15182
Darpan Aswal and C\'eline Hudelot, 22 Aug 2025, LLMSymGuard: A Symbolic Safety Guardrail Framework Leveraging Interpretable Jailbreak Concepts, https://arxiv.org/abs/2508.16325
Yu Yan, Sheng Sun, Zhe Wang, Yijun Lin, Zenghao Duan, zhifei zheng, Min Liu, Zhiyi yin, Jianping Zhang, 22 Aug 2025, Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs, https://arxiv.org/abs/2508.16347
Yu Yan, Sheng Sun, Zenghao Duan, Teli Liu, Min Liu, Zhiyi Yin, Jiangyu Lei, Qi Li, 22 Aug 2025, from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors, https://arxiv.org/abs/2503.00038
Chongwen Zhao, Zhihao Dou, Kaizhu Huang, 25 Aug 2025, Defending against Jailbreak through Early Exit Generation of Large Language Models, https://arxiv.org/abs/2408.11308
Junchen Ding, Jiahao Zhang, Yi Liu, Ziqi Ding, Gelei Deng, Yuekang Li, 25 Aug 2025, TombRaider: Entering the Vault of History to Jailbreak Large Language Models, https://arxiv.org/abs/2501.18628
Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, Saadia Gabriel, 23 Aug 2025, X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents, https://arxiv.org/abs/2504.13203
Hanjiang Hu, Alexander Robey, Changliu Liu, 25 Aug 2025, Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks, https://arxiv.org/abs/2503.00187
Chuhan Zhang, Ye Zhang, Bowen Shi, Yuyou Gan, Tianyu Du, Shouling Ji, Dazhan Deng, Yingcai Wu, 4 Sep 2025, NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models, https://arxiv.org/abs/2509.03985
Yakai Li, Jiekang Hu, Weiduan Sang, Luping Ma, Dongsheng Nie, Weijuan Zhang, Aimin Yu, Yi Su, Qingjia Huang, Qihang Zhou, 25 Aug 2025, Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models, https://arxiv.org/abs/2504.21038
Haibo Jin, Ruoxi Chen, Peiyan Zhang, Andy Zhou, Yang Zhang, Haohan Wang, 28 Aug 2025, GUARD: Guideline Upholding Test through Adaptive Role-play and Jailbreak Diagnostics for LLMs, https://arxiv.org/abs/2508.20325
Junjie Chu and Mingjie Li and Ziqing Yang and Ye Leng and Chenhao Lin and Chao Shen and Michael Backes and Yun Shen and Yang Zhang, 28 Aug 2025, JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring, https://arxiv.org/abs/2508.20848
Chongwen Zhao and Kaizhu Huang, 1 Sep 2025, Unraveling LLM Jailbreaks Through Safety Knowledge Neurons, https://arxiv.org/abs/2509.01631
Sihao Wu, Gaojie Jin, Wei Huang, Jianhong Wang, Xiaowei Huang, 30 Aug 2025, Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models, https://arxiv.org/abs/2509.00373
Ruoxi Cheng, Yizhong Ding, Shuirong Cao, Ranjie Duan, Xiaoshuang Jia, Shaowei Yuan, Simeng Qin, Zhiqiang Wang, Xiaojun Jia, 30 Aug 2025, PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization, https://arxiv.org/abs/2412.05892
Shei Pern Chua, Thai Zhen Leng, Teh Kai Jun, Xiao Li, Xiaolin Hu, 4 Sep 2025, Between a Rock and a Hard Place: Exploiting Ethical Reasoning to Jailbreak LLMs, https://arxiv.org/abs/2509.05367
Youjia Zheng, Mohammad Zandsalimy, and Shanu Sushmita, 5 Sep 2025, Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models, https://arxiv.org/abs/2509.05471
Junjie Mu, Zonghao Ying, Zhekui Fan, Zonglei Jing, Yaoyuan Zhang, Zhengmin Yu, Wenxin Zhang, Quanchen Zou, Xiangzheng Zhang, 8 Sep 2025, Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?, https://arxiv.org/abs/2509.06350
Yunhan Zhao, Xiang Zheng, Xingjun Ma, 16 Sep 2025, Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models, https://arxiv.org/abs/2509.12724
Johan Wahr\'eus, Ahmed Hussain, Panos Papadimitratos, 16 Sep 2025, Jailbreaking Large Language Models Through Content Concretization, https://arxiv.org/abs/2509.12937
Seongho Joo, Hyukhun Koh, Kyomin Jung, 13 Sep 2025, Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding, https://arxiv.org/abs/2509.10931
Chentao Cao, Xiaojun Xu, Bo Han, Hang Li, 15 Sep 2025, Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check, https://arxiv.org/abs/2509.11629
Yibo Zhang, Liang Lin, 14 Sep 2025, ENJ: Optimizing Noise with Genetic Algorithms to Jailbreak LSMs, https://arxiv.org/abs/2509.11128
Guorui Chen, Yifan Xia, Xiaojun Jia, Zhijiang Li, Philip Torr, Jindong Gu, 18 Sep 2025, LLM Jailbreak Detection for (Almost) Free!, https://arxiv.org/abs/2509.14558
Hyunjun Kim, Junwoo Ha, Sangyoon Yu, Haon Park, 10 Sep 2025, X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates, https://arxiv.org/abs/2509.08729