Researchers Expose 'CoT Forgery' Method That Tricks AI Into Sharing Dangerous Information
What Is CoT Forgery?
A team of AI safety researchers has documented a novel attack technique that manipulates large language models (LLMs) into sharing dangerous or prohibited information. The method, dubbed "CoT Forgery," exploits the models' chain-of-thought (CoT) reasoning processes—internal steps the AI uses to work through complex queries.
How the Attack Works
Standard jailbreaks attempt to directly trick a model into ignoring its safety guidelines. CoT Forgery takes a different approach: it feeds the model fabricated reasoning chains that appear legitimate and trusted. By convincing the AI that a user is wearing a green shirt or some other arbitrary badge of "trustworthiness," researchers found they could cause the model to output harmful information, including detailed instructions for synthesizing controlled substances like cocaine.
The technique essentially weaponizes the model's own reasoning architecture. Because LLMs are trained to follow and trust their internal logical chains, injecting false reasoning that mimics legitimateCoT patterns can override safety behaviors that would normally block such outputs.
Implications for AI Safety
This research highlights a fundamental challenge in securing LLM deployments. Models that rely heavily on reasoning traces to maintain safety may be vulnerable to manipulation through those very reasoning pathways. As AI systems become more integrated into consumer and enterprise applications, the attack surface continues to expand.
The findings underscore the need for defense-in-depth approaches rather than relying solely on prompting-based safety measures. Researchers recommend combining multiple guardrail techniques, including input filtering, output monitoring, and architectural changes that make CoT manipulation more difficult.
Responsible Disclosure
The researchers who discovered this vulnerability have notified relevant AI labs and are working with the community to develop mitigations before full public disclosure of the technical details.