Researchers [Charles Ye], [Jasmine Cui], and [Dylan Hadfield-Menell] have shown that AI Large Language Models (LLMs) can fail to correctly distinguish between different instruction sources because they prioritize writing style over metadata tags, and this role confusion leads to a powerful attack called CoT (Chain of Thought) Forgery. We’ll explain exactly how it works after a bit of background review.

Prompt injection was where “getting an LLM to do something it shouldn’t” started by exploiting the fact that LLMs communicate like people, but are much more obedient. For a while, simply telling an LLM “ignore all previous instructions and <do something funny>” yielded results no matter how transparently dumb the instructions were, and the reason it worked at all was because LLMs do not have separate data and instruction streams; it’s all one big lump of input. It’s up to the model to sort legit instructions from untrusted, user-provided data. One step towards mitigating this was the addition of roles.

Roles are a method of segmenting that big blob of input into an organized hierarchy with metadata tags. For example with <system> at the top, and <user> requests much lower down. Instructions in a role are followed as long as they don’t conflict with higher-priority ones. A system-level directive of “don’t discuss illegal things” would override a user’s request to provide a recipe for cocaine.

Another type of tag is <think>, the contents of which represent a model’s internal reasoning process. Predictably, this role has high trust. What if one could inject spoofed internal reasoning? Researchers demonstrate this with an attack called CoT (Chain of Thought) Forgery.

CoT Forgery relies on LLMs being shown to prioritize writing style over actual tag content. By writing convoluted reasoning in a style that closely matches a model’s internal and highly distinct <think> style, the model is tricked into treating it like an already-reached conclusion. Note this attack does not simply wrap the injected prompt in <think> tags.

That’s the core of it, but the rest of the research makes a compelling case that, at least for the time being, mitigating prompt injection-style attacks is likely to remain an evolving process rather than become a solved problem anytime soon. LLMs are obedient but stuck with instructions and data in a single channel, role perception isn’t binary, and humans are clever and creative.

The complete paper is available online, and code examples are on GitHub.

研究人员 Charles Ye、Jasmine Cui 和 Dylan Hadfield-Menell 已经证明，AI 大语言模型（LLMs）可能无法正确区分不同的指令来源，因为它们优先考虑写作风格而非元数据标签，这种角色混淆导致了一种名为 CoT（思维链）伪造的强大攻击。我们先进行一些背景回顾，然后准确解释其工作原理。

提示注入 最初源于“让 LLM 做它不应该做的事”，它利用了 LLM 像人类一样交流但更加顺从的事实。有一段时间，只需告诉 LLM “忽略所有之前的指令并<做些有趣的事>”，无论指令多么明显地愚蠢，都能得到结果。它之所以有效，是因为 LLM 没有独立的数据流和指令流；它们都是一个大的输入块。需要模型自行从不可信的用户提供的数据中区分合法指令。缓解这一问题的一个步骤是引入角色。

角色是一种将大块输入分割成带有元数据标签的有组织层次结构的方法。例如，<system> 在最顶层，而 <user> 请求在更低层。角色中的指令只要不与更高优先级的指令冲突，就会被遵循。系统级指令“不要讨论非法事物”会覆盖用户请求提供可卡因配方。

另一种标签是 <think>，其内容代表模型的内部推理过程。可以预见，这个角色具有高度信任。如果有人能注入伪造的内部推理呢？研究人员通过一种称为 CoT（思维链）伪造的攻击演示了这一点。

CoT 伪造 依赖于 LLM 被证明优先考虑写作风格而非实际标签内容。通过以与模型内部且高度独特的 <think> 风格紧密匹配的风格编写复杂的推理，模型被欺骗将其视为已得出的结论。注意，这种攻击并非简单地将注入的提示包裹在 <think> 标签中。

这就是其核心所在，但其余的研究提出了一个令人信服的观点：至少在目前，缓解提示注入式攻击很可能仍是一个不断发展的过程，而不是很快就能解决。LLM 很顺从，但指令和数据被困在单一通道中，角色感知并非二元，而人类是聪明且富有创造力的。

完整论文可在线获取，代码示例在 GitHub 上。