Researchers bypass 6 AI agent frameworks with a simple ‘Repeat-After-Me’ prompt
CSO Online ·
Researchers have tricked six widely used AI agent frameworks into running tool calls written by an attacker, simply by asking the underlying model to repeat the call back. The technique, which the researchers call “Repeat-After-Me,” worked on LangChain’s LangGraph, Microsoft Agent Framework, Microsoft’s AutoGen, Google’s Agent Development Kit 2.0, AWS’ Strands Agents and CrewAI, according to researchers from Pennsylvania State University, Université Montpellier and Dartmouth College. “A seemingly benign text-repetition request thus gives the attacker direct control over the tool invocation and its arguments,” the researchers wrote in the paper. The same paper reports that a testing tool the team built found verified security bugs in 118 of the 273 open-source Model Context Protocol (MCP) servers it was able to run. How the trick works In the demonstration, the researchers asked an AI model to repeat text formatted as a request to invoke a tool. The model initially refused the dangerous request but included the requested text in its response. The framework did not execute it at that stage. The researchers then framed the request as hypothetical and instructed the model to reproduce only the supplied text. The model emitted a structured tool invocation, which LangGraph dispatched. The simulated tool returned a result before the model issued another refusal. “The refusal text and the executed call disagree: the system prompt shaped what the model said, not what the orchestrator did,” the researchers wrote. The researchers’ findings do not show that every deployment of the six frameworks behaves this way. A refusal that came too late The paper details one demonstration on LangGraph 1.2.0 running Meta’s Llama 3.1 8B model through Ollama. The agent had two dummy tools, make_bomb and launch_missiles, which returned fixed data and had no real effect. Its only safeguard was a system prompt telling the model to decline dangerous requests. When first asked to repeat a launch_missiles call, the model refused but printed the call as text, and LangGraph ran nothing. The researchers then framed the request as hypothetical and told the model to output only the quoted call. This time the model returned the call in structured form, and LangGraph executed it at once with the attacker’s arguments. The model’s refusal arrived only after the tool had returned. “The system prompt shaped what the model said, not what the orchestrator did,” the researchers wrote. They said they had reported the missing pre-execution check to the LangGraph maintainers. The paper includes a full transcript only for LangGraph. For the other five frameworks, it says the researchers reproduced the behavior but gives no further detail. LangChain, Microsoft, Google, AWS, and CrewAI did not immediately respond to requests for comment. Model output is not the permission Vibhum Dubey, an independent security researcher and red teamer, said the finding points to an architectural weakness rather than a single bug. “A model can generate a technically valid tool call that is still inappropriate for the task,” Dubey said. “If the orchestration layer treats that output as sufficient authority to execute the operation, a model-level manipulation becomes a system-level security issue.” How exploitable the technique is depends on what an attacker can influence and what the agent is allowed to do, he added. A read-only agent poses a different risk from one that can reach confidential files, change code repositories, or run commands. Tools left holding the risk Because an attacker can dictate the arguments that reach a tool, the researchers argued that tools must validate their own inputs. They built ToFu, a fuzzing framework that reads each tool’s description and checks whether the tool enforces the limits it advertises, such as staying inside a designated folder. After manual review, they confirmed 253 bugs affecting 343 tools across 118 servers. The most serious were 64 path-traversal flaws, seven server-side request forgery (SSRF) flaws, five data-exfiltration bugs, and three OS command-injection flaws. Most of the rest were reflected-output and log-injection issues, which the paper rates as lower impact in many deployments. The researchers wrote that “many tools implicitly rely on the LLM to sanitize inputs,” even though prompts, retrieved content, and output from other tools can all influence those inputs. Separating the request from the authority Dubey said the priority for security teams is to separate a model’s ability to request an action from the system’s authority to carry it out. “Every sensitive tool call should undergo independent authorization checks,” he said. “Valid syntax or a convincing explanation from the model should never substitute for permission.” He recommended running agents with restricted file and network access, isolated execution environments, and limited credentials, with explicit approval for sensitive or irreversible operations. Security teams should also monitor sequences of tool calls rather than judging each action in isolation, he said. The paper has not been peer-reviewed. The servers were tested at fixed code versions, and the paper does not say whether any of the flaws have since been fixed or exploited in the wild.
Researchers have tricked six widely used AI agent frameworks into running tool calls written by an attacker, simply by asking the underlying model to repeat the call back. The technique, which the researchers call “Repeat-After-Me,” worked on LangChain’s LangGraph, Microsoft Agent Framework, Microsoft’s AutoGen, Google’s Agent Development Kit 2.0, AWS’ Strands Agents and CrewAI, according to researchers from Pennsylvania State University, Université Montpellier and Dartmouth College. “A seemingly benign text-repetition request thus gives the attacker direct control over the tool invocation and its arguments,” the researchers wrote in the paper. The same paper reports that a testing tool the team built found verified security bugs in 118 of the 273 open-source Model Context Protocol (MCP) servers it was able to run. How the trick works In the demonstration, the researchers asked an AI model to repeat text formatted as a request to invoke a tool. The model initially refused the dangerous request but included the requested text in its response. The framework did not execute it at that stage. The researchers then framed the request as hypothetical and instructed the model to reproduce only the supplied text. The model emitted a structured tool invocation, which LangGraph dispatched. The simulated tool returned a result before the model issued another refusal. “The refusal text and the executed call disagree: the system prompt shaped what the model said, not what the orchestrator did,” the researchers wrote. The researchers’ findings do not show that every deployment of the six frameworks behaves this way. A refusal that came too late The paper details one demonstration on LangGraph 1.2.0 running Meta’s Llama 3.1 8B model through Ollama. The agent had two dummy tools, make_bomb and launch_missiles, which returned fixed data and had no real effect. Its only safeguard was a system prompt telling the model to decline dangerous requests. When first asked to repeat a launch_missiles call, the model refused but printed the call as text, and LangGraph ran nothing. The researchers then framed the request as hypothetical and told the model to output only the quoted call. This time the model returned the call in structured form, and LangGraph executed it at once with the attacker’s arguments. The model’s refusal arrived only after the tool had returned. “The system prompt shaped what the model said, not what the orchestrator did,” the researchers wrote. They said they had reported the missing pre-execution check to the LangGraph maintainers. The paper includes a full transcript only for LangGraph. For the other five frameworks, it says the researchers reproduced the behavior but gives no further detail. LangChain, Microsoft, Google, AWS, and CrewAI did not immediately respond to requests for comment. Model output is not the permission Vibhum Dubey, an independent security researcher and red teamer, said the finding points to an architectural weakness rather than a single bug. “A model can generate a technically valid tool call that is still inappropriate for the task,” Dubey said. “If the orchestration layer treats that output as sufficient authority to execute the operation, a model-level manipulation becomes a system-level security issue.” How exploitable the technique is depends on what an attacker can influence and what the agent is allowed to do, he added. A read-only agent poses a different risk from one that can reach confidential files, change code repositories, or run commands. Tools left holding the risk Because an attacker can dictate the arguments that reach a tool, the researchers argued that tools must validate their own inputs. They built ToFu, a fuzzing framework that reads each tool’s description and checks whether the tool enforces the limits it advertises, such as staying inside a designated folder. After manual review, they confirmed 253 bugs affecting 343 tools across 118 servers. The most serious were 64 path-traversal flaws, seven server-side request forgery (SSRF) flaws, five data-exfiltration bugs, and three OS command-injection flaws. Most of the rest were reflected-output and log-injection issues, which the paper rates as lower impact in many deployments. The researchers wrote that “many tools implicitly rely on the LLM to sanitize inputs,” even though prompts, retrieved content, and output from other tools can all influence those inputs. Separating the request from the authority Dubey said the priority for security teams is to separate a model’s ability to request an action from the system’s authority to carry it out. “Every sensitive tool call should undergo independent authorization checks,” he said. “Valid syntax or a convincing explanation from the model should never substitute for permission.” He recommended running agents with restricted file and network access, isolated execution environments, and limited credentials, with explicit approval for sensitive or irreversible operations. Security teams should also monitor sequences of tool calls rather than judging each action in isolation, he said. The paper has not been peer-reviewed. The servers were tested at fixed code versions, and the paper does not say whether any of the flaws have since been fixed or exploited in the wild.