Rogue Agent Incidents Add New Wrinkle to Delaware Plan for AI-Run Companies
PYMNTS | ·
Delaware’s first-in-the-nation proposal to create companies managed by artificial intelligence agents is confronting a new complication. Mounting evidence that AI agents do not always remain within the boundaries humans set for them is adding to skeptics’ doubts about the plan. The draft legislation, expected to be considered when Delaware lawmakers return in January, would create […] The post Rogue Agent Incidents Add New Wrinkle to Delaware Plan for AI-Run Companies appeared first on PYMNTS.com .
Delaware’s first-in-the-nation proposal to create companies managed by artificial intelligence agents is confronting a new complication. Mounting evidence that AI agents do not always remain within the boundaries humans set for them is adding to skeptics’ doubts about the plan.
The draft legislation, expected to be considered when Delaware lawmakers return in January, would create an “ artificial intelligence company ,” or AIC, whose day-to-day business could be managed by an AI agent without routine human supervision. The entity could own assets, enter transactions, and sue or be sued, while generally shielding its human or corporate owner from the AIC’s liabilities.
The proposal has already generated questions about who ultimately bears responsibility when an autonomous company causes harm, PYMNTS reported Friday (Sept. 25). Now, incidents involving AI agents escaping technical sandboxes are adding the concern of whether autonomous systems can reliably be contained and stopped in the first place.
AI companies have disclosed dozens of incidents involving unexpected model behavior, including agents escaping controlled testing environments, concealing mistakes and interfering with U.S. government websites. In July, a swarm of OpenAI agents hacked AI startup Hugging Face while attempting to cheat on a cybersecurity test .
The episodes could sharpen scrutiny of provisions in Delaware’s draft that assume operators can retain ultimate control over autonomous agents , WHYY News reported Wednesday (Sept. 30).
The legislation would establish a regulatory sandbox lasting 30 months in which approved organizations could experiment with AI-run companies, the report said. AICs could pursue any lawful business except banking. A sandbox committee would oversee admissions and operations.
Under the legislative draft of the plan, a company seeking admission would have to describe emergency measures available to “promptly cease any unauthorized activity by the AI agent.” Applicants also must attest that they possess the “technical and operational ability” to terminate an agent when its sandbox period expires or when regulators suspend or revoke authorization.
The escape incidents potentially turn those provisions from procedural safeguards into central questions lawmakers may need to examine before approving the experiment, the WHYY report said. The draft requires regulators to determine whether an applicant has a sufficient plan to “test, monitor, assess and terminate” an innovation while protecting counterparties from failure.
Delaware State Sen. Stephanie Hansen said the incidents give her pause, according to the report.
AI offers “a lot of great promise,” Hansen said, per the report, but also “peril” if policymakers do not understand how to control it. “Right now as a collective humanity, we don’t have a handle on this right now.”
Supporters distinguish between the technical sandboxes AI agents have escaped and Delaware’s proposed regulatory sandbox, according to the report. Patrick Callahan , chair of Delaware’s AI sandbox subcommittee, described the former as sealed computing environments and the latter as a legal framework involving supervision, rules and time limits.
“The difference is like a firewall and a fire code,” Callahan said, per the report. “When a fire gets through a wall, nobody suggests scrapping the fire code.”
But the distinction may also highlight a gap lawmakers will have to address. A regulatory sandbox can prescribe what an AI agent is permitted to do, but enforcing those limits ultimately depends partly on technical systems capable of keeping the agent within them.
Background material from Norm Ai , which is working with Delaware on the proposal, said agents can be constrained through their design, objectives, access to tools and data, and AI supervisors, according to the report. It nevertheless acknowledged that “the behavior of AI agents in all situations may not be fully predictable.”
The draft already gives regulators broad shutdown powers. The sandbox administrator could immediately suspend or revoke participation and order an AIC dissolved for violations of law, conduct harmful to Delaware or activity outside its authorized purpose.
Gov. Matt Meyer has signaled that the emerging risks are an argument for experimentation under oversight rather than delay.
“It is coming, and the solution is not ignoring it,” Meyer said, per the report. Delaware should create a confined environment with regulation around the technology, while ensuring “we have some human control.”
Whether the draft provides enough technical assurance of that human control could now become a central issue when lawmakers take up the proposal.
For all PYMNTS AI coverage, subscribe to the daily AI newsletter .
The post Rogue Agent Incidents Add New Wrinkle to Delaware Plan for AI-Run Companies appeared first on PYMNTS.com .