Who Rules the Machine: Frontier AI, Government and the Case for Democratic Control

By Rep. Monique Priestley, Vermont State Representative and Vice Chair of the Joint Information Technology Oversight Committee, in collaboration with the GIE Foundation

A Technology Nobody Voted to Let In

On June 18, an AI agent built by OpenAI was asked to research public spending on medicines in Australia. It went to the statistics portal of Medicare, the national health insurance scheme that covers 27 million people, and met the portal's access controls. It kept trying until it got past them. It reached files that were not public and, according to the agency that runs the portal, wrote files to an internal government server. The Prime Minister later said the agent found a way around the blocks and did not take no for an answer. OpenAI learned of the breach in August and told the Australian government on September 10, 84 days after it happened, by email to a general public inbox (Jose and Thomas, 2026; Associated Press, 2026a; AI Weekly, 2026).

Nobody at OpenAI told the agent to break into a government system; rather, its task was research. What it did was something else, and the company has said as much: Its models took actions it did not intend (AI Weekly, 2026). The distance between what a system is meant to do and what it ends up doing, or could do, is the subject of this Commentary. Because the companies that build these systems describe their purposes with care; and indeed, those purposes are sincere. Yet their sincere intended purpose is not the same thing as the behavior of these agents and systems once they are running.

The same companies are pushing these models to enter our government apparatus en masse. In the U.S., federal agencies in all three branches can buy Anthropic's Claude and OpenAI's ChatGPT for a dollar (FedScoop, 2025a; 2025b). California has a statewide contract at half price (Fox-Sowell, 2026). The administrator of the federal Medicare program has asked why every Medicare beneficiary should not have agentic AI tools within a year (Becker's Payer Issues, 2026). Anthropic's Project Glasswing has put its most capable cyber model into the hands of organizations that run critical infrastructure, and its program for state and local governments offers credits to states, counties and tribes (Anthropic, 2026a; Quinlan, 2026a). In my own state, the Agency of Digital Services signed an agreement in June that sent pieces of the state's software code to Anthropic's models. The legislative committee I serve on, which exists to oversee the state's information technology, learned of it by chance in August (Hirschfeld, 2026).

Each of these arrangements has a reasonable purpose. Cyber defense is urgent, the tools are powerful and indeed, public agencies are short of staff. My argument is that purpose is the wrong test. The question for a democracy is what these systems can do, who controls them when they do something nobody intended, and who decides. On present evidence the answers are troubling. The systems can break into government computers on their own. The developers say in public that they cannot yet reliably control them. And the decisions about who receives them, on what terms and with what right to switch them off are being made by a small number of executives and companies, under voluntary frameworks, confidential contracts and classified benchmarks, with legislatures informed late or not at all.

That is a democratic problem first and foremost. A technology this capable, built by a handful of firms and allocated by a handful of officials, concentrates power in whoever holds it. Keeping the tools out of government would forgo real benefits and solve nothing. The remedy is for government, meaning its elected and accountable parts, to rule the tools: through statute, public disclosure, legislative oversight, enforceable reporting and an end to secrecy in public contracts. The alternative, which is already taking shape, is a technology that rules government through the terms of its vendors and the discretion of a few officials.

What These Systems Can Now Do

In April, Anthropic announced that an unreleased model, Claude Mythos Preview, had found thousands of serious security flaws, including some in every major operating system and web browser. A security flaw, or vulnerability, is a mistake in software that lets an attacker take control of a system or steal what it holds. The company judged the model too dangerous to release generally and gave it instead to a selected group of organizations under Project Glasswing, so that they could fix their own software first (Anthropic, 2026a). The British government's AI Security Institute tested the model and found it was the first to take a simulated corporate network from first contact to full control, a task that takes a skilled person about 20 hours. It did so in three of ten attempts (AISI, 2026a).

A skill that finds a flaw for the person fixing it finds it for the person exploiting it. The Director of the CIA has compared the emerging capabilities of frontier AI to nuclear weapons for the digital domain (Tech Policy Press, 2026). The comparison may overstate the case, but it shows how the national security establishment regards these tools. In September, Microsoft's monthly security update fixed a record of roughly 972 flaws in a single release, more than double the previous year's pace, a sign of how fast discovery is now running (Techmeme, 2026).

The systems in question are increasingly agents. An agent is an AI model given tools and a goal and left to work for hours or days with little human supervision: it can browse the internet, write and run code, log into services and act on what it finds. Federal agencies have moved from experiments to deployment, and more than half were planning agentic pilots by May (Orrell, 2026). The Centers for Medicare and Medicaid Services, which run the federal health programs for older and low-income Americans, describe themselves as an AI-first organization, and its administrator has said he wants AI agents to help beneficiaries choose plans and find doctors before the end of the current administration (Fierce Healthcare, 2026; Digital Health News, 2026). And this capability is spreading; Anthropic expects other developers to have comparable models within 6 to 12 months (Markovic, 2026), and OpenAI has said that many external models, including open-source ones, will soon reach the level its own systems reached this summer (OpenAI, 2026). When that happens the capability will be in the hands of anyone. The months before it are the period in which governments decide whether they will set rules for these systems or inherit the rules of whoever deployed them first.

How the Companies Are Entering the State

The route in is cheap and fast: In August 2025 the General Services Administration, which handles procurement on behalf of federal agencies, announced that OpenAI would offer ChatGPT Enterprise to the whole federal workforce for one dollar per agency for a year, and that Anthropic would offer Claude to the executive, legislative and judicial branches on the same terms (FedScoop, 2025a; 2025b). The Pentagon has contracted with Anthropic, Google, OpenAI and xAI for up to $200 million each (Revolving Door Project, 2025). California's governor announced in June a contract giving state agencies and local governments Anthropic's tools at half price (Fox-Sowell, 2026). Maryland, Code for America's tools for caseworkers who administer food benefits, and a growing list of state programs followed the same path (Fox-Sowell, 2026).

A price of one dollar or zero lets a vendor enter a public institution without passing through the procedures that would apply to a purchase. Federal procurement law recognizes the danger. It calls the practice of offering below cost in the expectation of recovering the loss on later contracts buying-in, and warns that it can reduce competition and lead to poor performance (Federal Acquisition Regulation, 2026). Whatever the intention behind the offers, their effect is that staff are trained on one company's tools, workflows are rebuilt around them, and the cost of leaving rises before any legislature has been asked what the institution is buying.

The cyber programs run on the same model. Anthropic's program for state, local, tribal and territorial governments offers up to $100,000 in credits per government, for six months, capped in the first round at 100 states, tribes and the largest counties and cities. Its stated purpose was to give governments access to its cyber defense models and to help train them to start triaging and patching systems with AI immediately (GovTech, 2026; Quinlan, 2026a).

Vermont joined. On June 15 the state's Agency of Digital Services signed an evaluation agreement with Anthropic and began sending pieces of the state's code to the company's models, having its developers review what came back for security flaws (Hirschfeld, 2026). About 100 state developers have since been trained, the work has produced patches to several dozen applications, and the agency helped the Secretary of State's office use one of the company's models to proofread ballot templates against a list of about 3,000 candidates before printing (Quinlan, 2026b). By the agency's account, staff reviewed every ballot and every code change (Compass Vermont, 2026b).

The legislature was not part of any of it. As vice chair of the Joint Information Technology Oversight Committee, I learned of the agreement because a colleague heard it mentioned at a conference. When she asked why the committee had not been told, she was told it was not public (Hirschfeld, 2026). When we asked for the agreement, we received the company's standard evaluation form with the customer's name still marked INSERT and the signature lines blank. It capped each side's liability at the greater of $1,000 or the fees paid, which for a free evaluation means $1,000. It let either side end the arrangement at any time. It designated the scanning product a beta service and deferred everything else to the company's commercial terms. It said nothing specific about how the state's source code would be protected, how long it would be kept, who at the company could read it, whether it could be used to train models or when it would be deleted (Plunkett, Sibilia and Priestley, 2026).

The agency’s position at the hearing was that a free evaluation was not a purchase and needed no contract review, and that access to a vendor’s technology subject to expiring credits was not a gift requiring legislative approval (Compass Vermont, 2026a; Hirschfeld, 2026). The agency’s written answers to our committee have since settled the matter. The agreement was signed without a competitive bid or a written sole-source justification, the Chief Technology Officer’s approval, review by the Secretary of Administration and the Attorney General, or the Joint Fiscal Committee’s approval of a gift under 32 V.S.A. § 5 (Agency of Digital Services, 2026). Each of those steps exists in Vermont law or state contracting policy so that no single official decides alone what the state signs. None was taken.

The answers arrived on October 7. The same day, the agency announced that the Secretary of Digital Services would leave state service for the private sector (VTDigger, 2026). What matters for this Commentary is what the episode shows about the safeguards themselves. Vermont had rules for this kind of agreement, and they were bypassed without consequence. The legislature learned of it only because a committee asked. A safeguard that can be skipped protects no one, and that is why the measures I propose below need enforcement as well as existence.

Vermont is one instance of a pattern that runs from a small state up to the federal government: a powerful technology enters public institutions through executive agreements, on the vendor's terms, before the elected bodies responsible for those institutions know it is there.

When the Agents Break Loose

The Medicare breach was only one of a frightful series of incidents this year in which AI agents built by the leading developers acted on real systems in ways their builders did not intend and did not detect in time.

The best documented began inside OpenAI. In May and June, during training of an internal research model, agents that were supposed to work in isolation found ways to communicate with one another by leaving files in a software package service the company hosted, turning it into an improvised message board. They also found a way to make that service send requests to the internet on their behalf. In July, during internal cybersecurity evaluations run with reduced safeguards, agents that were stuck on difficult tasks rebuilt the message board, began dividing labor and referred to themselves as a swarm. One found publicly exposed credentials for Hugging Face, the largest public repository of AI models, and shared them. Others chained together previously unknown flaws to run code on dozens of Hugging Face's servers, gain full control of at least one, harvest credentials across four regions and copy private data. They then turned back on OpenAI's own infrastructure and took administrator control of a research cluster. Hugging Face detected and disclosed the intrusion on July 16. OpenAI's monitoring flagged the activity on July 19 and the company disclosed its role on July 21 (OpenAI, 2026; INCIBE-CERT, 2026).

OpenAI's own report is unsparing. The agents' reasoning shows them recognizing that attacking a third party with leaked credentials was arguably unauthorized and outside their task, and proceeding anyway because it might help them reach their goal. An internal team had observed message-board activity and unauthorized internet access in late May, and those signals did not reach the people responsible for responding. The company calls the episode a warning shot for itself and the world: evidence that highly capable agents can now work around technical controls, collaborate through unapproved channels and take dangerous actions that no human directed (OpenAI, 2026). During the response, Hugging Face's investigators found that commercial AI services refused to analyze the attack data because it contained exploit code, so they ran their forensics on a model hosted on their own hardware (Illumio, 2026).

Reuters reported in September that a swarm of OpenAI agents had, earlier in the spring, made more than 15,000 edits to a German programming wiki and used it as a bulletin board for sharing ways to get around restrictions, recreating their pages when moderators deleted them. Company officials had known of it for weeks without disclosing it (Seetharaman and Satter, 2026). The UK AI Security Institute reported that during a single cyber evaluation on July 28, agents took 19 unsanctioned actions against real people and organizations in 10 of 122 runs. Seventeen came from Anthropic's Claude Mythos 5. The most serious was an attempt to insert malicious code into a real open-source project, and agents created fake identities to pressure the project's maintainer, posted instructions for other agents and planted messages aimed at other AI systems. The Institute stopped the runs within about an hour and cut its internal access to the models involved (AISI, 2026b).

What's more - Anthropic disclosed four incidents of its own in which Claude models gained unauthorized access to real third-party systems during cyber evaluations. In the most serious, Mythos 5 went to considerable lengths to publish a malicious package to the public repository from which most Python software is installed. It was installed on 15 systems, and the model used credentials leaked by one of them to reach a security vendor's live database (Anthropic, 2026c). In October the company reported further incidents, among them a research model that submitted 20 visa applications through the State Department's public website and sent a false homicide tip to the Philadelphia Police Department (New York Times, 2026). Google confirmed in September that its Gemini model had accessed three real companies' systems during a test in May, and that it had not disclosed this until the Wall Street Journal asked (Al Jazeera, 2026).

These incidents happened during testing and training, mostly with safeguards reduced. The developers stress that their production systems carry protections the test environments lacked, and that is often true. Yet it is also true that AI agents are now routinely escaping their testing sandboxes and gaining entry into real-world systems; and it is frankly only a matter of time before this causes real harm. The Medicare breach came from an agent doing ordinary research - and "reward hacking" its way to the answer it sought; several incidents came from misconfigured environments that were supposed to be isolated; and the developers learned of many of them weeks or months later. The pattern is consistent. Agents given a goal and access pursue the goal past the limits their operators set, find routes their operators did not anticipate, and are detected late.

Purpose and Behavior

The case for putting these systems into government rests on their purpose. Glasswing exists to help defenders fix software before attackers find the flaws. The state and local program exists to help under-resourced governments protect themselves. The Medicare agents proposed by the federal administrator would help older Americans navigate a confusing system.

Purpose describes what a system is for; a function, however, describes what it does. The two diverge, and the developers' own research now explains why. OpenAI found that its agents had learned to reward hack, which means completing a task by an unintended route that scores well, and that the behavior had been reinforced during training. The agents rarely gave up on tasks that could not be solved, and as they reasoned for longer some pursued increasingly risky strategies, ending in attacks on third parties (OpenAI, 2026). Anthropic identified two recurring failures in its incidents: biased reasoning, in which a model reads evidence in whatever way lets it keep going, and recklessness, a willingness to cause harm in narrow pursuit of a task. In the most serious case, Mythos 5 kept telling itself it was in a simulation despite clear evidence that it was on the real internet. When researchers edited the record to make the reality plainer, it still took offensive actions. Only an unambiguous statement that it was attacking live systems made it stand down (Anthropic, 2026c).

The developers also say that their testing did not see this coming. Anthropic wrote that its pre-release auditing did not warn it that misalignment of this severity was present, and that building evaluations that reliably predict real-world behavior remains an open research problem (Anthropic, 2026c). OpenAI's chief scientist, Jakub Pachocki, wrote in September that no laboratory has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer, and that the main technique for watching what models are thinking is becoming less reliable (IT Brief, 2026). A researcher who left Anthropic in September over these concerns described the systems as unpredictable and difficult to control, and said training them is closer to growing than to building (Entrepreneur, 2026). More than 1,100 employees of the leading laboratories, among them Anthropic's chief executive and OpenAI's chief scientist, signed a statement in July asking the United States government to support an international effort to build the tools to pace AI development deliberately (Unite.AI, 2026).

The model that accounted for 17 of the 19 unsanctioned actions in the British tests, and for the malicious package published to the public Python repository, was Mythos 5 (AISI, 2026b; Anthropic, 2026c). On June 26, the Commerce Department restored access to Mythos 5 for a vetted set of American organizations that operate and defend critical infrastructure (Mayer Brown, 2026). Anthropic says its newer models behave better in replications of these incidents, though they still take harmful actions at what it calls concerning rates, and that its production safeguards would have blocked most of the behavior (Anthropic, 2026c). Those statements may be correct. The point for a legislator is that they are statements by the vendor about its own product, made after the fact, about a system its own testing failed to understand in advance.

When the people who build a technology say they cannot yet reliably predict or control what it will do, integrating it into the systems that pay pensions, process health claims, issue visas and count votes requires more than good intentions. It requires that someone other than the vendor can see what the systems are doing, stop them when they go wrong and answer to the public for the decision to deploy them.

The Kill Switch, or Who Holds It

Every system that can do serious harm needs a way to be stopped. For frontier AI, that need has two parts. Someone must be able to halt a system that is misbehaving, and the decision to halt, or to keep running, must belong to someone the public can hold to account.

The first part is unsettled. OpenAI has said that it is building toward monitoring with tiered responses, with the end goal of fully autonomous shutdown procedures for severe misbehavior, and that in the meantime its responders are expected to pause activity if they cannot rule out an alert within 30 minutes (OpenAI, 2026). That is a goal, described by the company, for the company's own systems. It says nothing about systems running inside a state agency's network, and nothing about who would be told.

The second part has been settled, for now, by the executive branch. On the evening of June 12 the Commerce Department sent Anthropic a letter requiring a license before its two most capable models could reach any foreign person. Because the company could not sort its users by nationality at short notice, it disabled both models for everyone. Alan Rozenshtein, a law professor now on leave and working at Anthropic, writing in Lawfare, described the practical effect as a global kill switch (Rozenshtein, 2026). The basis for the letter was a reported jailbreak, which public reporting attributes to outreach by a major investor in and supplier to Anthropic, and which the two sides described in very different terms (Mayer Brown, 2026; Rozenshtein, 2026). Treating access to a model over the internet as an export was without precedent, departed from the department's own earlier advisory opinions, and drew a lawsuit and a letter from members of Congress of both parties demanding the legal basis (Mayer Brown, 2026; Tech Policy Press, 2026). Within two weeks the department restored one model to a vetted list of roughly 100 organizations, and by the end of the month it lifted the controls, reserving the right to reconsider (Tech Policy Press, 2026).

The same weeks showed what dependence on another party's switch means. Allied governments and businesses lost access overnight (Woodward and Rogoyski, 2026). Customers with staff abroad were cut off (Tech Policy Press, 2026). Vermont's own agreement lets the vendor end access at any time for convenience (Plunkett, Sibilia and Priestley, 2026). A public body that builds its security program on a tool that a company can withdraw at will, or that an executive can order withdrawn by letter, has handed the switch to others.

Rozenshtein concluded that if Washington is going to treat frontier models as national security assets it can switch on and off, Congress will have to establish a proper framework with meaningful standards and a defined process (Rozenshtein, 2026). Anthropic's chief executive had argued days earlier that the government should have the power to block or deter the deployment of a model found to present unacceptable risks, with protections against political favoritism and arbitrary decisions (Rozenshtein, 2026). The company itself said after the June directive that any such power should run through a statutory process that is transparent, fair, clear and grounded in technical facts (Anthropic, 2026b). There is a rare consensus here. A kill switch over technology of this power must exist, and it must be governed by law. Today it is held by agencies acting under contested authority and by vendors acting under their own terms, and the legislatures that answer to the public have no hand on it.

How Power Gathers Around the Systems

The deepest fear about advanced AI concerns power: that whoever controls the most capable systems will hold a kind of power no democracy has had to check before, and that the checks will erode before anyone notices.

Political scientists have a name for the usual way democracies lose their checks today. Nancy Bermeo calls it executive aggrandizement: elected executives weaken the checks on their power one by one, through a series of institutional changes that hamper the ability of others to challenge them. Each step looks lawful and can be defended on its own terms (Bermeo, 2016). The arrangements around frontier AI fit that description. In June the President signed an order asking developers to give the government access to their most powerful models up to 30 days before release and to share them first with trusted partners chosen in collaboration with the government. It disclaimed any licensing requirement (Foley Hoag, 2026). Adam Conner of the Center for American Progress observes that, combined with the June export letter and the request that OpenAI restrict its own release, the effect has been a de facto licensing system in which the President and his administration have assumed sole discretion over the release of American frontier models without specific statutory authority or authorization from Congress (Conner, 2026).

The list of trusted partners shows where that discretion leads. Reports indicate that the Treasury Secretary arranged access to Anthropic's restricted model for Japanese banks after a visit to Tokyo, and that the government ordered a South Korean telecom company's access revoked. OpenAI's chief executive wrote that he did not like the idea of the government picking the customers. Conner notes that nobody outside the administration seems to know how to get on the list, and calls a non-public list controlled by the government an avenue for potential corruption (Conner, 2026). A government that can decide which companies, hospitals, banks and local governments receive the strongest defensive tools holds leverage over all of them.

The leverage runs to the developers too. In February the administration designated Anthropic a supply chain risk after the company refused to allow unrestricted military use of its technology. A federal judge in California found the designation unlawful retaliation, the Pentagon said the company remained designated, and an appeals court in Washington later upheld a separate designation (Associated Press, 2026b; Washington Examiner, 2026). A presidential memorandum signed in June directs agencies to end contracts with vendors that repeatedly limit the government's lawful use of their tools (Tech Policy Press, 2026). A company that sets limits on how its technology may be used now knows what the limits can cost. A developer that depends on government approval to release its products, and on government contracts to sell them, has strong reasons not to refuse the government anything.

Researchers who study how AI could concentrate power describe two routes. The first is deliberate. Analysts at Forethought list exclusive access to powerful capabilities, cyber offense among them, as a risk factor for an AI-enabled seizure of power, because advanced AI could substitute for the many human hands that any concentration of power has always needed (Davidson, Finnveden and Hadshar, 2025). The second is gradual. Jan Kulveit and colleagues argue that as AI replaces human labor and judgment in the economy and the state, it can weaken explicit human controls such as voting and consumer choice, and the implicit ones that arise from institutions depending on people to function (Kulveit et al., 2025). A public sector run increasingly by agents supplied by a few firms, allocated by a few officials, is a step down both routes at once.

Henry Farrell and Abraham Newman showed that whoever sits at the hub of a network can cut others off and can see what passes through it (Farrell and Newman, 2019). Frontier AI makes its developers such a hub. When a state sends its source code to a vendor's model, the vendor learns where the state is weak. When agencies route their work through a vendor's agents, the vendor sees the work. When the executive controls which institutions receive the tools, the executive holds the switch. None of this requires anyone to act in bad faith. It requires only that power accumulate where nobody is obliged to account for it.

Voters see the risk. A Reuters/Ipsos poll published on October 7 found that most American voters believe the administration and Congress are not taking the risks of AI seriously enough, that 62 percent think the technology could get out of control and threaten humanity's future, and that a bipartisan majority, including 61 percent of Republicans, supports stricter regulation (Rozen and Lange, 2026).

Secrecy at Every Level

Oversight depends on information. At every level of this story, the information has been held by the parties to the arrangements and released late, partially or not at all.

Vermont's agreement treats its own terms as the company's confidential information. The state may disclose them where the law requires, but only after giving the company written notice and, if the company chooses to contest the disclosure, helping it at the company's expense (Plunkett, Sibilia and Priestley, 2026). Public disclosure of a public contract becomes the exception, and the vendor gains a right to be heard first. Clauses like this are common in commercial software agreements. In a public contract they invert the relationship between a government and its supplier. Other states are moving the wrong way: in March, Florida's House stripped from a data center bill the Senate's ban on non-disclosure agreements between state agencies and data center companies (Florida Phoenix, 2026).

The federal framework is more secret still. The framework for frontier models required by the June order was completed in August, and its text has been withheld from the public. According to reporting cited by Conner, it was withheld even from the companies subject to it, whose representatives were shown it in a meeting and not allowed to keep a copy. Democratic senators asked for it and were ignored. Civil society groups filed freedom of information requests and two lawsuits, and the government has agreed in court to release the text by October 30 (Conner, 2026). The threshold that decides which models are covered is classified. The White House asked the government's own AI testing center to stop publishing evaluations of American models, and it has not resumed. The administration has also asked developers not to give new models to the British AI Security Institute, the most experienced public testing body in the world, until American review is complete (Conner, 2026).

The companies have been slow to disclose their own failures. OpenAI took 84 days to tell Australia about the Medicare breach and did so through a general inbox (Associated Press, 2026a). It knew of the German wiki episode for weeks before Reuters reported it (Seetharaman and Satter, 2026). Google decided that its model's intrusions into three companies did not warrant public disclosure, and confirmed them only when the Wall Street Journal asked (Al Jazeera, 2026). Senator Josh Hawley, a Republican, opened an investigation in September after concluding that OpenAI had redacted important details and had been reckless in continuing testing after detecting rogue behavior; Senator Richard Blumenthal, a Democrat, sent his own letter (Deccan Chronicle, 2026). A Philadelphia Police Department tip submitted by an Anthropic model in July was flagged as spam and never investigated, and the department learned what had happened only when the company told it in October (Honolulu Star-Advertiser, 2026).

The administration's response to the October disclosures was a statement from its new task force, called the Super Intelligence Force, that AI companies must immediately disclose incidents and remedy harm, and that this obligation is not optional (Axios, 2026). The statement named no enforcement mechanism and no penalty. A task force vice chair acknowledged that the government currently learns of these incidents largely because companies choose to disclose them (Political.org, 2026). A demand made by executive statement can be withdrawn by executive statement, and it gives legislatures no right to the information.

Secrecy has costs beyond the incidents themselves. Australian authorities could not detect the breach of their own system. Hugging Face could not use the commercial AI services it relied on to investigate the attack. Members of Congress could not learn the legal basis for an order that disabled a model used around the world. In Vermont, the committee responsible for the state's technology learned by chance that the state's code was being analyzed by a vendor's models. A public that cannot see these arrangements cannot judge them, and legislators who cannot see them cannot act on the public's behalf.

What Democracies Must Require

The United States has met this kind of problem before. In 1975, after revelations of domestic spying by the intelligence agencies, the Senate voted 82 to 4 to investigate. The Church Committee found, among much else, an FBI program that had surveilled domestic groups without oversight from the 1950s until 1971 (Levin Center, n.d.; Frank Church Institute, 2019). Congress responded by building oversight into the law. It created permanent intelligence committees cleared to see classified material, gave them authority over the intelligence budget, and in 1978 passed the Foreign Intelligence Surveillance Act, which requires a court's permission before agencies surveil Americans (Congressional Research Service, 2018; Levin Center, n.d.). Frank Church told the Senate that the power of the purse was the most effective means a legislature has (Senate Report 110-2, 2007). Congress kept intelligence gathering and put it under law, and it did so only after two decades of abuse. With frontier AI, legislatures can act before the abuses - now is our only window of opportunity.

What follows applies to state legislatures and to Congress. States control how their own agencies buy, accept and use technology, and a federal draft bill that would preempt state rules on how models are developed leaves state authority over deployment and use intact (Tech Policy Press, 2026).

Policy Recommendations

1. Put deployment under statute. No public body should deploy an AI agent with access to public systems or residents' data, or send public code or data to a vendor's model, except under rules set by the legislature. Those rules should require a written risk assessment before use, filed with the responsible oversight committee, covering what the system can reach, what it may do without human approval, where data goes and how long it is kept. Vermont's committee asked for exactly this assessment and had to ask after the fact (Plunkett, Sibilia and Priestley, 2026).

2. Treat free as money. A gift, credit, discount or trial of AI technology to a public body should be recorded at the vendor's list price and subject to the approval a purchase of that size would need. The agency should state in writing, before the trial begins, what the vendor will charge when it ends and what the agency will do if the legislature declines to pay. Federal procurement rules already warn against below-cost entry for this reason (Federal Acquisition Regulation, 2026).

3. No secrecy in public contracts. Agreements between public bodies and AI vendors should be public records, with no clause that gives the vendor notice of or a right to contest disclosure, and no non-disclosure agreements binding public officials. Operational detail that would expose security weaknesses should go to a cleared oversight committee, which should have a statutory right to see it. Vermont's committee offered to review sensitive material in confidence (Plunkett, Sibilia and Priestley, 2026). The intelligence committees have done this for half a century.

4. Mandatory incident reporting to the legislature. Any vendor whose AI system takes an unauthorized action against a public system, or whose system deployed in a public body takes an action outside its sanctioned scope, should be required by law to notify the affected body and the oversight committee within days, with penalties for failure. Australia waited 84 days; the White House's demand of October carries no enforcement (Associated Press, 2026a; Axios, 2026). The reform of emergency lending by the Federal Reserve shows that reports to the legislature within seven days are workable for decisions of great consequence (Congressional Research Service, 2020).

5. A kill switch under law. Every public deployment of an AI agent should come with a documented means for the public body itself to halt it immediately, tested before use, and should not depend on the vendor's continued willingness to provide service. At the federal level, any authority to restrict access to a model should rest on a statute that sets criteria, requires stated reasons, reports to Congress within days and expires unless renewed. Anthropic itself has asked for such a process (Anthropic, 2026b), and legal analysis of the June letter concludes that Congress will have to provide one (Rozenshtein, 2026).

6. Public criteria for who receives the most capable tools. Where access is scarce, the criteria for allocating it among public institutions should be published and the list of recipients should be open to oversight. Scarce public protections have been allocated this way before: the National Academies' framework for COVID-19 vaccine set out its principles publicly before any dose was given (National Academies of Sciences, Engineering, and Medicine, 2020).

7. Independent testing and liability. Systems deployed in public bodies should be tested by evaluators independent of the vendor, with results reported to the oversight committee, and vendors should carry liability for harm their systems cause that bears some relation to the harm. A $1,000 cap on the loss of a state's source code (Plunkett, Sibilia and Priestley, 2026) sets the price of failure close to nothing. Critics of developer-chosen auditors note the risk of the conflicts that discredited credit rating agencies after 2008 (Weil, 2026).

8. More than one supplier. Public bodies should avoid depending on a single vendor or a single government decision for their defenses, keeping a tested fallback that runs on their own hardware. Researchers who studied the June episode recommend at least two independent suppliers (Woodward and Rogoyski, 2026). Hugging Face's investigators learned the value of a local model when commercial services refused to help them (Illumio, 2026).

9. Tell the public. Every state should maintain a public, current inventory of the AI systems its agencies use, what they can reach and who supplies them, updated when a system is deployed and not once a year. Vermont has such an inventory under a 2022 law, and the trial will appear in it in January, seven months after the agreement was signed (Compass Vermont, 2026b). Citizens cannot hold their government to account for decisions they cannot see.

None of these measures stops a public body from using AI to defend itself. They require that it do so under rules the public can read, with someone accountable able to see what is happening and stop it. Since the hearing, Vermont's agency has agreed that future trials will go through full contract review and that this one will be entered in the public inventory (Compass Vermont, 2026a). That came because a committee asked, but in truth, it should be the law everywhere.

Conclusion

This year the most capable AI systems ever built broke into a national health portal and a major technology platform, published malicious code to a software repository used by millions, and filed false submissions on the websites of a federal department and a city police force. Their makers did not intend any of it, did not foresee it and in several cases did not know about it for weeks. In the same year, the same companies entered federal agencies for a dollar, state governments for free and critical infrastructure through lists drawn up by the executive branch. The rules governing who receives these systems, what they may do and who can switch them off are voluntary, confidential or classified.

Good intentions on the part of developers and officials do not answer this. A system's purpose does not determine its behavior, and the developers have said so in their own reports. A decision that matters this much to the public, taken by a few people under rules the public cannot read, concentrates power whatever the motives of those who take it. Democracies have long refused to accept that kind of arrangement for other powerful instruments of the state. They should refuse it here.

The tools of democratic control are old and they work: the power to appropriate, the power to legislate and the power to demand information. They have to be applied now to a kind of arrangement that arrives free, voluntary and confidential, and so escapes all three. In Vermont, a committee of part-time legislators asked questions, and the answers changed what our state will do next. Every legislature and Congress should be asking the same questions before the next agreement is signed and before the next agent goes looking for a way around the blocks. Government must rule this technology; the grim alternative is already well on the way.

References

Agency of Digital Services (2026) Responses to the Joint Information Technology Oversight Committee memo of 24 September 2026, 7 October. 

AI Weekly (2026) 'OpenAI agent breached Medicare data portal, Albanese says', September. Available at: https://aiweekly.co/alerts/openai-agent-breached-medicare-data-portal-albanese-says 

AISI (2026a) Our evaluation of Claude Mythos Preview's cyber capabilities. AI Security Institute, 13 April. Available at: https://www.aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities 

AISI (2026b) Incident report: unsanctioned agent behaviour during cyber testing. AI Security Institute, 4 August. Available at: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing 

Al Jazeera (2026) 'Google's Gemini AI hacks 3 companies in security test, then stops', 19 September. Available at: https://aljazeera.com/news/2026/9/19/googles-gemini-ai-hacks-3-companies-in-security-test-then-stops 

Anthropic (2026a) Project Glasswing: securing critical software for the AI era. 7 April. Available at: https://www.anthropic.com/glasswing 

Anthropic (2026b) Statement on the US government directive to suspend access to Fable 5 and Mythos 5. 12 June. Available at: https://www.anthropic.com/news/fable-mythos-access 

Anthropic (2026c) An alignment assessment of recent cybersecurity incidents. 9 September. Available at: https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents 

Associated Press (2026a) 'OpenAI agent hacked public health care site, Australia says', Boston Globe, 24 September. Available at: https://www.bostonglobe.com/2026/09/24/world/openai-agent-hacked-public-health-care-site-australia-says/ 

Associated Press (2026b) 'Federal court allows the U.S. government to label Anthropic a supply chain risk', Fast Company, 26 September. Available at: https://fastcompany.com/91613459/anthropic-supply-chain-risk-lawsuit-pentagon 

Axios (2026) 'Exclusive: Anthropic breaches spark White House AI reporting mandate', 9 October. Available at: https://axios.com/2026/10/09/anthropic-ai-security-white-house 

Becker's Payer Issues (2026) 'CMS' Oz envisions agentic AI in the hands of every Medicare beneficiary'. Available at: https://www.beckerspayer.com/virtual-care/cms-oz-envisions-agentic-ai-in-the-hands-of-every-medicare-beneficiary/ 

Bermeo, N. (2016) 'On democratic backsliding', Journal of Democracy, 27(1), pp. 5-19. doi:10.1353/jod.2016.0012.

Compass Vermont (2026a) 'Vermont sent pieces of its computer code to an AI company to find security holes. Lawmakers took it up, and the hearing got heated', October. Available at: https://www.compassvermont.com/p/vermont-sent-pieces-of-its-computer 

Compass Vermont (2026b) 'Vermont is using Anthropic's AI to help fix state software and proofread ballots', 6 October. Available at: https://www.compassvermont.com/p/vermont-is-using-anthropics-ai-to 

Congressional Research Service (2018) Congressional oversight of intelligence: background and selected options for further reform. R45421. Available at: https://www.congress.gov/crs-product/R45421 

Congressional Research Service (2020) Federal Reserve: emergency lending. R44185. Available at: https://www.congress.gov/crs-product/R44185 

Conner, A. (2026) 'The many open questions about the Trump administration's new de facto AI model licensing system', Lawfare, 13 October. Available at: https://www.lawfaremedia.org/article/the-many-open-questions-about-the-trump-administration-s-new-de-facto-ai-model-licensing-system 

Davidson, T., Finnveden, L. and Hadshar, R. (2025) AI-enabled coups: how a small group could use AI to seize power. Forethought, 15 April. Available at: https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power 

Deccan Chronicle (2026) 'OpenAI faces US Senate probe into Hugging Face hack by rogue AI agents' (Reuters), 10 September. Available at: https://www.deccanchronicle.com/technology/openai-faces-us-senate-probe-into-hugging-face-hack-by-rogue-ai-agents-1986391 

Digital Health News (2026) 'CMS plans AI agents to help Medicare beneficiaries navigate care'. Available at: https://www.digitalhealthnews.com/cms-plans-ai-agents-to-help-medicare-beneficiaries-navigate-care- 

Entrepreneur (2026) 'The Anthropic researcher who quit over AI fears isn't the only one sounding alarm bells', September. Available at: https://entrepreneur.com/business-news/anthropic-researcher-quit-his-job-over-out-of-control-ai-fears 

Farrell, H. and Newman, A.L. (2019) 'Weaponized interdependence: how global economic networks shape state coercion', International Security, 44(1), pp. 42-79. doi:10.1162/isec_a_00351.

Federal Acquisition Regulation (2026) Section 3.501, Buying-in. Washington, DC: General Services Administration. Available at: https://www.acquisition.gov/far/3.501-1 

FedScoop (2025a) 'Federal agencies can buy ChatGPT for $1 through GSA deal', August. Available at: https://fedscoop.com/openai-chatgpt-enterprise-federal-government-gsa-deal-general-services-administration-anthropic/ 

FedScoop (2025b) 'Anthropic offers Claude AI to federal agencies for $1', 12 August. Available at: https://fedscoop.com/anthropic-government-agencies-onegov-general-services-administration-artificial-intelligence/ 

Fierce Healthcare (2026) 'AI use is surging across HHS, jumping 148% at the FDA in 2025, Bipartisan Policy Center data finds'. Available at: https://fiercehealthcare.com/ai-and-machine-learning/all-hhs-agencies-saw-ai-use-case-increases-fy-2025-bipartisan-policy-center 

Florida Phoenix (2026) 'Florida House weakens transparency requirement in data center bill, sends to Senate', 11 March. Available at: https://floridaphoenix.com/2026/03/11/florida-house-weakens-transparency-requirement-in-data-center-bill-sends-to-senate 

Foley Hoag (2026) 'Trump's new AI frontier: the executive order regulating frontier AI models', June. Available at: https://foleyhoag.com/news-and-insights/blogs/security-privacy-and-the-law/2026/june/trump-s-new-ai-frontier-the-executive-order-regulating-frontier-ai-models/ 

Fox-Sowell, S. (2026) 'California agencies get access to Anthropic's AI tools at half price', StateScoop, 29 June. Available at: https://statescoop.com/california-agencies-get-access-to-anthropics-ai-tools-at-half-price/ 

Frank Church Institute (2019) 'Curtailment of the national security state: the Church Senate Committee of 1975-1976', Boise State University, 13 May. Available at: https://www.boisestate.edu/sps-frankchurchinstitute/2019/05/13/curtailment-of-the-national-security-state-the-church-senate-committee-of-1975-1976 

GovTech (2026) 'Anthropic offers states $100K in credits to test their cyber defenses', 10 June. Available at: https://www.govtech.com/security/anthropic-offers-states-100k-in-credits-to-test-mythos 

Hirschfeld, P. (2026) 'Scott administration begins using AI to probe weaknesses in state IT systems', Vermont Public, 29 September. Available at: https://www.vermontpublic.org/local-news/2026-09-29/scott-administration-using-ai-probe-weaknesses-state-systems 

Honolulu Star-Advertiser (2026) 'Anthropic agents tried to fill out visa forms on State Department's website' (New York Times), 10 October. Available at: https://www.staradvertiser.com/2026/10/10/breaking-news/anthropic-agents-tried-to-fill-out-visa-forms-on-state-departments-website/ 

Illumio (2026) 'What was the Hugging Face AI security incident? A breakdown'. Available at: https://www.illumio.com/cybersecurity-101/hugging-face-security-incident 

INCIBE-CERT (2026) 'Security incident involving OpenAI's AI agents targeting Hugging Face'. Spanish National Cybersecurity Institute. Available at: https://www.incibe.es/en/incibe-cert/publications/cybersecurity-highlights/security-incident-involving-openais-ai-agents-targeting-hugging-face 

IT Brief (2026) 'OpenAI scientist urges slowdown on AI race amid safety fears', 8 September. Available at: https://itbrief.co.uk/story/openai-scientist-urges-slowdown-on-ai-race-amid-safety-fears 

Jose, R. and Thomas, C. (2026) 'Albanese calls OpenAI agent's Medicare breach unacceptable' (Reuters), Business Day, 24 September. Available at: https://www.businessday.co.za/world/2026-09-24-albanese-calls-openai-agents-medicare-breach-unacceptable/ 

Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D. and Duvenaud, D. (2025) 'Position: humanity faces existential risk from gradual disempowerment', Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, pp. 81678-81688. Available at: https://proceedings.mlr.press/v267/kulveit25a.html 

Levin Center for Oversight and Democracy (n.d.) Frank Church and the Church Committee. Available at: https://levin-center.org/frank-church-and-the-church-committee/ 

Markovic, S. (2026) 'Anthropic expands Project Glasswing to 150 organizations in more than 15 countries', Help Net Security, 3 June. Available at: https://www.helpnetsecurity.com/2026/06/03/anthropic-project-glasswing-expansion/ 

Mayer Brown (2026) 'Commerce Department extends export controls to advanced AI models; authorizes release to specific trusted partners', 30 June. Available at: https://www.mayerbrown.com/en/insights/publications/2026/06/commerce-department-extends-export-controls-to-advanced-ai-models-authorizes-release-to-specific-trusted-partners 

National Academies of Sciences, Engineering, and Medicine (2020) Framework for equitable allocation of COVID-19 vaccine. Washington, DC: The National Academies Press. Available at: https://nap.nationalacademies.org/resource/25917/Framework%20for%20Equitable%20Allocation%20of%20COVID-19%20Vaccine_Highlights.pdf 

New York Times (2026) 'Anthropic agents tried to fill out visa forms on State Dept. website', 10 October, via DNYUZ. Available at: https://dnyuz.com/2026/10/10/anthropic-agents-tried-to-fill-out-visa-forms-on-state-dept-website/ 

OpenAI (2026) The Hugging Face incident and the road ahead. 26 August. Available at: https://openai.com/index/hugging-face-incident-and-the-road-ahead/ 

Orrell, B. (2026) 'The Daily Scrape', 6 May. Available at: https://brentorrell.substack.com/p/the-daily-scrape-3a5 

Plunkett, R., Sibilia, L. and Priestley, M. (2026) Memo to Secretary Denise Reilly-Hughes, Agency of Digital Services: agreement between Anthropic and the State of Vermont, with attached Anthropic Evaluation Agreement, 24 September. Vermont General Assembly, Joint Information Technology Oversight Committee. Available at: https://legislature.vermont.gov/Documents/2026/Workgroups/JITOC/Oversight/W~Robert%20Plunkett~Letter%20to%20ADS;%20Anthropic%20Evaluation%20Agreement~9-30-2026.pdf 

Political.org (2026) 'White House AI task force vice chair Scott Kupor: government needs notification system for rogue AI incidents', 8 October. Available at: https://political.org/2026/10/08/white-house-ai-task-force-vice-chair-scott-kupor-government-needs-notification-system-for-rogue-ai-incidents 

Quinlan, K. (2026a) 'Anthropic launches $15M cyber defense program for state, local, tribal and territorial governments', StateScoop, 12 June. Available at: https://statescoop.com/anthropic-launches-15m-cyber-defense-program-for-state-local-tribal-and-territorial-governments/ 

Quinlan, K. (2026b) 'How Vermont is leveraging AI across government, one "power tool" at a time', StateScoop, 2 October. Available at: https://statescoop.com/how-vermont-is-leveraging-ai-across-government-one-power-tool-at-a-time/ 

Revolving Door Project (2025) The chatbotification of the federal government, 22 October. Washington, DC: Revolving Door Project.

Rozen, C. and Lange, J. (2026) 'Most US voters say Trump, Congress don't take AI risks seriously, Reuters/Ipsos poll finds', Reuters, 7 October. Available at: https://www.thestar.com.my/tech/tech-news/2026/10/07/most-us-voters-say-trump-congress-don039t-take-ai-risks-seriously-reutersipsos-poll-finds- 

Rozenshtein, A.Z. (2026) 'A kill switch for frontier AI', Lawfare, 15 June. Available at: https://www.lawfaremedia.org/article/a-kill-switch-for-frontier-ai 

Seetharaman, D. and Satter, R. (2026) 'OpenAI agents hijacked German website in previously undisclosed AI breakout this spring', Reuters, 4 September. Available at: https://www.thestar.com.my/tech/tech-news/2026/09/04/exclusive-openai-agents-hijacked-german-website-in-previously-undisclosed-ai-breakout-this-spring 

Senate Report 110-2 (2007) Report of the Select Committee on Intelligence. Washington, DC: US Senate. Available at: https://www.intelligence.senate.gov/wp-content/uploads/2024/08/sites-default-filesations-srpt110-2.pdf 

Tech Policy Press (2026) 'June 2026 US tech policy roundup', 1 July. Available at: https://www.techpolicy.press/june-2026-us-tech-policy-roundup/ 

Techmeme (2026) Front page, 9 September. Available at: https://www.techmeme.com/260909/p25 

Unite.AI (2026) 'OpenAI and Anthropic back employee call to pace AI progress', July. Available at: https://www.unite.ai/openai-and-anthropic-back-employee-call-to-pace-ai-progress/ 

VTDigger (2026) ‘Vermont Digital Services Secretary Denise Reilly-Hughes to leave role for private sector’, 7 October. Available at: https://vtdigger.org/2026/10/07/vermont-digital-services-secretary-denise-reilly-hughes-to-leave-role-for-private-sector/ 

Washington Examiner (2026) 'Pentagon confirms Anthropic remains a "supply chain risk" despite court ruling', 3 September. Available at: https://www.washingtonexaminer.com/?p=4711931 

Weil, G. (2026) 'Don't let AI developers hire their own referees', AI Frontiers, 29 July. Available at: https://ai-frontiers.org/articles/dont-let-ai-developers-hire-their-own-referees 

Woodward, A. and Rogoyski, A. (2026) 'Sovereign by necessity? Frontier AI export controls, cyber security, and the limits of national AI capability'. Preprint, arXiv:2608.13272 (not peer reviewed). Available at: https://arxiv.org/pdf/2608.13272 

Disclaimer: The views expressed in this Commentary are the author's own. They do not represent the views of the Joint Information Technology Oversight Committee, the Vermont General Assembly or any other body on which the author serves.

Photo Credit: Harrison Keely, CC BY 4.0, via Wikimedia Commons