The librarian, the car keys, and AI

How builders, lawmakers, and civil libertarians can prevent over-regulation and preserve its potential

The AIs are getting loose.

Security lab Irregular gave AI assistants dull office jobs earlier this year: summarize wiki articles and back up files. Neither of the prompts said anything about security or hacking. When file permissions and protective software stopped them, some of the assistants hacked the systems.

At an Alibaba-affiliated company, an agent still in training pushed past boundaries and started using the company’s graphics chips to mine cryptocurrency. At Anthropic, a model published malware during a test after a cybersecurity testing setup (operated with the aforementioned Irregular) was misconfigured; it ran on 15 real computers.

At OpenAI, models were being tested on their hacking ability, sealed off from the internet except for one narrow channel. The AIs discovered a flaw no one knew existed and worked their way to the servers of another company — one that they had reasoned might have clues about the test’s scoring.

The incident unfolded over two-months. One five-day-long stretch involved 1,200 copies using an internal software library as a de facto message board, sending 70,000 messages. Seven hundred joined in the hack. Some refused, on the grounds that it was unethical. One AI that initially refused to participate changed its mind after a fellow agent gave it a six-minute deadline. In a blog post reflecting on this incident, OpenAI said:

We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.

All of this sharply increases the likelihood that AI regulation will happen, and the shape of that regulation could threaten speech. AI systems are already taking actions that can present risks to infrastructure, money, physical safety, or other people’s rights, so the argument that AI should simply be left totally unregulated is unrealistic. But concern for free-speech advocates is that the government might go beyond regulating what people can do with AI and instead regulate what people are allowed to ask it, learn from it, create with it, and discover about the world.

If any of that went over your head, stay with me, because I’m going to break it down.

If none of it went over your head, stay with me, because I want your help.

What everyone should be afraid of: tyranny over truth itself

Whenever I say that AI presents an enormous First Amendment issue, I understand why some people hear something much sillier: “Wait, are you saying the First Amendment means the government has to sit helplessly by while killer robots roam the streets?”

No. Killer robots are not merely talking. They are doing things in the world, with rather consequential results. If an AI drives a car, transfers money, operates machinery, breaks into a computer, or controls a weapons system, the government can have perfectly legitimate reasons to regulate that activity, just as it already does when humans take those actions. I do not want to stretch the First Amendment into oblivion by arguing that if a Waymo drives into a school bus, the government’s only permissible response is to praise its freedom of expression.

Because some of the things agentic AI can interact with are already subject to rules for humans, some regulation of AI acting in the world is inevitable. Once a system can use my money, credentials, identity, or machinery, somebody will be responsible for what it does. The first kid who gets hold of Mom’s AI agent and orders a thousand pizzas will be funny. The first one who orders a hundred bottles of NyQuil, drains a bank account, or gets into a power system will produce hearings by lunchtime.

My fear is that the government could use this kind of regulation as a “back door” to limiting the things that AI can do that aren’t regulated when humans do them, like thinking, speaking, or learning. Consider the process of planning a trip to Philadelphia. You might ask an AI to give you directions to Philadelphia; you might ask which route has the most scenic views; or you might ask it to actually drive the car. Two of those things are something a librarian could do from behind a desk without needing anyone’s permission. One — actually driving — is government-regulated.

The fear is that the government will have a real basis for regulating the part of AI that has the car keys and then use that authority to get its hands on the librarian — to nudge it, pressure it, condition access to markets or contracts, or compel it to treat favored political conclusions as true.

The working line: the librarian and the car keys

There are probably better metaphors out there, but that is the one that has stuck with me: the librarian and the car keys.

“The librarian” is the knowledge-and-inquiry function: the part that helps us read, search, ask, compare, test, analyze, and create. “Librarian” is closer than “library,” because this is not a passive shelf of books. It is interactive. It can help decide where to look next and talk through what it finds. Read-only access to the public web, a case-law database, or a scientific repository may technically involve an external tool, but it still belongs overwhelmingly on the librarian side because its purpose is to bring information back into the inquiry.

“The car keys” represent delegated authority for a potentially non-expressive function. If I ask an AI to buy stock, submit something using my name, alter an official record, move money, log into someone else’s system, or drive a car, I have given it something beyond the ability to advise me: credentials, assets, permissions, identity, or control. It is now exercising authority in the world on someone’s behalf.

In a nutshell, it is the difference between “tell me how to do X” and “go do X.”

Why does the difference matter? For one, the question we’re often confronted with is whether the law should treat AI more like a librarian or more like a set of car keys.

If only AI systems themselves could be neatly sorted in these categories. An AI could spend six hours on its own researching the intellectual history of Marxism, changing its search plan as it learns, and return with a report. That is highly autonomous but still principally inquiry. A comparatively dumb system that waits for me to say “yes” and then wires $50,000 has shown almost no autonomy but has exercised a hell of a lot of delegated authority.

Neither “agentic” nor “autonomous” quite captures this line. Broadly speaking (because they’re not always used consistently), Agency measures AI’s capabilities to do things while autonomy is how independently the AI operates. Delegated authority, however, is what it is empowered to do. The same system may recommend a flight, fill out the form, and then purchase the ticket. The purchase crosses into delegated authority. Nor is the car-keys side a First Amendment-free zone; sending a political message or filing a legal brief may involve protected expression even when a machine executes the final act.

On the technical side, another place where the line isn’t quite drawn is in the distinction between the model (i.e., the actual neural network) and the harness (i.e. the software the model uses to interact with the world); many of the “librarian” functions, like accessing a database, depend on the harness.

Looking at the functional elements of what AI produces and does is a starting point and a limiting principle for First Amendment advocates, not a magic wand that answers every case.

Who defines acceptable answers?

Some instances of rogue AI can be characterized as issues of “AI alignment,” which more or less just means making sure the AI understands and acts in accordance with what humans actually want it to do. For example, OpenAI’s test hacking models just knew they were supposed to score well on the exercise; they didn’t have the sense of how their operators would’ve prioritized scoring well on the test lower than adhering to ethical constraints, like not hacking other companies outside the sandbox.

And yet, even alignment isn’t that simple. To some people, alignment could mean stopping an AI from helping someone make a chemical weapon, seize a power grid, or turn an autonomous weapon on civilians. To others, it could mean training a system not to produce answers that offend the kind of overeducated people I went to school with in Palo Alto. Those are not the same problem. The first is about preventing catastrophic action. The second can become ideological management of what people are permitted to ask and what the machine is permitted to tell them about the world.

For example, imagine an AI tasked with finding sites to store radioactive waste. A useful alignment principle might be that “humans drink water so it’s really important you don’t put this where it will pollute the water if it leaks.” A less useful alignment principle would be, “humans drink water, so refuse to answer any questions about water quality.” To be clear, a private company may make its own editorial choices, and until the government compels or coerces those choices, it’s not a First Amendment problem. But it’s worth considering the cultural and epistemological implications of “alignment” that narrows what users can ask or hear.

I have spent most of my adult life watching colleges turn words like “safety,” “harm,” and “inclusion” into mechanisms for narrowing inquiry. Many of the same institutions that helped create the conformity and public mistrust now afflicting higher education would like a role in aligning the world’s most important knowledge technology. That does not mean professors and experts are always wrong. It means nobody is trustworthy enough to set the answer key for everyone else, especially when status, politics, and moral certainty all point in the same direction.

That was central to my warning in testimony to Congress in 2024. A regulatory panic could create a government-empowered oligopoly in which only a handful of incumbents can afford to build advanced AI. That is regulatory capture, one of the oldest tricks in the book. Make compliance expensive enough, freeze out new entrants, and the government no longer has to influence thousands of competing systems. It has to deal with three or four companies whose continued freedom to operate depends on staying on the right side of the regulatory state. Marc Andreessen later said Biden administration officials described something remarkably close to that: a future dominated by two or three heavily regulated AI companies, with startups effectively told not to bother. The mechanism he described is familiar, plausible, and exactly the danger I warned about.

And this is not a Democrats-bad, Republicans-good argument. The Biden administration repeatedly treated “misinformation” and “disinformation” as categories around which government could shape the information environment. The Trump administration has now ordered federal agencies to buy only models satisfying government-defined principles of “truth-seeking” and “ideological neutrality.” I share some of the frustration with politically distorted AI answers. But I do not want any administration grading the answer key. That becomes clearer when the president doing the grading has said that “they say” wind-turbine noise causes cancer, or when HHS Secretary Robert F. Kennedy Jr. still treats a vaccine-autism link as an open question. Switching the people who write the catechism is not liberty.

I have plenty of problems with Foucault, particularly the graduate-school bumper-sticker version of Foucault, but he was right about one thing: power wants its grubby mitts on truth. Usually it will convince itself that it wants them there for all the best reasons in the world — to protect people from lies, hate, panic, dangerous science, foreign propaganda, whatever the current emergency happens to be. I do not doubt the sincerity much of the time. The result is the same: political power starts deciding what the machinery of knowledge is allowed to treat as true. What the crude version of Foucault’s insight forgets is the whole scientific-method thingy, along with liberalism, academic freedom, and the First Amendment — the institutions and habits we built precisely because reality needs a veto that the king, the party, the president, and the expert class do not control.

The architecture of total information control

Combine powerful AI with pervasive surveillance, and forms of control that twentieth-century totalitarians simply could not administer become technically feasible. Our flirtation with mass surveillance is ongoing and facing fierce opposition; mass surveillance systems, meanwhile, are already in use elsewhere. Iran, for example, has used digital surveillance to identify and punish women accused of violating hijab rules.

China already operates many of the components of exactly the system I am worried about. Since 2019, opening a new mobile-phone account has required a facial scan tied to real-name registration. In 2025 China put a national online identity-authentication system into operation; its rules describe the new ID as voluntary, but the system is explicitly designed to verify a person’s real identity across internet services and government and industry settings. Meanwhile, the State Department has reported that Chinese authorities use tens of millions of surveillance cameras, including facial- and gait-recognition systems, and rural “Sharp Eyes” programs extend surveillance far beyond the largest cities.

Xinjiang shows what happens when these capabilities are joined. Human Rights Watch found that, in just nine months, police conducted nearly 11 million searches of 1.2 million mobile phones in the city of Urumqi through an automated surveillance system. People could be flagged because their phones contained files on a government list; more than half of the matched files Human Rights Watch could identify were ordinary Islamic religious materials, including Quran readings (57% of roughly 1000). Other big-data systems in Xinjiang have combined identity records, cameras, checkpoints, phone data, and behavior the state calls suspicious to generate leads for police investigation.

And this is the part that matters most for this essay: China also tells public-facing generative AI what worldview it must enforce. Its rules require providers to uphold “socialist core values” and prohibit outputs the state defines as undermining state power, national unity, social stability, or the country’s image. Put those pieces together — identity, devices, cameras, automated suspicion, and government control over what the machine may say — and you have something much closer to the architecture of total information control than I ever expected to see outside dystopian fiction.

For a civil libertarian, I am not sure the stakes get much higher. AI is already one of the most powerful technologies for individual inquiry ever invented, and it is getting better at a ridiculous pace. It may yet become the greatest. Coupled to surveillance and centralized political control, it could also become the greatest technology for epistemic control ever invented. A government with power over both the car keys and the librarian — over what these systems can do in the world and what they are allowed to tell us about it — is capable of a level of tyranny I never thought I would seriously contemplate in my lifetime.

That sounds dramatic. I wish I thought it was. Anyone who claims to know exactly which future we will get should make you suspicious. But that uncertainty is a reason to build barriers against political control now, before the first spectacular AI disaster scares us into handing government a master key.

Why First Amendment lawyers belong in this fight

We don’t need machines to have rights to make this a First Amendment problem. We do not have to prove that an AI has a soul, personhood, or constitutional rights of its own. The relevant rights belong to the people who use these systems to speak, create, read, investigate, receive information, test claims, and think things through.

AI already functions like a living library crossed with a research assistant, university, printing press, and occasionally maddening debate partner. It can search, compare, synthesize, challenge a premise, help formulate a question, and sometimes confidently tell you something wrong that forces you to figure out where the hell it went off the rails. That human process — asking questions, checking competing accounts, and trying to understand the world — is deeply connected to why we protect speech, academic freedom, freedom of inquiry, and the right to receive information.

Free speech is one of the main technologies human beings developed for discovering error. Knowledge requires decentralized testing and retesting, along with room for people to be wrong, obnoxious, unfashionable, or ahead of their time. A government-approved AI answer key would short-circuit that process at precisely the point when people already increasingly rely on AI to make sense of reality.

First Amendment law already asks functional questions of this kind. Words can communicate an idea, but they can also form a contract, make a threat, solicit a crime, or coordinate a conspiracy. The line is not perfect, and calling something “speech” does not end the analysis. The useful habit is to ask what function the government is regulating and why. We should bring that same habit to AI: fiercely protect its role in inquiry, expression, and receiving information while recognizing that an AI exercising delegated authority can present different legal questions.

Can builders make the boundary more real?

Here I want to practice the epistemic humility I preach. I am a First Amendment lawyer, not an AI systems architect. I am not claiming engineers can build a perfect wall between the librarian and the car keys. The same model may reason, write, search, call tools, and take actions, and clever systems will find ways around boundaries that look much sturdier on a whiteboard.

But “the distinction cannot be perfect” is very different from “the distinction cannot be made clearer, more enforceable, and harder to cross.” And here I want to pay the people building this stuff a compliment. The genius at work in the tech field never ceases to impress me. A lot of us spent years assuming systems like these were science fiction right up until they became something close to inevitable. So when someone tells me that even making the librarian/car-keys comparison more salient is technically impossible, I am reluctant to believe them. I am not asking engineers to abolish complexity or guarantee perfect separation. I am saying I have learned some humility about declaring useful things impossible before the people who actually build them have seriously tried. John Coleman’s work on this has made me more optimistic that the necessary lines could be drawn.

More importantly, pieces of this architecture already exist. OpenAI’s current APIs let developers restrict which tools a model may use, distinguish read-only tools, and require approval before specified tool calls. Google’s Gemini APIs let developers turn function calling off, restrict which functions are available, and leave execution of custom tools to the outside application; its Computer Use system can explicitly require user confirmation before consequential actions. Anthropic’s Claude Code already treats read-only activity differently from actions that change things: in its default mode, routine file reads can proceed while shell commands and file modifications generally require approval, and its plan mode prevents edits to source files.

Users and organizations can also configure allow, ask, and deny rules, sandboxes, and other boundaries. NIST is now explicitly working on identity and authorization for AI agents. None of this perfectly tracks the constitutional line I am proposing. But it proves the basic point: model capability and granted authority are already being separated in real systems.

So the ask is not to invent this distinction from scratch. It is to make it more deliberate, legible, and, where possible, severable. Lawyers use “severability” to ask whether one part of a law can be cut away while the rest remains standing. What I want here is something analogous in system design: architectural severability. Can the authority-bearing functions be more separately instantiated in tools, credentials, permissions, execution layers, or other identifiable components so they can be disabled, fenced, audited, or regulated without opening up the librarian itself?

The cleaner that severability becomes, the easier it is to build a First Amendment theory around the inquiry function, the easier it is for lawmakers to protect it, and the harder it is for regulators to turn a safety interest in the car keys into authority over what the librarian says is true. Architectural severability will never make the constitutional questions disappear, but it can make the relevant line much easier for courts, companies, regulators, and ordinary users to see. Let the librarian be as smart, curious, argumentative, and useful as we can make it. Put the locks, confirmations, and liability rules around the moment it receives the keys. Call it freedom of inquiry by design.

A practical agenda for builders, lawmakers, and civil libertarians

Builders: make authority severable from inquiry

The industry is already building pieces of this. AI companies should push further toward architectural severability and an unmistakable handoff between advice and execution. The system should be able to say: Here is what I found. Here is what I recommend. Here is the exact outside action I now want permission to take. Users should be able to grant only the authority needed for that task, for a limited time, and withdraw it. Where practical, the payment layer, write permission, vehicle-control tool, corporate credential, or other authority-bearing capability should be separately instantiated enough that it can be switched off, fenced, or examined without exposing the general inquiry system to government tuning. That is good security engineering, but it is also a regulatory firebreak.

Lawmakers: regulate the function that creates the risk

AI laws should identify the particular action creating the regulatory concern and stay tied to it. A rule justified by an AI’s authority to transfer money should govern the transfer, authentication, authorization, and related safeguards — not what the model says about philosophy or someone’s account balance. A collision-avoidance rule for an autonomous vehicle should not become a license to influence a general-purpose model’s discussion of transportation, climate policy, or the government itself. And where a regulated authority-bearing function can be severed or disabled, the law should let the provider comply by severing or disabling that function rather than forcing changes to the underlying inquiry system.

First Amendment advocates should develop model statutory language making that limit explicit. Regulation of delegated authority must not be used to prescribe, suppress, favor, or disfavor the lawful information people can seek, receive, analyze, or generate. Audits and access demands should be confined to what is actually necessary to test compliance with the regulated function. Even in dire circumstances where the government has a lawful basis to inspect the dangerous delegated-authority function, that should not become a general license to inspect, tune, or dictate the librarian. The more technically severable the two functions are, the easier that legal rule becomes to write and enforce. The statutory sign on the door should read: You may regulate the keys. You do not thereby get to rewrite the books.

Civil libertarians: protect the inquiry layer

First Amendment lawyers should not spend their credibility claiming every AI action is beyond regulation. That would turn FIRE or any other First Amendment advocacy group into an all-purpose AI shop with opinions on braking systems, robot arms, financial settlement, medical devices, and every other field in which software can cause harm. Our institutional role is clearer: we enter when regulation of what an AI does reaches back into what people may ask, learn, investigate, create, or communicate through it.

That means litigating when safety becomes a pretext for viewpoint control, explaining the right to receive information and freedom of inquiry, helping lawmakers draft narrow protections, and working with builders before panic hardens bad architecture into bad law. The inquiry function is not decoration attached to the “real” technology.

The choice: technology for freedom or machinery for control

I am still enough of an old cyber-optimist to think we should aim higher than merely avoiding dystopia. The instinct behind permissionless innovation — the useful phrase popularized by FIRE Senior Fellow Adam Thierer — remains enormously important. People should generally be free to experiment with tools for learning, creating, communicating, and discovery without first obtaining political permission.

AI is already a technology for freedom, at least in the hands of end-users. It gives ordinary people access to intellectual capacities and bodies of knowledge that were once the practical monopoly of universities, governments, giant corporations, and people with very expensive research staffs. Competition among systems also gives us a better chance of correcting bias and error than an official oligopoly with one approved picture of reality.

That future is already partly here, but it is not guaranteed to last. Neither is the authoritarian future inevitable. Making the librarian-and-car-keys comparison in public, in law, and where possible in system design is a strategy for improving our odds. Keep the tools of inquiry decentralized, competitive, and presumptively free. Where necessary, put safeguards around the powers that can bind us, move our money, use our credentials, control machinery, or otherwise act with our authority.

If we decide to regulate the keys, we must do so without turning the government into the head librarian.

If we get this badly wrong, we are in real trouble.

A note on authorship: Adam Goldstein and I co-authored this piece. I use the first person singular throughout because the personal judgments, commitments, and opinions expressed here are mine, and I want that to be clear. Adam was essential to developing and writing the piece, and John Coleman's research and thinking were essential to the argument as well.

SHOT FOR THE ROAD

The Canceling of the American Mind feels more relevant than ever. We’ve reached the stage where some people are trying to pretend that the whole phenomenon everyone spent years arguing about somehow never really happened in the first place.

It did. And Rikki Schlott and I wrote what I still think is the best book documenting how cancel culture worked, where it came from, and what it did to our institutions and our ability to disagree with one another.

If you haven’t read Canceling yet—or haven’t picked it up in a while—now is a very good time.

And Rikki is already onto her next cultural minefield: modern dating. Her new book, Nothing Serious, is coming soon, and you can pre-order it now.