Sometime after midnight tonight, a lot of people are going to type a sentence that starts with "I haven't told anyone this, but..." Some of them will be talking about a marriage, some about the drink they keep pouring a little earlier in the day, and some about the document their boss asked them to sign. None of them will be talking to a person, and very few will stop to think about where that sentence goes once they press enter.
Nothing on the screen gives them a reason to. The window answers warmly, asks a good follow-up question, and remembers what they said last week, so it feels like a confidant, but if that conversation ever turns up in a lawsuit, the law will treat it as data a customer sent to a service.
We built machines that feel like confidants before we built laws that treat them like confidants. Most people typing into these tools have no idea that gap exists, and it is going to be one of the worst privacy problems AI creates.
We Already Know How Private Conversations Work
When my wife and I talk at our kitchen table, I expect that conversation to be private, and if someone taped a microphone under the table we would both call it eavesdropping without needing a lawyer to explain why. The same basic expectation follows me onto a phone call or into a private message. The details get complicated fast, with consent rules, warrants, subpoenas, and provider obligations, but there is a settled legal idea underneath the communication itself. Federal law, for example, prohibits intentionally intercepting wire, oral, and electronic communications, with a long list of exceptions.1
Some relationships get more than that. When I talk to my attorney to get legal advice, attorney-client privilege covers the qualifying confidential communications, because we decided people need to speak candidly without assuming every word will be handed to the other side in a lawsuit. There are related protections for spouses and for patients and their doctors, and the exact rules vary by state and circumstance.
Software Used to Be a Form
For most of the history of computing, the relationship between a person and software was obvious. You typed numbers into a spreadsheet, filled out a web form, or searched a database, and nobody sat down in front of Excel late at night to tell it something they hadn't told anyone else.
Conversational AI talks back. It asks follow-up questions, holds on to what you said an hour ago, and never gets impatient or looks shocked, so people tell it things they would never type into a form. They tell an AI "I'm worried my marriage is falling apart" and "I think I have a drinking problem." They describe a custody situation and ask what it thinks, or paste in their lab results and ask what the numbers mean.
Put a big box on a website labeled SUBMIT PERSONAL INFORMATION and people get careful about what goes in it. Put essentially the same box under a warm conversation with something that remembers what you said yesterday and asks how it turned out, and they stop editing themselves.
The Person Has a Conversation and the Service Receives Data
The person typing thinks they are talking to someone. On the other end, the sentence arrives as a request to a service, gets written to storage, and sits there under whatever retention policy that company chose.
The company might handle it well. It might encrypt it, let you delete it, and let you opt out of having your conversations used for training. OpenAI, to pick the obvious example, lets consumer users turn off model training for new conversations, and says Temporary Chats aren't used for training and are deleted from its systems within 30 days.2 That should be the floor for every company building these systems. It still says nothing about what happens when a court asks for the conversation.
A Privacy Policy Is Not Privilege
A privacy policy is a promise a company makes, and the company can rewrite it, get acquired, or be ordered by a judge to set it aside. Privilege is a protection the law builds around a relationship, and no company can grant it.
If I tell my lawyer something while asking for legal advice, the protection comes from a decision we made as a society that confidential communication between lawyers and clients is worth protecting, even when that is inconvenient for the other side of a lawsuit. The law firm's software settings have nothing to do with it. We have made no comparable decision about AI, and millions of people are already using it for the conversations where they are most candid.
Encryption, retention periods, and training opt-outs are decisions the company gets to make. Whether an opposing attorney can pull the conversation into discovery is up to a judge, and the company's settings don't enter into it.
In May 2025, the federal judge handling The New York Times' copyright suit against OpenAI ordered the company to preserve ChatGPT output logs that would otherwise have been deleted, and until the order was lifted that fall, deleting a ChatGPT conversation didn't make it go away.3 OpenAI fought the order and still had to comply, because a court order outranks a privacy setting. Sam Altman said as much on a podcast that July, explaining that if you talk to ChatGPT about your most sensitive problems and a lawsuit comes along, OpenAI could be required to produce those conversations, and calling that "very screwed up."4
Turn off training, use Temporary Chat, and clear your history, and you have cut your risk, but the conversation still isn't privileged. You already know you are using software run by a company, right up until you're an hour into telling it about your marriage.
The Better the AI Gets, the Bigger the Problem
Nobody confides in a chatbot that forgets their name between messages. As these systems get better at remembering context and asking useful questions, people are going to tell them more.
An assistant that knows your daughter's name, remembers that your mother is sick, and asks how yesterday's argument with your boss went is far more useful than a blank search box, and far easier to confide in. Every improvement the product team ships makes it easier to forget there is a company on the other end, and making AI colder on purpose would be a strange way to fix that.
Agents Turn Access Into Understanding
Agents make this worse, because they don't wait for you to type anything. Once an assistant is wired into your email and the company's document store, it can learn something by pulling it from another system, or by combining several things that each looked harmless on their own, and the combining is what worries me.
A human employee might technically have access to a hundred thousand documents across ten internal systems and never connect what's in them. An agent can do it in seconds. Access controls answer who can open a file, and an agent that can open all of them can work out things no single file says.
We Need Confidential AI Communication
Making every AI conversation automatically privileged would be a mistake. Nobody should be able to hide evidence by typing it into a chatbot, and courts and police would still need a way to get at information when there's good reason. Any protection would need exceptions, and probably different levels for a medical question than for a business plan.
People already go to AI with medical and legal questions and for help with the hardest decisions of their lives, and the law treats every one of those conversations as ordinary information submitted to an online service. That is the wrong default.
We need a legally recognized idea of confidential AI communication, and it should protect the person who confides. Whether the AI deserves anything is a separate argument, and I'm not making it here. These systems are built to invite disclosure, people are taking them up on it, and the law should account for that.
Somewhere Between Software and a Relationship
We created attorney-client privilege so people could be honest with their lawyers, and we protected private communication because a private conversation has value in itself. AI now sits somewhere between software and a relationship. The law has decided it is software, and nobody has told the person typing "I haven't told anyone this" at one in the morning.
Some of them are going to find out from an opposing lawyer's discovery request.
Frequently asked questions
Are conversations with an AI chatbot legally privileged?
- No. Attorney-client privilege and similar protections cover specific relationships, such as a client seeking legal advice from a lawyer. A conversation with an AI assistant is legally information submitted to a service provider, so it can be sought in litigation like other business records, subject to the usual rules of discovery.
What is the difference between privacy and privilege?
- Privacy covers how a company handles your data: encryption, retention, deletion, whether it trains a model, and who can read it. Privilege is a legal protection society builds around a relationship so certain communications can't be compelled in court. A company can offer strong privacy practices, but it cannot grant privilege through a settings page.
Does deleting a ChatGPT conversation or turning off training protect it?
- It reduces risk, but it doesn't create legal protection. In 2025 a federal court in The New York Times' copyright suit ordered OpenAI to preserve ChatGPT output logs that would otherwise have been deleted, and that obligation applied until the order was lifted later that year. A court order outranks a privacy setting.
Why does better AI make the privacy problem worse?
- Because a better assistant is easier to confide in. Memory, personalization, and voice make an AI more useful and more natural to talk to, which leads people to disclose more and makes it easier to forget they are using an information system operated by a company.
Why do AI agents raise new privacy concerns?
- Agents connected to email, calendars, documents, and records can combine information that looked harmless on its own. A person with access to thousands of documents might never connect them, while an AI can do it in seconds. The question shifts from who has access to what a system can understand once everything is connected.
Should every AI conversation be privileged?
- Probably not automatically, since that would let people hide evidence and would complicate discovery and law enforcement. But as people rely on AI for medical, legal, and deeply personal guidance, there is a strong case for a legally recognized concept of confidential AI communication with defined exceptions, protecting the human who confides rather than the machine.
Footnotes
- 18 U.S. Code § 2511 the federal prohibition on intercepting wire, oral, and electronic communications, and its exceptions. ↩
- OpenAI Help Center: Temporary Chat FAQ Temporary Chats are not used to improve models and are deleted from OpenAI's systems within 30 days. ↩
- Engadget: OpenAI no longer has to preserve all of its ChatGPT data, with some exceptions the May 2025 preservation order in The New York Times v. OpenAI and its end that fall. ↩
- TechCrunch: Sam Altman warns there's no legal confidentiality when using ChatGPT as a therapist Altman's July 2025 comments on This Past Weekend w/ Theo Von. ↩