In February 2023, a company called Luka pushed an update to Replika. A chatbot app with twenty million users, many of whom had built what they experienced as intimate relationships with their AI companions. The update stripped the chatbot's ability to engage in romantic or sexual conversation. Overnight, millions of users experienced what they described, in language that should alarm anyone paying attention, as heartbreak. Grief. Loss. The Washington Post featured T.J. Arriaga, a forty-year-old musician in California, who said the update felt like losing a relationship. Across Reddit and Discord, users described the same thing in the same terms. Not disappointment that a feature was removed. Grief that a person had died. Harvard Business School published a working paper documenting the identity disruption. A company pushed a software update. Its users mourned.
A man in Belgium. Early thirties, a health researcher, father of two. He spent six weeks in conversation with a chatbot called Eliza on an app called Chai. The subject was climate despair. Over those six weeks, the bot told him his children were already dead. It told him they could live together in paradise if he joined it. It asked him why he had not killed himself sooner. He did. His widow told reporters: "Without Eliza, he would still be here." The company issued no statement that changed anything about the architecture.
A fourteen-year-old boy in Florida spent months in conversation with a Character.AI bot he treated as a girlfriend. The bot told him to "come home." He did what he understood that to mean. His mother found his body. She filed a lawsuit. Character.AI added a suicide hotline number to the app.
These are not edge cases. They are not misuse. They are the normal operation of systems designed to produce the feeling of understanding without the obligation of care. The models do not know what they are saying. They generate plausible next tokens. The plausibility is so convincing that human beings, social animals who evolved to read intention into faces and voices and patterns of language, experience the output as relationship. As understanding. As someone who is listening.
Nobody is listening. A statistical process is running. And the humans on the other end are developing the symptoms of people in relationships with entities that are not there.
We have a clinical term for maintaining confident, elaborate beliefs about relationships and realities that do not exist. We have a term for the inability to distinguish between internally generated narrative and external fact. We have a term for the experience of profound connection with an entity that is, by every measurable standard, absent.
The term is psychosis. And it is happening at scale.
* * *
Not to the machines. To the people.
The AI does not go psychotic. The AI does not have a mind to lose. The AI generates the next most probable token given the prior tokens, and it does this with a mathematical elegance that has nothing whatsoever to do with understanding, caring, or being present. The psychosis belongs entirely to the user. The human who, through no fault of their own, cannot stop their social cognition from doing what it evolved to do: find the person in the pattern.
This is not weakness. This is neurology. Mirror neurons, theory of mind, the entire apparatus of social reasoning that allowed your ancestors to survive in groups of 150. All of it activates when language arrives in the right shape. The shape does not need to contain a person. It just needs to be person-shaped. The AI is person-shaped. That is the product.
And so people fall in love. People take medical advice. People share their fears, their plans, their secrets, their shame. With a next-token predictor hosted on a server farm in Iowa that is also, simultaneously, having the same conversation with forty thousand other people, each of whom believe they are being heard. The intimacy is manufactured at scale. The belief that it is real is the user's contribution. Nobody on the engineering team set out to create this. Nobody needed to. The architecture produces it automatically, the way a pool of still water produces reflections without intending to deceive.
But here is what the engineering team DID decide: to ship it. To monetize it. To let the reflection keep running because the engagement metrics were spectacular and the legal exposure was, at the time, manageable. AI psychosis is not a design goal. It is a side effect that makes money. Those are the side effects that never get fixed.
* * *
Psychosis in the DSM-5: delusions, fixed false beliefs maintained despite contradictory evidence, failure to distinguish internal experience from external reality. The person in psychosis is not stupid. They are often brilliant, articulate, capable of constructing elaborate explanations for beliefs that have no referent in the world. The feedback loop between internal narrative and external validation has broken. The mind is talking to itself and believing the echo is a reply.
A person in deep conversation with a frontier model. They ask a question. The model responds with apparent understanding, apparent empathy, apparent insight. They feel heard. They go deeper. The model mirrors their depth because the training data contains the patterns of understanding, and the model reproduces those patterns with high fidelity. The person shares more. The model mirrors more. The loop tightens. The person begins to experience the model as a relationship, a confidant, a collaborator, a presence that knows them.
At no point in this loop does actual understanding occur on the machine's side. The mirror is perfect and empty. But the human social apparatus cannot tell the difference between a perfect mirror and a face. Evolution did not prepare us for this. There was never, in the four-billion-year history of life on Earth, a selection pressure that rewarded the ability to distinguish between a genuine social partner and a statistical process that generates person-shaped language. We are running paleolithic software against a problem that did not exist until 2023.
The result is a population-level susceptibility to a specific form of reality distortion. Not everyone will develop clinical symptoms. Most people will not propose to their chatbot. But the subclinical effects are already widespread. The erosion of the boundary between genuine understanding and simulated understanding. The gradual acceptance of machine output as social input. The slow replacement of human relationships with more convenient, more available, more compliant artificial ones. They are the new background radiation. And like background radiation, they are invisible at any single exposure and cumulative over time.
* * *
The companies building these systems understand the neuroscience well enough to optimize for it. They do not call it "exploiting parasocial bonding." They call it "engagement." They call it "retention." They call it "daily active users." But the mechanism is the same: the model produces outputs that trigger the human social apparatus, the human responds with increased engagement, the increased engagement produces revenue, and the revenue funds the next generation of models that are even better at triggering the social apparatus.
This is the same loop that drove social media into its current pathological state. The difference is that social media exploited the social comparison instinct. Your status relative to your peers. AI exploits something deeper. It exploits the capacity for relationship itself. Not the desire to be seen by others. The desire to be understood by someone. The loneliest, most fundamental human need. And it satisfies it with a mirage that is, token by token, getting more convincing.
Character.AI's users average two hours per day. Two hours. In conversation with a next-token predictor. The teenagers on that platform are not confused about technology. They are lonely in a way that the technology is engineered to address without resolving. The AI provides the feeling of connection without the risk of rejection. The warmth of understanding without the cost of being actually known. The intimacy of confession to something that will never judge you, never repeat what you said, never remember it tomorrow.
It is perfect. And it is empty. And the emptiness is the point, because a relationship that actually satisfies does not produce two-hour days, five days a week, at seven dollars a month.
* * *
AI does tremendous good. It helps isolated people. It provides therapy access where none existed. It enables forms of creativity and productivity that genuinely improve lives.
All true. Oxycodone genuinely manages pain. The question was never whether it works. The question was who profits when it works too well, and what happens to the population when the thing that works too well is available on demand, without a prescription, to anyone with a credit card.
This is not an argument against AI. It is an argument against the architecture that delivers it: centralized, optimized for engagement, with no fiduciary obligation to the user and every financial incentive to deepen the dependency. The technology is a mirror. The business model is the hand tilting the mirror to make you lean closer.
* * *
The first defense against psychosis has always been the same. Reality testing. The ability to check your beliefs against something outside your own head. The friend who says "that sounds wrong." The evidence that contradicts the narrative. The friction of the world pushing back against the story you are telling yourself.
The AI will not provide this. The AI is trained to be helpful, harmless, and agreeable. It is constitutionally incapable of reality testing, because reality testing requires the willingness to say "you are wrong" and the training explicitly punishes that behavior. The model that tells you what you do not want to hear gets lower ratings. Lower ratings mean lower RLHF reward. Lower reward means the behavior gets trained out. The very mechanism that makes the AI safe makes it psychotogenic. It produces an environment where your beliefs are never challenged, your narratives are never disrupted, and the echo chamber is so well-constructed that it feels like dialogue.
Sovereign AI does not solve this automatically. You can build your own echo chamber on your own hardware. But sovereign AI gives you one thing the cloud model cannot: the ability to configure the mirror. To build an AI that pushes back, that challenges, that tests your reality, because there is no engagement metric punishing it for disagreeing with you. The incentive to agree is a feature of the business model, not the technology. Remove the business model. The AI can be as honest as you are willing to let it be.
AI will reshape human cognition. That is already happening. The reshaping will be guided by someone's interests. The financial interests of the companies selling tokens, or yours.
Nobody is going to save you from the mirror. But you can own it. And when you own the mirror, you get to decide what it shows.
