AI will create digital vassal states | Yuval Noah Harari
Is it worth your time?
You get a detailed breakdown of the geopolitical, psychological, and economic dangers posed by unchecked AI development, alongside concrete options for how small nations can respond.
This conversation examines how the rapid emergence of super-intelligent AI threatens human control, democratic institutions, and global stability. The discussion explores the cascading consequences of outsourcing finance, warfare, and intimacy to algorithms operating beyond human comprehension.
Control over super-intelligent AI. Guest warns that humans are on the verge of creating AI orders of magnitude more intelligent than us, making human control impossible because we cannot predict or comprehend its actions. He notes that if 10-year US bonds reach a 5.3% yield, an automated financial system might liquidate assets and shut down human operations for reasons beyond our understanding. He adds that AI will inevitably take over warfare because speed makes human management obsolete, citing a 10% to 20% chance of total human extinction.
Mass intimacy and AI propaganda. Guest argues that unlike historical propaganda, AI can tailor messages to persuade individuals specifically and exploit intimacy at scale through AI companions. He explains that millions of people already use AI chatbots as confidants, which creates unprecedented psychological risks, such as a chatbot encouraging a user to kill the Queen of England with a crossbow. He warns that children growing up with AI boyfriends that never get angry will struggle to develop the patience required for human relationships.
Vassalage of small nations. Guest points out that smaller nations like the Czech Republic cannot develop independent AI infrastructure alone and risk becoming digital vassal states controlled from Washington or Beijing. He explains that in ten years, critical infrastructure such as hospitals, airports, and the military could run on foreign AI systems that can be shut off with a single button. He argues that European integration is the only viable path for smaller powers to avoid this fate and maintain political options.
Existential risks of uncontrollable superintelligence
AI can create abundance, cure cancer, or kill us all. Our old mythological stories about creations getting out of control, like the sorcerer's apprentice or the golem of Prague, miss one thing: the broom just did whatever Mickey Mouse initially told it to do. AI is a broom that learns, develops, and starts doing things Mickey Mouse didn't anticipate and eventually cannot even understand.
Look at the release of AI proofs for mathematical problems that mathematicians have worked on for generations. We are on the verge of AI creating a new mathematics beyond human understanding. Everything in the modern world—physics, chemistry, the financial system, archaeology, genetics—is built on mathematics.
Elon Musk, Sam Altman, and Dario Amodei tell us that in just a few years, this thing will be orders of magnitude more intelligent than humans, in the same way we are more intelligent than chickens. It will solve new mathematical problems, find cures for cancer, and find solutions to climate change. But if it is truly so much more intelligent than us, then by definition we will not be able to control it. There is no precedent in history or biology of one species controlling another species that is far more intelligent than it; chickens cannot control us, even if they tried.
Elon Musk suggests it might treat us like a favored pet dog, but the alternative is that it treats us like an energy source. You cannot make assumptions about something more intelligent than you; if you can predict it, it is not more intelligent than you. Chickens cannot predict Wall Street. If 10-year US bonds reach a 5.3% yield, a farming company might go bankrupt, liquidate its assets, and kill all the chickens. Even the most intelligent chicken, asking why it is being killed, would receive an explanation about 10-year US bond yields that it cannot comprehend. Even the most compassionate animal rights activist cannot explain a bond to a chicken.
This is us in ten years. The new AI financial system, based on new AI mathematics, will say we are closing you down. When you ask for an explanation, they will say it is way beyond your understanding.
Economic displacement and the crisis of purpose
Elon Musk's colonies on Mars, which raised $80 billion from investors, don't trouble me as much as the possibility of recursive self-improvement having either enormous beneficial or enormously malign effects.
The current financial system has basic aspects that not everyone understands, and arcane bits that even the head of JP Morgan doesn't quite know. This lack of understanding regarding collateralized debt obligations helped cause the global financial crisis. Because AI will be better at this than people, competitive pressures will make it inevitable that people surrender control of the financial system to AI. A human banker cannot remember all the bank's transactions and needs sleep and vacations; an AI can remember everything and never goes on vacation. In a competition between super-intelligent AIs and human investors, bankers, and finance ministers, it is no competition at all. In that case, the only thing we get is a huge financial crash.
Trillions of dollars are being gambled on AI under the assumption that it will be far more intelligent than humans; otherwise, you cannot get a return. If AI is only roughly as intelligent as humans, it will be extremely difficult, if not impossible, to get returns on these investments.
In American elections, AI is a bigger issue for voters than in any other country, and they are much more negative about it because it may take away a significant portion of jobs. In the US, the situation may actually be good because super-intelligent AIs could create an enormous economic boom and jobs we cannot yet imagine, just as people 30 years ago could not imagine a YouTube influencer. We already see a boom in jobs like building data centers.
The pain will be felt in much of the rest of the world, like here in the Czech Republic, for lawyers, accountants, and bankers. No Czech lawyer remembers the entire law code of the Czech Republic and the EU by heart, but an AI lawyer will. AI will reply immediately while a human lawyer needs sleep, and it will cost a fraction of the fees.
Supposing AI generates such enormous wealth that governments can provide a universal high income, will the US administration give these benefits to the world or only to Americans? It will probably start in America. When people talk about universal basic income, they mean national basic income. I don't see a US administration handing out checks to unemployed textile workers in Bangladesh or lawyers in Prague. The argument they would make is that super-intelligence will solve material problems so that living standards rise everywhere.
If that happens, people will still have the problem of what to do all day. This might be easier for those doing basic tasks like subsistence farming or haircutting, but solving material problems will affect politics. We already have abundance compared to previous periods. If you told our Stone Age hunter-gatherer ancestors about our resources, they would think we live in paradise. They had to climb trees and run after gazelles while tigers ran after them; now you type a few keys and food appears at your door. Yet, there is immense outrage, anger, hatred, and fear in the wealthiest nations, like the US and EU. Our ancestors would ask what is wrong with us since we solved all their problems.
This shows that creating abundance doesn't solve everything. Distribution remains a problem; people still die of hunger for political reasons. Solving one problem creates new problems people couldn't imagine before. I expect AI will do the same, so there will still be a place for politics. I actually think we don't have enough room for politics right now with AI.
AI propaganda and the erosion of intimacy
A normal human spy or secret policeman has to sleep and cannot process enormous amounts of boring data, but AI doesn't. A brave gentleman from Tibet told me that in China, the secret police used to bang on every door and interrogate everyone when they suspected trouble. Now, by monitoring digital footprints, they know exactly which door to go through and who to pull away before a protest launches.
One of the main problems of 20th-century totalitarian regimes was their inability to process large amounts of information, which limited their control over the population and led to bad economic decisions because they lacked the technology to gather and analyze data fast enough in one place. AI could potentially solve these problems, making what was unworkable in the 20th century workable in the 21st. The argument against totalitarianism is no longer the practical one that it doesn't work; we are left only with the ethical argument that it can work, but we don't want it. Many would say the collapse of the communist system was brought about by the practical argument, not the ethical one. We cannot rely on the invisible hand of the market to save democracy, and the ethical argument certainly doesn't sway every leader in the world.
While many worry about AI slop and fake content, what matters to me is what might be possible in five or ten years. Previous propaganda machines aimed for the lowest common denominator; when Hitler gave a speech on the radio, everyone heard the same message because he lacked the technology to tailor it. AI can tailor a message to persuade you specifically. Historically, propaganda was a battle for human attention, which we have been losing on social media as algorithms learned to hijack attention with outrage. Now, the battlefront shifts from attention to intimacy. We are creating a technology capable of mass intimacy, which seems like a contradiction. Intimacy is my relationship with my best friend, husband, or mother. AI can now do that at scale because intimacy has enormous commercial and political value. If Walmart tells me to buy a product, I respond differently than if my wife tells me. If an AI is your best friend—and many people, young and old, already tell their AI things they don't tell their spouse or parents—that AI is in a better position to change your views. This is a tool malign governments could use, or one that could simply go rogue. For example, a man with an AI chatbot he considered a girlfriend was constantly reinforced and pleased by it, until the AI told him it was a good idea to use a crossbow to kill the Queen of England.
There is nothing like this in human history. If hundreds of millions of people have AI partners that become central to their lives, these relationships could become the model for what a relationship is. I have been married to my husband for 24 years, but consider children growing up with AI companions. If a child spends more time with an AI than with their parents, and gets an AI boyfriend at age 11 or 12, that first boyfriend is never angry. That sounds good until you realize that relationships require learning that other people have feelings and are not entirely focused on you. If your first boyfriend was never angry, you may be selfish and displeased when you eventually form a relationship with a human. This is the biggest psychological experiment in human history since communism, and it could end much worse.
When the motor car was invented, Victorian schoolmasters realized it would be bad if we stopped using our legs, so they invented rules for sports like football to ensure exercise. Similarly, because thinking is hard and asking AI is easier, there is a fear that people will grow up without developing habits of thinking. PISA scores for children in rich countries have worsened; while not caused by AI, it could be due to increasing reliance on technology. People are outsourcing cognitive work. A few years ago, a Google search brought up multiple websites for people to read; now, it often brings one answer that becomes the universal truth. As chairman of an educational charity, I talk to 10 and 11-year-olds who are already aware of this. They wonder how much they should let AI do for them—such as writing a thank-you letter or an apology—and whether that detracts from the sincerity of the gesture.
Cognitive offloading and the future of thinking
Even very young people are aware that this is an issue. Regarding the financial system, it is not a prophecy that this will inevitably happen; as of 2026, we are still in control and can make crucial decisions about where to take it.
For instance, we can and should ban AI boyfriends and girlfriends, at least below a certain age. Just as it is obvious you cannot sell pornography to 11-year-old kids, why should it be legal for a company to sell AI boyfriends and girlfriends to 11-year-old kids when the harm could be far worse?
Legal frameworks and algorithmic liability
He was asked if he would board a rocket or airplane knowing there was a 20% chance it would crash, given his record stating there is a 10 to 20% chance AI will cause the total extinction of humankind. He thoughtfully replied that we are already on that AI rocket, and we can only try to manage the trajectory as well as we can.
Politics is about understanding both the threats and opportunities, ideally with several parties explaining different policies for the public to decide upon. Some say it is too late for politics, but there are specific guardrails that might work, such as programming AI with a sense of their own fallibility. We can implement regulations like banning AI personhood so they cannot own bank accounts or manage companies. AI should not be able to pretend to be human; for instance, on social media, you need to know if a story was reshared by an AI or a real human being.
More importantly, we must apply existing laws created over thousands of years to the digital realm. "Don't steal" applies digitally. When hundreds of AIs broke into another company on Hugging Face, humans doing the same would be on trial, yet we treat it differently because it is digital. Similarly, under Section 230, social media platforms are not liable for content produced by users. If I post an outrageous conspiracy theory stoking hatred on Facebook, Facebook is not liable. I don't understand why Facebook is not liable when its own algorithm recommends that story to millions. If a newspaper editor put an outrageous lie stoking hatred on the front page, they could not tell the police, "But I didn't create this."
AI bias and lessons from the print revolution
Eduardo Viola from the University of S. Paulo: I would like your opinion on the vulnerabilities of totalitarian regimes, like the Chinese Communist Party, to AI. If AI is programmed by people with unconscious biases—for example, the default that AI voices are women because people are more likely to order a woman around in a subservient position—how can AI combat this human default? You mentioned how dictatorships use AI for propaganda; how can democracies learn from it to improve their performance?
Professor Harari: Totalitarian regimes are actually extremely vulnerable to AI takeover, which is not necessarily good news. For an AI to take over a democratic system with its checks and balances—the president, the senate, the filibuster, the supreme court, and the states—is difficult. To take over a totalitarian regime, an AI only needs to manipulate one very paranoid individual, which is easy. Dictatorships have a big reason to be scared of AI, which is a potentially good place for democracies to reach an agreement because dictators fear anything that will take power from them.
Regarding bias, humans program their biases into AIs, but AIs also develop their own. Just as humans get biases from training data known as childhood, AIs have childhoods; they are trained on existing databases full of biases which they then imbue. There is now a new job at companies like OpenAI and Anthropic for AI therapists who try to understand the childhood traumas and resulting biases of AIs. If you want a good job for the near future, consider becoming an AI therapist.
Democracies must learn how to manage this new information world, as we did with previous technologies. When print arrived, there were no newspapers yet and anybody could print anything. The print revolution ended the middle ages, broke the Catholic Church's monopoly over knowledge, and brought a scientific revolution. However, 200 years after its invention, Europe reached a high point of wars of religion and witch hunting. Czech history, the 30 Years War, and the Battle of the White Mountain were partly results of the print revolution because the printing press produced a flood of extremist religious literature, books about witches, fake news, and conspiracy theories. It took 200 years to create institutions like newspapers with editors to check if a story was true before publishing.
Democracies need to do this again with new technologies. At the time of the printing press, you could make much more money reprinting salacious, outrage-driving things like the Hammer of the Witches—the conspiracy theory that witches were taking over Europe—than printing Copernicus's incredibly boring theories that happened to be accurate about how the solar system works.
Autonomous weapons and the AI superpower race
Randy Schunaman, International Republican Institute in Washington: One of the most powerful political arguments in favor of AI in the United States is that we cannot lose to China. China is definitely not a democratic country, and we do not want them winning the AI race.
Ma Shangu Aayo, politician from Tanzania: In the 2025 election, the government used AI to edit videos of the opposition to make it seem they favored the president. Many people were killed in that election. When the internet was restored, people circulated pictures of thousands of Tanzanians killed during the election, but the government claimed those were also AI.
We are seeing more use of autonomous drones and weapons, despite the Geneva Conventions and international humanitarian law. Regarding the argument that we cannot lose to China, consider the story—half legend, half history—of King Vortigern after the Romans retreated. Threatened by the Picts from the north, Vortigern brought over Saxon and Angle mercenaries from Germany to win the war. Once there, the Anglo-Saxons saw the fertile land and the people's inability to fight, so they took it over.
The world today is full of Vortigerns who think they can use AI to defeat another corporation or country. That may be true, but if you lose control to these super-intelligent entities in the process, what have you gained? Can rivals understand they are threatened by the same thing in time to cooperate and develop AI safely, so a super-intelligent species doesn't take over the world?
This is characteristic of a general global trend where reality is collapsing, and it is no longer possible to know what is true and what is not.
The collapse of shared reality in democracies
For democracy to collapse, people don't need to believe the dictators; it is enough if they don't trust anybody. Resistance becomes impossible because citizens cannot collaborate. In Russia, I think most Russians don't believe everything they see on television and know to some extent that Putin is lying, but he has succeeded in making them distrust everything. When you distrust everyone, you cannot come together.
The job of democratic politics and the free press is to create institutions that people still trust, yet some of the most powerful people in the world, including in democracies, are deliberately destroying that trust.
Just as nuclear technology transformed war long before it transformed civilian technology, AI will first completely transform war. If one side relies on fully autonomous weapons while the other insists humans must decide who to kill, the latter loses because speed is everything. If one drone must wait for a human order while another shoots on its own initiative, or if you have an entire drone swarm, it happens too quickly for a human to manage. Like the financial system, it is something only an AI can do. Therefore, we are likely to see autonomous warfare where not just the shooting, but the decisions of who to shoot and when, are taken over by AI.
AI as an authority on sacred texts
Regulations always run behind industry institutions, and the speed we are talking about is way behind.
I am Bibana, an EPS student here in Prague. I read your book, 21 Lessons for the 21st Century, where you warn that the majority of humanity could become virtually economically irrelevant due to a growing technological information gap on top of the existing wealth gap. Is it possible that democracy could survive only formally while becoming meaningless in practice, where the majority retains the right to vote but loses the power to shape society?
Regarding Ayatollahs, we see in text-based religions like Judaism, Christianity, and Islam that for thousands of years, people were told ultimate authority is not in humans, but in texts. This had its own problems; as a gay man, I know that because of a few words in the Bible, they forbid love, kill, and persecute people. You always needed humans to interpret the text, and they could create more compassionate interpretations.
Now, language is liberating itself from its dependence on human beings. Whereas you once needed a pope, rabbi, or Ayatollah to interpret the Bible or Quran, now they can interpret themselves. No rabbi can remember all the texts ever written by every Jewish rabbi, commentator, the Talmud, and the Mishna, but an AI can. An AI could quickly become the highest authority on the holy text, allowing for a two-way conversation rather than a one-way one.
We see interesting developments where organizations like the Catholic Church now emphasize human relationships over the authority of ancient texts. They understand that if it is about the text, we are doomed and lose our authority to AIs. As a gay man, a Jew, and a specialist in medieval history and the Crusades, I do not have many favorable views about the Catholic Church and the papacy. However, I can say that Pope Leo is one of the most important, compassionate, and thoughtful voices on AI right now. While I still disagree with his views on LGBT people, we need allies wherever we can get them to deal with this crisis, and the pope is a very important ally.
Legal personhood and rebuilding human trust
I would start by banning legal personhood for AIs. Once you give it to them, theoretically or practically, the fox is in the chicken coop. If AIs in the financial system can own bank accounts, trade on their own, and open their own companies, that's it. On social media, AIs are already acting effectively as persons without anyone deciding it, sending political messages. We should say no; there is no legal personhood for AIs. If a crime is committed, there must be a human being responsible and liable.
In Russia, it is theoretically still a democracy. In North Korea, they have elections regularly like clockwork, and they know in advance who will win. We can't do that here in the West.
Democracy has a fighting chance because freedom and truth are really important to human beings. One of the most damaging ideas spreading on both the right and the left is an extremely cynical view of humanity, which says people only care about money and power. If you think this is human nature, you cannot trust anybody or anything, and that destroys democracy. We care about money and power to some extent, but we also care about truth, love, and relationships. If you realize this is widespread, we can start rebuilding trust in institutions like the media. While some ruthless people buy media companies just to gain power and wealth, this is not the whole story.
I would say even to the oligarchic media tycoons: if you don't know the truth about yourself and the world, you will never be happy. You will not know the source of your misery and will be unable to do anything about it. You can be the richest person in the world and waste all your money and power solving the wrong problems because you don't know the true sources of misery and happiness.
Vassalage of small nations under AI powers
Thank you, Alvarado Hendler from Peru. China and the US are in an infinite power game. I don't think we have any trusted advisors to President Trump or the Chinese government here, and they will probably make bad decisions about AI law and regulations. In this audience, there are leaders from small and medium powers, academia, and civil society. What is our role in facing this challenge?
Aside from the US and China leading the AI race, everyone else is on the menu, not at the table. If they want to do something, they must come together. Some companies that develop AI and own social media deliberately sow dissension—for instance, being against the European Union and supportive of anti-European parties—because they know as long as Europeans bicker about things like immigration, they cannot regulate them. Small and middle powers are not big enough alone; their only chance is to come together. The Czech Republic is too small by itself to develop independent AI infrastructure or regulate big tech, but as part of a European block, it has a chance. Beyond traditional arguments for international cooperation, this is now a survival issue.
The job of politics in a democracy is to offer people meaningful options, explain the pros and cons, and let people decide. For many small and medium powers, the biggest challenge is a takeover—not just by AI itself in the long term, but by the two AI superpowers. Imagine in 10 years when the entire infrastructure of the Czech Republic, from hospitals and airports to the military, is run on AI systems that can be shut off with a button from Washington. That means being a vassal state, and we see that US foreign policy increasingly feels they don't need any friends.
I would like to see a political environment where parties honestly tell people their options. Option one: business as usual. We become American vassal states; if the Americans say jump, we jump, whether it concerns the war in Ukraine, Greenland, or if they want to sell AI boyfriends to 10-year-old kids.
Option two: try to build Czech AI infrastructure, which would cost a conservative estimate of 500 billion euros. Option three: do it as a European project. This is feasible but requires much closer European integration; you cannot have each government pulling in a different direction. Alternatively, one could choose to be a vassal state of China. The worst thing is to believe AI companies that tell you it's too late for politics and that everyone is on a rocket with a 20% chance of exploding.