The arrival of AGI | Shane Legg (co-founder of DeepMind)
Intro
The guest reflects that human intelligence will not be the upper limit of what is possible, warning that superintelligence will dwarf human intelligence and structurally transform the economy and society, which makes it necessary to consider how to structure this new world.
The guest explains that current artificial intelligence systems are showing much more than just small sparks of AGI, pointing out that his definition of minimal AGI is an artificial agent that can at least perform the cognitive tasks that humans typically can. He likes this bar because setting it lower feels like failing expected cognitive tasks, while setting it much higher would exceed what many people can do.
The guest notes that current systems are uneven, operating much better than humans at speaking 150 languages and possessing phenomenal general knowledge, but still failing at continual learning over extended periods and in visual reasoning, such as understanding perspective between cars or counting connections in a network graph.
The guest expects these weaknesses to be addressed over a number of years as metrics improve, noting that while it will take time to work through the long tail of human cognitive abilities, systems will become increasingly capable at professional levels in coding, mathematics, and general knowledge.
The guest outlines that making models more reliable requires a combination of methods, including collecting specific visual reasoning data and implementing algorithmic and architectural changes like episodic memory systems and retrieval processes for continual learning.
The guest expands on his spectrum of definitions, placing minimal AGI at the lowest level, which he estimates is about two years away, representing the point where an AI stops failing in ways that would surprise us in a human.
The guest clarifies that minimal AGI does not mean we fully understand how to reach extraordinary feats of human cognition like inventing physics theories or writing literature, but once AI achieves the full spectrum of human cognition, we reach full AGI.
The guest defines the level beyond human cognition as artificial superintelligence, or ASI, describing it vaguely as an AGI that has become so capable that it is far beyond what humans can reach.
The guest recalls that when he and Ben Goertzel discussed a name for systems with general capabilities rather than specialized tasks, he suggested adding the word general to create artificial general intelligence because it rolled off the tongue.
The guest explains that the term quickly shifted from a subfield of study into a category of artifacts that needed a definition, noting that Mike Gubrud had actually used a similar industry-focused definition in a 1997 nanotech security conference paper.
The guest feels no regret about coining the term because it provided a necessary way to refer to genuinely general AI systems as distinguished from narrow advanced systems like AlphaFold or AlphaGo.
The guest observes that different people adapted the term through various lenses, with some viewing AGI through the lens of future societal transformation and economic growth, while he views it as a historical point in time.
The guest defines this historical moment as the threshold where AI systems belong in a similar category to human intelligence because they can perform typical human cognitive tasks, even if they are not geniuses like Mozart or Einstein.
The guest stresses the usefulness of a clear definition to clear up confusion caused by differing timelines, noting that many people who argue about whether AGI is three or fifteen years away are simply using different definitions for the same underlying expectations.
Defining artificial general intelligence
Evaluating alternative benchmarks, the guest notes various proposed definitions for artificial general intelligence, such as humanities final exam language model benchmarks, performing tasks as a chef in a new kitchen, or turning a small sum into a million dollars. He considers the financial metric a narrow economic perspective better suited to a specialized trading algorithm rather than general intelligence. True generality, he argues, is the remarkable flexibility of the human mind to handle many diverse tasks. Systems failing basic cognitive functions expected of almost anyone remain unsatisfactory. To operationalize his definition, he proposes a suite of tasks with known typical human performance levels where any failure prevents meeting the definition. Passing this initial battery leads to a second, adversarial phase where a team of people spends months probing the artificial intelligence to find cognitive tasks where it fails. If no such failure cases emerge after extensive testing, the system effectively reaches the threshold for practical purposes.
Discussing future perceptions of artificial intelligence, the guest anticipates that systems will become so generally capable across numerous dimensions that people will simply refer to them by product names like Gemini nine while recognizing their broad capabilities in writing poetry, playing invented card games, doing math, and translating. Before reaching that stage, however, mapping capabilities across different dimensions remains crucial for society to navigate powerful machine intelligence. Intelligence cannot be reduced to a single dimension because systems might display superhuman capability in certain areas while remaining fragile and weak in others. Understanding this distribution helps identify both opportunities and misapplications, moving beyond oversimplified dialogues that treat systems as either omnipotent or overhyped. Human intelligence exhibits a similar jagged profile where individuals excel in languages, math, or music while struggling elsewhere.
Shifting to ethics, the guest identifies a key aspect as whether artificial intelligence can understand ethical behavior, analyze potential actions robustly, and reason reliably about the ethics of its conduct. He favors an approach called system two safety, referencing Daniel Kahneman's distinction between fast, instinctive system one thinking and deliberate system two thinking. Just as humans facing difficult situations must pause beyond gut instincts to analyze complex nuances and consequences against moral norms, advanced models can utilize reasoning chains. When presented with ethical dilemmas like lying to save someone's life, these thinking models can analyze the situation and apply rules thoughtfully. By making this reasoning tight and grounded in desired ethics, systems could theoretically become more ethical than humans by applying rules consistently at a superhuman level.
Addressing the challenge of grounding these models in human ethics, the guest acknowledges that human ethics lacks a single universal standard, varying significantly across people, cultures, and regions, requiring models to understand these different expectations using the global data they absorb. Current agents are built by training large models on vast worldly data, resulting in relatively static objects that do not learn much new. This paradigm is shifting toward systems incorporating more learning algorithms, becoming more agentic by executing multi-step tasks like writing software or planning trips, and eventually becoming embodied in robotics. Through ongoing interaction and experience rather than static initial datasets, these systems will tighten their connection to reality, even though the foundational data ultimately originated from human experiences.
Ethics of AGI
Legg explains that these systems achieve a grounding in reality through ongoing experience. When considering whether an artificial intelligence can become more ethical than humans, he notes that while absolute certainty is impossible in complex real-world actions, surgeons also cannot guarantee one hundred percent safety in operations. Instead, we must test systems, balance benefits against risks, and actively monitor them after deployment to roll them back if failure rates exceed acceptable levels. Pre-deployment testing includes checking whether models can be tricked into helping create bioweapons or assisting with hacking, with mitigations adjusted based on capability levels. Interpretability methods, particularly within deliberate reasoning models, allow researchers to look inside the system to check its reasoning. Legg distinguishes between outcomes and intentions, noting that we judge human actions differently depending on whether harm was intentional or an accidental side effect of trying to do the right thing, and the same principles apply to evaluating artificial agents.
Shifting to broader societal impacts once capable artificial general intelligence arrives, Legg discusses the open question of machine consciousness. When presenting a concrete scenario of an advanced embodied agent that learns, remembers its history, and talks about its own consciousness, leading experts remain unsure whether it is conscious. While he doubts current models possess consciousness, he believes that in the future, people will strongly disagree about whether advanced systems are conscious in the absence of a universally accepted scientific definition or measurement. Another major question is whether we will transition from general intelligence to a superintelligence far beyond human capability. Comparing human biological brains operating at low energy consumption, slow signal propagation speeds, and low channel frequencies to data centers operating at megawatt scales with optical speeds and massive bandwidth, he argues that human intelligence is not the upper limit of what is possible, just as humans are outrun by cars or outseen by the Hubble telescope.
Because global competitive dynamics make stopping superintelligence difficult, Legg stresses the urgent need for system two safety to make superintelligence super ethical, applying its advanced capabilities to moral decision-making. Addressing the societal consequences of human intelligence being dwarfed by superintelligence, he warns that the traditional economic model where people trade physical and mental labor for access to resources will no longer function, requiring new systems for distributing wealth, even as the total production of goods and services grows massively. He recalls telling university vice chancellors that because artificial intelligence will perform a significant fraction of cognitive labor within a decade, every academic department from law and engineering to politics, medicine, and finance must rethink its field. Just like the industrial revolution, this technological shift brings immense opportunities alongside complex societal effects.
Societal and economic transformation
The guest compares the current situation to March 2020, when experts warned of an incoming pandemic while people continued visiting pubs and football games. He explains that people struggle to believe massive changes are coming because most extreme predictions fizzle out, but argues that when underlying fundamentals drive a shift, people must take the possibility seriously and think carefully about what AGI means.
Looking at the next few years, the guest does not expect immediate massive disruptions. Instead, he sees AI systems transitioning from helpful tools into performing genuinely valuable economic work, though unevenly across different fields. In software engineering, for example, a team that previously needed one hundred engineers might soon require only twenty, who will use advanced AI tools to increase their productivity and take on meaningful work.
As this transition unfolds, the guest expects public discussions to shift from viewing AI as a fun novelty for planning holidays or helping children with homework to recognizing it as a structural force changing the economy and society. He believes this development could spark a golden age by dramatically increasing production and advancing science, freeing humans from unnecessary labor. However, he stresses that this capability is only beneficial if society figures out how to translate it into individual and collective flourishing, noting that entry-level employees and graduates will feel the initial impacts in the labor market.
Future of work
For the short to medium term, the guest points out that many jobs will remain protected. A physical role like plumbing relies on robotics that are not yet capable and will take a long time to become price competitive. In contrast, elite cognitive work commanding high compensation—such as high-powered legal mergers, advanced finance, machine learning engineering, and mathematics—is vulnerable because it can be done remotely over the internet using standard computer interfaces. Other roles may stay protected if they rely on a distinct human personality, as people value knowing a real person is behind the work.
The guest urges academic departments across universities to take AGI seriously, observing that many experts still view current systems through the lens of past limitations, ignoring how rapidly capabilities compound year over year. He suggests that the general public often grasps the reality of AI better than domain experts, easily recognizing advanced language, mathematics, and problem-solving skills without getting caught up in specialized denial.
Confirming his long-held prediction, the guest reaffirms his view that there is a fifty-fifty chance of reaching minimal AGI by 2028, a timeline he has maintained consistently since publishing it on his blog in 2009.
Potential for human flourishing
The guest estimates that full AGI will arrive three, four, five, or six years after minimal AGI, placing it comfortably within a decade. When asked if this knowledge makes him feel nihilistic, he responds that the situation presents an enormous opportunity. Just as the Industrial Revolution harnessed energy to perform mechanical work and create widespread wealth, algorithms and computation can now handle cognitive labor to generate wealth, new technologies, and novel medicines.
The central challenge, according to the guest, lies in capturing these benefits while managing risks and costs. He acknowledges the profound difficulty of imagining a future world where artificial intelligence truly helps humans flourish, noting that the question touches on philosophy, economics, psychology, and ethics, and requires many more thinkers to explore what a positive future might look like.