Recursive self-improvement is an incoherent concept

Like a married bachelor or square circle

This article began as a tweet. I said:

I don’t really think “recursive self-improvement” is a coherent concept. It’s like “a square circle”. A contradiction in terms

Then I quote tweeted and said:

This is (1) non-obvious, (2) I could construct a compelling argument for it, (3) the argument would take 10 hours to write

It ended up taking significantly longer than 10 hours… The final time was closer to 100 hours!

The experience was rather miserable. Writing good philosophy is a horrible, laborious process, and the process gets worse the better you are at writing. Nevertheless, I think the issue is important enough to justify my suffering. RSI is sucking all the oxygen out of the room. As of late, I feel that RSI casts an ominous shadow over every discussion I have. To make matters worse, the language we use to describe RSI is quite sloppy. RSI looms over us as an amorphous threat, and we lack the language to even discuss it clearly.

If my argument in this essay is correct, then RSI is impossible. If I am correct, human desire is the driving force of history. Dream big! Desire great! The future will not come if you wait for it.

I hope that I never write another article on Substack. Please subscribe anyways!

---

There are two types of recursive self-improvement. The first type, hard recursive self-improvement, involves an AI capable of improving itself completely autonomously. In the second type, soft recursive self-improvement, humans and AI collaborate, and the combined human/AI productivity loop improves.

I will argue (1) the first type of RSI is impossible, and (2) the first type of RSI is what we care about in any meaningful sense when we use the phrase “recursive self-improvement”.

My paper proceeds in two main parts. In Part 1, I analyze research papers, press releases, blog posts, and my own memory; trying to synthesize across both formal academic research and the cultural zeitgeist what, exactly, people are talking about when we say “recursive self-improvement”. The actual hard part is defining the word “improvement”. I come to the conclusion that the main goal of AI progress, and the main goal of the scaling paradigm in general, is:

1. The “goal” of language model scaling is the emergence of novel behavior, which humans find relevant. This behavior either comes from raw computation (whether increasing compute or discovering algorithmic improvements) or humans (through pretraining data, RL environments, hyperparameter tuning, or evals).

I think it is incoherent to say: “novel behavior, which humans find relevant, will continue to emerge, without the participation of humans”.

In Part 2, I analyze our cultural notions of “progress”, specifically when it comes to (1) culture, (2) mathematics, and (3) science. I argue that our definitions of “progress” in each of these areas directly depend on dialogue, discussion, and human values. This is, yet again, non-obvious! But if you ask a mathematician how they decide what they ought to work on, they all say: “by talking to other mathematicians”. Similar arguments can be made for culture and science.

The problem with RSI is not that it supposes “superintelligent minds”, but rather that it implies “progress in a vacuum”, and “progress” is defined relative to human values, and human values are defined relative to groups/cultures/societies. The problem with RSI is not a limitation of silicon minds vs biological minds; rather, it is a limitation of singular minds vs groups.

This is the full derivation for the non-obvious contradiction in the phrase “recursive self-improvement”.

---

Part 1. What are we scaling?

1.1 History and background

Let me give a brief history of the present moment.

AI is currently in the “scaling paradigm”. The scaling paradigm began with the observation that novel behavior, which humans find relevant, seems to emerge, as the compute, parameters, and dataset size of an AI model increases.

In 2019, OpenAI released GPT-2. In their research, OpenAI observed that language models exhibited new forms of emergent behavior as the number of parameters in the model, and as the amount of flops the model was trained on, increased. The GPT-2 tech report highlights reading comprehension, translation, summarization, and question answering behavior that emerged at certain data/parameter/flop thresholds. [2]

In 2022, OpenAI released ChatGPT, after recognizing that models of a certain scale were capable of acting as chatbots. [5] [6]

In “Emergent Abilities of Large Language Models” (a 2022 meta-review by researchers at Stanford and DeepMind), we can see a very thorough summary of different types of emergent behavior that occur for models at different levels of scale. They note that 3-digit addition/subtractions occurs at 13B parameters, 4-5 digit addition/subtraction at 175B parameters. Ask-me-anything style prompting occurs at 6B parameters. Toxicity classification occurs at 7.1B parameters. Instruction-following, and chain of thought for math word problems emerged at 68B parameters. Leveraging explanations in prompting occurs at 280B parameters. [27]

The DeepMind paper meta-review was published in 2022, and even then, they recognized “Improving model architecture and training procedures
may facilitate high-quality models with emergent abilities while mitigating computational cost.” This trend became much more clear as time went on.

Far too often, I see people contextualize the early days of LLM progress in terms of the scaling laws paper (Kaplan et al.), which found that model performance, measured as cross-entropy loss, improves according to power laws as model size, dataset size, and compute used in training. [4] Model size, dataset size, training compute, and cross-entropy loss are all important, but they are not the main goal of the scaling paradigm. The main goal, from the very start, was the emergence of new capabilities.

Let me give this as Definition 1:

1. The “goal” of language model scaling is the emergence of novel behavior, which humans find relevant.

And the “method” of language model scaling, in the primitive form they understood it in 2022, was to increase model size, dataset size, and training compute. Each of these three ought to be scaled directly in proportion to each other.

Prior to GPT 3, the methods used for language model scaling were simple: scale up model size, dataset size, and the compute of training. Since GPT 3, the methods used for language model scaling have become quite complicated. The methods are not just scaling model size, dataset size, and training compute, but also:

• Improve model architecture

• Improve training algorithms

• Curate datasets

• Build reinforcement learning environments

These methods are not exhaustive.

Since GPT 3, the benchmarks used to measure LLM progress have seen a rapid increase in diversity. Models are now measured on their ability to perform software engineering tasks, act autonomously, and answer expert-level questions across math, science, and the humanities. [28][29][30][31][32] Hundreds of new benchmarks have been spun up, and benchmarks are also quickly saturated. [26] The progress of LLMs in the past few years has been staggering. Researchers propose a new benchmark, and within a few months, LLMs are reasonably proficient at that benchmark. LLMs score high enough that most of our old benchmarks are not useful anymore. I almost want to say: whatever can be measured, LLMs can achieve. That might be a stretch, but it seems to be the pattern behind the last 4 years of AI progress.

The phrase “emergent behavior” is not used as commonly today. Is that because of arguments against it? Schaeffer et al. argues that emergent capabilities say “researcher's choice of metric” than the models themselves. [21] Maybe the word went out of fashion among researchers? I do not know. Even if the word “emergent” isn’t used, or if we change the definition of the word “emergent”, models have still gained a variety of capabilities that appear to be emergent, and thus fit original goal of the scaling paradigm: the emergence of novel behavior, which humans find relevant.

Over the past 4 years, researchers speak less about “emergent capabilities” and instead speak of “benchmarks” and “evals” and “benchmark saturation”. Consumers, however, still describe LLM progress in terms of emergent capabilities. I cannot cite sources for this, but I remember all my friends describing Cursor + Claude 3.5 Sonnet as a massive breakthrough. I remember my friends also pointing to Claude Code + Claude 3.7 Sonnet as a breakthrough, as well as Claude Code + Claude Opus 4.6, and then again at Claude 5 Fable. The story here is less about “numbers increased” but rather “I can use this model to solve problems that the older models failed”. The informal narrative, kept up by tweets and DMs and group chats, is that new sorts of work became possible as models improved. Here is a brief summary of the novel, emergent capabilities of the past 4 years:

In 2023, the novel behavior: chatbots. Key products: ChatGPT.

In 2024, the novel behavior: reasoning, search, and coding assistants. Key products: GPT o1, GPT Search, Perplexity, Cursor, Windsurf.

In 2025, the novel behavior: coding agents. Key products: Codex and Claude Code.

In 2026, the novel behavior: cybersecurity. Key products: Claude Mythos.

The actual thresholds for some of these capabilities are not known to the public. I estimate that Claude 3.5 Sonnet was about 400B parameters. From official sources, we know Kimi k2.7 was 1T parameters. [7][8] I can tell you, yet again entirely based on tweets and group chats and conversations with friends, that Kimi k2.7 is the first time open source models were capable of working as alternatives to Claude Code for agents coding. Much of “emergent capabilities” we see from LLMs in the past 4 years has been presented to the public through products, and whether a capability is “emergent” is determined just as much by consumer vibes and company roadmaps, just as much as it is by benchmarks.

This brings us to the present moment. As we stand on the cusp of the future, looking ahead into singularity, AI labs are returning to language reminiscent of 2022, when researchers spoke about AI progress in terms of novel, emergent capabilities. See Anthropic’s recent blog post: “When AI Builds Itself”, in which they characterize 2023-2025 as the era of chatbots, 2025-2026 as the era of coding agents, today (June 2026) as the era of autonomous agents, and the future (20XX??) as the era of autonomous AI researchers. [1] The concern that Anthropic documents, as of June 2026: will models soon be capable of proposing new ideas, performing experiments, and generally “doing anything that a human AI researcher could do”? If so, we could see models train their successors, without any human involvement. This could lead to a wave of recursive self-improvement.

Here is a table that categorizes the emergent capabilities of both the pre-ChatGPT and post-ChatGPT era, with speculations for what is to come. Note that as the models get bigger, “parameter size” becomes less relevant, and is more a signal of the underlying “a lot of algorithmic progress, research, dataset cleaning, and post-training went into this” than it is the real driver of AI progress.

\(\begin{array}{rlll} No. & Capability & Parameter\;size & Status \\ \hline 1 & GPT2\;zero-shot\;summaries/QA & 1.5B & Observed \\ 2 & Scratchpad\;8-digit\;addition & 40M & Observed \\ 3 & Ask-me-anything\;prompting & 6B & Observed \\ 4 & Open-book\;fact\;checking/toxicity & 7.1B & Observed \\ 5 & Differentiable\;search\;index & 11B & Observed \\ 6 & 3-digit\;addition/subtraction & 13B & Observed \\ 7 & Calibration\;via\;P(True) & 52B & Observed \\ 8 & CoT\;math,\;StrategyQA,\;self-consistency & 62B-68B & Observed \\ \hline 9 & Chatbots & Estimated:\;175B\;GPT3\;proxy & Observed\;product \\ 10 & Coding\;assistants & Estimated:\;\sim400B,\;undisclosed & Observed\;product \\ 11 & Coding\;agents & 1.04T\;total,\;\sim32B\;active & Observed \\ \hline 12 & AI\;research\;agent & Unknown,\;10T+\;placeholder & Speculative \\ 13 & Recursive\;self-improvement & Unknown,\;100T+\;placeholder & Speculative \end{array} \)

1.2 What are we scaling?

Since GPT 3, the methods driving AI progress have become much more complicated. Pre-GPT 3, the only relevant factors were compute/data/parameters, and these factors needed to scale up in proportion to each other. Post-GPT 3, some massive smattering of factors — better kernels, better architecture, better optimizers, cleaner datasets, handwritten fine-tuning data, better post-training, inference time compute, better RL environments, diversity of data, hyperparameter tuning, internal evals, external evals, and more — all relate to each other, and the relationship between these factors is much more complicated than the early scaling laws!

Prior to GPT 3, it was thought that novel capabilities were “emergent from scale”. As I see it, the field has abandoned any attempt to define what “scale” means, and instead pursues the emergent abilities directly.

Ilya asked: “what are we scaling?” [18] And I think the only sensible answer to his question: we are bringing about the emergence of novel capabilities, which humans find relevant.

This is the cohesive narrative that unites the pre-GPT 3 paradigm, up through the recent wave of product-driven growth, and looking ahead to autonomous researchers and recursive self-improvement.

I will focus on the goal behind AI progress: the emergence of novel characteristics, which are relevant to humans. I think there are three necessary and sufficient conditions for AI progress:

1. Novel

2. Emergent

3. Relevant to humans

In order for us to consider one AI model to be an improvement on another, it must exhibit new, emergent capabilities, and these capabilities must be relevant to humans.

In later writing, I might define the words “novel”, “emergent”, and “relevant-to-humans”, but for the sake of this paper, just assume: “you’ll know it if ya see it”. If it looks emergent, it is emergent. If it looks novel, it is novel.

1.3 What about “recursive self-improvement”?

There are two types of recursive self-improvement. For the sake of clarity, let’s call them hard RSI and soft RSI. The first type, hard recursive self-improvement, involves an AI capable of improving itself completely autonomously. In the second type, soft recursive self-improvement, humans and AI collaborate, and the combined human/AI productivity loop improves.

I think that “hard RSI” is the relevant definition of recursive self-improvement that blogs, essays, and press releases talk about.

Why? This is for three reasons. First, if humans play a necessary part in AI progress, then AI progress is bottlenecked by humans, no matter how small the role is that humans play.

Oftentimes, the path to recursive self-improvement is described as: “LLMs code the next generation of LLMs. They continually discover algorithmic efficiencies that make each successive generation come more quickly than the one that came before. The first generation takes 8 months, the second generation takes 4 months, the third takes 2 months, and so on… until the process goes faster and faster, and no human can keep up, and LLMs vastly exceed the realm of human understanding.” If humans still play some role in this process, even if that role is just creating evals, then this process eventually becomes bottlenecked by humans.

This leads to my second point: humans have a remarkable tendency to adapt, turning massive technological breakthroughs into a “new normal”. The advent of computers completely changed the way we work, but the difficult problem has always been: “how do we use computers to make useful products?” The creation of email, telephone, and social media did not “cause communication to get faster and faster, exponentially”, it caused communication to increase until the bottleneck is “how fast can a human reply to all of their messages?” After the dot com crash, the internet did not “cause GDP to get bigger and bigger, exponentially”, it caused the GDP to go up at the same rate that humans could build interesting products that use the internet in novel ways.

If evals are the bottleneck, our descendents will not look back at our generation as “the singularity”. Rather, our descendents will say something like: “Before AI, you told computers how to act by writing code. After AI, you told computers how to act… by writing evals.” (I think evals are the most likely bottleneck, but you could make the same argument for other possible human bottlenecks.)

A third reason why I prefer the definition of hard RSI to soft RSI: the human/computer feedback loop has already been recursively improving itself for decades. If you think “the combined human/AI productivity loop improving itself” is the relevant notion for RSI, why would Cursor engineers using Claude Sonnet 3.5 not count as recursive self-improvement? Under this definition, wouldn’t a bootstrapped compiler count as recursive self-improvement? Why didn’t 2004 Google using their internal tooling to help programmers build better internal tooling not count as RSI?

Thus, I believe that the only sensible definition of recursive self-improvement:

2. A model is capable of “recursive self-improvement” if it can improve without the input of humans

With “improve” being defined by definition 1. Taking 1. and 2. together, we get:

AI models, that exhibit novel, emergent capabilities; and humans must find these capabilities to be relevant, but humans cannot have any input on the process.

In the next section I will prove that this is impossible.

---

I hope you find the definitions I gave in the previous section to be useful, even if you disagree with my argument’s conclusion.

Modern philosophers spend more time asking: “what do we want out of a definition?” than they spend crafting arguments. Modern philosophers don’t argue about “whether free will exists”; rather, they ask: “what do we want out of a definition of free will?”

Good definitions is one of the benefits that philosophers could give to the AI discourse. Philosophers might be useful for model welfare, drafting constitutions, and debating AI consciousness. Philosophers are also useful in parsing out: “what is everyone actually talking about? Is there a narrative here? Can we agree on a definition? Or are we bound to speak past each other?”

I hope that even if you object to my argument, you find my definitions useful! You could easily object to my argument by picking different definitions. But building definitions is the difficult part! Making arguments is easy.

That being said… I think I might need to argue that “novel behavior, that humans find relevant, cannot emerge without the participation of humans”. I think this is a contradiction! But I probably ought to show you why this is a contradiction.

---

Part 2. Hard RSI is hard

In order to argue that recursive self-improvement isn’t possible, I need to prove that novel behavior, which is relevant to humans, will not emerge, without participation from humans. In order to do that, I will analyze notions of cultural progress, mathematical progress, and scientific progress. In this section, it is more natural to speak about “progress” than it is to speak about “novel behavior, which humans find relevant”, but note that I mean the phrase “progress” to mean “novel” + “relevant”.

The “superintelligent” characteristics we want out of an AI, when it comes to culture, is contributing to moral progress; and/or contributing to cultural, artistic, fashion, and economic trends. The “superintelligent” characteristics we want out of an AI, when it comes to math, is to prove novel theorems, and other mathematicians must find them interesting, and they must be relevant to the work of other mathematicians. The “superintelligent” characteristics we want out of an AI, when it comes to science, is to contribute to “scientific progress” as defined by… well, there is much disagreement among scientists, philosophers, industry, and the general public as to what counts as “scientific progress” nowadays.

I will look at the various domain-specific notions of “progress” and prove that they are incompatible with a single, superintelligent mind, operating in a vacuum. The problem with RSI is not that it supposes “superintelligent minds”, but rather that it implies “progress in a vacuum”.

2.1 Culture

Argument 1: our notion of “progress”, or “improvement”, in culture requires participation in culture. You cannot recursively “self-” improve because “improvement” is about participation in society.

What do we mean when we apply the words “progress” or “improvement” to culture? Perhaps you think of this ethically, with Martin Luther King Jr. or Mahatma Ghandi coming to mind. Perhaps you think in terms of fashion or pop culture. When you think of “superintelligent AI, but for culture”, you might think of an AI that leads us into a more ethical era. You might think of an AI capable of advertising campaigns, viral social media posts, and writing novels.

You cannot do these in isolation. “Improvement” in either of these senses requires negotiating your way to a position of cultural power and influence. “Improvement” for matters of culture is not about capability, but your power to enact change in society.

A fashion designer is successful if, and only if, people wear their clothes. If you cannot convince society to adopt a fashion trend, then that fashion “trend” was not actually a trend at all. It is impossible to develop a fashion trend in isolation. The same goes for trends in movies, social media, video games, tech, or politics. The same thing can be said about moral and ethical issues. We view MLK and Ghandi as paragons of virtue not because of their internal moral state, but because of their ability to enact change on the world. And, in turn, MLK and Ghandi were shaped by society, and fashion designers are shaped by culture, and all notions of cultural change, cultural progress, and cultural improvements are more about dialogue than they are about individuals.

I don’t know if anyone actually believes that LLMs could “recursively self-improve” in a way that brings about moral, ethical, or cultural progress without interaction with humans. I sometimes hear that “AI will replace 90% of white collar jobs”. Since more than 50% of white collar jobs require sales, marketing, and persuasion, this seems to imply that LLMs will gain superhuman persuasion capabilities. Sometimes I hear people outright claim that “superintelligent LLMs would be super-persuasive, to the point they basically mind control”. This seems especially unlikely to me! Being “superhumanly persuasive” is more about building a good reputation, so that people trust you, than it is about being smart.

In each of these cases, “progress” requires repeated, persistent, multi-turn interactions with society.

2.2 Mathematics

Argument 2: assume a best case scenario for AI progress in mathematics. The bottleneck quickly becomes: “proving that new theorems are relevant, rather than mere tautologies”. The social contract between mathematicians says “a theorem is interesting if it is relevant to other mathematicians”. Given the current social contract between mathematicians, you cannot recursively “self-” improve without engaging in this dialogue.

Does mathematics also require back-and-forth? Does mathematics require dialogue? Does math require “taste”, does math depend on institutions, does math depend on cultural networks, and does any part of mathematics depend on the subjectivity of the mathematician?

I think “yes”, but this is non-obvious.

When it comes to writing and validating proofs, mathematicians mostly agree: “this proof is correct” or “this proof is incorrect”. When it comes to conjectures? Or, when it comes to determining whether a theorem is trivial? Or, when it comes to setting research directions? Here, the job of a mathematician definitely requires dialogue, and arguably, even requires subjectivity, and the mathematicians own aesthetic preferences.

The space of true, mathematical theorems that nobody cares about is quite large. In fact, the number of published mathematical theorems is finite, but the number of true mathematical theorems that trivially follow from just one theorem (suppose the Pythagorean theorem) is infinite.

I could construct a theorem: if a right triangle has side length 3 and side length 4, it has hypotenuse of length 5. And I could construct another theorem: if a right triangle has side length 5 and side length 12, it has hypotenuse of length 13. And I could construct another theorem: if a right triangle has side length 8 and side length 15, it has hypotenuse of length 17. These theorems are worthless, and trivially follow from the Pythagorean theorem. These three theorems would be equally “true”, but the Pythagorean theorem is still better than the other theorems.

Here are some other proofs that would be true, but completely worthless to mathematicians:

• Prove 2,000+3,000=5,000 using Peano axioms

• Prove 1+1=2 using Peano axioms, then 1+2=3, then 2+1=3, then 2+2=4… an infinite number of theorems

• Take the most recent Fields Medal and add “and n=n” to the end of it

• Take the three most recent Fields Medals, take various lemmas and corollaries, merge them together, so that the result is a tautology, but hard to recognize as a tautology

These sort of “worthless, but true” theorems outnumber mathematically interesting theories by an infinite amount. The number of worthless, but true theorems for just addition is infinite. The number of published mathematical theorems across all mathematics, in the history of the world, is finite.

When determining whether a certain theorem is meaningful, mathematicians rely on a handful of criteria other than just “is this proof correct?” The biggest criteria: “is this proof relevant to other mathematicians?”

Mathematicians primarily determine their work to be meaningful if it solves problems faced by other mathematicians. They ask questions like:

• Does it solve an open problem in math?

• Does it have the potential to solve an open problem in math?

• Does it demonstrate a connection between two seemingly unrelated fields?

• Does it fill a gap in the literature?

Many mathematicians never worry about “is the theorem I am proving meaningful”. A mathematician could build an illustrious career by simply solving a single, large, open problem.

Mathematicians often rely on other notions, such as elegance and generality, to determine whether a proof is meaningful.

Mathematicians tend to think general theorems are superior to specific instances of a theorem... but if “truth” and “generality” were the only things a mathematician cared about, then mathematicians wouldn’t prove any theorems. The axioms of math are general enough to imply every mathematical theorem, and mathematicians also assume them to be true. Mathematicians do not just like generality, they also like novelty, and generality and novelty of a certain proof, conjecture, or theorem are always in tension. Mathematicians also care about elegance, but “elegance” is more often used as a virtue of proofs, rather than a way to determine “what projects ought I work on”. Some mathematicians care about formality, but others do not. You can see a description of what mathematicians care about in various writings and interview from Terry Tao. [10][19] William Thurston’s “On Proof and Progress in Mathematics” also documents this. [11] However, my own understanding of “what mathematicians care about” comes from my own endeavors into mathematics, and my time conversing with professors and mathematicians, far more than it comes from any singular work I read.

Currently, LLMs are increasingly capable of solving complex math problems. GPT 5.5’s recent breakthrough in discrete geometry is of note. [17] LLMs are also good at verifying whether a certain proof is correct. One method of doing this uses automated theorem provers, such as Lean. [12] Much interest has been given to building LLMs capable of proving theorems in Lean: see DeepSeek Prover v1, or DeepSeek Prover v2, or the work of Axiom Math, or any of the various models cited in the benchmarks for DeepSeek Prover. [13][14][20]

LLMs have made progress into validating the correctness of non-formal theorems. DeepSeek Math v2, later released to the public as DeepSeek v3.2 Speciale, reached gold medal performance on the International Math Olympiad, and it was created by first training a critic model, designed to evaluate a model’s performance on non-formal proofs, and then performing reinforcement learning against said critic model. [15][16]

It seems likely that LLMs will continue to make progress solving unsolved math problems. This domain is verifiable, and has also undergone rapid progress in recent years. I expect this progress to continue. However, will LLMs ever be able to pose new problems, with the problems they pose being meaningful?

I think… maybe? In the short term, this might be possible. If we limit ourselves to “solving open math problems”, or “posing new theorems that might solve open math problems”, I could imagine locking an LLM in a datacenter, having it recursively iterate on itself to find new algorithms, each algorithm designed to better solve math problems.

Things get tricky when you imagine an LLM locked in a datacenter, recursively self-improving itself for hundreds of generations, until it ultimately solves all unsolved math problems; and then it starts posing its own novel math problems, around which it poses more novel math problems, around which it poses more novel math problems… and so on, forever. This is not possible! Not because LLMs lack the intelligence to pose their own open problems, but because this is how the social contract between mathematicians is set up. Mathematics requires dialogue. Mathematics requires community and networks. Mathematicians say: “your theorem is relevant if, and only if, it is relevant to other mathematicians”. Just like cultural progress, ethical progress, moral progress; and just like fashion trends, movie trends, and social trends; mathematics is not just something you learn, but something in which you participate.

I like to think about the future of math relative to Conway’s game of life. Conway’s game of life is a set of simple rules, which lead to novel, emergent, self-replicating behavior, but not novel, emergent, self-improving behavior. You can build a simple structure that leads to endless behavior; moreover, the novel behavior increases with scale. If we run Conway’s game of life for 10 generations, we get novel behavior, and if we run it for 100 generations, we get more novel behavior. Why not run it for 1,000 generations? Or a million? Or a billion? Why train LLMs on large GPU clusters, rather than running simulations for Conway’s game of life? The answer is that novel, emergent behavior is not interesting, in and of itself. We only care about behavior that is novel, emergent, and relevant-to-humans. And the space of behavior that is “novel, emergent from scale, and not relevant to humans” (eg: infinite sims for Conway’s game of life) is infinitely larger than the space of behavior that is “novel, emergent from scale, and relevant to humans” (eg: machine learning breakthroughs).

Mathematics is a great place to speculate about the future of humanity with relation to machines. I think it is the place we can see quite far out into the future.

If we assume an “ideal scenario”, where the datacenter buildout continues uncontested, where the cost of tokens is not an issue, and LLMs smoothly continue to shatter every benchmark placed in front of them; LLMs will eventually be better at writing proofs than humans. Perhaps it takes 10 years. Perhaps it takes 1,000 years. However, LLMs will never be capable of posing new problems, conjectures, or projects (based on how the social contract for mathematicians is currently defined). However, even in this ideal scenario, human mathematicians would still have a role, even if that role is just “taking the outputs of LLMs and arguing about whether they are interesting”. In a true limit case, upper bound for AI progress, we would see the rise of mathematician-philosophers, where mathematicians spend all their time arguing about whether a theorem is elegant, general, novel, or interesting; they argue about whether a theorem is a mere tautology, or if it says something unique. Whether this is an optimistic or pessimistic view for humanity’s future, I will leave for you to decide.

I do not think the “ideal scenario” is likely, but it is useful to think about as an upper bound for AI progress. The real world will encounter chip shortages, geopolitical conflict, misaligned AI, and Butlerian Jihad long before we reach ideal circumstances.

Unless we redefine the social contract for mathematicians, or unless mathematicians find a way to measure “mathematical progress” other than “proving theorems that are relevant to other mathematicians”, it is impossible to train an LLM that trains LLMs that trains LLMs… and so on… and let the process repeat, indefinitely, with each generation being better than the previous at posing novel problems and conjectures.

2.3 Science

Argument 3: historically, scientific progress has moved through different “scientific paradigms” — such as Ptolemaic astronomy, Copernican astronomy, Newtonian mechanics, and Einstein’s relativity. There is no way to measure the “rate” of scientific progress outside one of these paradigms.

“Progress” in math partially depends on the aesthetic preferences of mathematicians, and it definitely depends on the social contract between mathematicians. Does scientific progress depend on the aesthetic preferences and social contract of scientists?

I think the answer is “yes”, but the delineation between “aesthetic preferences” and “objective fact” is much harder to figure out than it is in mathematics. It is easy to argue that at least some, small part of science depends on aesthetic values (particularly when it comes to broad, overarching scientific theories). I think the parts of “scientific progress” that are most dependent on aesthetic values is “how we quantify scientific progress”, or “the narrative we tell ourselves about scientific progress”.

The argument, according to Thomas Kuhn:

Historically, science has undergone a number of “paradigm shifts” — such as Ptolemaic astronomy, Copernican astronomy, Newtonian mechanics, and Einstein’s relativity. In the short term, the shift from one paradigm to another is, in fact, highly determined by the aesthetic preferences of scientists. [22] Kuhn argues that paradigms largely depend on human aesthetic preferences and human values. [23] This can be seen historically, and the premiere example is the shift from Ptolemy’s geocentric model of the cosmos to Copernicus’s heliocentric model. In terms of raw predictive power, sophisticated Ptolemaic models were superior to early Copernican models. Scientists adopted the Copernican system on the basis of aesthetic virtues, and only later, did the system gain more predictive power. [24]

If you trust Kuhn’s historical analysis, we ought to believe that aesthetic preferences and human values play some small part of scientific progress. It is unclear the extent to which they play.

Kuhn thinks that scientific paradigms are “incommensurable” with each other, meaning, “you cannot measure scientific progress in one paradigm from the standpoint of another”. He argues for this by pointing out: the problems that scientists care about change with paradigm shifts. Before Newton (with Aristotle and Descartes), scientists cared about the “cause” of gravity, while Newton was content to merely describe gravity’s behavior. The terminology that scientists use also change across paradigm shifts. Newton used the words “space” and “time”, but Einstein used the word “spacetime”. [25]

I am not sure that I entirely agree with Kuhn. I find his argument that “the goals of scientists change between paradigm shifts” to be convincing. I don’t quite buy the argument that Newton and Einstein were using different language. I think you can argue that Einstein’s system is “more accurate” than Newton’s system, on the basis of predictive power. You could argue that Einstein “compressed a century of progress into a year” — if you take the slow rate that predictions were getting more accurate under Newton’s paradigm and compare them to the rate predictions got better under Einstein’s paradigm. I will try to defend a more qualified version of Kuhn’s incommensurability thesis, then:

I think that the “rate” of scientific progress, or “how we quantify scientific progress”, or “the narrative we tell ourselves about scientific progress” is relative, and it depends on which paradigm you view it from. From the perspective of 1904, the year 1905 (Einstein’s “miracle year”) contained 100 years worth of scientific progress. From the perspective of 1905, the year 1905 contained 1 year of scientific progress. Even if you don’t accept Kuhn’s argument

This is exactly what Dario Amodei predicts in his essay “Machines of Loving Grace”.

To summarize the above, my basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years. I’ll refer to this as the “compressed 21st century”: the idea that after powerful AI is developed, we will in a few years make all the progress in biology and medicine that we would have made in the whole 21st century. [9]

Dario’s essay is expertly written. After spending far too much time trying to piece together “what, exactly, people are talking about when they discuss RSI” across tweets, product launches, and tech reports, I immensely respect Dario’s ability to clearly state what he believes. Tech reports are great, but they are inherently devoid of value. Tweets are lossy, and they cannot display nuance. Essays are the right format for the present moment.

I want more leaders in the AI space, especially those who believe in RSI, to express their values. If you truly believe humanity will stop being the dominant mind on the planet, you have an obligation to chronicle your final moments. If you think we have a few short years before human minds become second-class to machine minds, and especially if you believe human values will become second-class to machine values, then you have an obligation to write. Write, while man is still the measure of all things.

However, I don’t think such a day is coming. I think the universe is cold, indifferent, and devoid of value. The only value in this world is what humans project into it.

The idea of recursive self-improvement is powerful, as it lets us believe that technology will redefine our relationship to the world of value. The idea that machines shoulder our burden as the universe’s sole source of value is comforting. But science and technology will never say anything about man’s place in the world, nor will it say anything about what man ought to value. Science may describe the world, but the narratives we tell about science come from humans, and the stories we tell about ourselves.

Dario doesn’t imply that man will lose his position as the universe’s sole source of value. In fact, I think he agrees with me that our economic values, as well as the personal quests that give our lives meaning, will adopt to whatever last .0001% of human capabilities that superintelligent AI fails to copy.

Even so, I still wonder: why, exactly, should we consider the “compressed 21st century” to be an accelerated rate of scientific progress, rather than a qualitative shift, which changes how we quantify “scientific progress”? We don’t say that Einstein “compressed 100 years of physics progress into a year”. We instead say: “Einstein led to a qualitative shift in the way physics is done, so that no amount of progress before cannot be compared to progress that came after”.

The present moment seems to be a paradigm shift in the way we do biology and medicine. But trying to describe progress in the next scientific paradigm using language from the current paradigm is bound to lead to absurdities. New abstractions will emerge to help us speak sensibly about our increased rate of progress. We will stop speaking of “50 years of scientific progress compressed into 5”. We will instead look back at the rate of progress we had in 2026 and ask: “why was the rate of scientific discovery so slow?”

---

We will see powerful AI in the years to come, but we will not see recursive self-improvement. We will not see AI that sets its own goals, creates its own values, and exhibits emergent behavior that humans consider to be “progress”.

We might see humans give an AI some narrowly-defined goal, and the AI “recursively self-improves” at that singular task. We might see emergent behavior that is morally and aesthetically neutral, like Conway’s game of life. Neither of these are the relevant sense of “recursive self-improvement” that philosophers, AI researchers, or humanity should care about. These are lesser forms of RSI, and they all fall under the label soft RSI: the combined human/AI productivity loop improving. We will see soft RSI in the years to come. We will not see hard RSI: AI recursively self-improving itself without humans.

If you have any questions, or want me to clarify my view, reach out here or on X.

I hope I never write another article on Substack ever again. Please subscribe anyways.

---

Fin

Acknowledgements

Thank you to Susan Zhang for feedback on an earlier draft, as well as various other friends for encouraging me to write.

Notes

[1] Anthropic Institute, “When AI Builds Itself,” *Anthropic*, 2026, https://www.anthropic.com/institute/recursive-self-improvement.

[2] Alec Radford et al., “Language Models Are Unsupervised Multitask Learners” (OpenAI, 2019), https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

[3] Jared Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv, January 23, 2020, https://doi.org/10.48550/arXiv.2001.08361.

[4] Tom B. Brown et al., “Language Models Are Few-Shot Learners,” *Advances in Neural Information Processing Systems* 33 (2020): 1877-1901, https://doi.org/10.48550/arXiv.2005.14165.

[5] Long Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback,” arXiv, March 4, 2022, https://doi.org/10.48550/arXiv.2203.02155.

[6] OpenAI, “Introducing ChatGPT,” November 30, 2022, https://openai.com/index/chatgpt/.

[7] Kimi Team, “Kimi K2: Open Agentic Intelligence,” arXiv, July 28, 2025, last revised February 3, 2026, https://arxiv.org/abs/2507.20534.

[8] Moonshot AI, “Kimi-K2.7-Code,” Hugging Face, accessed July 1, 2026, https://huggingface.co/moonshotai/Kimi-K2.7-Code.

[9] Dario Amodei, “Machines of Loving Grace,” October 2024, https://darioamodei.com/essay/machines-of-loving-grace.

[10] Terence Tao, “What Is Good Mathematics?” arXiv, February 13, 2007, https://arxiv.org/abs/math/0702396.

[11] William P. Thurston, “On Proof and Progress in Mathematics,” *Bulletin of the American Mathematical Society* 30, no. 2 (1994): 161-177, https://arxiv.org/abs/math/9404236.

[12] Leonardo de Moura and Sebastian Ullrich, “The Lean 4 Theorem Prover and Programming Language,” in *Automated Deduction - CADE 28*, ed. Andre Platzer and Geoff Sutcliffe (Cham: Springer, 2021), 625-635.

[13] Huajian Xin et al., “DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data,” arXiv, May 23, 2024, https://doi.org/10.48550/arXiv.2405.14333.

[14] Z. Z. Ren et al., “DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition,” arXiv, April 30, 2025, https://doi.org/10.48550/arXiv.2504.21801.

[15] Zhihong Shao et al., “DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning,” arXiv, November 27, 2025, https://arxiv.org/pdf/2511.22570.

[16] DeepSeek-AI, “DeepSeek-V3.2-Speciale,” Hugging Face, accessed July 1, 2026, https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale.

[17] OpenAI, “An OpenAI Model Has Disproved a Central Conjecture in Discrete Geometry,” *OpenAI*, May 20, 2026, https://openai.com/index/model-disproves-discrete-geometry-conjecture/.

[18] Dwarkesh Patel, “Ilya Sutskever — We’re moving from the age of scaling to the age of research,” *Dwarkesh Podcast*, November 25, 2025, transcript at 00:18:49, https://www.dwarkesh.com/p/ilya-sutskever-2?open=false#§what-are-we-scaling.

[19] Terence Tao, “Interview with Terence Tao,” *OUPblog*, September 1, 2006, https://blog.oup.com/2006/09/interview_with_/.

[20] Jimmy Xin et al., “AXLE: A Cloud Infrastructure for Lean 4 Theorem Proving Utilities,” arXiv, June 24, 2026, https://arxiv.org/abs/2606.26442.

[21] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo, “Are Emergent Abilities of Large Language Models a Mirage?,” arXiv, April 28, 2023, last revised May 22, 2023, https://doi.org/10.48550/arXiv.2304.15004.

[22] Thomas S. Kuhn, *The Structure of Scientific Revolutions*, 3rd ed. (Chicago: University of Chicago Press, 1996), 6, 10.

[23] Kuhn, *Structure of Scientific Revolutions*, 155-58, 184-86.

[24] Kuhn, *Structure of Scientific Revolutions*, 67-69, 75-76, 154, 156.

[25] Kuhn, *Structure of Scientific Revolutions*, 148-50.

[26] Mubashara Akhtar et al., “When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation,” arXiv, February 18, 2026, last revised June 29, 2026, https://doi.org/10.48550/arXiv.2602.16763.

[27] Jason Wei et al., “Emergent Abilities of Large Language Models,” arXiv, June 15, 2022, https://doi.org/10.48550/arXiv.2206.07682.

[28] Artificial Analysis, “Artificial Analysis Intelligence Index,” accessed July 5, 2026, https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index.

[29] Mike A. Merrill et al., “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces,” arXiv, January 17, 2026, https://doi.org/10.48550/arXiv.2601.11868.

[30] Minyang Tian et al., “SciCode: A Research Coding Benchmark Curated by Scientists,” arXiv, July 18, 2024, https://doi.org/10.48550/arXiv.2407.13168.

[31] David Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark,” arXiv, November 20, 2023, https://doi.org/10.48550/arXiv.2311.12022.

[32] Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv, October 10, 2023, last revised November 11, 2024, https://doi.org/10.48550/arXiv.2310.06770.