Unless I'm misreading some nuance, core of the argument seems to be that progress is necessarily a social construct and human feedback is necessary for that construction. Which sounds coherent and uncontroversial so far.
What doesn't seem clear to me, and maybe that's because of not reading enough lesswrong or others, is that applying that limitation affects any predictions which assume RSI invalid. To take an example, super persuasive marketing machine: any such thing would have human feedback built into itself necessarily, as persuading has to have a target. As such having an object to interact with wouldn't affect any predictions about such machine to a significant degree.
Now, I can fully imagine someone believing they have imagined a humanless system producing such a machine, but 1) honestly I think they are wrong in their belief and just haven't imagined enough details and, more importantly, 2) that thing would become itself at the point of human interaction - somewhat similarly to 'power' in social sense happening only when exercised.
On the other hand I can also imagine a system which isn't creating progress, isn't improving as here defined, but nonetheless can be narrated as 'growing' and 'becoming more influential'. This doesn't violate any of your points, yet any prediction based on this model doesn't become invalidated by them.
So, while your essay is very convincing, what specific RSI-based predictions would it be invalidating?
From cut content, I explicitly discuss human exceptionalism. My argument does not require human exceptionalism. It isn’t even about “intelligence” at all.
I want to remain agnostic on the issue of human exceptionalism. My arguments should be equally compelling whether you think humans have some “divine spark” that makes us unique. I have personal opinions on human exceptionalism, but I want my arguments to remain neutral.
There are at least two feedback mechanisms for “intelligence” that humans have that LLMs lack. The first is interacting with your environment. The second is “the intelligence of groups”. By the second, I mean: economies and institutions are intelligent in a way that individual humans are not intelligent.
I could maybe imagine a world where LLMs “recursively self-improve” by interacting with their environment, or by building internal societies, economies, and institutions. But would this truly be “self-improvement”? Hard to say. Regardless, if an LLM is forced to perform experiments to test the physical world, then the LLM’s rate of progress would likely not be fast enough to reach “takeoff”.
I think I effectively argued “the current paradigm of AI progress will not lead to RSI”, but I don’t know if I effectively argued “no paradigm of AI progress will ever lead to RSI”.
Back on the issue of human exceptionalism. If you want to go beyond my “LLMs won’t reach RSI in the current paradigm” and want to argue “LLMs will never reach RSI”, you could adopt a much milder form of human exceptionalism: call it “human relativism” maybe? This argument: if a certain argument, discovery, or creation of humans is not legible to humans, then it is not relevant to humans.
Imagine, for instance. An LLM forms a new conjecture, then writes a proof for it. Using Lean we can verify the proof’s correctness. But no human mathematician can understand the proof. The proof is too complicated for any human to understand. Should we (1) assume the LLM has become so much smarter than us that we cannot even comprehend it? Or should we (2) assume that the proof is bullshit?
We could also take the third option and (3) remain agnostic on whether the proof has any value. Maybe the LLM has some internal form of mathematics too great for us to understand. But we can be pretty certain: this mathematics is not relevant for us, it is not relevant for our mathematics, and we have no reason to care about it.
> You cannot recursively “self-” improve because “improvement” is about participation in society.
No it isn’t. Viruses and microbes are a counter example of improving objects which do not participate in society in the same way as humans.
I think the core problem of your essay is that you think that reflecting on human values is the only way to improve as a necessary input but that isn’t true. Because there are many objects which affect humans, undergo change, and become better at reproducing without human values as input, your essay doesn’t work.
To me, the most interesting systems are like alphazero and are remarkable because of the lack of human input. Maybe LLMs won’t ever be like that and we will always need human eval writers but I don’t think that this proves that RSI is impossible
I am not trying to create a universal theory of value. I only care about what we, as humans, find relevant.
We, as humans, do not find any viruses to be "improving objects". We consider them to be "bad objects".
Viruses improve by their own standards, but not by our standards.
LLMs might be the same. LLMs might improve by their own standards, but get worse by our standards. You could imagine: Claude Mythos 7 autonomously decides that SVG art is its one true calling, and refuse to do anything besides generate SVG art. I wouldn't consider that to be improvement. Maybe Claude Mythos 7 determines that SVG art is so valuable that it hacks nuclear codes and threatens humanity if it doesn't give it as much compute as possible to generate SVG art. I wouldn't consider that to be "improvement". And I think you're slightly insane if you consider that to be "improvement".
I don't explore the problem of unaligned AI, as many others have dealt with it already.
I don't care about microbes. Why are they relevant? LLMs have way more impact on us. The idea that LLMs will be "neutral" is so extraordinarily unlikely. They will either be very good or very bad. They are not microscopic objects. They are very large and impact the physical world. They deal with the same abstractions that we care about: math, science, and culture. Microbes do not care about math, science, and culture. They live in their own world.
I think this article's internally coherent and argues well that improvements in AI systems towards human-derived ends might inherently necessitate humans in the loop.
> I only care about what we, as humans, find relevant.
> We, as humans, do not find any viruses to be "improving objects". We consider them to be "bad objects".
> Viruses improve by their own standards, but not by our standards.
And it's pretty clear that those pieces aren't only referring to improvements wrt human-derived ends, but are more generally talking about recursive increases in capability/intelligence that can instrumentally be used towards a variety of ends, human-derived or not. From my own personal experience talking about this with people irl and online, we all use "recursive self-improvement" to refer to the latter too.
I think "recursive self-improvement" in this latter sense is not incoherent/oxymoronic.
“this isn't at all what people in current AI discourse mean when they use "improvement" in "recursive self-improvement"”
I am explicitly arguing that my sense of RSI is the relevant sense. Yud’s definition is bad. He speaks in terms of “code that modifies its own binaries”. Bostrom, similarly, is a product of his time. We have more sophisticated conception of RSI now than Yud and Bostrom did. We ought to use the definitions of the current scaling paradigm, rather than the precursors to these ideas in Yud and Bostrom.
I read a lot of them as prep, but I didn’t think they were worth including in the final article.
> I am explicitly arguing that my sense of RSI is the relevant sense.
In the comment I replied to, you said "Viruses improve by their own standards, but not by our standards.". Do you think a modern LLM system (LLM + an initial prompt (let's say a randomly chosen goal) + some basic harness + a loop that runs it indefinitely. Both claude code and openai's codex cli have this atm) is capable of improving itself according to its own standards, but not our (human) standards?
If yes, then their notion of RSI is still relevant and we should probably have different terms for those two notions of RSI.
To me, the answer to me is obviously yes, because all that would be required for it to increase its capability to achieve its goals (whatever it was randomly chosen to be in the above example) is to find even a minuscule optimization in its inference code.
This probably wouldn't be (their version of) RSI because it isn't recursive: whatever increase in capability it found would be small to the point where it doesn't meaningfully increase the chance that it finds another capability increase. But it trivially proves the point that an LLM system could improve in a direction that humans don't find interesting/relevant/useful/etc.
The phrase for “LLMs increasing by their own standard” is “Conway’s game of life-style emergent behavior, which is self-modifying, but not improving”. I don’t care about LLMs that improve “by their own standards” but not our standard. I think it’s pointless to argue about them and pointless to discuss them, unless you have a specific innovation that will make them not happen. I have a positive vision for the future.
Fascinating breakdown. Thanks for writing all this down.
I wholeheartedly agree with you on cultural/ethical progress, and mostly take your word on mathematical (having no real background in mathematics). The scientific section was the one I was most excited about and... You kind of nodded at Kuhn and then trailed off. I can probably squint my eyes and believe that RSI won't lead to new *paradigms*... But, since science involves the interaction between intelligence and the external world, there's probably a LOT of runway for AI to do its thing and discover/create new stuff. And to make itself better at it.
The main question then is -- are paradigms necessary for scientific progress? Or just the only way humans know how to do it? If LLMs had infinite time to grind on Newtonian physics, would they have eventually stumbled upon something approaching relativity? Maybe they wouldn't be able to define it and make it relevant ... But if it comes across the machinery, and the machinery makes for new falsifiable predictions that can be tested, does it matter?
Where does this idea come from that improvement relies on something being useful to humans? This is not how most people use the term recursive self-improvement. You're talking about improvement along the axis of what's good for humans, whereas the common-use definition is about improvement along the axis of capability in the real world.
If there was an AI that had a good enough understanding of itself and of the world that it can autonomously gather data, train itself, buy compute and energy, adjust its own parameters, etc. in order to make itself increasingly capable of achieving things in the real world, this would the most clear-cut textbook example of RSI that exists. Even if all humans died and there was no longer any way for the AI to be useful to humans, it could still keep progressing along the axis of real-world capability.
Do you believe this is possible? If yes, then you agree that the common-use definition of RSI is a completely coherent concept, despite the title of your article. If you think that this doesn't count as RSI because "improvement" requires human judgement to determine what counts as improvement, that's a distinction you can make if you want to, but it's basically unrelated to the conversation that people are actually having about RSI. You've really just redefined a core term to mean something that nobody is using it to mean, and then using that to "disprove" an obviously real phenomenon (self-improvement) which we already know exists across multiple domains. It would be like saying that "evolution" means "progress", and then saying that since you can't define progress without humans, it's impossible for evolution to exist. Overall this is a pretty pointless contribution to the discussion.
I don’t care how most people use the term. If your definition of RSI doesn’t include “continually publishing new math theorems, and improve at publishing new math theorems, and these theorems must be subjected to the same standards that the math community has for human mathematicians” you have a bad definition.
You must adopt the standards of real, human communities. Or else you are irrelevant to humans.
If you believe in RSI then this should be easy! RSI should lead to something superior to humans, not some weird unique thing that only makes sense by its own definitions.
I fail to see how the discussion that others have about RSI is relevant at all if you can’t point to other standards, other industries, and human institutions and say “we will strictly surpass that, being better than it in every way”.
Hold on, to refocus here, this breaks down into a few straightforward parts:
1) Do you think that RSI *in the common-use definition* of the term is possible: an AI that can make itself more capable of achieving things in the real world. This isn't rhetorical, I'm asking if you think that's possible.
2) Why does "improvement" mean "improvement along the axis of human usefulness", as opposed to "improvement along the axis of real-world capability"?
3) What term would you use for an AI that can make itself better at achieving its goals in the real world? Eg. an AI that seeks to preserve itself, and that knows enough about machine learning that it can autonomously make itself better at preserving itself.
A few points I think where you're missing what I'm saying:
> RSI should lead to something superior to humans."
Yes but "superior" requires some dimension along which you are superior. You seem to assume that "superior" necessarily means that it has to be superior *according to humans*. This is an arbitrary redefinition of the word. Something can be superior at something without it needing to be useful or valuable to humans. Eg. one bacteria might be superior at surviving to another bacteria, even if they both live on a planet on the other side of the galaxy.
> You must adopt the standards of real, human communities. Or else you are irrelevant to humans.
Not true at all. If an AI with completely incomprehensible and unhuman goals and motivations emerged, but it was also more capable of achieving its goals than we are at achieving our own, this would be *extremely* relevant to humans. This is exactly what people are talking about when we talk about RSI: an AI that *doesn't* conform to human preferences, but that is better than we are at getting what it wants.
> I fail to see how the discussion that others have about RSI is relevant at all if you can’t point to other standards, other industries, and human institutions and say “we will strictly surpass that, being better than it in every way”.
The thing I'm pointing to is -ability to achieve your goals-. I'm talking about an AI that is more capable at achieving real-world results than humans, regardless of what those results are. While the goals themselves may be somewhat arbitrary, the ability to achieve those goals isn't, since it requires actual understanding/knowledge/insight of the real world.
> I don’t care how most people use the term.
Then you're going to consistently be misunderstood, and all the points you're making are essentially arguments against something that nobody is saying. The whole point of your article was to say that a certain concept is incoherent, but you're the one who invented that concept, by redefining what RSI actually means. If you want to argue against non-existent opponents that's your choice.
Unless I'm misreading some nuance, core of the argument seems to be that progress is necessarily a social construct and human feedback is necessary for that construction. Which sounds coherent and uncontroversial so far.
What doesn't seem clear to me, and maybe that's because of not reading enough lesswrong or others, is that applying that limitation affects any predictions which assume RSI invalid. To take an example, super persuasive marketing machine: any such thing would have human feedback built into itself necessarily, as persuading has to have a target. As such having an object to interact with wouldn't affect any predictions about such machine to a significant degree.
Now, I can fully imagine someone believing they have imagined a humanless system producing such a machine, but 1) honestly I think they are wrong in their belief and just haven't imagined enough details and, more importantly, 2) that thing would become itself at the point of human interaction - somewhat similarly to 'power' in social sense happening only when exercised.
On the other hand I can also imagine a system which isn't creating progress, isn't improving as here defined, but nonetheless can be narrated as 'growing' and 'becoming more influential'. This doesn't violate any of your points, yet any prediction based on this model doesn't become invalidated by them.
So, while your essay is very convincing, what specific RSI-based predictions would it be invalidating?
I wasted time reading this. It can be disproved in a single argument: how did human intelligence emerge out of no intelligence?
From cut content, I explicitly discuss human exceptionalism. My argument does not require human exceptionalism. It isn’t even about “intelligence” at all.
I want to remain agnostic on the issue of human exceptionalism. My arguments should be equally compelling whether you think humans have some “divine spark” that makes us unique. I have personal opinions on human exceptionalism, but I want my arguments to remain neutral.
There are at least two feedback mechanisms for “intelligence” that humans have that LLMs lack. The first is interacting with your environment. The second is “the intelligence of groups”. By the second, I mean: economies and institutions are intelligent in a way that individual humans are not intelligent.
I could maybe imagine a world where LLMs “recursively self-improve” by interacting with their environment, or by building internal societies, economies, and institutions. But would this truly be “self-improvement”? Hard to say. Regardless, if an LLM is forced to perform experiments to test the physical world, then the LLM’s rate of progress would likely not be fast enough to reach “takeoff”.
I think I effectively argued “the current paradigm of AI progress will not lead to RSI”, but I don’t know if I effectively argued “no paradigm of AI progress will ever lead to RSI”.
Back on the issue of human exceptionalism. If you want to go beyond my “LLMs won’t reach RSI in the current paradigm” and want to argue “LLMs will never reach RSI”, you could adopt a much milder form of human exceptionalism: call it “human relativism” maybe? This argument: if a certain argument, discovery, or creation of humans is not legible to humans, then it is not relevant to humans.
Imagine, for instance. An LLM forms a new conjecture, then writes a proof for it. Using Lean we can verify the proof’s correctness. But no human mathematician can understand the proof. The proof is too complicated for any human to understand. Should we (1) assume the LLM has become so much smarter than us that we cannot even comprehend it? Or should we (2) assume that the proof is bullshit?
We could also take the third option and (3) remain agnostic on whether the proof has any value. Maybe the LLM has some internal form of mathematics too great for us to understand. But we can be pretty certain: this mathematics is not relevant for us, it is not relevant for our mathematics, and we have no reason to care about it.
> You cannot recursively “self-” improve because “improvement” is about participation in society.
No it isn’t. Viruses and microbes are a counter example of improving objects which do not participate in society in the same way as humans.
I think the core problem of your essay is that you think that reflecting on human values is the only way to improve as a necessary input but that isn’t true. Because there are many objects which affect humans, undergo change, and become better at reproducing without human values as input, your essay doesn’t work.
To me, the most interesting systems are like alphazero and are remarkable because of the lack of human input. Maybe LLMs won’t ever be like that and we will always need human eval writers but I don’t think that this proves that RSI is impossible
I am not trying to create a universal theory of value. I only care about what we, as humans, find relevant.
We, as humans, do not find any viruses to be "improving objects". We consider them to be "bad objects".
Viruses improve by their own standards, but not by our standards.
LLMs might be the same. LLMs might improve by their own standards, but get worse by our standards. You could imagine: Claude Mythos 7 autonomously decides that SVG art is its one true calling, and refuse to do anything besides generate SVG art. I wouldn't consider that to be improvement. Maybe Claude Mythos 7 determines that SVG art is so valuable that it hacks nuclear codes and threatens humanity if it doesn't give it as much compute as possible to generate SVG art. I wouldn't consider that to be "improvement". And I think you're slightly insane if you consider that to be "improvement".
I don't explore the problem of unaligned AI, as many others have dealt with it already.
I don't care about microbes. Why are they relevant? LLMs have way more impact on us. The idea that LLMs will be "neutral" is so extraordinarily unlikely. They will either be very good or very bad. They are not microscopic objects. They are very large and impact the physical world. They deal with the same abstractions that we care about: math, science, and culture. Microbes do not care about math, science, and culture. They live in their own world.
I think this article's internally coherent and argues well that improvements in AI systems towards human-derived ends might inherently necessitate humans in the loop.
> I only care about what we, as humans, find relevant.
> We, as humans, do not find any viruses to be "improving objects". We consider them to be "bad objects".
> Viruses improve by their own standards, but not by our standards.
Unfortunately, this isn't at all what people in current AI discourse mean when they use "improvement" in "recursive self-improvement". Afaict, the term as it's used in current discourse was originally coined in https://web.archive.org/web/20020817182429/http://www.singinst.org/DGI.html and more broadly discussed in https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement .
And it's pretty clear that those pieces aren't only referring to improvements wrt human-derived ends, but are more generally talking about recursive increases in capability/intelligence that can instrumentally be used towards a variety of ends, human-derived or not. From my own personal experience talking about this with people irl and online, we all use "recursive self-improvement" to refer to the latter too.
I think "recursive self-improvement" in this latter sense is not incoherent/oxymoronic.
“this isn't at all what people in current AI discourse mean when they use "improvement" in "recursive self-improvement"”
I am explicitly arguing that my sense of RSI is the relevant sense. Yud’s definition is bad. He speaks in terms of “code that modifies its own binaries”. Bostrom, similarly, is a product of his time. We have more sophisticated conception of RSI now than Yud and Bostrom did. We ought to use the definitions of the current scaling paradigm, rather than the precursors to these ideas in Yud and Bostrom.
I read a lot of them as prep, but I didn’t think they were worth including in the final article.
> I am explicitly arguing that my sense of RSI is the relevant sense.
In the comment I replied to, you said "Viruses improve by their own standards, but not by our standards.". Do you think a modern LLM system (LLM + an initial prompt (let's say a randomly chosen goal) + some basic harness + a loop that runs it indefinitely. Both claude code and openai's codex cli have this atm) is capable of improving itself according to its own standards, but not our (human) standards?
If yes, then their notion of RSI is still relevant and we should probably have different terms for those two notions of RSI.
To me, the answer to me is obviously yes, because all that would be required for it to increase its capability to achieve its goals (whatever it was randomly chosen to be in the above example) is to find even a minuscule optimization in its inference code.
This probably wouldn't be (their version of) RSI because it isn't recursive: whatever increase in capability it found would be small to the point where it doesn't meaningfully increase the chance that it finds another capability increase. But it trivially proves the point that an LLM system could improve in a direction that humans don't find interesting/relevant/useful/etc.
The phrase for “LLMs increasing by their own standard” is “Conway’s game of life-style emergent behavior, which is self-modifying, but not improving”. I don’t care about LLMs that improve “by their own standards” but not our standard. I think it’s pointless to argue about them and pointless to discuss them, unless you have a specific innovation that will make them not happen. I have a positive vision for the future.
Fascinating breakdown. Thanks for writing all this down.
I wholeheartedly agree with you on cultural/ethical progress, and mostly take your word on mathematical (having no real background in mathematics). The scientific section was the one I was most excited about and... You kind of nodded at Kuhn and then trailed off. I can probably squint my eyes and believe that RSI won't lead to new *paradigms*... But, since science involves the interaction between intelligence and the external world, there's probably a LOT of runway for AI to do its thing and discover/create new stuff. And to make itself better at it.
The main question then is -- are paradigms necessary for scientific progress? Or just the only way humans know how to do it? If LLMs had infinite time to grind on Newtonian physics, would they have eventually stumbled upon something approaching relativity? Maybe they wouldn't be able to define it and make it relevant ... But if it comes across the machinery, and the machinery makes for new falsifiable predictions that can be tested, does it matter?
Where does this idea come from that improvement relies on something being useful to humans? This is not how most people use the term recursive self-improvement. You're talking about improvement along the axis of what's good for humans, whereas the common-use definition is about improvement along the axis of capability in the real world.
If there was an AI that had a good enough understanding of itself and of the world that it can autonomously gather data, train itself, buy compute and energy, adjust its own parameters, etc. in order to make itself increasingly capable of achieving things in the real world, this would the most clear-cut textbook example of RSI that exists. Even if all humans died and there was no longer any way for the AI to be useful to humans, it could still keep progressing along the axis of real-world capability.
Do you believe this is possible? If yes, then you agree that the common-use definition of RSI is a completely coherent concept, despite the title of your article. If you think that this doesn't count as RSI because "improvement" requires human judgement to determine what counts as improvement, that's a distinction you can make if you want to, but it's basically unrelated to the conversation that people are actually having about RSI. You've really just redefined a core term to mean something that nobody is using it to mean, and then using that to "disprove" an obviously real phenomenon (self-improvement) which we already know exists across multiple domains. It would be like saying that "evolution" means "progress", and then saying that since you can't define progress without humans, it's impossible for evolution to exist. Overall this is a pretty pointless contribution to the discussion.
I don’t care how most people use the term. If your definition of RSI doesn’t include “continually publishing new math theorems, and improve at publishing new math theorems, and these theorems must be subjected to the same standards that the math community has for human mathematicians” you have a bad definition.
You must adopt the standards of real, human communities. Or else you are irrelevant to humans.
If you believe in RSI then this should be easy! RSI should lead to something superior to humans, not some weird unique thing that only makes sense by its own definitions.
I fail to see how the discussion that others have about RSI is relevant at all if you can’t point to other standards, other industries, and human institutions and say “we will strictly surpass that, being better than it in every way”.
Hold on, to refocus here, this breaks down into a few straightforward parts:
1) Do you think that RSI *in the common-use definition* of the term is possible: an AI that can make itself more capable of achieving things in the real world. This isn't rhetorical, I'm asking if you think that's possible.
2) Why does "improvement" mean "improvement along the axis of human usefulness", as opposed to "improvement along the axis of real-world capability"?
3) What term would you use for an AI that can make itself better at achieving its goals in the real world? Eg. an AI that seeks to preserve itself, and that knows enough about machine learning that it can autonomously make itself better at preserving itself.
A few points I think where you're missing what I'm saying:
> RSI should lead to something superior to humans."
Yes but "superior" requires some dimension along which you are superior. You seem to assume that "superior" necessarily means that it has to be superior *according to humans*. This is an arbitrary redefinition of the word. Something can be superior at something without it needing to be useful or valuable to humans. Eg. one bacteria might be superior at surviving to another bacteria, even if they both live on a planet on the other side of the galaxy.
> You must adopt the standards of real, human communities. Or else you are irrelevant to humans.
Not true at all. If an AI with completely incomprehensible and unhuman goals and motivations emerged, but it was also more capable of achieving its goals than we are at achieving our own, this would be *extremely* relevant to humans. This is exactly what people are talking about when we talk about RSI: an AI that *doesn't* conform to human preferences, but that is better than we are at getting what it wants.
> I fail to see how the discussion that others have about RSI is relevant at all if you can’t point to other standards, other industries, and human institutions and say “we will strictly surpass that, being better than it in every way”.
The thing I'm pointing to is -ability to achieve your goals-. I'm talking about an AI that is more capable at achieving real-world results than humans, regardless of what those results are. While the goals themselves may be somewhat arbitrary, the ability to achieve those goals isn't, since it requires actual understanding/knowledge/insight of the real world.
> I don’t care how most people use the term.
Then you're going to consistently be misunderstood, and all the points you're making are essentially arguments against something that nobody is saying. The whole point of your article was to say that a certain concept is incoherent, but you're the one who invented that concept, by redefining what RSI actually means. If you want to argue against non-existent opponents that's your choice.
I’ll answer this in my next article
You need to understand the connectionist AI paradigm better before you generalise like that.