The Case Against LLMs Recursively Self Improving Their Way To AGI
Why can't LLMs keep self-improving by themselves until they reach AGI
Welcome to Infinite Curiosity, a weekly newsletter that explores the intersection of Artificial Intelligence and Startups. Tech enthusiasts across 200 countries have been reading what I write. Subscribe to this newsletter for free to receive it in your inbox every week:
I recently came across an interesting paper titled The Matrix: A Bayesian Learning Model for LLMs. It's by Vishal Misra and Siddhartha Dalal from Columbia University.
On one side, AI doomers keep asserting that LLMs will self improve their way to AGI and destroy the world. So using this paper as the basis, Vishal Misra posted a simple thread on X to show why LLMs cannot recursively self improve their way to AGI. I wanted to dig in and find out for myself why that’s the case.
The paper's title starts with "The Matrix". What is this matrix?
LLMs generate text based on probabilistic principles. The matrix here is a conceptual tool to understand how LLMs function. It encodes the probabilities of transitioning from a given sequence of tokens to a next possible token.
To get a bit more technical, the matrix is an abstract representation of a multinomial transition probability matrix. Each row of the matrix represents a specific sequence of tokens. And each column within that row represents the probability of each possible next token that could follow the given sequence. This matrix effectively encapsulates the entire potential output of the LLM based on its training.
Wait a minute, did you say entire potential output? Wouldn't that be enormous?
Yes! It's absurdly huge. The example used in the paper says that if we have 50,000 tokens. And let's say the prompt size is 8,000 tokens. Now we need to list all possible combinations of 8,000 tokens. Let’s think of it as 8,000 empty spaces and each space can be filled with 50,000 unique tokens. So the number of unique rows we're looking at is 50,000 x 50,000 x 50,000 x ..... (8,000 times) = 50,000 ^ 8,000. The size of the matrix then becomes 50,000 x 50,000 ^ 8,000. This number is bigger than all the atoms in the entire universe.
The way LLMs work is that they try to represent this giant probability matrix in a compact way. Now that we know what the matrix does, let's go to the next step.
Why can't LLMs self improve their way to AGI?
The paper makes a number of interesting points. I’m going to summarize the points based on what’s relevant to our discussion here. LLMs are modeled as approximating the enormous matrix we just talked about. Each row represents the multinomial distribution of possible next tokens given a sequence of previous tokens. No LLM can fully represent or learn this matrix due to its enormous size and complexity.
Since the matrix is so large, LLMs approximate it using the finite dataset they're trained on. This training set only captures a subset of all possible text. This limits the model's knowledge to what has been seen in the data.
Text inputs are converted into embeddings, which are vector representations. The transformation from embeddings to probability distributions smoothly interpolates the probabilities based on the embeddings. But it does not extrapolate beyond that.
Text generation by LLMs is treated under a Bayesian learning framework. What does it mean? It means that the generation process involves updating probability distributions based on input prompts. It is constrained to the data the model was trained on and its approximations of the probability matrix. So the model's ability to learn or adapt is confined within these learned distributions. And cannot extend beyond them without external inputs.
LLMs adjust their outputs based on the input prompts within the context of what they have learned during training. This is called in-context learning. It's a form of adaptation to new inputs based on old data, but not an improvement of the model's core capabilities.
Recursive self-improvement would require the model to step outside its predefined probabilistic constraints and generate new knowledge beyond its training data. It is structurally incapable of doing it. Even if an LLM could modify its underlying architecture or training algorithms (which they can't without human intervention), it is still bounded by the initial design and the data it was exposed to. Improvements in capabilities are fundamentally limited to refinements within these bounds rather than true self-evolution.
In essence, LLMs operate within a closed system defined by their training data and structural design. They don’t have the capability to automatically expand beyond these boundaries and embark on a journey of recursive self-improvement.
If you're a founder or an investor who has been thinking about this, I'd love to hear from you. I’m at prateek at moxxie dot vc.
If you are getting value from this newsletter, consider subscribing for free and sharing it with 1 friend who’s curious about AI: