7/20/26
Laird Stewart
7/20/26
One common case for AI's transformative impact is this: human intelligence is a biological computation limited by our skull size. Silicon computation can simulate our brains' computation (or execute better algorithms) with no such limit. If intelligence scales with computation resources, AIs will blow past humans in intelligence. Sam Altman invoked something similar on Tyler Cowen's podcast on Nov. 5, 2025:
Tyler Cowen: Whether or not you agree with it, what is the best this-is-not-a-bubble argument. Is it just the insatiable demand for compute?
Sam Altman: There’s a lot of arguments I’m tempted to give, but I think the intellectually most interesting one is we have no idea how much past human-level intelligence can go and what you can do with it as it does. There’s all the arguments that everyone has made. The one I would like to see people talk about much more is “How are you even supposed to think about vastly superhuman intelligence and the economic impacts of that?”
I dislike this appeal because it leads an undiscerning audience to draw an incorrect conclusion. In this case: AI's economic utility will scale with intelligence and blow past that of humans. This conclusion implicitly assumes a linear relationship between intelligence and impact. However, these relationships aren't linear. Intelligence's impact is limited by other inputs into the decision making process (e.g., data, knowledge) and bounded by fundamental constraints (e.g., information theory, chaos theory). I'll describe my framework of intelligence as a black-box and go into detail about these constraints.
Consider the game of chess. Over many years, humans identified strong opening positions. When AlphaZero came along in 2017, it favored some openings over others, but didn't come up with totally new ones. By following known opening theory human masters can still compete. In the end-game, there are known endgame tablebases which, for positions with few enough pieces, give the result (win, draw, loss). For simple enough endgames, human masters can play to perfection. In the mid-game, once we have left theory and the position is still too complex to exist in a tablebase, AlphaZero's advantage over humans is the most pronounced. However, even then, in each position there are generally a few moves that both the human master and AI would consider. It is rare that an AI plays a move no human would consider, and even rarer that it plays a move no human would understand. As an analogy to human affairs, we can think of three classes of decision making
The first category could involve things like the structure of a democratic government or business – we know which designs tend to work, and it's unlikely there is a much better way to run either. The mid game is like the act of scientific research, business decision making, or military strategy. In most “positions” there is a set of choices which human experts and AI would both find, AI could often pick the best move, but only rarely finds a "move" which humans wouldn't have thought of. The third category involves things like using the simplex algorithm to solve a linear programming problem or applying Kalman filter for linear state estimation.
There isn't an infinite runway for intelligence to improve here – it boils down to occasionally making a better decision. There are only so many decisions a scientist or CEO makes in the course of their career, so the impact is important but bounded. An important question becomes what percentage of real-world decisions are akin to mid-game positions (which I won't examine here).
Let's now consider how an intelligence makes "mid-game" decisions. I'll describe a simple framework of intelligence as a starting point. I'll use Eliezer and Soares's conceptualization of intelligence from If Anyone Builds It, Everyone Dies (IABIED) as a starting point:
In our view, intelligence is about two fundamental types of work: the work of predicting the world, and the work of steering it.
“Prediction” is guessing what you will see (or hear, or touch) before you sense it. If you’re driving to the airport, your brain is succeeding at the task of prediction whenever you anticipate a light turning yellow, or a driver in front of you hitting their brakes.
“Steering” is about finding actions that lead you to some chosen outcome. When you’re driving to the airport, your brain is succeeding at steering when it finds a pattern of street-turns such that you wind up at the airport, or finds the right nerve signals to contract your muscles such that you pull on the steering wheel.”
I'll add that "prediction" could include counterfactual predictions (if I had taken the highway would I be there by now?), conditional predictions (if I brake on this patch of snow will my tires slip?), and non-observed outcomes (is the coolant circulating?).
I like this conception, but I think something is missing. How would you categorize proving a conjecture? Perhaps manipulations are "actions" you use to steer towards a proof. A bit of a stretch. How about proposing a conjecture to begin with?
You could argue that math is a tool in service of the ultimate goal of prediction or steering, so trying to categorize these things isn't well posed to begin with. But we don't re-discover Newton's physics every time we design a new bridge. To draw from and contribute to our existing knowledge base is an important feature of intelligence. My dictionary, for instance, defines intelligence as "the ability to acquire and apply knowledge and skills". Therefore I'd add the creation of knowledge as a third category called Creation. I use knowledge here to encompass useful information and ideas. It could include the synthesis of data (e.g., the GDP of Delaware summarizes data about the whole economy) or scientific theories and mathematical proofs.
Eliezer describes intelligence by its outputs because explaining what it is is much harder. Following suit, I'll extend his conception by proposing data and knowledge as inputs (and add my new "Creation" as an output).
Data comprises observations of the real world. This could include sensory perception (i.e., the flip side of "prediction") or information gathered for analysis (e.g., the evidence brought before the judge in a trial or the results of a clinical trial). Data differs from knowledge in that it must be gathered from the real world.
Intelligence is that active process which turns knowledge and data into a prediction, action, or more knowledge (creation). Like Eliezer I won't go into how intelligence works or what it is.
To ground things I'll briefly describe LLMs in this framework. Knowledge and data are provided to the LLM in two ways: encoded in the network's parameters via pre-training and provided in context at inference time. The latter can be provided in the prompt or retrieved by the model via an action like a call to a search engine or database. Actions are mediated via MCP: the model outputting tokens denoting a specific tool along with JSON parameters. Prediction operates at two scales. At the micro-scale, the model predicts the next token in a sequence. At the macro-scale, the model can construct predictions as a plain language response. For example, "I give Trump a 50% chance of winning the election". Creation operates only at this macro-scale of responses.
Today, LLMs can only compound knowledge (the top arrow in the diagram above) via in-context learning. For example with a scaffold and MCP actions which read/write from a markdown file. The alternative, to update weights directly with new knowledge, is called continual learning and is not routinely used. New data can also only be ingested via in-context learning. Again, a scaffold could be built around this, and is what autonomous laboratory startups like Periodic labs are doing. The alternative to in-context learning is to include the new knowledge in the training run of the next model, but this can take weeks to months.
With this framework we can discuss the limits on intelligence (in particular, its outputs creation, prediction, and action). There are two limiting factors: the input knowledge/data and provable bounds in the generation of knowledge, predictions, and actions.
Knowledge begets knowledge. An Einstein born in Newton's time wouldn't discover relativity because the mathematical building blocks and physics sophistication were not yet in place. Similarly, Kepler needed Tycho Brahe's measurements of elliptical orbits to develop his laws. If we sent a baby Elon Musk to ancient Greece, they would not become space-faring. Similarly, an advanced intelligence today may accelerate our scientific progress, but not leapfrog us to antimatter engines.
Data generation is limited by the number and quality of our sensors deployed in the real world (e.g., the radius of our largest telescope). It is also bottlenecked by how quickly we can set up and run experiments. This is already the case with drug discovery, see What will it take for AI to change drug discovery?
Next, there are fundamental constraints on what knowledge can be created and what outcomes can be predicted. Here are a few that come to my mind (though not exhaustive).
This isn't to say that AI can't generate knowledge, only to point out which limits exist. As I'm writing this Fable just disproved the Jacobian Conjecture. With these limits in mind, we can revisit claims about scaling intelligence. Here are Eliezer's "advantages machines have over biological brains" from IABIED
Sheer speed. ... Even if it took 1,000 transistor operations to do the work of a single neural spike, and even if artificial intelligence was limited to modern hardware, that implies human-quality thinking could be emulated 10,000 times faster on a machine
Copy-and-paste abilities. ... it takes twenty years or longer to grow a single new human and transfer into them a tiny fraction of all human knowledge... Artificial intelligences will eventually inhabit a different world, one where genius could be replicated on demand.
Faster improvements. ... GPUs (...) improve, and the algorithms running on those chips improve, much much much faster than the human species can evolve larger hips.
Larger memories. ... In terms of storage space, [the human brain] defeats most laptops... modern AIs are trained on a significant part of the entirety of human knowledge, and retain a significant portion of [it]
Higher-quality thinking. ... There are many well-measured cases of how humans' minds fall prey to systematic errors. (For example, “motivated skepticism”: the tendency to look for arguments against conclusions you don’t like, but not against ones you do like...
Self-experimentation and self-rewriting capabilities. AIs could make copies of their minds, perform experiments on them, and (if needed) restore the originals from backups. ... AIs could try building slightly different versions of their own minds to see if they perform better...
These advantages speed up the "intelligence" cloud in the middle of the diagram, but run up against the fundamental limits described above. Insofar as the creation of knowledge requires real-world measurements (e.g., validating a new physics theory) the top loop is bounded as well. Real world actions, experiments, measurements will become the bottleneck of progress, Amdahl's law style.
Humans have become only marginally more intelligent than our ancestors, thanks to better diets and more intensive schooling. If a 21st-century family adopted an ancient Greek baby, it would likely do fine academically. Sending Einstein back to ancient Greece as a baby would not jump-start the industrial revolution. Most human progress can be attributed to the compounding of knowledge rather than increasing human intelligence. Here is Eliezer from IABIED
Human intelligence is the source of all our power, all our technology. Even within the human species, small differences in how long groups spend accumulating technological prowess translates into military advantages on the order of "We have guns and they do not."
Therefore AI safety advocates should be particularly concerned with rogue deployments which can clandestinely expand their sensing network, run experiments, and expand their knowledge base.