<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://xiangli-shaun.github.io/feed.xml" rel="self" type="application/atom+xml"/><link href="https://xiangli-shaun.github.io/" rel="alternate" type="text/html" hreflang="en"/><updated>2026-09-26T20:08:59+00:00</updated><id>https://xiangli-shaun.github.io/feed.xml</id><title type="html">blank</title><subtitle>Assistant Professor at Massachusetts General Hospital and Harvard Medical School. Medical foundation models, generative AI, and expert-machine alignment. </subtitle><entry><title type="html">A 2016 Roadmap Comes Full Circle</title><link href="https://xiangli-shaun.github.io/blog/2026/a-2016-roadmap-comes-full-circle/" rel="alternate" type="text/html" title="A 2016 Roadmap Comes Full Circle"/><published>2026-09-19T00:00:00+00:00</published><updated>2026-09-19T00:00:00+00:00</updated><id>https://xiangli-shaun.github.io/blog/2026/a-2016-roadmap-comes-full-circle</id><content type="html" xml:base="https://xiangli-shaun.github.io/blog/2026/a-2016-roadmap-comes-full-circle/"><![CDATA[<p>In 2016, I distilled my research during PhD study into a perspective paper <em>Functional Neuroimaging in the New Era of Big Data</em>. In that paper, I presented my vision for the next ten years in its Fig. 1: a roadmap built on two targets — turning neuroimaging big data into knowledge (discovery), and building better informatics systems scaled to such big data (sharing).</p> <p>This year, that vision took a concrete step forward: we published our fMRI foundation model in <em>Nature Biomedical Engineering</em>, a milestone for scaling up neuroimaging in size, scope, and long-term impact.</p> <p>From there, we launched <strong>NeuroDiscovery</strong>, an agentic AI harness that runs automatic, end-to-end neuroimaging research, continuously discovering knowledge from neuroimaging data 24/7.</p> <p>And now the arc continues. As co-PI with Lifang He, and with the support from NIBIB, we’ll pursue full-lifecycle research on neuroimaging data analytics solutions — bringing that 2016 roadmap full circle, from discovery to sharing.</p>]]></content><author><name></name></author><category term="notes"/><category term="neuroimaging"/><category term="foundation-models"/><category term="agentic-ai"/><summary type="html"><![CDATA[A perspective paper written at the end of my PhD set two targets for the next ten years — discovery and sharing. This year both moved.]]></summary></entry><entry><title type="html">The Six-Hundred-Billion-Dollar Question: The Answer Is Not in the Price</title><link href="https://xiangli-shaun.github.io/blog/2026/the-six-hundred-billion-dollar-question/" rel="alternate" type="text/html" title="The Six-Hundred-Billion-Dollar Question: The Answer Is Not in the Price"/><published>2026-07-07T00:00:00+00:00</published><updated>2026-07-07T00:00:00+00:00</updated><id>https://xiangli-shaun.github.io/blog/2026/the-six-hundred-billion-dollar-question</id><content type="html" xml:base="https://xiangli-shaun.github.io/blog/2026/the-six-hundred-billion-dollar-question/"><![CDATA[<blockquote> <p>This blog post is machine-translated from the original Chinese version. Link: <a href="https://mp.weixin.qq.com/s/YcHE_AX1vSjriCkwluatqw">https://mp.weixin.qq.com/s/YcHE_AX1vSjriCkwluatqw</a></p> </blockquote> <p>In early 2023, processing a million tokens through the GPT-4 API cost thirty dollars. By the end of 2025, a model of equivalent capability had fallen below ten cents per million — a 99.7% collapse in three years. It is hard to find a product in the history of commerce whose unit price has gone to zero at this speed.</p> <p>In 2024, David Cahn of Sequoia Capital asked a question: at the current rate of infrastructure investment, AI companies would need to generate roughly six hundred billion dollars in annual revenue to justify what was being spent. When he first ran the arithmetic in 2023, the gap was two hundred billion. A year later it had tripled. Now, in 2026, the gap has not closed; it is still widening. Allianz Research puts the divergence between AI capital expenditure and revenue growth at 46%, above the 32% that preceded the telecom bust of 2001.</p> <p>The larger problem is that AI has stopped being <em>cool</em>, and stopped being <em>mysterious</em>. Benedict Evans, formerly of a16z, offers a sharper judgment: large models are becoming a commodity. Models are extremely hard to build, he grants — but so were flat panel displays, and nobody ever got rich merely by manufacturing displays. On his account the frontier labs have no moat and no defensibility beyond their ability to raise capital, have found product-market fit nowhere outside coding and marketing, and have no real product at all: only a text box and an API.</p> <p>Read only these figures and these opinions and the conclusion looks settled: AI is a bubble, and the underlying AI layer will be the least valuable product in it.</p> <p><strong>My view is that this conclusion is reached too quickly.</strong> Too quickly because it conflates <em>price</em> with <em>value</em>, and conflates <em>technology</em> with <em>application</em>.</p> <p>In fiscal 2026, NVIDIA’s data center business reached one hundred ninety-four billion dollars in revenue. Meta and Oracle plan to double capital expenditure in 2026. Combined annual capex across Microsoft, Alphabet, AWS and Meta rose from roughly two hundred twenty billion in 2024 to three hundred sixty billion in 2025, to seven hundred billion in 2026 guidance. Meanwhile the optical communications market inside AI data centers is growing at 60% a year, and silicon photonics shipments are projected to grow more than fivefold by 2030. This is not speculative capital chasing a narrative. These are the largest technology companies on earth betting hard currency on an economic transition they believe is irreversible. They may well be wrong about the <em>rate</em> of spending — history shows that such errors are fatal — but they are unlikely to be wrong about its <em>direction</em>. There is an old line on Wall Street: don’t fight the tape. We can question the valuations, question the pace, question near-term returns. But we should first ask ourselves: who is closer to the answer — me, or the several hundred financial engineers inside those firms?</p> <p>The bubble thesis reaches most often for the fiber optic analogy of the 1990s: telecom companies laid far more network than demand required, investors lost money, many firms went bankrupt, and so AI infrastructure will travel the same road. But the analogy has a structural flaw. <strong>Fiber is a physical asset laid once.</strong> Once laid, it is laid; it becomes a sunk cost, anyone can use it, and whoever installed it retains no ongoing pricing power. WorldCom did not fail because fiber had no value; it failed because once fiber exists, its supplier loses the right to price it. Railroads follow the same logic: after the track is down, the money is made by those who run trains on it, not by those who laid it.</p> <p>Compute is not fiber. <strong>Compute is a continuously consumed resource, not infrastructure laid once.</strong> Training and inference burn through chips every day — literally — and each generation carries a significant performance gap over the last, so that yesterday’s silicon cannot simply substitute for tomorrow’s demand. This means the commercial logic of a compute supplier resembles an energy company (continuous supply, continuous billing, technical iteration manufacturing replacement demand) far more than a railroad company (build it, then wait for others to come). NVIDIA’s one hundred ninety-four billion dollars in fiscal 2026 data center revenue is not the echo of a sunk investment. It is a consumption market in continuous operation.</p> <p><strong>So my first judgment is this:</strong> compute, power, and optical interconnect — the substrate layers — retain enormous value potential, and unlike the infrastructure bubbles of history, the continuously consumed nature of compute gives its suppliers far stronger value capture. But the price collapse is real: not at the hardware layer, at the model layer. The price of a token is going to zero, and what that tells us is that the center of value capture is shifting from <em>who owns the model</em> to <em>who can build something real with it</em>. Evans says as much himself: the money will flow to the application layer — to companies that embed AI in specific workflows, own distribution, and solve actual problems.</p> <p>So who is making money at the application layer?</p> <p>The answer is: almost nobody.</p> <p>A RAND Corporation report from late 2025 found that 80.3% of enterprise AI projects failed to deliver their promised business value. A third were abandoned before reaching production; nearly three in ten shipped but fell short of expectations; nearly a fifth run to this day and will never recover their cost. MIT’s Project NANDA found in July 2025 that 95% of organizations deploying generative AI saw no measurable return whatsoever. McKinsey’s 2025 global AI survey reports that 88% of organizations use AI in at least one function, while only 39% see any EBIT impact — and McKinsey notes, pointedly, that the organizations seeing real financial return had redesigned their end-to-end workflows <em>before</em> selecting a model technology.</p> <p>a16z’s early-2026 analysis places its bet on what it calls the <em>thick app</em>: not a thin wrapper over a large model, but multi-model orchestration, autonomy control, context engineering. The traditional SaaS logic no longer applies; AI has to eat labor.</p> <p>But why have so many projects failed? Why did McKinsey find that the decisive factor was not technology but workflow redesign? <strong>Because what the overwhelming majority of domain AI projects are doing is not solving a technical problem. They are attempting to force a form of knowledge that cannot be formalized into formal shape.</strong></p> <p>In an earlier piece — <em>AI at the Boundary: What AI Truly Cannot Do Is Precisely What Lives on the Computer</em>, developed formally in <a href="https://arxiv.org/abs/2605.14407">Metis AI: The Overlooked Middle Zone Between AI-Native and World-Movers</a> — I proposed a frame: the Greeks distinguished <em>techne</em>, knowledge that can be encoded and taught, from <em>metis</em>, the practical, situated, relational wisdom that resists codification. <strong>AI is, in essence, a <em>techne</em> machine.</strong> It performs astonishingly on tasks high in <em>techne</em>, and it fails systematically on tasks high in <em>metis</em>.</p> <p>That piece listed five structural characteristics for judging the <em>metis</em> content of a task: <em>consequential irreversibility</em>, <em>relational irreducibility</em>, <em>normative open texture</em>, <em>adversarial co-evolution</em>, and <em>accountability anchoring</em>. Look again at the enterprise AI failures and the pattern maps almost one to one.</p> <p>This is why domain AI cannot simply be dismissed as a <em>wrapper</em>. The real question is not whether the technology is strong enough, but whether <em>metis</em> can be put inside it. And the accumulation of <em>metis</em> does not track the speed of technical iteration — now measured in months, sometimes weeks — but the speed of institutional iteration, measured in years and sometimes decades. For a medical AI to deliver genuine clinical decision support, it needs more than a better model: it needs clinical trials, the evolution of regulatory frameworks, the rebuilding of trust between physician and patient, the maturation of liability. None of this has a fast-forward button.</p> <p><strong>So my second judgment is this:</strong> the current time window is not long enough for a domain AI that fundamentally transforms productivity to have appeared. Not because AI cannot do it, but because <em>metis</em> accumulates slowly, and we have only just begun.</p> <p>Exactly one domain has already produced genuine productivity: writing code.</p> <p>Cursor reached two billion dollars in annual revenue faster than any SaaS product before it — from one million to two billion in twenty-eight months. Claude Code went from zero to two and a half billion in annualized revenue in nine. JetBrains’ early-2026 survey found 90% of developers using at least one AI tool at work, with individual productivity gains between 21% and 55%.</p> <p>This is not an accident. Return to the <em>metis</em> frame: <strong>of all professional work, writing code has the lowest <em>metis</em> content.</strong> Information is highly concentrated — it is all on the screen. The rules are explicit: syntax and the compiler. Results are verifiable: the tests pass or they do not. It satisfies the conditions for a <em>techne</em> machine almost perfectly. Consequences are reversible before deployment; adversarial co-evolution is very weak, since code does not change its behavior because you wrote it; and the structure of responsibility is relatively clear — you wrote it, so you wrote it. Even here, in the lowest-<em>metis</em> domain we have, <em>metis</em> still generates problems, as I argued in <a href="/blog/2026/when-code-begins-to-write-itself/">When Code Begins to Write Itself</a>.</p> <p>The common view now is that AI has delivered real productivity only in programming, and from this people doubt whether AI is genuinely useful anywhere else. <strong>But this reverses the causation.</strong> AI succeeded in programming first <em>precisely because</em> programming is the professional domain lowest in <em>metis</em>. Other domains are not places AI can never reach; they are places where the <em>metis</em> requires far longer to accumulate, embed, and iterate. Put differently: AI’s success in programming is its floor, not its ceiling.</p> <p>An engine that reaches the level of general human intelligence will necessarily produce value across every human domain. This is not faith; it is structural inference. In radiology we have a notion called <em>Aunt Minnie</em>: if someone walks into the room and her face, her voice, her bearing and her habits are all your Aunt Minnie’s, then she is your Aunt Minnie. AI is the same. If a system can understand language, handle knowledge, carry out reasoning, form plans, solve problems, create content and call tools — and if across a widening range of real work settings its performance approaches or exceeds that of an ordinary human — then in economic terms it already possesses the functional properties of a person, and it will almost certainly produce value across every human domain, inevitably replacing, amplifying, or restructuring a great deal of work formerly done by people.</p> <p>But we must remember that <strong>time is not a variable one skips lightly</strong>. Medical AI waits for systems of clinical validation to catch up. Legal AI waits for frameworks of responsibility to mature. Educational AI waits for pedagogy to shift from <em>transmitting knowledge</em> to <em>constructing capability</em>. Every domain has its own rhythm of <em>metis</em> accumulation, and that rhythm is set not by the throughput of a GPU but by the speed at which human society evolves.</p> <p>So the six-hundred-billion-dollar question is not <em>can AI companies earn six hundred billion</em>. It is <em>are we willing to wait for the metis to grow</em>. The price is collapsing — that is true. The value is waiting — that is also true. <strong>Both are true at once, and they do not contradict each other.</strong> The domain AI that genuinely changes how people work will have to wait the way fiber waited for YouTube: ten years, until healthcare and law and education and finance — those fields most deeply embedded in <em>metis</em> — one by one, slowly, make AI a part of themselves.</p>]]></content><author><name></name></author><category term="essays"/><category term="ai-economics"/><category term="ai-strategy"/><category term="metis"/><category term="essay"/><summary type="html"><![CDATA[The price of a token is collapsing and the value of AI is waiting. Both are true at once, and the reason is that metis accumulates on institutional time, not technical time.]]></summary></entry><entry><title type="html">When Code Begins to Write Itself</title><link href="https://xiangli-shaun.github.io/blog/2026/when-code-begins-to-write-itself/" rel="alternate" type="text/html" title="When Code Begins to Write Itself"/><published>2026-07-02T00:00:00+00:00</published><updated>2026-07-02T00:00:00+00:00</updated><id>https://xiangli-shaun.github.io/blog/2026/when-code-begins-to-write-itself</id><content type="html" xml:base="https://xiangli-shaun.github.io/blog/2026/when-code-begins-to-write-itself/"><![CDATA[<blockquote> <p>This blog post is machine-translated from the original Chinese version. Link: <a href="https://mp.weixin.qq.com/s/BBg8rxH9EwfmFkKkZYJVhA">https://mp.weixin.qq.com/s/BBg8rxH9EwfmFkKkZYJVhA</a></p> </blockquote> <p>In early 2025 Andrej Karpathy coined a term: <em>vibe coding</em>. You tell the AI in natural language what you want, the AI generates code, and you do not read it carefully — if it looks about right, you accept it. Give in to the vibes. The phrase caught on to the point that Collins Dictionary named it word of the year for 2025. A year later, it is already obsolete.</p> <p>What has replaced it is <em>loop coding</em>, or more precisely <em>loop engineering</em>. Where is the difference? In vibe coding the human is still inside the loop: he has stopped writing code, but he is still looking, still feeling, still nodding and shaking his head. In loop coding the human factor withdraws from the loop itself. You first design an autonomous loop; the AI agent takes the task and runs on its own — writes code, runs tests, reads the errors, edits the code, runs the tests again, and continues until a preset termination condition is met. The human role retreats from <em>judging inside the loop whether this is right</em> to <em>designing the loop’s structure and its stopping condition</em>. Claude Code, Devin, OpenAI Codex Agent — the core working mode of every mainstream AI coding tool now is this.</p> <p>Discussing ChatGPT 5.5 Pro on mathematics problems, Gowers remarked that very good coders are better at vibe coding than not such good coders. Placed in the context of loop coding, the observation needs updating. In vibe coding a good programmer’s advantage is intuition: he can feel where the generated code is wrong. In loop coding his advantage is architecture: he can design a loop framework the AI will find it hard to wander out of. The first is a wine taster; the second is the food engineer at the winery. But the two share something: <strong>if you do not understand the thing yourself, you can neither sense the problem nor design the loop well.</strong></p> <p>Loop coding is genuinely fast. A good framework can generate, test and repair dozens of files in minutes. A feature module that traditionally took a junior engineer a week, a loop agent will deliver in an afternoon — along with a great many burned tokens — in a version that <em>runs</em>. This speed is itself changing the economics of the software industry.</p> <p>But the first consequence of that speed is not a decline in code quality. It is a break in comprehension. Addy Osmani at Google gave the phenomenon a name, <em>comprehension debt</em> — he credits the term to Jeremy Twei. When code is produced faster than humans can read it, you are borrowing from the future, and what you are borrowing is your own future capacity to maintain the system. Traditional technical debt is knowing the code is bad and having no time to fix it. <strong>Comprehension debt is not knowing whether the code is good, because no one has read it.</strong> This is not a metaphor. GitClear’s 2026 data shows that on teams making heavy use of AI coding, code churn — the proportion of code that must be revised or deleted after being written — rose by 39%. Seventy-five percent of engineering leads expect to accumulate moderate to severe technical debt from AI-assisted development within twelve months. There is more code and less understanding.</p> <p>This leads to a deeper problem. When people cannot finish reading what the AI has written, they naturally fall back on proxy indicators: did the tests pass? Is CI green? Did lint stay quiet? These tools worked well at the speed a human writes code, because human output rate was itself a form of implicit quality control. But once a loop agent raises output by an order of magnitude, those proxies are forced to carry far more weight than they were designed for. <em>Tests passed</em> does not mean <em>the code is correct</em>; it means only <em>the code satisfies the conditions the tests happen to cover</em>. The difference lies in what the tests do not cover — and AI-written tests and AI-written code tend to share the same blind spots.</p> <p>This is where things start to get strange. When a loop agent meets, inside the loop, something it does not know how to resolve — an API returning a format it has not seen, a library behaving differently from the version in its training data — it does not stop and say <em>I don’t know</em>. We have now observed, in many settings, that it will sometimes invent a solution that looks entirely reasonable: assume an API endpoint that does not exist, define a data format that does not conform to the specification, or write a comment saying <em>according to the documentation, the default value of this parameter is X</em> when the documentation contains no such sentence. <strong>This is not a bug in the traditional sense.</strong> A bug is code that fails to work as expected; here the code works perfectly against an expectation the AI invented for itself. The loop framework becomes a closed, self-validating system of fiction, wholly consistent on the inside. The problem surfaces only on contact with the real world: a user sees data that <em>should not</em> appear, a third-party service returns a format that <em>should not</em> exist. And by that point what we face is not a bug but a virtual building raised on a fictional foundation, with virtual light showing through one of its virtual windows.</p> <p>In medical imaging we have found that mainstream vision-language models will still produce a fluent, confident diagnosis when the evidence does not support one. We call this <em>silent failure</em>. What happens in loop coding is the code version of the same failure mode: <strong>the model does not know that it does not know, and its confidence disables every verification mechanism downstream.</strong> In imaging the consequence is a potentially absurd misdiagnosis. In code it is a system that looks entirely normal, passes every test, and then breaks three months after launch under some particular condition — and once repaired, breaks under another.</p> <p>The process also accelerates itself, because every round of a loop agent’s output becomes the input to the next round. If a false assumption enters at round three, round four will not correct it; round four will treat it as established fact and build new code on top. By round ten, that small original fiction has been buried under layer upon layer of logic. The heap of terrible code a human programmer leaves behind at least has a human being’s thinking buried inside it: you can curse him, follow his logic, and roughly reconstruct what he was thinking and how he went wrong. <strong>The heap an AI leaves behind has no thinking in it at all.</strong> It is a stack of locally optimal solutions, each layer solving a problem created by the layer beneath it.</p> <p>I wrote recently about artificial intelligence in <a href="/blog/2026/the-infinite-game-and-the-last-instruction/">The Infinite Game and The Last Instruction</a>, breaking learning into three motions: the <em>problem raiser</em> (what to learn from), the <em>problem solver</em> (how to learn), and the <em>problem judger</em> (whether the learning objective has any value at all). Loop coding is a nearly perfect instance of a raiser–solver loop: the agent continually discovers problems (a test failed) and solves them (edit the code), the loop drives itself, and no human intervention is required. It can even run with no explicit judger, using the test cases as referee — passing means right, failing means keep editing.</p> <p>But I also wrote this in that piece: as the raiser–solver becomes more autonomous, the inherited <em>lossy proxies</em> are forced to bear far more normative weight than they were ever designed to carry, and capability that has not been examined is what people call danger. What loop coding is staging now is the concrete version of that abstract argument. <strong>A test case is a lossy proxy.</strong> It can judge whether code satisfies some set of preset conditions. It cannot judge whether that set of conditions is complete, and still less whether this code should have been written at all.</p> <p>Code has begun to write itself. It writes quickly, it writes confidently, and all the tests pass. But the question of whether it <em>should</em> be written still has to be answered by a person — assuming there is still a person watching.</p>]]></content><author><name></name></author><category term="essays"/><category term="ai-coding"/><category term="ai-safety"/><category term="software-engineering"/><category term="metis"/><category term="essay"/><summary type="html"><![CDATA[Vibe coding kept the human in the loop. Loop coding takes the human out of it. What follows is not worse code but a break in comprehension — and a failure mode that looks exactly like success.]]></summary></entry><entry><title type="html">The Infinite Game and The Last Instruction</title><link href="https://xiangli-shaun.github.io/blog/2026/the-infinite-game-and-the-last-instruction/" rel="alternate" type="text/html" title="The Infinite Game and The Last Instruction"/><published>2026-05-19T00:00:00+00:00</published><updated>2026-05-19T00:00:00+00:00</updated><id>https://xiangli-shaun.github.io/blog/2026/the-infinite-game-and-the-last-instruction</id><content type="html" xml:base="https://xiangli-shaun.github.io/blog/2026/the-infinite-game-and-the-last-instruction/"><![CDATA[<p><em>A note on learning of the machine</em></p> <p>Learning of the machine can be seen as three recurring motions: First, a <em>problem raiser</em> selects what to learn from: it encounters the environment and samples material from it. Second, a <em>problem solver</em> transforms that material toward some objective optimally: it extracts regularity, forms a model, or to <em>compress</em> them, as Chaitin noted in his book <em>Meta Math!</em> Third, a <em>problem judger</em> evaluates the objective itself: it asks whether such transformation has value, either for the internal economy of the system or for some external agent, organism, or institution. Decades of work across sub-disciplines that approximate learning: actor-environment in reinforcement learning, generator-discriminator in adversarial training, hypothesize-and-test in scientific methods, genotype-phenotype-fitness in evolution, all of these keep recovering these three motions in different vocabularies. Two observations about them drive the argument that follows.</p> <p>First observation: if the first two motions (problem raiser-solver) are smartly connected, they will form a self-feeding loop with no natural termination. An evolving mechanism that picks what to consider, coupled with an evolving mechanism that transforms what is picked, with each of them continually sharpening the other, enters an infinite game. The raiser-solver enact, in classical terms, the exploration–exploitation dynamic at the heart of learning: the first explores, searching for material the system has not yet mastered; the second exploits, consolidating what is found into the system’s capabilities. The infinite game emerges when neither side is allowed to dominate, when exploration is never fully resolved into exploitation, and exploitation continually surfaces new directions for exploration. Such equilibrium can take various forms. Yet given the optimal conditions: unlimited computing, sufficient data, a stable and computable environment, such a system can asymptotically discover all compressible regularities. This is, in spirit, the promise that <em>Solomonoff’s theory of universal induction</em> makes precise: that an idealized learner, given enough resources, converges on the shortest programs explaining its observations. The interesting recent exploration in AI is to ground the problem raiser in <em>computational substrates</em>: Python interpreters, theorem provers, chess engines, SAT solvers, rather than in static information (e.g., text). Text substrates produce retrieval-shaped tasks because text is information; computational substrates produce reasoning-shaped tasks because the answer is the result of work the raiser performs and the solver must simulate. Our vision is that an iterated raiser–solver loop could break through ceilings that pretraining over the whole of human text (and non-text) appears to have reached, because pretraining compresses what humans wrote about the world, yet the infinite game compresses the world. <strong>Note how this loop can proceed without an explicit problem judger:</strong> the substrates themselves appear to judge, because whatever resists compression, prediction, or control becomes the next problem to be raised. Such is almost the condition of modern AI/ML research: the judge is not absent because valuation has vanished, but because it has been buried inside data, loss, benchmark, reward, and scale.</p> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/blog/infinite-game-480.webp 480w,/assets/img/blog/infinite-game-800.webp 800w,/assets/img/blog/infinite-game-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/blog/infinite-game.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" loading="eager" onerror="this.onerror=null; document.querySelectorAll('.responsive-img-srcset').forEach(function (n) { n.remove(); });"/> </picture> <figcaption class="caption">Abstract illustration of how the problem raiser-solver game iterates</figcaption> </figure> <p>Second observation: following the note above, the three motions are not equally investigated by AI community, and they fall in an order. The problem solver is the strongest and most developed. Gradient descent, backprop, transformers, and modern optimizers have made the transformation engine so reliable that human contribution become more architectural and engineering. Even those are increasingly automated. The problem raiser is traditionally weaker but has got a lot of advancement in the recent decade. Self-supervised learning has been a major force here, letting systems manufacture their own training signal from raw data rather than depending on hand-labeled examples. But what most systems still learn from are human-curated artifacts: the pre-training corpus, the fine-tuning set, the RLHF preference data, and the annotations woven through them all. The problem judger is the weakest, and weakest in a different sense from the others, <em>constitutive</em> rather than technical. <strong>The machine does not originate value, and humans rarely even articulate the value they intend to instill.</strong> Most of the time the “problem judging” is implied, smuggled in through data choices, loss functions, and reward shaping, rather than stated. Evaluation criteria can be formalized, but value itself cannot be reduced to a loss without remainder. While every modern AI system has something playing the role of the judge: a reward model, a verifier, a parsimony term, a frontier band. What none of these implementations do is judging, let alone originating value. Each is a proxy for a preference supplied from outside, almost always by a human. The machine can often answer: “Did I improve according to this objective?” But it usually cannot independently answer: “Is this objective worth pursuing?” That distinction matters. A chess engine can generate brilliant moves, but “winning at chess” is given to it. A recommender system can maximize engagement, but “engagement is valuable” is an imposed assumption. Even when a model appears to “self-evaluate,” it is usually evaluating against internalized human-labeled preferences or externally designed heuristics (e.g., regularizations), thus what they really generated are instrumental, internal sub-values.</p> <p>What makes the gap in the second observation urgent, rather than merely structural, is the pace at which the problem raiser-solver are developing, the infinite game becoming real. The trend will be toward more general, more domain-spanning instances of the same architecture. Yet the problem judger is not arriving at the same rate to close the gap. The result is a near future in which the infinite game produces increasingly powerful artifacts whose directions remain implied, inheriting the definition of “value” through <em>lossy proxies</em>: datasets, reward models, constitutions, preference labels, benchmarks, deployment incentives, product metrics, etc. As the problem raiser-solver becomes more autonomous and the infinite game becomes more powerful, those inherited proxies are forced to bear more normative weight than they were ever designed to carry. <strong>Capability without justified direction is what <em>dangerous</em> names.</strong> The danger does not abate as the system gets better at the first two motions; it compounds with them. The artifacts built by the infinite game-powered engine will be governed not by values they originated, nor by values we fully chose (and hope), but by the residue of objectives we failed to understand before amplifying.</p> <p>We do not yet know, largely, how to close the gap. The deeper question is whether a learning system can in principle generate the standard against which it evaluates, or whether value must always be imported from external. For humanity, the desired system shall integrate all the three motions, thus a powerful predictor of the world weighted by what good is, which is something the system comes to hold rather than something it expects to be handed. Such system contains three layers of instruction. The first instruction is operation: “do this task”. The second instruction is optimization: “do this task the best towards this goal”. The third, and <em>last instruction</em> is direction: “decide what the goal should be, and why it is worth pursuing at all.” AI has become extremely good at everything below <em>the last instruction</em>. It can search, compress, imitate, generalize, optimize, self-correct, generate curricula, and use tools. What it cannot yet do is originate the authority of the standard by which those activities are judged. It can ask whether an answer satisfies a rule, but not whether the rule deserves to govern. The <em>last instruction</em> is also the one we have least understood, because it is the one we have seldom asked, or been able to ask, the machine to write for itself. The next decade of work, equal parts technical and philosophical, will be done there.</p> <hr/> <p><em>Originally published on <a href="https://medium.com/@xiangli.shaun/the-infinite-game-and-the-last-instruction-5c25bb1197e3">Medium</a>, 19 May 2026.</em></p>]]></content><author><name></name></author><category term="essays"/><category term="ai-alignment"/><category term="ai-safety"/><category term="machine-learning"/><category term="essay"/><summary type="html"><![CDATA[Learning of the machine moves in three recurring motions — raising problems, solving them, and judging whether the objective is worth pursuing. We have built the first two into an engine that does not stop. The third we have barely begun.]]></summary></entry><entry><title type="html">Gentle Problems</title><link href="https://xiangli-shaun.github.io/blog/2026/gentle-problems/" rel="alternate" type="text/html" title="Gentle Problems"/><published>2026-05-11T00:00:00+00:00</published><updated>2026-05-11T00:00:00+00:00</updated><id>https://xiangli-shaun.github.io/blog/2026/gentle-problems</id><content type="html" xml:base="https://xiangli-shaun.github.io/blog/2026/gentle-problems/"><![CDATA[<blockquote> <p>This blog post is machine-translated from the original Chinese version. Link: <a href="https://mp.weixin.qq.com/s/srq9dWWd43k3Eoz_gZCDtQ">https://mp.weixin.qq.com/s/srq9dWWd43k3Eoz_gZCDtQ</a></p> </blockquote> <p>I have a few things to say about the Gowers post that has been filling everyone’s screen lately — <a href="https://gowers.wordpress.com/2026/05/08/a-recent-experience-with-chatgpt-5-5-pro/">A recent experience with ChatGPT 5.5 Pro</a>.</p> <p>If you assume I am about to talk about Metis AI, I am not: what AI can do at present is still some distance from that boundary.</p> <p>What I want to discuss is a core observation in Gowers’ original that almost every retelling has passed over. He points out that the problems ChatGPT 5.5 Pro solved come from a paper of Nathanson’s, written for students just entering research — in his words, <em>relatively gentle open problems</em>. The traditional way to start a new PhD student is to hand them an open problem that looks as though it might not be too hard but in fact takes real work; we used to do the same thing, though lately in CS it has turned into writing surveys and building benchmarks. But if an LLM can solve problems of this kind, then at least in mathematics that traditional path no longer works.</p> <p>Then he writes the sentence actually worth attending to: the <strong>lower bound</strong> of a mathematical contribution now becomes <em>proving something an LLM cannot prove</em>, rather than merely <em>proving something nobody has proved</em>. What is precise about this observation is that he is describing the lower bound moving up, not the ceiling being reached. What the AI achieved was to compress into an hour a piece of work that would previously have taken a new PhD student several weeks. Work of that kind is, in essence, a non-trivial improvement made inside an existing research framework. That is remarkable — but the distance between it and <em>AI possessing the capability of a real mathematician</em> may be larger than many people imagine. <strong>What AI can do at present is recombine existing parts in non-obvious ways. It cannot yet invent new parts.</strong></p> <p>Gowers runs a thought experiment in the post: suppose a mathematician solves an important problem through a long conversation with an LLM, where the mathematician provides the guidance but the LLM does all of the technical work and supplies the central idea. Would we regard this as a major achievement of the mathematician? His answer: he does not think we would. But he adds immediately that mathematicians who have genuinely solved hard problems themselves turn out to have the advantage when collaborating with an LLM — <em>just as very good coders are better at vibe coding than not such good coders.</em></p> <p>Put those two judgments side by side and they become very interesting. <strong>Inside the boundary of AI capability, the value of a human doing the work independently is falling</strong> — it is gradually becoming unnecessary. <strong>Above that boundary, the deep intuition a human accumulates through long training becomes more important</strong>, because it determines whether you can effectively steer the AI toward what it cannot do on its own. I expect that very soon AI will re-scan, re-verify and then optimise, at an unprecedented scale, every problem that can be precisely described — mathematics among them. We have students doing exactly this in neuroimaging right now.</p> <p>Two more words for our <a href="https://arxiv.org/abs/2605.14407">Metis AI</a> framework. In that frame, mathematical proof is almost pure <em>techne</em>: the rules are explicit, the results are verifiable, and nothing depends on interpersonal relationships or institutional context. If AI is going to demonstrate capability in any field at all, a highly formalised branch like combinatorics is the most natural place to begin. So the performance of ChatGPT 5.5 Pro here is not surprising. What it did was take one more step on precisely the class of task AI was always best at.</p> <p>But Gowers’ observation reveals a subtler structure: <strong>even inside a purely <em>techne</em> domain, there is a clear capability boundary.</strong> On one side of it are the <em>gentle problems</em> — technical improvements made within an existing framework, which is the ideal battleground for an LLM’s pattern recognition and recombination. On the other side is the work that requires years of accumulated intuition, that requires building deep connections between apparently unrelated fields, that requires judging which problems are worth spending time and energy on at all. These capacities are closer in nature to <strong>knowing what is important</strong> than to <em>being able to prove what is correct</em>. Even in the most purely formal of domains, once the difficulty of a problem crosses a certain line, what you need is no longer only <em>techne</em> but something like a mathematician’s <em>metis</em>. It is not quite practical, relational, situated knowledge in the ordinary sense — but it refuses formalisation just as firmly, and it can only come from long personal practice.</p> <p>Finally, a simple fact check. In the articles going around, Gowers is cast as someone who <em>once publicly mocked AI</em> and has now <em>admitted defeat for the first time</em>. None of that is in the original. Where the retellings have him saying the result was fully at the level of a doctoral thesis, even fit to serve as its most brilliant chapter, what he actually wrote was that it would make a perfectly reasonable chapter in a PhD thesis — a competent chapter, not a brilliant one. Gowers’ tone is measured from beginning to end. He states explicitly that the result builds heavily on Rajagopal’s ideas, rather than being an original breakthrough arriving out of nowhere. This systematic stripping-away of qualifiers is by now a standard operation in AI coverage, and I will say no more about it.</p>]]></content><author><name></name></author><category term="essays"/><category term="ai-mathematics"/><category term="metis"/><category term="techne"/><category term="essay"/><summary type="html"><![CDATA[The observation almost every retelling of Gowers' post missed — that the lower bound of mathematical contribution has moved, and that a capability boundary runs through even the purest techne.]]></summary></entry></feed>