Cheap Oracles

On Mastodon these days you keep running into a particular kind of reassurance: AI is a bubble, it'll blow over, and if you treat it as somebody else's problem, it will politely remain one. I think all three parts are wrong, but the third is the interesting one. Useful technologies have produced bubbles before, and the crash never un-invented the railway. Whether a technology affects you doesn't depend on whether you use it. It depends on whether it changes the cost of getting work done.

What chess tells us about the models

The most common dismissal of language models goes like this: they can't actually do anything, they've just read everything. What comes out is, at best, the average of what humans have already written. Without us, they'd be nothing.

That objection has a history, and you can go visit it in chess. There, too, human experience looked like the essential raw material: a century of tournament practice, opening theory, the accumulated understanding of generations of players. Engines were built out of exactly this knowledge — evaluation functions designed by humans, with criteria weighted by humans.

At the end of 2017 came AlphaZero. The system was given the rules of the game, but no human games, opening theory, or hand-written evaluation criteria. It played against itself and learned from the results. The final published account, in Science a year later, reports that after roughly nine hours of training it beat the 2016 TCEC champion version of Stockfish, with 155 wins and 6 losses out of a thousand games.

It's worth being precise about what got refuted there. Not "computers are faster than humans" — we've known that since Deep Blue. Something narrower and more surprising: superhuman chess did not require human games as training material. The archive of human chess was not the ceiling. It wasn't even the foundation. It was a detour.

Why did this work in chess, of all places? Because chess supplies cheap, exact feedback. The rules say whether a move is legal, whether a position is terminal, and who won. The outcome signal is sparse — it arrives at the end of a game — and turning it into good moves still costs a great deal of computation. But it is unambiguous and available whenever you want to run another game. A system can generate training material without asking a human to judge each attempt.

That distinction matters for the rest of this essay. By an oracle I mean any repeatable mechanism that scores an attempt. Oracles differ in cost, latency, coverage, and noise. Chess has an exact outcome oracle. Software tests are fast but incomplete. Human reviewers are broad but slow and inconsistent. A second model is cheap and fast, but only a critic — not ground truth.

That the engine to beat today is once again called Stockfish changes none of the central result. Current Stockfish evaluates positions with a neural network, while the AlphaZero-inspired Leela Chess Zero learns through self-play. Human games remain useful; they are no longer the limiting resource.

The bitter lesson

In 2019, Rich Sutton described why this pattern isn't a one-off but a constant running through 70 years of AI research: methods that scale with compute — search and learning — eventually beat methods built around human expertise. The expertise helps in the short term and becomes a constraint in the long term, across chess, Go, speech recognition, and computer vision. Sutton calls this the bitter lesson, because of where it stings: what lost, each time, was the part we considered ourselves indispensable for.

Halevy, Norvig and Pereira had made a complementary argument in 2009. In "The Unreasonable Effectiveness of Data" they describe how simple methods with lots of data can beat sophisticated methods with little data. Sutton's emphasis is scalable search and learning; theirs is the leverage of large datasets. Both warn against mistaking today's handcrafted advantage for a permanent one.

Software has known this from its own practice for longer than AI research has had a name for it. In 2001, arguing on the kernel mailing list that Linux is evolution rather than design, Linus Torvalds warned: "don't EVER make the mistake that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That's giving your intelligence much too much credit." Eighteen years before Sutton, and not about AI at all — the same two ingredients, parallel search and a feedback cycle, and the same sting for our self-image.

You can read the bitter lesson as the latest in a series of decentrings. The Earth is the center of the universe — it is not. We are the crown of creation — a branch on the tree. We are masters in our own house — most of what we do is prepared by processes we never get to watch. We are not even the protagonists of heredity; seen from the genes' side, we are the vehicle. Each of these demotions stung, and each of them stuck. The bitter lesson files the next one: our explicit cleverness is not the engine we took it for. Systems that turn computation and feedback into performance have repeatedly outrun the knowledge we tried to build into them by hand — and we are slowly running out of benchmarks with which to reassure ourselves that nothing comparable is at work in the models.

Code has an oracle

Which makes the question the same for every domain: what feedback is available, and what does a query cost?

For code, the answer is: quite a lot of feedback, much of it cheap. Does it compile? Do the tests pass? Did the benchmark regress? These checks do not prove that the program does what the task demands — tests are incomplete and specifications vague — but they reject a useful fraction of bad attempts quickly and automatically. Code sits closer to the chess side of the line.

Coding agents exploit exactly this property. They search the codebase, try something, read the error messages, and try again. This is not training in AlphaZero's sense: the model's weights do not change every time a test fails. But the feedback still lets the agent search over attempts at runtime. The amount of compute burned to imitate this ordinary loop can look absurd, especially with computer use, where the model perceives the screen through screenshots. The economic comparison is not between an agent and an efficient program. It is between the cost of the loop and the cost of the human attention it replaces or amplifies.

That trade is the oldest one in the business. Nobody writes operating systems in assembler, even though compilers arrived producing machine code that was plainly worse than what a good human wrote by hand. They also arrived to exactly the objection agents hear today: how am I supposed to trust machine code I didn't write? Nobody has asked that question in decades. We have traded machine efficiency for human leverage with every generation of languages, and it has paid off every time.

Andrej Karpathy named where this ladder points back in 2017: Software 2.0 — instead of writing every instruction, you specify desirable behavior and let optimization find the parameters. Coding agents still produce ordinary source code, but they echo the same inversion at the tool level: describe the result, generate an attempt, and let the environment judge it.

How far this can go when someone actually means it is shown in Bun's account of its rewrite. Jarred Sumner describes a mechanical port of the JavaScript runtime from Zig to Rust: more than a million lines added, about fifty Claude Code workflows, around 64 Claude instances at peak, and eleven days of work, closely monitored by one engineer. He puts the pre-merge usage at about $165,000 at API prices and estimates — his numbers, not a controlled comparison — that a manual port would have occupied three engineers with full context for about a year. The realistic alternative, he says, was not a slower port but no port at all.

The important part is why it worked. Bun's test suite is written in TypeScript and does not depend on the runtime's implementation language. It could keep asking whether the Rust code behaved like the Zig code. The tests did not suddenly become cheaper. Producing and revising candidate patches did. The Bun port doesn't prove that agents can build arbitrary software; it shows what becomes possible when a large codebase already has strong, language-independent feedback.

The same calculation exists at small scale. My python-podcast.de pet project ran the Podlove player for years — the standard in the German podcasting scene, and for my case the wrong tool: a Vue app in its own iframe, with many dependencies, its own design system, and features I never needed. Writing my own player was still never seriously on the table, because the feasibility study alone would have been a project: perhaps two weeks of work to discover whether the idea held up at all. You don't run many experiments at that price. You postpone them indefinitely.

Then my weekly limits reset, I had a day of full token budget, and very little to lose. One /goal prompt and one day of an agent grinding through the loop produced the first usable version. The site now runs a dependency-free player in plain TypeScript that has since grown transcripts, chapters, and persistent playback. Its current component bundle is roughly 12 KiB gzipped including CSS, plus a small site-specific persistence layer, and the pages pass their Core Web Vitals checks. The player isn't the story. The story is the price of the experiment — down from two uncertain weeks to a day of compute. With that price falls a whole category of reasons not to try things.

Where the mechanical checks stop, I add another kind of feedback: a separate model whose job is to attack the result. Bun made adversarial review a fixed step; I do something similar across vendors and ask models from different providers to check one another's work. What was a home-grown practice a year ago now has official tooling — OpenAI ships a Codex plugin for Claude Code with normal and adversarial review modes. Notice the role reversal: the model is no longer only the thing being judged. It has become an oracle itself — a cheap and noisy one, whose mistakes may correlate with the first model's even across vendors, and which is never ground truth. But cheap beats perfect when perfect is not on offer.

The price of feedback

The line doesn't run between creative and mechanical work. It runs along the cost and quality of feedback. The expensive part was never just producing an answer; it was knowing whether the answer was any good.

At one end are chess, parts of code, and increasingly parts of mathematics. With Lean, a small kernel can check whether a proof term establishes a formal statement, and mathlib provides a growing body of formalized mathematics. Once the statement and its assumptions have been formalized, checking a proof is cheap and exact. Deciding whether the formal statement captures the theorem you intended is still human work.

In the middle are domains where feedback exists but each query is slow, costly, or noisy. A drug is tested in trials that run for years, under ethical constraints, and return statistical evidence rather than certainty. Human preference feedback belongs here too: people can compare outputs, but their judgments vary, cost money to collect, and tend to reward broadly shared preferences. The language models themselves straddle this spectrum. Pretraining runs on an oracle that grades itself for free — the next token is its own answer key — which is why it could scale to everything ever written; the preference feedback bolted on afterwards is slow, expensive, and averaging. Optimize for the center of a distribution and you get the center of a distribution. The edges quietly disappear.

At the other end are domains where you can hardly put the question at all. You cannot run a country twice and compare two economic policies. Natural experiments sometimes provide variation that is as good as random, but you cannot usually manufacture them on demand. This oracle answers when it feels like it. With the question of whether a novel is good, disagreement joins the cost: the readers contradict one another, and there may not even be a single metric to converge on. Maybe literature stays human work longer than software not because it is harder, but because nobody knows what to optimize against.

Software architecture sits somewhere in the middle. Whether a boundary was well chosen often shows up months later, after requirements have changed — feedback slow and expensive enough that architecture is, in my experience, where agents still fail first. I would not bet on the limitation staying put. METR's experiments estimate that, across its suite of software-related tasks, the length of task a frontier agent can complete with 50 percent reliability doubled roughly every seven months over the six years it studied. That is a benchmark trend, not a forecast. But the history of these models is a history of falling oracle prices, and "the models cannot do long work" is an unstable place to stand.

The heap we're sitting on

So what does this mean in practice? First, something uncomfortable about code we'd long since signed off on.

In my recent work, current models have repeatedly found real bugs in well-kept, handwritten codebases. Not style nits — bugs. The code was fine, measured by what a human review catches. It no longer is, measured by what a model finds in it. The code has not deteriorated. What changed is the cost of examining it, and with it the standard against which leaving it unexamined is judged.

The popular word for bad machine-made code is "slop," and it is usually meant as a label for whatever fell out of a chat window. But slop is not a property of origin. Slop is a difference: between the code as it stands and what the best affordable check would find in it. Call it verification debt. The uncomfortable property of this debt is that it grows on its own — with every model generation, without anyone touching a line. The code stays still while the waterline rises, and every codebase sits a little lower in the water. Including the well-kept ones.

Not using the tools doesn't stop any of that. It only means you are the last to find out where your code stands. That's the illusion of not being affected: colleagues, clients, and competitors can apply the cheaper verification whether you do or not. Taking the bitter lesson seriously means giving up the comfort of believing that refusal is a strategy.

What follows is work: rebuild processes, automate verification, and learn where agents help and where they still need supervision. Small teams have one real advantage here: low organizational latency. A large company may have more compute, more data, and dedicated platform teams, but it also has contracts, committees, and migration costs. A solo developer or a team of three can redesign a workflow in an afternoon. Employees face a separate incentive problem when they are expected to buy the tools or learn the new practice on their own time while the gains accrue elsewhere. People who can keep part of the gain have a clearer reason to lead the change.

A lot of this is still human work. The load-bearing word is still.