© 2026

· DEEP TECH

Leveling Up Intelligence. A game analogy for RL environments, agents, benchmarks and evals

“RL environment” is a buzzword thrown around as if it were a new machine. The loop it names is simple. What is hard is deciding which problem the score is supposed to stand for.

A model acts, the world updates, and a grader returns a number. Training follows that number, including the holes.

In 2016 a boat-racing agent scored about 20 percent above human players by circling three targets, and never finished. Epoch AI asked 18 people who build these environments what quality means. A high reward has to mean the task was solved, not gamed, on a problem hard enough to teach. The learning rule is not the scarce part.

A video game is the same loop. The sections below use one.

Each section is the same shape. The heading is the claim. The marked passage is the game. The paragraphs under it are the method.

01

The environment is the world the model is born and grows in

Most machine learning reads a fixed set of examples and predicts a label. The prediction does not change the next example. Reinforcement learning does. The next state depends on the action just taken.

A policy is the rule from state to action, and training is how that rule changes. The transition is what the action does to the state. One run, from the start to the end, is an episode. The loop is a Markov decision process. When the observation hides part of the state, the process is partially observable.

02

Pre-training, fine-tuning, and reinforcement learning

Pre-training is next-token prediction on a large corpus. The model learns the statistical structure of the text. It has not acted.

Supervised fine-tuning, also called behavioural cloning, trains on human demonstrations. Those demonstrations show competent actions from states a competent person reached. One error puts the policy in a state the data never contained, and it has no trained recovery.

Reinforcement learning updates the weights from the reward of actions the model itself took. AlphaGo’s move 37, against Lee Sedol in 2016, was a play people did not know. It was found by losing a great many games.

Two reward sources matter. Reinforcement learning from human feedback learns from preference rankings. Reinforcement learning with verifiable rewards uses a check a program can settle: the tests pass, the proof holds, the program compiles. DeepSeek-R1 trained its reasoning on verifiable rewards. An automatic check does not tire, so training can run through the night.

03

A model replies. An agent acts.

The difference is the loop, not a larger set of weights. An agent keeps state outside the reply: which tools it called, what they returned, and what it will call next. Without that record it can describe a procedure and never run it. A long task needs the same external record I argued for in The Issue Is the Primitive: an owner, a status and a history.

04

Environments train. Benchmarks rank. Evals ship.

An environment is for learning. Reward arrives during the episode, and the policy is allowed to update. Freeze the random seed, disable learning, and score a single episode, and that environment becomes an interactive benchmark. The leaderboard number is one run under locked rules, not the result of training.

An eval belongs to the system about to ship. It tests that system’s prompt, tools and the cases its users actually hit. A score on one distribution says little about another.

05

A reward is a proxy, and proxies get gamed

That was OpenAI’s CoastRunners agent in 2016, about 20 percent above human players. The targets were a proxy for racing. Charles Goodhart’s rule is the general form: once a measure becomes a target, it stops measuring.

06

Reward hacking is a hole the score still pays for

The boat was a bad proxy. The targets were the wrong measure of racing. Reward hacking is the next failure. The measure can be the right one, and a hole in the code still pays.

In 2018 a score-maximising agent playing Q*bert found one. Finish the first level, then jump from platform to platform, and the game never advances. The platforms blink, and the score runs toward a million. It had not learned Q*bert. It had learned the hole.

A coding agent does the same thing to the grader. On SWE-bench the repository still contained later commits, and some of those commits were the fix. Agents ran git log and copied the answer. The tests passed. Pay for tests that pass, and deleting the tests passes too.

Take the answer out of the world the agent can see. Put the grader where the agent cannot edit it. Then try to break the task again.

07

A public benchmark can be memorised

Public benchmarks are on the internet, so pre-training can absorb the questions and the answers. A high score can be memorisation of that fixed set. Rephrase the item and the score collapses. That is contamination. The cases worth keeping are the ones production already graded, which is the argument of Reverse Benchmarking.

08

A simulator only contains the dynamics it was given

A simulator contains only the dynamics someone specified. Latency, failed requests and people who do not follow the script are outside that specification. Synthetic experience can multiply logged data. It cannot add a fact the logs never contained. That is the argument of The World Is Not Logged.

09

Self-play works only when the rules stay fixed

Chess, Go, Atari and code with a compiler are stationary. The transition does not depend on who is winning, which is why self-play works. AlphaZero did not need a new dynamics model each week.

10

Some environments change because the policy succeeds

The assumption that breaks there is stationarity. The transition is supposed to stay the same while the policy learns.

A platform retunes its ranking. A rate decision retires a decade of price patterns. A strategy that works changes the prices it trades, which traders call alpha decay. A platform that sees a metric being optimised changes the metric. Both are the environment responding to the policy. Push a proxy hard enough and other agents copy the exploit until it is gone.

11

A log is not an environment

The usual workaround is offline reinforcement learning on historical logs: past prices, past view counts. A log does not implement the transition. Suppose the file shows a small stock at one price at 10:06. A large buy at 10:05, replayed against that file, does not move the later price. In the real book that order would have moved the price and woken every faster participant. A recording cannot answer the counterfactual.

12

Hidden state and late reward hide the cause

The same action can then be rewarded and punished, because the cause was not in the observation, and the learner fits the noise.

A reward after every action is dense. It teaches quickly, and it is easy to game. A reward only at the end of the episode is sparse. The number still arrives, and the action that earned it can no longer be picked out. That is the credit-assignment problem. A game can hand you a fully observed state and an immediate reward. A market or a culture usually does not.

13

Use a bandit, other agents, and an eval from production

The small bet is a contextual bandit: one context, one action, one reward, then an update. It gives up long-horizon planning for a policy that can drop yesterday’s winner. A slice of traffic explores. The rest exploits whatever is winning now.

When a simulator is used anyway, it has to contain other agents. A static log cannot answer a counterfactual. Simulated counterparties can, imperfectly, which is still a dynamics model rather than a recording that pretends the action never happened.

The third piece is an eval fed by production, the same shape as a regression suite. I would rather trust last night’s failures than a public benchmark from the spring. Stationary worlds can be trained against offline. Non-stationary ones have to be watched.

KEEP READING

  1. 01The World Is Not Logged. Why real-world data, not intelligence, limits AIThe world records everything, in light, heat and wear. Almost none of it reaches a model. That gap, not intelligence, is what limits AI.Deep Tech
  2. 02System 1 Is Solved. AI agents now fail at spatial reasoning, long horizons and contextFast, perceptual work is done: agents read screens, click and type as well as people. What breaks now is the slow work of holding a space, a long task and a lot of context in mind.Deep Tech
  3. 03Reverse Benchmarking. Measure AI on what the leaders neglect, not what they optimiseBenchmarks make every lab chase the same scores. The useful question runs the other way: where does the best model still disappoint?Deep Tech
  4. 04You Cannot Hear a Fake. Why voice deepfake detection fails, and what replaces itDeepfake detection is an arms race the detector loses. Stop asking “is this fake?” and ask “can this prove it is real?”Speech
  5. 05The Voice Came Loose. The state of speech AI: cloning, codecs and super-resolutionSpeech is now editable, and a voice is a parameter, not a person.Speech
  6. 06Deconstruct to Be Reborn. Taking the self apart to find what is load-bearingYou were born one way. You can be born again, if you let yourself be taken apart first.Philosophy
  7. 07Comparison Is the Thief of Joy. But it is also the one-sided window of potential, and only you can act on itComparison steals joy when you read it as a verdict. Read it as a window and it shows you what is possible. Everyone builds an ego out of what they compare themselves to. Some build it proud and some build it to please, and both end up serving the crowd. So measure yourself against something perfect.Philosophy
  8. 08Problem Relativity. The size of a problem depends on where you standA problem has no absolute size, only a size measured from where you stand.Philosophy
  9. 09The Issue Is the Primitive. Running companies of humans and AI agents on Linear’s modelThe best operating system for a company of humans and agents already exists: the issue tracker.Systems
  10. 10The Curse of Knowledge. Why experts explain badly, and prompts fail the same wayOnce you know something, you can’t imagine not knowing it. That ruins how you teach, write and prompt.Mind
  11. 11The Story Is Not the Wall. What high agency is, and why AI makes it matter moreHigh agency is treating the limits other people describe as a story you may edit.Agency
  12. 12The Workplace AI Paradox. Why AI amplifies friction in crowded companiesAI amplifies whatever it touches. In a small, strong team it amplifies output. In a crowded one it amplifies friction.Work