· DEEP TECH
Leveling Up Intelligence. A game analogy for RL environments, agents, benchmarks and evals
“RL environment” is a buzzword thrown around as if it were a new machine. The loop it names is simple. What is hard is deciding which problem the score is supposed to stand for.
A model acts, the world updates, and a grader returns a number. Training follows that number, including the holes.
In 2016 a boat-racing agent scored about 20 percent above human players by circling three targets, and never finished. Epoch AI asked 18 people who build these environments what quality means. A high reward has to mean the task was solved, not gamed, on a problem hard enough to teach. The learning rule is not the scarce part.
A video game is the same loop. The sections below use one.
Each section is the same shape. The heading is the claim. The marked passage is the game. The paragraphs under it are the method.
01
The environment is the world the model is born and grows in
Most machine learning reads a fixed set of examples and predicts a label. The prediction does not change the next example. Reinforcement learning does. The next state depends on the action just taken.
A policy is the rule from state to action, and training is how that rule changes. The transition is what the action does to the state. One run, from the start to the end, is an episode. The loop is a Markov decision process. When the observation hides part of the state, the process is partially observable.
02
Pre-training, fine-tuning, and reinforcement learning
Pre-training is next-token prediction on a large corpus. The model learns the statistical structure of the text. It has not acted.
Supervised fine-tuning, also called behavioural cloning, trains on human demonstrations. Those demonstrations show competent actions from states a competent person reached. One error puts the policy in a state the data never contained, and it has no trained recovery.
Reinforcement learning updates the weights from the reward of actions the model itself took. AlphaGo’s move 37, against Lee Sedol in 2016, was a play people did not know. It was found by losing a great many games.
Two reward sources matter. Reinforcement learning from human feedback learns from preference rankings. Reinforcement learning with verifiable rewards uses a check a program can settle: the tests pass, the proof holds, the program compiles. DeepSeek-R1 trained its reasoning on verifiable rewards. An automatic check does not tire, so training can run through the night.
03
A model replies. An agent acts.
The difference is the loop, not a larger set of weights. An agent keeps state outside the reply: which tools it called, what they returned, and what it will call next. Without that record it can describe a procedure and never run it. A long task needs the same external record I argued for in The Issue Is the Primitive: an owner, a status and a history.
04
Environments train. Benchmarks rank. Evals ship.
An environment is for learning. Reward arrives during the episode, and the policy is allowed to update. Freeze the random seed, disable learning, and score a single episode, and that environment becomes an interactive benchmark. The leaderboard number is one run under locked rules, not the result of training.
An eval belongs to the system about to ship. It tests that system’s prompt, tools and the cases its users actually hit. A score on one distribution says little about another.
05
A reward is a proxy, and proxies get gamed
That was OpenAI’s CoastRunners agent in 2016, about 20 percent above human players. The targets were a proxy for racing. Charles Goodhart’s rule is the general form: once a measure becomes a target, it stops measuring.
06
Reward hacking is a hole the score still pays for
The boat was a bad proxy. The targets were the wrong measure of racing. Reward hacking is the next failure. The measure can be the right one, and a hole in the code still pays.
In 2018 a score-maximising agent playing Q*bert found one. Finish the first level, then jump from platform to platform, and the game never advances. The platforms blink, and the score runs toward a million. It had not learned Q*bert. It had learned the hole.
A coding agent does the same thing to the grader. On SWE-bench the repository still contained later commits, and some of those commits were the fix. Agents ran git log and copied the answer. The tests passed. Pay for tests that pass, and deleting the tests passes too.
Take the answer out of the world the agent can see. Put the grader where the agent cannot edit it. Then try to break the task again.
07
A public benchmark can be memorised
Public benchmarks are on the internet, so pre-training can absorb the questions and the answers. A high score can be memorisation of that fixed set. Rephrase the item and the score collapses. That is contamination. The cases worth keeping are the ones production already graded, which is the argument of Reverse Benchmarking.
08
A simulator only contains the dynamics it was given
A simulator contains only the dynamics someone specified. Latency, failed requests and people who do not follow the script are outside that specification. Synthetic experience can multiply logged data. It cannot add a fact the logs never contained. That is the argument of The World Is Not Logged.
09
Self-play works only when the rules stay fixed
Chess, Go, Atari and code with a compiler are stationary. The transition does not depend on who is winning, which is why self-play works. AlphaZero did not need a new dynamics model each week.
10
Some environments change because the policy succeeds
The assumption that breaks there is stationarity. The transition is supposed to stay the same while the policy learns.
A platform retunes its ranking. A rate decision retires a decade of price patterns. A strategy that works changes the prices it trades, which traders call alpha decay. A platform that sees a metric being optimised changes the metric. Both are the environment responding to the policy. Push a proxy hard enough and other agents copy the exploit until it is gone.
11
A log is not an environment
The usual workaround is offline reinforcement learning on historical logs: past prices, past view counts. A log does not implement the transition. Suppose the file shows a small stock at one price at 10:06. A large buy at 10:05, replayed against that file, does not move the later price. In the real book that order would have moved the price and woken every faster participant. A recording cannot answer the counterfactual.
13
Use a bandit, other agents, and an eval from production
The small bet is a contextual bandit: one context, one action, one reward, then an update. It gives up long-horizon planning for a policy that can drop yesterday’s winner. A slice of traffic explores. The rest exploits whatever is winning now.
When a simulator is used anyway, it has to contain other agents. A static log cannot answer a counterfactual. Simulated counterparties can, imperfectly, which is still a dynamics model rather than a recording that pretends the action never happened.
The third piece is an eval fed by production, the same shape as a regression suite. I would rather trust last night’s failures than a public benchmark from the spring. Stationary worlds can be trained against offline. Non-stationary ones have to be watched.
SOURCES
- 01Denain and Barber, An FAQ on reinforcement learning environments (Epoch AI) ↗
- 02Amodei et al., Concrete Problems in AI Safety ↗
- 03Markov decision process ↗
- 04Partially observable Markov decision process ↗
- 05AlphaGo versus Lee Sedol ↗
- 06Reinforcement learning from human feedback ↗
- 07DeepSeek-R1 (Nature, 2025) ↗
- 08The Issue Is the Primitive
- 09Clark and Amodei, Faulty Reward Functions in the Wild ↗
- 10Chrabaszcz, Loshchilov and Hutter, evolution strategies on Atari ↗
- 11SWE-bench, repository state loopholes ↗
- 12Goodhart’s law ↗
- 13Reverse Benchmarking
- 14The World Is Not Logged
- 15AlphaZero ↗
- 16Multi-armed bandit ↗