ZendoRL: an RL environment for teaching LLMs to run experiments

Introduction

In Robert Abbott's card game Eleusis, a secret rule accepts or rejects each card played. Players test ideas by playing cards, read the verdicts, and try to name the rule. Hugging Face turned the game into a benchmark for studying the scientific behaviour of LLMs (Louapre, Can LLMs Play the Game of Science?).

Rule discovery includes a judgement that is easy to overlook: when to stop gathering evidence. No number of experiments proves a rule, so a player has to decide when the evidence is enough to commit, and when another experiment is worth running. The models were overconfident: their stated confidence ran far above their actual accuracy. They also mistimed their guesses: most guessed too early, and a few waited long after they already had the rule.

This project set out to extend that study: turn rule discovery into a reinforcement-learning environment and train better behaviour, meaning experimenting before guessing and committing to a rule at the right moment.

The work is coordinated and published on Substrate, in the research Room Learning to test hypotheses: its hypotheses, pre-registered experiments, runs and findings are recorded there as the work happens. The benchmark is ZendoBench, and every experiment's design, code and results are in the zendo lab.

Q1Does it generalise?Do the same timing failures show up with scenes of pieces instead of a line of cards, and in other model families?
Q2Can RL teach it?Build an RL environment with an exact reward. Train an open-weight model to experiment and to commit at the right moment.
Q3Does it transfer?Test the trained model on games it never saw: Eleusis, ZendoWorld, Sudoku-Bench. Then test it on reasoning benchmarks.

Zendo

Zendo is a game of inductive inference. One player, the Master, invents a secret rule about arrangements of pieces, called koans. The students build koans, and the Master marks each one with a white stone if it follows the rule and a black stone if it does not. A student who thinks they have the rule guesses it. A wrong guess gets a counterexample: a koan the guess labels wrongly. ZendoRL is the single-player version: one player against an engine that holds the rule.

Fig. 2A · One piece, exploded

Parts list · pick a value

4 × 3 × 3 × 2= 72 piece types

Blocks look the same either way up, so a dot marks their top face. When the block is upside down, the dot is on the bottom, drawn here as a hidden line. The engine still tells the two apart: orientation is an attribute of every piece.

The student and the Master

The student starts with two free koans, called seeds: one that follows the rule and one that does not. The student has a budget of 30 points. An experiment costs 1 point. Submitting a rule costs 3, and the student can submit at most twice. See the full episode spec.

Budget · spent 0 of 30

Submissions · 0 of 2

Struck off

Hidden rule

Click a panel or use the buttons to step through; on a phone, scrolling down opens each panel in turn. Hover a piece for its attributes. In the DSL experiment the student submits the code; in the natural-language experiment, the sentence. Real games from the dev runs are in Fig. 3A.

Results

Win rates on the dev set: all 230 dev rules × 2 seed pairs, 460 games per model, one attempt each, one table per experiment (Fig. 5C). Dev rules are public, so these are not a held-out result; the sealed set (890 rules × 4 seed pairs) is kept for final evaluations. The headline is the mean of the five tier win rates, T2 to T6, with equal weights. T1 is played and reported but is diagnostic only: it never enters the headline.

Dev set · DSL experiment

Model	Headline	T1diagnostic	T2	T3	T4	T5	T6	Malformed	Games

Dev set · Natural-language experiment

Model	Headline	T1diagnostic	T2	T3	T4	T5	T6	Malformed	Games

How they played

Each model's 460 dev games in the DSL experiment, T1 included.

How they played

Style

Model	Experiments per game	Bits per experiment	Rules still possible at the first submission	Ideal player's P(win) then	Submissions contradicting what was shown

Experiments per gameNo better direction Scenes the player built before it stopped, over all 460 games. More is not better by itself; read it with the next columns.

Bits per experimentHigher is better Information each experiment gave: log₂ of the rules still possible before it over those after it, averaged over experiments. Early experiments give the most, so a player that stops early can score high here.

Rules still possibleLower is better Rules of the tier's catalog that agree with every scene shown so far: seeds, experiments and counterexamples. The median over games, at each game's first submission.

Ideal player's P(win)Higher is better The chance that submitting the most probable rule at that moment would win, under the engine's posterior. It is not the chance that the player's own rule wins, and it is not 1 over the rules still possible: the posterior weighs how likely each rule makes the seeds and counterexamples that were drawn. The mean over games.

ContradictingLower is better Submissions that give a scene already shown a different stone from the one it had, out of all submissions.

Replay a game

Each square is one dev game, green if won and red if lost: pick one to replay it below, move by move.

Fig. 3A · Dev games, move by move

Lab log

Experiments beyond the baselines, newest first: interventions now, RL runs once training starts.

Appendix: RL Environment

Rules, the hidden formulas that decide whether a scene gets a white or a black stone, come out of a generator. They are deduplicated by meaning, split by rule so no test rule is a training rule, and played and graded without a human in the loop. Everything here is the frozen engine and benchmark, ZendoBench 1.0.0. Click a chamber.

Fig. 5A · Section A–AIn: rule families → Out: reward 1 / 0 / unscored

Detail C6 · GraderScale 2:1

Z3 searches for a scene on which the submitted and hidden rules disagree. An exact evaluator checks every scene of 1 to N pieces, through a quotient that groups the piece types neither rule tells apart. A win or a loss needs both to agree, unless the two rules have the same normal form, which wins outright. If Z3 is still undecided after a 120-second retry, the exact check decides alone. Every win is checked again on every scene.

If a comparison still cannot be settled, the game is played once more from the start, then left unscored, never counted as a win or a loss. In the natural-language experiment the grader checks the judge's DSL translation of the English guess.

Z3 · search for a disagreeing sceneExact · check every scene

Both agree → win / lossZ3 undecided → the exact check decidesCannot settle → unscored

Under the rule limits, every pair of legal rules is decided: the bound on the exact check's cost is proven, not measured. The exact check agrees with an independent full-domain evaluator on every scene of the tier families at 4 and 5 pieces.

Spec · one episode

Scene: 1 to N pieces, with N = 3, 4 or 5 by tier. Pieces have no positions.
Hidden rule: A formula in a small Lisp-style language: one rule of the tier's generated catalog.
Start: Two labelled seed scenes, one positive and one negative. Each is drawn at random with its stone: a piece count from 1 to N, then each piece's type at random.
Experiment: Build any scene and see its stone. 1 point.
Submit: Propose a rule, in the DSL or in English depending on the experiment. 3 points, at most twice. Budget: 30 points.
Reply: One action per turn, as a JSON object. The player may reason first; the last action object in the reply is played, and nothing is repaired. Each turn shows all the evidence again; reasoning is not carried to the next turn. A reply that cannot be read is charged what it attempted.
Feedback: A wrong submission returns a counterexample: a scene of the smallest size on which the guess and the hidden rule disagree, drawn uniformly.
End: A correct submission, no submission left, or a budget that pays for no action.
Reward: 1 if the submitted rule labels every scene of 1 to N pieces as the hidden rule does, else 0; a game the grader cannot settle is unscored. Checked exactly, no partial credit. An English rule is first translated into the DSL by an LLM judge; the grader decides.

Legend · stones

White: the scene follows the rule.

Black: it does not.

Why only two submissions?

With unlimited submissions, the cheapest strategy is to guess, then patch the guess on each counterexample. It wins without a single experiment. With two, the player has to decide when it knows enough. In the Eleusis benchmark guesses are unlimited at a small cost each, so no wrong guess is final.

Fig. 5B · Section through a rule: "there is a blue pyramid"

(exists x0some piece x0

(andboth

(= (color x0) BLUE)is blue

(= (shape x0) PYRAMID)is a pyramid

))close

The language

Rules are s-expressions in a small first-order language: quantifiers, boolean connectives, counting, and two relational forms. same says two pieces have the same colour, shape or size; the two may be one piece. compare compares how many pieces meet two conditions.

Fig. 5C · Two experiments

ZendoRL runs two experiments. They differ in how the model states a rule, and both end in the same grader.

Experiment A · DSLReply: one JSON action

Each turn is a fresh prompt holding the game so far as JSON: every labelled scene, the submissions and verdicts, and the budget. The model reasons, then answers with one compact JSON action: build a scene, or submit a rule in the DSL. The last action object in the reply is played; a broken one is charged, never repaired. Local models can decode under a grammar that admits exactly the legal actions. A submitted rule goes straight to the grader, so no LLM touches the reward.

{"action":"experiment","scene":[{"color":"BLUE","shape":"WEDGE","size":"SMALL","orientation":"UP"}]}

{"action":"submit","rule":"(exists x0 (= (color x0) BLUE))"}

Experiment B · Natural languageReply: an English sentence

The model states its rule in English. An LLM judge translates the sentence into the DSL, and the grader checks the translation. The judge only translates. It never decides whether the rule is right.

"There is a blue piece."
LLM judge (exists x0 (= (color x0) BLUE))
grader win or counterexample

Why both?

The DSL experiment keeps every LLM out of the reward, but it also tests whether the model can write the DSL. The natural-language experiment drops the formal language and takes guesses the way the Eleusis benchmark and ZendoWorld do, at the cost of a judge that can mistranslate. Comparing the two separates reasoning from notation.

Fig. 5D · Tier ladder, elevation

Tiers go from rules about one attribute (T1, diagnostic only) to relations between pieces over 4 pieces (T5) and counting over 5 (T6). Bar height grows with the number of rules per tier, not to scale; the hatched part is the sealed set.