Opening Rankings from Engine Self-Play Are Less Certain Than They Look: Sampling Error, Draws, and Strength Dependence in Strength-Limited Stockfish Games

Introduction

An opening sets the position from which the rest of a chess game develops. Players and coaches often assume that an opening's practical value depends on who plays it. Sharp, theory-heavy systems may reward preparation and calculation, while quieter ones may forgive more error. This suggests a testable question: does the relative performance of an opening change with playing strength? One possible explanation is complexity. Openings that gain relative to others at higher strength might demand more precise play. Testing that would require a reliable pattern of strength dependence and a measure of complexity defined without reference to outcomes (§ 7).

Human game databases cannot answer this cleanly, because players choose openings for reasons of experience, preparation, style, and reputation. Strength-limited engine self-play gives more control. The same forced first move can be repeated against an opponent at the same nominal setting, without either side choosing or preparing. The cost is that a strength-limited engine's errors are not human errors, and its nominal settings are not human ratings. This paper therefore studies controlled engine behavior. It does not say which openings are best for people.

Study 1 analyzed 17,400 Stockfish 15.1 self-play games from 29 valid runs. At each of three nominal settings, the engine played ten forced White first moves and ten forced Black replies to 1.e4, with 290 games per opening-by-setting cell. The leading opening changed with strength. For White, 1.e4 led at the lowest setting, 1.d4 at the middle, and 1.Nf3 at the highest. For Black, 1...e5 led at the lowest and 1...c5 at the two higher settings. Two of these leads came down to a single game. Taken at face value, the changes suggest that the best opening depends on strength.

A change in the leader is not evidence that the opening effects changed. When several openings perform about equally and each rests on a few hundred games, chance alone can reorder them. We therefore ask two questions: do differences among openings change systematically with strength (an opening-by-strength interaction), and how many games are needed to rank close alternatives reliably? We call a ranking reliable if the same ordering would reappear in new games.

A null result is informative only if the analysis could have detected a real effect. We therefore use a Connect Four positive control to check detection of a large opening main effect, and sensitivity simulations to examine power for smaller effects, including chess opening-by-strength interactions (§ 4.2 and § 4.4). Replication and validation studies with complete game records address the limits of Study 1.

In brief, the results are as follows. No statistically reliable opening-by-strength interaction was found. Rankings from a few hundred games are unstable, and the design detects only fairly large single-opening effects. The 1.d3 deficit was not confirmed in new games. Survey-selected replies showed no statistically reliable advantage over free engine choice. The paper also makes four contributions: it reconciles and checks the Study 1 summaries, tests interactions and quantifies rank uncertainty, checks the analysis pipeline and extends the work to a newer engine, and proposes a protocol for future preregistration.

Background

Chess research shows that expertise changes how players perceive positions, not only how far they calculate, as de Groot [6], Chase and Simon [5], and Gobet and Simon [12] found. Error also depends jointly on skill and position difficulty [3]. These findings make a strength-dependent opening effect plausible, but they concern human players. Stockfish's UCI_LimitStrength option sets a target UCI_Elo value [23]. At reduced strength, it evaluates several candidate moves at shallow depth and applies a random bias that can favor a weaker move [24]. This error process differs from human error [17], and engine strength also depends on search depth and hardware [8]. The nominal values in this study are therefore labels for ordered settings, not human ratings or portable measures of strength.

Two outcome measures are used. Win percentage counts a draw as a non-win. Expected score gives one point for a win, half a point for a draw, and zero for a loss. If draws become more common at higher strength, win percentages can fall even when the balance between the colors barely changes. Connect Four provides a positive control for detecting a large opening main effect. The game is solved: under perfect play, the first player wins only by opening in the center column [2]. The effect size under the study bot is estimated from its games.

Finally, Study 1 was examined before its hypotheses and analyses were fully settled, so its inferential findings are exploratory. Preregistration fixes hypotheses before outcomes are seen [18], and flexible analysis can produce apparently meaningful findings [11, 15]. Study 4 tests the exploratory 1.d3 finding on games played after the hypothesis was fixed. Studies 5 and 6 examine related opponent-strength questions using a newer engine and a fixed node budget. § 7 sets out a protocol to be registered before new confirmatory games are played.

Methods

Study overview

The research program consists of six chess studies, a Connect Four positive control, and a separate robustness run. Study 1 provides the original strength-limited self-play dataset. The positive control tests whether the analysis detects a known large opening effect. Studies 2–4 address three limitations of Study 1: the lack of game records, uncertainty about how far apart the strength settings really were, and the exploratory status of the 1.d3 finding. Studies 5 and 6 instead hold Black at one nominal strength while varying White's strength; they use a newer engine, a fixed node budget, independent evaluation, and forced multi-move variations. Table 1 summarizes the designs.

Study	Purpose and comparison	Scheduled games	Outcome and status
Study 1	Ten White first moves and ten Black replies at three Stockfish 15.1 strength settings, with both sides at the same setting	18,000	Designated-color win (exploratory)
Study 2	Study 1 design repeated with complete game records and 30 games per cell	1,800	Wins and expected score (replication and robustness)
Study 3	Direct matches among the three settings, with colors alternated	800	Stronger setting's expected score
Study 4	New games testing whether 1.d3 scores below the other White moves (the test also uses the White-opening games from Study 2)	1,800 additional	White win and expected score (hypothesis fixed in advance but not publicly registered)
Study 5	Reply survey, followed by independent evaluation of the selected replies, a fixed book, and free Stockfish	6,960 survey and 5,220 evaluation	Black expected score (separate selection and evaluation data)
Study 6	Eighteen forced multi-move variations against two White strength settings	10,440	Black expected score (exploratory)
Connect Four	Seven forced opening columns, with the weakest bot on both sides	2,100	First-player win (positive control)

Note. Study 1 analyzed 17,400 of 18,000 scheduled chess games, and Connect Four analyzed 2,030 of 2,100, because one run's output was lost. A separate Studies 2–4 robustness run used a second computing environment and different randomized game orders; it is not a primary study.

Study 1: design and data

Engine configuration and treatments

Study 1 games were generated by the simulation module of a web-based game platform, using the configuration in source-code commit 5872d85. The server ran Stockfish 15.1 with UCI_LimitStrength enabled, one thread, a 1 MB hash table, and a fixed search time of 10 milliseconds per move. Nominal UCI_Elo values of 1320, 2000, and 2500 were requested. Because 1320 is below the minimum allowed by this release, the lowest setting actually ran at the 1350 floor. The values are treated as labels for three settings, not as calibrated Elo ratings. Games ran on a shared free-tier processor, so the number of positions searched within the time limit depended on the hardware.

Each run contained 600 chess games, 200 at each setting (Supplementary Figure S1). At each setting, 100 White-opening games forced one of ten White first moves, ten games per move. A separate set of 100 Black-reply games forced 1.e4 for White and then one of ten Black replies, again ten games per reply. Apart from the forced move, both sides used the same strength setting. The full list of treatments and simulator labels appears in Supplementary Table S2. The White treatments therefore fixed one move, while the Black treatments fixed the first two plies (half-moves). These are initial commitments, not complete named opening variations.

The engine received the current position as a FEN string on every move, rather than the full move history. Games ended by checkmate, stalemate, insufficient material, fivefold repetition, or the 75-move rule. Games that reached 300 plies were stopped and scored as draws. Supplementary Table S1 gives the full configuration, including concurrency, seed handling, library behavior, and source-code locations.

Runs, exclusions, and data recovery

Thirty runs were planned. The output of one completed run (Run 7) was lost before it could be recorded, so no data from it were analyzed. This left 29 valid runs, 17,400 games, and 290 games in each of the 60 opening-by-setting cells. No values were imputed and no other runs were excluded.

Study 1 did not save game records. Cell counts were reconstructed from archived screenshots of each run's summary and bar charts and summed across runs. Setting-level summaries were transcribed independently, giving 87 run-by-setting records. Run 12 at the 2000 setting lacked its summary panel, so its draw rate is missing. The six setting-by-color totals matched exactly.

This rules out large systematic errors, but not errors that cancel out within a run and setting.

Study 1: outcomes and analyses

Primary outcome and central estimand

The primary outcome for each cell was the designated-color win probability. This is the probability that White won after a forced White first move, or that Black won after a forced Black reply. Draws stayed in the denominator because Study 1 lacked per-opening draw counts. The central question (the estimand) was whether the differences among openings changed across settings, which is the opening-by-strength interaction. This is different from an overall change in win rate that affects most openings together. For each opening, its relative change was also summarized by a difference-in-differences contrast: its change from the lowest to the highest setting, minus the average change of the other nine openings of the same color.

Separate binomial logistic models were fitted for White and Black. An additive model with opening and strength was compared, by likelihood-ratio test, with a model that also included their interaction. Strength was treated as categorical. As a focused sensitivity analysis, a second model allowed each opening's effect to change linearly across the three ordered settings. Opening-specific contrasts used normal approximations, with Holm adjustment across the ten openings of each color. The less conservative Benjamini–Hochberg adjustment ([4]) was also reported as a sensitivity analysis.

Secondary and diagnostic analyses

Cell estimates use 95% Wilson intervals ([28, 1]). Within each color and setting, heterogeneity among the ten openings was tested with a $2 \times10$ Pearson chi-square test. Because the range between the highest and lowest win rates has no simple reference distribution, its null distribution was simulated from 100,000 samples of ten equal-probability binomial cells. Holm's ((1979)) procedure adjusts the six within-setting tests. Ranking reliability is the between-opening variance divided by the total variance, so values near 0 mean the ordering is mostly noise.

The 1.d3 contrast was tested with a Mantel–Haenszel common odds ratio stratified by setting ([16]), with a test of effect homogeneity. Because 1.d3 was selected as the weakest move after inspection, a Bonferroni adjustment for choosing one of ten candidates was also applied.

Rank uncertainty was evaluated with 20,000 parametric bootstrap samples drawn from the observed cell proportions ([7]). Because this bootstrap is centered on the observed rates, it favors the observed leaders, so its probabilities are optimistic. An empirical-Bayes shrinkage analysis gave a less optimistic alternative.

Run-level summaries provided independent replications of each setting-level rate. The displayed draw rate combined both sets, so White's first-move advantage could not be recovered exactly. If $W$ is White's win rate in the White set, $B$ Black's win rate in the Black set, and $D$ the combined draw rate, the advantage lies between $W - B - D$ and $W - B + D$ (Appendix A).

Power was estimated by normal approximation and by simulation (§ 4.2). Analyses used Python 3.12 with NumPy, pandas, SciPy ([27]), and statsmodels ([21]), with fixed random seeds.

Positive control and follow-up studies

Connect Four positive control

Each run also included 70 Connect Four games, ten per forced first-move column, with the weakest bot on both sides. This gives 2,030 games, or 290 per column. They were analyzed with chess main-effect procedures: Wilson intervals, a chi-square homogeneity test, a center-versus-other-columns contrast, and ranking reliability. This control does not test opening-by-strength interactions.

Study 2: replication with complete records

An independent Python program using python-chess 1.11 ((Fiekas, n.d.)) reproduced the platform's game loop and recorded every move list, result, termination reason, and game length. Study 2 used Stockfish 15.1 with the same requested settings, 10-millisecond move limit, FEN-only input, one thread, 1 MB hash, and 300-ply cap. Thirty games were played in each of the 60 Study 1 cells, for 1,800 games in all, in randomized order. The study ran on a dedicated modern laptop, which would be expected to search more positions in 10 milliseconds than the shared Study 1 server. Study 2 therefore tests whether the relative opening conclusions hold under the same software settings, not whether absolute outcome rates reproduce across hardware. Because Study 2 also used an independently written game loop, any change in absolute rates could be due to either the hardware or the implementation.

Study 3: actual strength gaps

Direct matches tested whether the settings really played at different strengths. Colors alternated in 200 games between the lowest and middle settings, 200 games between the middle and highest settings, and 400 games between the lowest and highest settings. The stronger setting's score (wins plus half of draws) was converted to an Elo difference using the logistic formula, with intervals based on the score's standard error. The widest nominal pairing received more games because an earlier run had produced an unexpectedly small estimated gap. This allocation therefore depended on earlier data.

Study 4: prospective test of the 1.d3 deficit

Before any follow-up games were played, an earlier version of this manuscript fixed the hypothesis that 1.d3 would score below the other nine White moves. It also fixed the test (a one-sided Mantel–Haenszel test stratified by setting) and the significance level ($\alpha= .05$). The hypothesis was not entered in a public registry. Study 4 added 60 games per White move per setting, or 1,800 games. The test combined these games with the 900 White-opening games from Study 2, giving 90 new games per move per setting. This gave about 70% power if the true effect matched the exploratory estimate. Because that estimate was the most extreme of ten candidates, the true deficit is probably smaller, and power for a smaller deficit is lower. Monte Carlo simulation of the prespecified test, at a 40% win rate for the comparison moves, gave about 36% power for a four-point gap and 24% for a three-point gap. The test used the original 10-millisecond configuration rather than the more demanding configuration proposed in § 7.

Independent robustness run

A separate run of the same program on a single-core cloud processor, with different randomized orders, is reported in § 4.4 and excluded from the primary analyses. It reveals instability from execution or hardware but cannot rule out errors shared by both runs.

Studies 5 and 6: fixed-compute extensions

Studies 5 and 6 used Stockfish 19 [25] with UCI_LimitStrength, one thread, a 16 MB hash table, and a fixed budget of 3,000 nodes per move. A node limit makes the amount of search independent of processor speed. Standard draw rules, including threefold repetition, were applied. Games that reached 400 plies without a result were censored (set aside) rather than scored as draws. Black always played at nominal 2000, and White played at either 1800 or 2000. These studies therefore ask whether Black's best opening choice depends on the opponent's strength. Cells were scheduled in 29 randomized blocks of ten games, and differences between blocks were used for inference. The absolute rates in these studies cannot be compared directly with those of Studies 1–4.

Study 5: selection and independent evaluation

White's first move was forced to 1.e4, 1.d4, or 1.c4. In a survey phase, four prespecified Black replies were compared for each White move at both White settings: 24 cells of 290 games, or 6,960 games. For each of the six combinations of White move and setting, a fixed rule selected the reply with the highest smoothed expected score. The resulting set of choices (the ”learned policy”) was then frozen. An independent evaluation crossed the three White moves and two White settings with three ways of choosing Black's reply: the learned policy, a prespecified fixed book, and letting Stockfish choose freely. This produced 18 new cells and 5,220 games. The two prespecified primary contrasts compared the learned policy with the free engine at each White setting, weighting the three first moves equally and applying Holm adjustment across the two tests.

Study 6: forced multi-move variations

Study 6 forced 18 multi-move sequences from six opening families: four Sicilian, four Ruy Lopez, four Queen's Gambit, two French, two Caro-Kann, and two English variations. Each variation was tested against White at 1800 and at 2000, producing 36 cells of 290 games and 10,440 scheduled games. The outcome was Black's expected score among completed games. Pairs within the same family were compared with paired $t$ tests on the 29 block differences, with Holm adjustment across 42 strength-specific comparisons. The 21 variation-by-strength interaction tests were adjusted separately. These comparisons were exploratory. Full sequences, schedules, candidate policies, and configuration fingerprints are provided in the supplementary material.

Reproducibility

The Zenodo archive [10 https://doi.org/10.5281/zenodo.23357406] contains reconstructed Study 1 summaries, game records for the follow-up studies, protocols, and analysis code. The Declarations describe its contents.

Results: Opening Performance, Ranking Uncertainty, and Validation

Sections 4.1–4.3 cover Study 1 outcomes, strength interactions, and ranking uncertainty. Section 4.4 reports validation and replication; Section 4.5 presents fixed-compute extensions involving unequal-strength opponents.

Data integrity and setting-level outcomes

Data integrity

Study 1 contained 290 games in each of the 60 opening-by-setting cells. White won 1,478, 1,316, and 1,230 of the 2,900 White-opening games at the 1320, 2000, and 2500 settings. Black won 1,281, 1,140, and 1,128 of the corresponding Black-reply games. These six totals matched the independently transcribed run summaries exactly, and no duplicate run records were found. This agreement makes a large systematic transcription error unlikely.

Wins, draws, and first-move advantage

As the nominal setting increased, designated-color win rates fell and draws became much more common (Table 2 and Figure 1). White's win rate in the

Setting	White wins, White set	Black wins, Black set	Draws	Implied White expected scorea
1320	51.0% [48.9, 53.1]	44.2% [42.1, 46.2]	1.1% [0.8, 1.4]	53.4%
2000	45.4% [44.0, 46.7]	39.3% [37.7, 40.9]	12.8% [11.9, 13.7]	53.0%
2500	42.4% [40.2, 44.6]	38.9% [37.2, 40.6]	16.2% [15.2, 17.2]	51.8%

White-opening set fell from 51.0% at 1320 to 45.4% at 2000 and 42.4% at 2500. The decline from 1320 to 2500 was consistent across runs, $t(28) = 5.49$, $p < .001$. Black's win rate in the Black-reply set fell from 44.2% to 39.3% and 38.9%. Over the same settings, the pooled draw rate rose from 1.1% to 12.8% and 16.2% (Kruskal–Wallis $p < .001$, and $p < .001$ for 2000 versus 2500).

Note. Bracketed values are 95% t intervals across runs. The 2000 draw estimate is based on 28 runs because one run summary was unavailable. ^aAssumes equal draw rates in the White-opening and Black-reply sets at each setting.

The rise in draws explains most of the drop in designated-color wins. If the White-opening and Black-reply sets are assumed to have equal draw rates within a setting, White's expected score changes much less than its win rate: 53.4%, 53.0%, and 51.8%. The lower win rates at higher settings should therefore not be read as a matching decline in the quality of White's positions. Study 2 later shows that this absolute draw pattern did not reproduce on different hardware with an independently written game loop.

At 1320, where draws were rare, the available summaries bound White's first-move advantage between 5.7 and 7.9 percentage points, and its expected score between 52.8% and 53.9%. At 2000 and 2500, the bounds include zero, because Study 1 did not record draw counts separately for the two sets. Assuming equal draw rates gives first-move-advantage estimates of 6.1 and 3.5 points (Supplementary Table S9). The complete Study 2 records give direct, though less precise, estimates (§ 4.4).

Setting separation and run-level dispersion

Run-to-run dispersion ratios ranged from 0.50 to 1.41, and none of the nine overdispersion tests reached .05 (Supplementary Table S10). The largest, for White wins at 2500, gave $p = .075$. These checks support the binomial models at the setting level, but they cannot detect dependence confined to one opening. Study 3 is the stronger test of strength ordering.

Opening differences and strength dependence

Differences within settings

Uncertainty at the level of individual openings was large (Supplementary Figures S2 and S3). With 290 games per cell, a typical 95% Wilson interval spans about 11 percentage points. The observed ranges among White moves were 12.1, 13.4, and 13.8 points at the three settings, compared with 8.3, 8.6, and 10.3 points among Black replies. In simulations where all ten openings had the same true win probability, the typical observed range was about 9 points, and a range near 13 points occurred about 5% of the time (Supplementary Figure S4). The White spreads are therefore somewhat larger than chance alone would usually produce, while the Black spreads are consistent with chance.

Separate homogeneity tests pointed in the same direction but were not conclusive (Supplementary Table S4). The White tests gave $p = .191$, $.077$, and $.029$ at 1320, 2000, and 2500. The Black tests gave $p = .590$, $.287$, and $.454$. None remained significant after Holm adjustment across the six tests (smallest adjusted $p = .174$). Estimated ranking reliability for White rose across the settings from .28 to .42 and .52, but for Black it was .00, .17, and .00. Even where some White moves appear to differ, sampling error still accounts for much of the observed ordering.

Central interaction test

The main analysis asked whether the differences among openings changed across strength settings. Strength had a clear main effect on designated-color wins for both colors (Table 3). After adjusting for strength, the White moves also differed overall, LR $\chi^2(9) = 26.19$, $p = .002$, but the Black replies did not, LR $\chi^2(9) = 15.01$, $p = .091$. Much of the White opening effect comes from 1.d3 (see below). These main effects describe average differences among openings. They do not show that the openings' relative performance changes with strength.

Color	Term tested	LR $\chi^2$	df	p
White	Strength main effect	43.99	2	< .001
White	Opening main effect	26.19	9	.002
White	Opening × strength (general)	20.40	18	.311
White	Opening × linear strength trend	6.89	9	.649
Black	Strength main effect	20.60	2	< .001
Black	Opening main effect	15.01	9	.091
Black	Opening × strength (general)	12.09	18	.843
Black	Opening × linear strength trend	7.88	9	.546

Note. Strength is tested against an intercept-only model, opening against a strength-only model, and each interaction against the additive model. The linear-trend test allows each opening's logit effect to change linearly across the three ordered settings.

No statistically reliable opening-by-strength interaction was found. For White, the general interaction test gave $p = .311$ and the focused linear-trend test gave $p = .649$. For Black, the corresponding values were .843 and .546. The additive models describe the cell counts adequately. In other words, the data are consistent with all openings shifting together on the logit scale as the setting changes (Supplementary Figure S5). This does not prove that all interaction effects are zero. Realistic small differences remain unresolved with these sample sizes.

To estimate what the omnibus tests could have detected, their power was approximated with a normal approximation. This assumed independent interaction effects with a common standard deviation across the 30 opening-by-setting cells of each color, and a mean win probability of 45%. Under these assumptions, the 18-df tests would have had roughly 10%, 35%, and 77% power for interaction standard deviations of 1, 2, and 3 percentage points, and about 98% power at 4 points. A single opening that departs from the others is harder to detect. Its contrast would need to be roughly 12 to 16 points to reach 80% power. A simulation using binomial logistic likelihood-ratio tests gave similar power values of 11%, 36%, 74%, and 93%. For a single opening, however, the omnibus test is weak. A simulation of one opening with a 12-point effect detects it in only about 39% of studies (Table 4). The approximations above describe diffuse interactions across many openings, and they should not be read as power for one opening.

True effect (pp)	Connect Four: $\chi^2$ homogeneity test	Connect Four: center contrast	Study 1: single-opening contrast (uncorrected)	Study 1: omnibus interaction LR test
0 (no effect)	5.1%	6.3%	4.5%	4.3%
2.2	6.7%	10.8%	—	—
3	8.8%	16.1%	10.0%	7.8%
5.6	23.2%	42.0%	18.3%	10.0%
8	41.6%	71.3%	45.5%	20.8%
11.2	77.4%	94.4%	—	—
12	—	—	75.0%	38.8%
16.9	99.0%	100.0%	—	—
22.5	100.0%	100.0%	—	—

The opening-specific comparisons lead to the same conclusion (Figure 2 and Supplementary Tables S5 and S6). The only absolute change from 1320 to 2500 that survived Holm adjustment was the decline for 1.Nc3, from 51.4% to 39.0% (−12.4 points, 95% CI [−20.5, −4.4], adjusted $p = .025$). Relative to the average change of the other White moves, however, its decline was only −4.3 points (95% CI [−12.8, 4.2]). The largest relative changes were +6.8 points for 1.Nf3, +6.4 for 1.b3, and +7.8 for Black's 1...c5, but every interval included zero. No difference-in-differences contrast survived Holm or Benjamini–Hochberg adjustment.

Exploratory 1.d3 finding

Much of the White opening main effect came from 1.d3. It was the lowest-scoring White move at 1320 and 2500 and the second lowest at 2000. Pooled across settings, 1.d3 won 40.0% of 870 games, compared with 46.9% for the other nine moves (Supplementary Table S7). The stratified common odds ratio was 0.75 (95% CI [0.65, 0.87], $p < .001$), with no sign that the effect differed across settings ($p = .523$). The result stayed significant after correcting for selecting the weakest of ten moves. Because the contrast was identified after looking at Study 1, it is treated as exploratory and tested prospectively in § 4.4.

Ranking uncertainty and required sample size

Uncertainty in the observed leaders

Which opening came first was much less stable than the overall opening effects. In the parametric bootstrap, the observed White leader ranked first in 29% of replicates at 1320, 74% at 2000, and 68% at 2500 (Supplementary Figure S6). Because these simulations are centered on the observed rates, they favor the observed leaders. After empirical-Bayes shrinkage, the corresponding probabilities fell to .21, .53, and .54. For Black, the estimated between-opening variance was zero at 1320 and 2500, which gives every reply a .10 probability of being first. At 2000, the observed Black leader's probability was .21. Most 95% rank intervals covered at least six of the ten positions (Supplementary Tables S7 and S8).

Spearman rank correlations between settings ranged from .17 to .61 for White and from .16 to .71 for Black. These modest and variable correlations do not by themselves show that the underlying rankings changed. When true differences between openings are small relative to sampling error, observed rankings can shift a great deal even if the true ordering stays the same.

Games required to identify the leader

Simulations showed what this uncertainty means for sample size (Figure 3). The results can be reproduced by simulating ten openings whose true win probabilities are normally distributed, with a mean of 45% and a between-opening standard deviation of 2, 3, or 4 percentage points. With the current 290 games per opening and a 3-point standard deviation, the opening with the highest observed rate was truly the best in about 44% of 20,000 simulations. The figure was about 33% with a 2-point and 52% with a 4-point standard deviation. Reaching an 80% chance of identifying the best opening required about 5,000 games per opening when the standard deviation was 3 points, and about 10,000 when it was 2 points. Even at 20,000 games per opening, the observed leader was wrong in 7% to 15% of simulations. These values depend on the stated model and should not be treated as universal thresholds.

The between-opening standard deviations implied by the Study 1 reliability estimates (§ 4.2) were about 1.8 to 3.0 points for White and 0 to 1.3 points for Black. The 3-point scenario is therefore near the top of the White estimates and above the Black estimates. Where true differences are smaller, leaderboards are even less reliable than the 44% figure suggests.

A less demanding goal is easier to reach. With 290 games per opening and a 3-point standard deviation, the truly best opening was among the observed top three in 78% of simulations. The evidence therefore supports drawing up a shortlist of plausible leaders more readily than naming a single best opening.

Validation and replication

Connect Four positive control

The first player won 63.1% of games from the center column (95% CI [57.4, 68.5], or 183 of 290 games). The other six columns ranged from 32.1% to 45.2%. The homogeneity test across columns was significant, $\chi^2(6) = 64.47$, $p < .001$. The center column exceeded the mean of the other six by 22.5 points (95% CI [16.5, 28.5]). Ranking reliability was .91, and in a parametric simulation the center column led in every replicate (Figure 4). Because each run contributed only ten games per column, the center column had the highest score, counting ties, in 20 of the 29 runs. The control therefore demonstrates that the opening main-effect procedures can detect a large effect at this sample size; it does not directly validate the chess opening-by-strength interaction test. We also examined where detection fails for smaller effects. We calibrated simulations to the observed Connect Four rates (40.6% for the other six columns, 63.1% for the center) and diluted the center effect by known fractions, giving true effects of 22.5, 16.9, 11.2, 5.6, and 2.2 points. In the same way, we simulated the Study 1 design with a single opening whose effect at the highest setting differs from the others. Table 4 gives the detection rates.

Note. Connect Four rows at 0, 3, and 8 points use 2,000 simulated studies each. The other Connect Four rows use 4,000 studies each. Study 1 rows use 400 studies per effect, so Monte Carlo error is about 2 to 3 points. The Study 1 contrast uses an uncorrected 5% test. The 5.6-point row for Study 1 uses the 5-point simulation. With a Bonferroni correction over ten openings, the 12-point Study 1 power falls to about 44%. The 0-point rows show that the center contrast is slightly liberal (6.3%) under its normal approximation.

Study 2: complete-record replication

Study 2 did not reproduce Study 1's absolute setting-level rates (Supplementary Table S11). At 1320, draws rose from 1.1% to 18.8%, White wins fell from 51.0% to 43.0%, and Black wins fell from 44.2% to 35.7%. At 2500, draws rose from 16.2% to 24.2%. At 2000, the two studies were similar. The Study 2 draw rates therefore did not rise steadily with strength. They were 18.8%, 12.7%, and 24.2% at the three settings. Study 2 differed from Study 1 in both processor and implementation (an independently written game loop, § 3.4). These comparisons show that absolute rates depend on the full configuration, but they cannot isolate hardware as the cause.

The Study 2 draw pattern is unlikely to be explained by sampling noise alone. Of the 113, 76, and 145 draws at 1320, 2000, and 2500, 46, 36, and 72 ended in insufficient material, 22, 13, and 33 by fivefold repetition, and 35, 23, and 30 at the 300-ply cap. The 2000 setting has fewer draws than either neighbor. The $\chi^2$ test against 1320 gives $p = .004$, and the test against 2500 gives $p < .001$. The draws are therefore mostly endgame endings, and the setting that resolves the most games decisively is the middle one. The data show how games end but not why. This pattern motivates a hypothesis for the proposed confirmatory study, which will record termination reasons and game lengths for every game.

The complete records show no repeated games, since no two shared their first 20 plies, and draw-rate tests across the ten forced moves were not significant after adjustment ($p$ from .132 to .937). Direct estimates of White's first-move advantage were 4.2 points at 1320 (95% CI [−3.0, 11.4]), 4.3 at 2000 ([−3.1, 11.8]), and 2.5 at 2500 ([−4.5, 9.5]).

Opening effects did not differ between the two studies for White, $\chi^2(9) = 9.29$, $p = .411$, or for Black, $\chi^2(9) = 7.10$, $p = .627$. No opening-by-strength interaction was found in Study 2 alone (White $p = .543$, Black $p = .120) or in the combined data (White $p = .407$, Black $p = .680$). Expected-score models gave the same result.

Study 3: validation of strength ordering

Direct matches confirmed that each setting beat the one below it under the study conditions (Supplementary Table S13). The 2000 setting won all 200 games against the lowest setting. The 2500 setting scored 93.8% against 2000, which corresponds to an estimated 470-point difference (95% CI [401, 577]). Against the lowest setting, however, 2500 scored 93.9% over 400 games, an estimated 474-point difference ([417, 555]). This is far below the nominal gap of 1,150 points. The pairwise results do not fit a single scale. Against the lowest setting, the 2000 setting scored 100% while the 2500 setting scored only 93.9%, so the nominally weaker setting did better against the same opponent. The 2000 setting won all 200 of its games against the lowest setting, whereas the 2500 setting won 369 of 400, drew 13, and lost 18. All 18 losses ended in checkmate, and they were long, averaging 155 plies. Games against the lowest setting were also much longer at 2500 (116 plies on average) than at 2000 (74). These long-game losses raise the possibility that game conversion, rather than only opening play, contributed to the unexpected pairwise scores. We cannot identify the mechanism from these data. One candidate is the random move selection used at limited strength, which can matter more in long games. Whatever the cause, the inconsistent pairwise performance margins do not support interpreting the nominal labels as calibrated distances on one Elo scale. The comparisons support treating the settings as ordered experimental conditions, not as literal or uniformly spaced Elo ratings.

Study 4 and the separate robustness run

In the 2,700 White-opening games played after the 1.d3 hypothesis was fixed, 1.d3 won 37.4% of its 270 games, compared with 40.0% for the other moves (Supplementary Table S14). The prespecified one-sided Mantel–Haenszel test did not reach the threshold: common odds ratio 0.90, 95% CI [0.69, 1.16], $p = .208$. Expected scores were 46.5% and 49.7% ($p = .134$). The Study 1 deficit was therefore not confirmed in new games, although the new estimate pointed in the same direction. The interval includes both no effect and the Study 1 estimate. The follow-up therefore neither confirms nor rules out a deficit as large as the one in Study 1, and it does not show that 1.d3 is equivalent to the other moves. Pooling these data with the separate robustness run gave a similar odds ratio of 0.91 (95% CI [0.75, 1.09]), one-sided $p = .146$, but that combined analysis was not prespecified.

The separate robustness run gave setting-level rates within 4.2 points of the laptop run, found no interaction (White and Black $p = .521$), and did not confirm the 1.d3 deficit (odds ratio 0.91, one-sided $p = .249$). Because both follow-up runs used the same program, it is a robustness check rather than an independent software replication.

Fixed-compute extensions

Study 5: independent evaluation of selected replies

The Study 5 survey produced 6,955 completed games, and the independent evaluation produced 5,216 of 5,220 scheduled games, with censored games excluded. The learned policy chose different replies by opponent strength after 1.e4 and 1.d4, and chose 1...c5 at both settings after 1.c4 (Supplementary Table S15). Several leading survey estimates were close, so the selected replies were themselves uncertain.

Replies selected from the survey showed no statistically reliable advantage over letting Stockfish choose freely (Figure 5 and Supplementary Table S16). Against White at 1800, the learned policy's Black expected score was 3.32 points lower than the free engine's (95% CI [−7.38, +0.74]). Against White at 2000, it was 3.93 points lower ([−8.69, +0.84]). Both Holm-adjusted $p$ values were .213. The fixed book did not differ detectably from the free engine, and all six opening-specific estimates were negative but included zero (Supplementary Table S17). The study therefore neither shows that selection helps nor shows that it harms.

Study 6: forced multi-move variations

Study 6 recorded 10,434 completed games of 10,440 scheduled (3,679 White wins, 4,881 Black wins, and 1,874 draws, with six censored). Black's mean expected score was 64.55% against White at 1800 and 46.97% against White at 2000 (Figure 6 and Supplementary Table S18). This 17.6-point shift with White's setting dominates the figure. Seven repeated move sequences in two Caro-Kann Gurgenidze cells changed that variation's score by less than one point.

uled games per point; six games censored overall). Horizontal rules separate opening families, and the dotted line marks 50%.

Two of the 42 exploratory within-family comparisons survived Holm adjustment (Supplementary Table S19). Against White at 2000, the Najdorf Sicilian scored 13.68 points above the Scheveningen (adjusted $p$ = .0013, multiplicity-adjusted 99.88% CI [+3.73, +23.62]). Against White at 1800, the Queen's Gambit Declined scored 13.28 points above the Slav (adjusted $p$ = .0053, 99.88% CI [+2.49, +24.07]). These estimates are larger than the estimated differences among first moves in Studies 1 and 2. They are, however, the largest of 42 exploratory comparisons and are therefore likely to overstate the true differences. Forced multi-move lines are also a different kind of treatment from single first moves.

No variation-by-strength interaction survived adjustment across the 21 tests, and the smallest adjusted $p$-value was .433. Study 6 therefore finds two exploratory differences among complete variations that survive adjustment, but no statistically reliable evidence that their relative performance changes with the opponent's strength.

Discussion

Principal finding

The leading opening changed across strength settings, but there was no statistically reliable evidence that the best opening changed with strength. Neither interaction test was significant for either color, no opening-specific change survived correction, and the follow-up studies agree. The Connect Four control and the sensitivity analysis show that the pipeline finds large effects, but that effects of a few points would often be missed (Table 4). The changes in leaders therefore do not establish strength dependence, and modest interactions remain possible.

Why rankings and win rates mislead

The observed leader is sensitive to sampling error. At a between-opening standard deviation of three points, the observed leader after 290 games per opening was truly best in only about 44% of simulated studies (Figure 3). A ranking is therefore an estimate with uncertainty, not a directly observed property of an opening. Win rates also move with draw rates. In Study 1, draws rose sharply at the higher settings, so wins for both colors fell together while expected score changed little. Absolute rates also changed on different hardware and with a different game loop (Study 2), so they should be read as properties of the full configuration.

Selection, complexity, and the 1.d3 result

Replies selected because they scored best in the survey did no better than letting Stockfish choose freely in new games. This is the pattern expected from choosing the maximum of several noisy estimates. The survey did not include the free-engine comparison, however, so other explanations remain possible. Selection and evaluation data should always be kept separate when an experiment is used to recommend an opening.

The complexity hypothesis is not tested here, because the completed studies found no reliable pattern of changing relative performance for complexity to explain. Classifying openings as complex after seeing their rankings would be circular, so a valid test needs a measure defined in advance (§ 7). The 1.d3 result shows the same need. Its Study 1 deficit of about seven points shrank to about two to three points in new games, below the prespecified threshold.

What this means for choosing an opening

The study does not identify an opening that suits a particular level, and it gives no basis for recommending a first move or reply to players of any rating. The leader at each setting changed within the range that sampling noise produces, and the best survey replies did no better than free engine choice. The settings are also not human ratings (§ 6.4). Even a reliable engine-level interaction would describe Stockfish, not club or tournament players.

One possible extension is that humans could adopt the engine's tendencies, meaning the openings the engine scores well with at a given setting. This is untested, and there are reasons for caution. Human players study some openings intensively and neglect others, so an opening the engine favors may be unfamiliar to people, or already heavily analyzed and well defended. Engine tendencies also did not transfer even within the engine, as § 5.3 shows. Testing the idea would require human games with controls for preparation and selection, such as randomized assignment of openings in a controlled setting or a carefully matched comparison of human results. Until such a test supports them, claims of a level-specific best opening should be treated with caution.

Limitations

Data provenance

Study 1 was reconstructed from screenshots of summary displays rather than from game records. The per-cell totals matched independently transcribed summaries exactly, which makes a large systematic error unlikely, but errors that cancel out among openings within a run cannot be excluded. Study 1 also lacks move sequences, per-opening draw counts, and termination reasons. Study 2 supplies complete records under the same software settings, but it is much smaller and ran on different hardware with an independently written game loop.

Strength conditions

The nominal labels are ordered experimental settings, not calibrated Elo ratings. Stockfish 15.1 calibrates UCI_Elo under much longer time controls than the 10 milliseconds per move used here, so actual strength was probably below the nominal values, and the gaps need not match the labels. The platform also departed from standard play: the engine received the position without the move history, and fivefold rather than claimable threefold repetition applied. These choices may have changed game length and draw frequency.

Statistical and confirmatory limits

The design has limited sensitivity to small interactions (Table 4). Multiplicity adjustments reduce false positives within families of tests but do not turn exploratory selection into confirmation. The Study 4 hypothesis was fixed in an earlier version of this manuscript but was not publicly registered. The reply sets and variations in Studies 5 and 6 were chosen by the authors, so those contrasts remain exploratory.

Scope

The findings apply to the tested Stockfish releases, strength mechanisms, search budgets, hardware, and forced treatments. Nominal settings are not measures of human skill, so the results should not be applied directly to human players. Studies 1–5 forced only a first move or reply, and Study 6 covered 18 sequences. Studies 5 and 6 used a newer engine release, a node budget, different draw rules, and an asymmetric strength design, so they extend rather than replicate the original evidence.

Proposed Confirmatory Study

Rationale

The completed studies leave small strength-dependent effects unresolved. The proposed confirmatory study tests a prespecified opening-by-strength interaction, evaluates a complexity measure defined without reference to outcomes, and reassesses the 1.d3 result under a reproducible configuration. It will also include a real small-effect check, in which known handicaps are implanted in actual games, to confirm that the analysis detects effects of the size predicted by the sensitivity simulations. No data from this study have been collected or analyzed. The protocol will be registered on OSF Registries before any confirmatory games are generated.

Planned hypotheses

Table 5. Planned hypotheses for the proposed confirmatory study.

Note. Entries marked as still to be resolved are not confirmatory specifications until they are finalized in the registration.

Design and sample size

The design keeps the ten White first moves and ten Black replies. It uses 2,100 games per opening-by-setting cell at the lowest and highest settings, and 700 at the middle setting. For an 8-point interaction contrast, about 1,215 games per cell give 80% power at $\alpha= .05$, and about 2,060 at $\alpha= .005$ (Figure 7). The complete design requires 98,000 games. Games will be randomized, each will have a logged seed, and the archive will keep complete PGN records, termination reasons, and ply counts.

Complexity index and analysis

For each forced move, 500 positions will be sampled from full-strength continuations and analyzed at a fixed node budget. The index combines breadth (the number of moves within 50 centipawns of the best move), narrowness (the proportion of positions with only one such move), and cost (the second-best move's evaluation loss, capped at 300 centipawns). It is standardized as $(z_{\mathrm{narrow}} + z_{\mathrm{cost}} - z_{\mathrm{breadth}}) / 3$. The primary model, the multiplicity correction, and the exclusion rules will be fixed in the registration. A run will be excluded only for a documented technical failure identified without looking at its outcomes.

Conclusion

Across 17,400 strength-limited Stockfish games, a Connect Four positive control, replication and validation studies, and two fixed-compute extensions, the observed opening leaders changed more readily than the estimated differences among openings. No completed analysis found a statistically reliable opening-by-strength interaction. That does not show that all interactions are absent, and the sensitivity analysis shows that the design would often miss effects of a few points. The exploratory 1.d3 deficit was not confirmed in new games, and replies selected from one set of engine games did no better than free engine choice. A leaderboard built from a few hundred games should be treated as a noisy estimate, not a stable hierarchy. Reporting uncertainty, testing interactions directly, keeping complete records, documenting the full configuration, and separating selection from evaluation data would make future engine-opening comparisons more credible.

Declarations

Data and code availability

The archived record at Zenodo [10 https://doi.org/10.5281/zenodo.23357406] contains the transcribed Study 1 data (cells.csv and run_summaries.csv), the Connect Four data, the original screenshots, the replication program, complete game records for Studies 2–4 and the robustness run in JSON Lines format with move lists, game-level CSV and PGN records for Studies 5 and 6, protocols, schedules, analysis outputs, and the analysis and figure scripts. The source code is also on GitHub at https://github.com/FLICK11-hash/gaming-platform. Supplementary Table S20 gives the Connect Four column counts.

Column	First-player wins / 290	Win % [95% CI]
1	114	39.3 [33.9, 45.0]
2	121	41.7 [36.2, 47.5]
3	118	40.7 [35.2, 46.4]
4 (center)	183	63.1 [57.4, 68.5]
5	131	45.2 [39.5, 50.9]
6	130	44.8 [39.2, 50.6]
7	93	32.1 [27.0, 37.6]

Author contributions

Conrad Flick: Conceptualization, Methodology, Software, Investigation, Data curation, Formal analysis, Visualization, Writing – original draft, and Writing – review & editing. Jaysen Jensen: Writing – review & editing, including editorial restructuring and revision for clarity and accessibility. Jingyu Huang: Methodology, Software, Investigation, Formal analysis, Data curation, and Writing – original draft for Studies 5 and 6. Mario Camara Paslar: Writing – review & editing, including revision of the manuscript and recommendations about its presentation and interpretation.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors declare no potential conflicts of interest with respect to the research, authorship, or publication of this article.

Acknowledgments

The authors thank Carey Main of the University of Chicago, who mentored Conrad Flick through a University of Chicago summer mentorship, for her guidance, for suggestions that shaped parts of the project, and for contributing portions of the simulation platform code.

AI-assisted tools disclosure

Claude (Anthropic) was used to assist with grammar and wording, figure preparation, and code generation. Its code contributions included portions of the simulation platform, as recorded in the repository's commit history, and the replication program and statistical analysis scripts. Claude also assisted with drafting and revising portions of the manuscript. The independent robustness run was executed by Claude on a cloud processor and is reported separately from the primary analyses. Codex (OpenAI) was used during an earlier revision to assist with editorial restructuring, language revision, HTML formatting, and the placement of tables and figures. In the most recent revisions, Claude shortened the manuscript, drafted the human-copying paragraph in § 5.4, wrote the sensitivity simulations and the explanations of the Study 2 and Study 3 patterns, restored the Connect Four control, rebuilt the tables, regenerated the supplementary exhibits from the archived scripts, and prepared the LaTeX source. The authors conducted all primary experiments and take full responsibility for the content of the article.

Appendix A. Derivation of the First-Move Advantage Bounds

Consider one strength setting, with 100 games in the White-opening set (set 1) and 100 games in the Black-reply set (set 2). Let $W$ be White's win percentage in set 1, $B$ Black's win percentage in set 2, and $d_1$ and $d_2$ the draw percentages in the two sets. Every game ends in a White win, a Black win, or a draw. Black therefore wins $100 - W - d_1$ percent of set 1, and White wins $100 - B - d_2$ percent of set 2.

Across the pooled 200 games, White's win percentage is $[W + (100 - B - d_2)] / 2$ and Black's is $[B + (100 - W - d_1)] / 2$. Their difference is

If the displayed draw rate $D$ is the draw percentage over all 200 games, then $D = (d_1 + d_2) / 2$, with $d_1 \ge0$ and $d_2 \ge0$. The term $(d_1 - d_2) / 2$ is largest when $d_2 = 0$ and $d_1 = 2D$, and smallest in the reverse case. Therefore

When $d_1 = d_2$, $\Delta= W - B$. White's expected score in the pooled games is White's win percentage plus $D / 2$. This simplifies to $(W + 100 - B) / 2 + (d_1 - d_2) / 4$, and it therefore lies within $(W + 100 - B) / 2 \pm D / 2$.

Averaging these bounds across runs gives the values in Supplementary Table S9. Because the bounds are linear in $W$, $B$, and $D$, the bound for the mean equals the mean of the run-specific bounds. Tighter bounds could be obtained by capping $d_1$ and $d_2$ based on the observed spread of run-level draw rates, but the analyses do not rely on such caps.

Reply	Wins / 870	Win % [95% CI]	Deviation from setting mean (pp)	Rank 1320	Rank 2000	Rank 2500
1...c5 (Sicilian)	383	44.0 [40.8, 47.3]	+3.2	1–10	1–7	1–5
1...e5 (Open Game)	377	43.3 [40.1, 46.6]	+2.5	1–7	1–9	1–9
1...e6 (French)	376	43.2 [40.0, 46.5]	+2.4	1–9	1–7	1–9
1...d5 (Scandinavian)	359	41.3 [38.0, 44.6]	+0.5	1–9	2–10	1–10
1...Nc6 (Nimzowitsch)	354	40.7 [37.5, 44.0]	−0.1	3–10	1–8	1–10
1...d6 (Pirc-style)	350	40.2 [37.0, 43.5]	−0.6	1–9	2–10	2–10
1...g6 (Modern)	345	39.7 [36.5, 42.9]	−1.1	1–10	1–9	4–10
1...Nf6 (Alekhine's)	343	39.4 [36.2, 42.7]	−1.4	1–9	3–10	3–10
1...c6 (Caro-Kann)	332	38.2 [35.0, 41.4]	−2.6	3–10	5–10	2–10
1...a6 (St. George)	330	37.9 [34.8, 41.2]	−2.9	4–10	4–10	2–10

Supplementary Material

The supporting research package is available through the project repository and the permanent Zenodo archive, which is the version of record for the deposited materials. It contains the Study 1 transcriptions (cells.csv and run_summaries.csv) and original simulator screenshots, the replication program and engine-build instructions, complete Study 2–4 and robustness-run records in JSON Lines format with move lists, Study 5–6 game-level CSV and PGN records with protocols, schedules, and analysis outputs, and the analysis and figure scripts.

Supplementary tbl:S1–S19 follow the order of the paper, from configuration and design through the six completed studies. Table S20 gives the Connect Four positive-control data. Supplementary Figure S1–S7 follow in the same order. Exhibits that already appear in the main text or appendices are not repeated here. Values in Table S8–S14 were produced by the archived analysis scripts, values in Table S15–S18 come from the archived Study 5 and 6 analysis outputs, and Table S20 comes from the archived Connect Four analysis.

Move	Wins / 870	Win % [95% CI]	Deviation from setting mean (pp)	Rank 1320	Rank 2000	Rank 2500
1.d4 (Queen's Pawn)	425	48.9 [45.5, 52.2]	+2.6	1–9	1–4	2–10
1.Nf3 (Reti)	422	48.5 [45.2, 51.8]	+2.3	1–9	3–10	1–4
1.c4 (English)	420	48.3 [45.0, 51.6]	+2.0	1–8	1–8	2–9
1.e4 (King's Pawn)	419	48.2 [44.9, 51.5]	+1.9	1–8	2–10	1–7
1.b3 (Nimzo-Larsen)	416	47.8 [44.5, 51.1]	+1.6	2–10	1–8	1–7
1.e3 (Van 't Kruijs)	412	47.4 [44.1, 50.7]	+1.1	1–9	1–8	2–9
1.g3 (King's Fianchetto)	396	45.5 [42.2, 48.8]	−0.7	1–9	2–9	3–10
1.f4 (Bird's Opening)	390	44.8 [41.6, 48.1]	−1.4	1–9	2–9	3–10
1.Nc3 (Van Geet)	376	43.2 [40.0, 46.5]	−3.0	1–9	6–10	4–10
1.d3 (King's Indian Attack)	348	40.0 [36.8, 43.3]	−6.3	8–10	4–10	6–10

White first move	White setting	Selected Black reply	Training score (%)
1.e4	1800	1...e5 (Open Game)	68.88
1.e4	2000	1...c5 (Sicilian)	50.34
1.d4	1800	1...d5 (Queen's Pawn Game)	67.35
1.d4	2000	1...f5 (Dutch)	50.34
1.c4	1800	1...c5 (Symmetrical English)	64.46
1.c4	2000	1...c5 (Symmetrical English)	50.51

Supplementary Tables

tbl:S1. Engine and simulator configuration used for all Study 1 runs.

Item	Value
Engine and version	Stockfish 15.1 (Debian 12 package), driven over UCI by a Go server
Strength limiting	UCI_LimitStrength = true. UCI_Elo requested at 1320, 2000, and 2500. Because 1320 is below the 15.1 minimum, the lowest setting played at the 1350 floor.
Search budget per move	Fixed move time of 10 ms (go movetime 10), 1 thread, 1 MB hash
Position input	Current position sent as FEN on each move. No move history was sent, so the engine could not see repetitions.
Concurrency	Two engine processes shared across settings, with two games in progress at a time
Randomness and seeds	No user-set seeds. Variation came from the engine's internal random move selection at limited strength and from timing.
Game termination	Checkmate, stalemate, insufficient material, fivefold repetition, and the 75-move rule, applied automatically by the notnil/chess library v1.10.0 (notnil, n.d.). Games reaching 300 plies were scored as draws.
Forced moves	White-opening set: White's first move forced and Black's reply chosen by the engine. Black-reply set: 1.e4 forced for White, then Black's reply forced. Connect Four positive control: the first player's column forced, with the weakest bot on both sides.
Games per run	10 per forced move × 20 moves × 3 settings = 600 chess games, plus 70 Connect Four games (10 per forced column) as a positive control.
Hardware	Render free-tier web service (shared CPU, about 512 MB RAM), with the website hosted on Vercel
Data collection	Summer 2026, under the configuration of commit 5872d85 (August 2, 2026), before the platform went offline in August 2026. Results were recorded by screenshot.

Note. Values are taken from the platform source code (internal/engine/stockfish, internal/domain/research, internal/research/simulate, and the Dockerfile). The chess library does not apply threefold repetition or the 50-move rule automatically, so games can continue past them until one of the listed conditions is met.

Forced move	Simulator label	Set
1.e4	King's Pawn	White
1.c4	English	White
1.d4	Queen's Pawn	White
1.Nf3	Reti	White
1.e3	Van 't Kruijs	White
1.Nc3	Van Geet	White
1.g3	King's Fianchetto	White
1.f4	Bird's Opening	White
1.b3	Nimzo-Larsen	White
1.d3	King's Indian Attack	White
1...e5	Open Game	Black
1...d5	Scandinavian	Black
1...e6	French	Black
1...Nf6	Alekhine's	Black
1...d6	Pirc-style	Black
1...g6	Modern	Black
1...c5	Sicilian	Black
1...c6	Caro-Kann	Black
1...Nc6	Nimzowitsch	Black
1...a6	St. George	Black

Note. In the Black-reply set, every game began with 1.e4 followed by the forced reply. Labels appear in parentheses in the text for continuity with the simulator output.

Setting	White wins, White set (%)	Black wins, Black set (%)	Draws (%)	Black wins, White set (%)a	White wins, Black set (%)a	White expected score (%)a
1320	51.0 [48.9, 53.1]	44.2 [42.1, 46.2]	1.1 [0.8, 1.4]	47.9	54.7	53.4
2000	45.4 [44.0, 46.7]	39.3 [37.7, 40.9]	12.8 [11.9, 13.7]	41.8	47.9	53.0
2500	42.4 [40.2, 44.6]	38.9 [37.2, 40.6]	16.2 [15.2, 17.2]	41.4	44.9	51.8

Note. Bracketed values are 95% t intervals across runs. The 2000 draw-rate estimate uses 28 runs because the Run 12 summary was unavailable. ¹Implied values that assume equal draw rates in the two sets at a setting. Appendix A gives bounds that do not require this assumption.

Color	Setting	Pooled win %	$\chi^2(9)$	p	Observed range (pp)	Null median	Null 95th pct. (range)	p (range)	$\sigma_f$ (pp)	Reliability
White	1320	51.0	12.41	.191	12.1	9.0	13.1	.116	1.8	0.28
White	2000	45.4	15.56	.077	13.4	9.0	13.1	.043	2.5	0.42
White	2500	42.4	18.58	.029	13.8	8.6	13.1	.030	3.0	0.52
Black	1320	44.2	7.45	.590	8.3	9.0	13.1	.623	0.0	0.00
Black	2000	39.3	10.84	.287	8.6	8.6	12.8	.536	1.3	0.17
Black	2500	38.9	8.82	.454	10.3	8.6	12.8	.264	0.0	0.00

Note. Null distributions of the range come from 100,000 simulations with equal win probabilities. $\sigma_f$ is the method-of-moments estimate of the between-opening standard deviation, and reliability is $\sigma_f^2/(\sigma_f^2+\sigma^2)$. Holm-adjusted p-values for the six $\sigma_f^2$ tests are all $\ge.17$.

tbl:S5. White first moves: win percentages, absolute change from 1320 to 2500, and difference-in-differences (DiD) relative to the other nine moves.

Move	1320	2000	2500	Change [95% CI]	p	DiD Holm [95% CI]	p	Holm p
1.Nf3 (Reti)	52.4	43.1	50.0	−2.4 [−10.5, +5.7]	1.000	+6.8 [−1.7, +15.4]	.119	1.000
1.b3 (Nimzo-Larsen)	49.0	48.3	46.2	−2.8 [−10.9, +5.4]	1.000	+6.4 [−2.1, +15.0]	.140	1.000
1.d3 (King's Indian Attack)	42.4	41.4	36.2	−6.2 [−14.1, +1.7]	.376	+2.6 [−5.8, +11.0]	.542	1.000
1.e4 (King's Pawn)	54.5	43.4	46.6	−7.9 [−16.0, +0.2]	.221	+0.7 [−7.9, +9.2]	.874	1.000
1.e3 (Van 't Kruijs)	52.4	47.2	42.4	−10.0 [−18.1, −1.9]	.077	−1.6 [−10.1, +6.9]	.711	1.000
1.f4 (Bird's Opening)	50.0	45.2	39.3	−10.7 [−18.7, −2.6]	.065	−2.4 [−10.9, +6.1]	.583	1.000
1.c4 (English)	54.1	47.6	43.1	−11.0 [−19.1, −2.9]	.065	−2.8 [−11.3, +5.8]	.526	1.000
1.g3 (King's Fianchetto)	51.0	45.5	40.0	−11.0 [−19.1, −3.0]	.065	−2.8 [−11.3, +5.7]	.524	1.000
1.d4 (Queen's Pawn)	52.4	52.8	41.4	−11.0 [−19.1, −3.0]	.065	−2.8 [−11.3, +5.8]	.525	1.000
1.Nc3 (Van Geet)	51.4	39.3	39.0	−12.4 [−20.5, −4.4]	.025	−4.3 [−12.8, +4.2]	.321	1.000

Reply	1320	2000	2500	Change [95% CI]	p	DiD	Holm [95% CI]	p	Holm p
1...c5 (Sicilian)	43.4	43.4	45.2	+1.7 [−6.4, +9.8]	1.000	+7.8 [−0.7, +16.3]	.073	.733
1...Nc6 (Nimzowitsch)	41.0	41.7	39.3	−1.7 [−9.7, +6.3]	1.000	+3.9 [−4.5, +12.4]	.358	1.000
1...a6 (St. George)	40.3	35.2	38.3	−2.1 [−10.0, +5.9]	1.000	+3.6 [−4.8, +11.9]	.405	1.000
1...c6 (Caro-Kann)	41.4	34.8	38.3	−3.1 [−11.1, +4.9]	1.000	+2.4 [−6.0, +10.8]	.573	1.000
1...e6 (French)	45.9	43.1	40.7	−5.2 [−13.2, +2.9]	1.000	+0.1 [−8.4, +8.6]	.979	1.000
1...d5 (Scandinavian)	46.2	38.6	39.0	−7.2 [−15.3, +0.8]	.462	−2.2 [−10.6, +6.3]	.613	1.000
1...d6 (Pirc-style)	45.5	37.6	37.6	−7.9 [−15.9, +0.1]	.363	−3.0 [−11.4, +5.5]	.493	1.000
1...e5 (Open Game)	48.6	41.4	40.0	−8.6 [−16.7, −0.6]	.288	−3.7 [−12.2, +4.8]	.391	1.000
1...g6 (Modern)	43.8	40.3	34.8	−9.0 [−16.9, −1.0]	.238	−4.1 [−12.5, +4.3]	.336	1.000
1...Nf6 (Alekhine's)	45.5	36.9	35.9	−9.7 [−17.6, −1.7]	.174	−4.9 [−13.3, +3.5]	.256	1.000

Note. Rank intervals come from 20,000 parametric bootstrap replicates centered on the observed proportions. They therefore give an optimistic picture of rank certainty (§ 3.3).

Note. Rank intervals come from 20,000 parametric bootstrap replicates centered on the observed proportions. They therefore give an optimistic picture of rank certainty (§ 3.3). Every Black-reply game began with 1.e4.

Setting (pp)	FMA, equal draws	Runs with FMA > 0	t test across runs	Assumption-free FMA bounds (pp)	Implied White expected score (%)	Expected-score bounds (%)
1320	6.8	21 of 29	$t(28) = 4.47$, $p < .001$	[5.7, 7.9]	53.4	[52.8, 53.9]
2000	6.1	26 of 29	$t(28) = 5.59$, $p < .001$	[−6.8, 18.9]	53.0	[46.5, 59.3]
2500	3.5	20 of 29	$t(28) = 2.66$, $p = .013$	[−12.7, 19.7]	51.8	[43.7, 59.9]

Setting	Outcome	Mean (%)	SD (pp)	Range (%)	Dispersion ratio	p (overdispersion)
1320	White wins, White set	51.0	5.56	42–68	1.24	.181
1320	Black wins, Black set	44.2	5.37	32–53	1.17	.247
1320	Draws, both sets	1.1	0.67	0–2	0.83	.720
2000	White wins, White set	45.4	3.51	35–50	0.50	.988
2000	Black wins, Black set	39.3	4.25	31–48	0.76	.816
2000	Draws, both sets	12.8	2.34	8–18	0.98	.491
2500	White wins, White set	42.4	5.86	32–53	1.41	.075
2500	Black wins, Black set	38.9	4.52	29–46	0.86	.676
2500	Draws, both sets	16.2	2.64	10–21	1.02	.429

Note. Expected variance is $p(1 - p)/n$, with $n = 100$ for win rates and $n = 200$ for draw rates. Ratios above one indicate more run-to-run variation than independent sampling would predict.

Setting	Outcome	Study 1 (%)	Study 2 (%)	Difference, Study 2 − Study 1 [95% CI]
1320	White wins, White set	51.0	43.0	−8.0 [−13.9, −2.1]
1320	Black wins, Black set	44.2	35.7	−8.5 [−14.2, −2.8]
1320	Draws, both sets	1.1	18.8	+17.7 [14.6, +20.9]
2000	White wins, White set	45.4	41.7	−3.7 [−9.6, +2.2]
2000	Black wins, Black set	39.3	38.7	−0.6 [−6.4, +5.1]
2000	Draws, both sets	12.8	12.7	−0.2 [−3.0, +2.6]
2500	White wins, White set	42.4	37.0	−5.4 [−11.2, +0.3]
2500	Black wins, Black set	38.9	35.7	−3.2 [−8.9, +2.5]
2500	Draws, both sets	16.2	24.2	+8.0 [4.4, +11.5]

Note. Study 2 contributes 300 games per set and setting (600 for draws). Intervals use a normal approximation.

Setting	Checkmate (%)	Insufficient material (%)	Fivefold repetition (%)	75-move rule (%)	Stalemate (%)	300-ply cap (%)	Draws, White / Black set (%)	Mean plies (SD)
1320	81.2	7.7	3.7	0.8	0.8	5.8	15.7 / 22.0	159.3 (58.6)
2000	87.3	6.0	2.2	0.2	0.5	3.8	14.0 / 11.3	148.2 (57.7)
2500	75.8	12.0	5.5	0.3	1.3	5.0	25.3 / 23.0	161.3 (61.0)

Note. Percentages are based on the 600 games at each nominal setting. Games reaching the 300-ply cap were scored as draws.

Pairing (weaker vs. stronger)	Games	Stronger setting W–D–L	Stronger setting score	Estimated Elo gap [95% CI]	Nominal gap
1320 vs. 2000	200	200–0–0	100.0%	Not estimable (perfect score)	650
2000 vs. 2500	200	180–15–5	93.8%	470 [401, 577]	500
1320 vs. 2500	400	369–13–18	93.9%	474 [417, 555]	1,150

Note. Estimated gaps use the logistic Elo formula. A perfect score gives an unbounded estimate. Nominal gaps use the 1350 floor for the lowest setting.

Setting	1.d3 win %	Other moves win %	1.d3 expected score (%)	Other moves expected score (%)
1320	38.9	39.0	47.2	48.3
2000	41.1	42.1	48.9	49.2
2500	32.2	38.8	43.3	51.5
All settings	37.4	40.0	46.5	49.7

Note. Each White move has 90 games per setting (270 in total for 1.d3 and 2,430 for the other moves). Expected score counts a draw as half a win. Prespecified one-sided Mantel–Haenszel test: common odds ratio 0.90, 95% CI [0.69, 1.16], $p = .208$. Test of equal odds ratios across settings: $p = .650$.

Note. Black played at nominal 2000 throughout. The scores are shrinkage-adjusted training estimates used for selection, not estimates from the independent evaluation. All six selected cells had 290 completed outcomes.

White setting	Contrast	Estimate (pp)	95% CI	p	Holm p
1800	Learned − engine	−3.32	[−7.38, +0.74]	.109	.213
1800	Fixed book − engine	−2.23	[−6.41, +1.96]	.297	—
2000	Learned − engine	−3.93	[−8.69, +0.84]	.106	.213
2000	Fixed book − engine	+0.46	[−3.79, +4.71]	.832	—

Note. Negative estimates favor letting Stockfish choose freely. Intervals and p-values use block-level normal approximations. Holm adjustment applies only to the two prespecified learned-policy contrasts.

Table S17. Study 5 difference between the learned policy and freely choosing Stockfish, by White first move.

White first move	White setting	Completed games (learned / engine)	Difference (pp)	95% CI
1.e4	1800	290 / 288	−0.99	[−7.99, +6.01]
1.d4	1800	290 / 290	−5.00	[−12.01, +2.01]
1.c4	1800	289 / 290	−4.08	[−10.86, +2.70]
1.e4	2000	290 / 290	−4.66	[−12.18, +2.87]
1.d4	2000	290 / 290	−4.31	[−11.57, +2.95]
1.c4	2000	289 / 290	−2.77	[−10.13, +4.58]

Note. Game-level normal-approximation intervals are unadjusted. Censored games are excluded rather than scored as draws.

Family	Variation	Against White 1800 (%) [95% CI]	Against White 2000 (%) [95% CI]
Sicilian	Najdorf	66.6 [61.7, 71.6]	52.8 [47.5, 58.0]
Sicilian	Dragon	65.5 [60.7, 70.4]	46.0 [40.9, 51.2]
Sicilian	Classical	61.4 [56.3, 66.6]	45.3 [40.3, 50.4]
Sicilian	Scheveningen	60.7 [55.5, 65.9]	39.1 [33.9, 44.3]
Ruy Lopez	Morphy	62.1 [57.0, 67.1]	47.8 [42.6, 52.9]
Ruy Lopez	Berlin	68.8 [63.9, 73.7]	42.8 [37.6, 48.0]
Ruy Lopez	Steinitz	61.4 [56.3, 66.5]	42.8 [37.6, 47.9]
Ruy Lopez	Classical	65.9 [60.9, 70.8]	47.2 [42.2, 52.3]
Queen's Gambit	Declined	71.7 [67.1, 76.4]	51.2 [46.1, 56.3]
Queen's Gambit	Accepted	68.7 [63.7, 73.7]	48.3 [43.2, 53.3]
Queen's Gambit	Slav	58.4 [53.4, 63.5]	48.4 [43.3, 53.6]
Queen's Gambit	Chigorin	60.9 [56.0, 65.7]	46.0 [40.8, 51.2]
French	Classical	66.6 [61.6, 71.5]	50.5 [45.2, 55.8]
French	Winawer	64.8 [59.8, 69.9]	48.8 [43.7, 53.9]
Caro-Kann	Classical	62.4 [57.3, 67.5]	43.4 [38.4, 48.5]
Caro-Kann	Gurgenidze	56.7 [51.5, 62.0]	43.1 [37.9, 48.3]
English	Symmetrical	68.4 [63.6, 73.3]	50.7 [45.5, 55.8]
English	King's English	70.9 [66.2, 75.6]	51.2 [46.0, 56.4]

Note. Black played at nominal 2000. Each cell contains 290 scheduled games, and six cells have 289 completed outcomes because one game was censored. Intervals are descriptive and not simultaneous.

White setting	Contrast	Difference (pp)	Holm-adjusted p	99.88% CI
2000	Sicilian Najdorf minus Sicilian Scheveningen	+13.68	.0013	[+3.73, +23.62]
1800	Queen's Gambit Declined minus Slav	+13.28	.0053	[+2.49, +24.07]

Note. The paired tests use 29 block differences with 28 degrees of freedom. The intervals use a Bonferroni level of .05/42. Both contrasts are exploratory.

Note. Center-column contrast: 22.5 points (95% CI [16.5, 28.5]). Homogeneity across columns: $\chi^{2}(6) = 64.47$, $p < .001$. Ranking reliability 0.91. The center column had the highest score, counting ties, in 20 of 29 runs.

Supplementary Figures