Teaching AI Agent to Play Poker Strategically Through Preference Optimization

Introduction

Legal poker actions can still be strategically poor, making poker a useful testbed for strategic decision-making under hidden information. CFR provides approximate equilibrium strategies in finite two-player zero-sum games [1], while systems such as Pluribus [2] motivate evaluation in more complex multiplayer settings.

We first compare a static policy with three frozen compact language models: SmallThinker, Qwen3, and Gemma-3. SmallThinker performs best among them in exact games, yet remains weak on six-max agreement.

We therefore compare Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO) [4], Group Relative Policy Optimization (GRPO) [5], and Group Sequence Policy Optimization (GSPO) [6], and propose Strategic Preference Optimization with KL-anchored Equilibrium (SPOKE). SPOKE combines teacher-relative value penalties, KL anchoring, risk constraints, and structured rationale supervision to improve strategic quality without sacrificing teacher-consistent behavior.

Proposed Method

Each record contains the player’s information state $s$, legal action-size menu $A(s)$, teacher distribution $\pi_T$, optional action values $Q_T$, risk features, and rationale tags $t$. Prompts exclude opponents’ hidden cards. Canonical action strings receive model likelihood scores, normalized over $A(s)$ to form $\pi_\theta$; a separate output predicts structured tags (see Figure 2).

We operationalize the objective as follows. Let $[x]_+ = \max(x,0)$ and $V_T = \mathbb{E}_{a\sim\pi_T} Q_T(s,a)$. Define

The first penalty discourages individual below-reference actions; the second penalizes an average value shortfall. These are teacher-relative proxies, not cumulative CFR regret. With $m_s$ indicating validated value targets, we optimize the policy using

Here $c$ is a risk cost, $B_s$ its budget, and $\ell_{\mathrm{tag}}$ is binary cross-entropy over rationale tags. KL anchoring discourages collapse of the teacher’s mixed policy. We use value-based terms only where validated ($m_s = 1$ for Kuhn/Leduc and $m_s = 0$ for six-max), with hyperparameters selected on the development split.

After restricting the policy to legal actions, RangeDial applies a bounded preference shift

where $b$ is 1 for raise, 0 for check/call, and −1 for fold; $\alpha\ge0$ sets the strength. Increasing $d$ favors raising without adding illegal actions.

Experimental Setups

Data and evaluation. The record set holds 5,183 decisions after merging 670 repeated six-max spots (4,883 spots from 5,553 observations), with teacher exploitability 0.001486 in Kuhn (10,000 CFR iterations) and 0.010877 in limit Leduc (2,000 iterations). Splits precede prompt augmentation and group information sets and public histories to prevent leakage: train/dev/test are 6/3/3 (Kuhn), 189/45/54 (Leduc) and 3,449/705/729 (six-max). Full chance enumeration and best responses support exact profile value, NashConv, and exploitability on the declared small games (over all information sets).

Six-max evaluation uses 100-big-blind preflop decisions with label-smoothed, Pluribus-derived behavioral actions. The primary behavioral endpoint is held-out top-1 agreement; teacher KL is secondary. Legality is checked separately.

Models and controls. The static baseline samples uniformly from legal actions in Kuhn and Leduc and uses a fixed preflop action chart in six-max. Frozen SmallThinker-4B-A0.6B [3], Qwen3-0.6B [7], and Gemma-3-1B-IT [8] were each evaluated once on the same held-out sets, task prompt, and legal menu.

caption: Zero-shot controls vs. optimization methods (SmallThinker-4B-A0.6B). Three-run means: frozen controls are single runs. Bold: best per metric.

:::

Action probabilities use first-token scores for fold/call/raise; these serve as zero-shot controls.

We then train SFT, DPO [4], GRPO [5], GSPO [6], and SPOKE as a SmallThinker-4B-A0.6B policy over 4,096 hashed prompt features, where DPO, GRPO, and GSPO are single-step versions of those objectives, and evaluate them under the same protocol with three runs (seeds 13/42/73).

Sensitivity analyses. KL-weight and RangeDial sweeps use the same three runs. SFT receives identical tag supervision. Component-removal ablations have no measured results, so the contribution of each SPOKE term cannot be isolated.

Preliminary Diagnostics and Limitations

tbl:1 shows that zero-shot capability does not transfer uniformly across evaluation settings. SmallThinker improves over the static baseline in Kuhn (0.326 vs. 0.458) and Leduc (1.767 vs. 2.374), yet reaches only 0.159 six-max top-1 agreement. Qwen3 and Gemma-3 exhibit different failure modes, indicating that no frozen model performs consistently well across both strategic and behavioral endpoints.

The optimization objectives reveal a similar trade-off. SFT achieves strong six-max agreement (0.754) and low teacher KL (0.523), while GRPO and GSPO improve Leduc exploitability to 1.239 and 1.331 but increase KL to 2.357 and 2.354. DPO performs poorly under this protocol.

SPOKE provides the strongest combined result, best on all four metrics. Variation across three runs remains small, although the sample size is insufficient to establish statistical superiority.

Sensitivity analysis shows that KL weighting controls the trade-off between behavioral agreement and teacher divergence while Leduc exploitability remains relatively stable. RangeDial further changes aggression without introducing illegal actions.

Conclusion

Frozen compact language models demonstrated inconsistent performance in six-max evaluations and exact-game evaluations. We proposed SPOKE, integrating teacher-relative value penalties, KL anchoring, risk constraints, and structured rationale supervision. Across three runs of a SmallThinker-4B-A0.6B policy, SPOKE achieved the lowest mean exploitability in Kuhn (0.213) and Leduc (1.081), and the highest six-max top-1 agreement (0.775), while maintaining teacher KL comparable to SFT. These findings provide preliminary evidence that strategic performance and teacher-consistent behavior can be jointly improved. However, the limited number of runs, the absence of component ablations, and the evaluation of behavioral-only at six-maximums warrant further validation.

Acknowledgments

This research has received funding support from the NSRF via the Research and Innovation Acceleration Agency for Competitiveness and Area Development (RCAD) (Program Management Unit for Frontier Brainpower and Future Industries) [grant number B13T690002-1].

References