betaFeedback

WorksComputer Science

Strategic Competition and Convergence in LLM Leaderboards

Abstract

Leaderboards are a popular way to compare large language model (LLM) capabilities and can influence how model developers invest resources. We ask when competition on leaderboards leads developers to produce models with similar capability profiles, which we call monoculture, or similar performance levels. We study this by formulating leaderboard competition as a game in which model developers invest resources across capabilities to maximize their win rate. We show that similar production functions can induce similar capability profiles, while comparative advantage can sustain distinct capability profiles. We then study convergence in performance levels and show that diminishing returns can lead to an increasingly crowded competitive frontier. Using data from LM Arena and OpenRouter, we find that developers improve more in categories where they lag competitors, while still retaining category-specific advantages. At the same time, we find that the frontier of models becomes increasingly crowded, alongside increasingly diversified model usage patterns. Our results suggest that increasing competition does not need to produce interchangeable models, but can instead sustain competitive models with distinct strengths.

Introduction

Leaderboards are one of the main ways in which large language models (LLMs) are measured and can influence how models are developed [3]. While benchmark-style leaderboards remain popular, newer head-to-head leaderboards compare models based on aggregated human preferences [7]. This changes strategic incentives for developers, as a model’s rank depends on competitors’ capabilities.

A natural question is whether this competition pushes developers toward similar performance levels, capability profiles, or both. By providing developers with a common competitive target, shared evaluation may encourage them to invest in the same capabilities, potentially producing similar capability profiles or monoculture [19]. By contrast, differences in developer strengths across capabilities can create incentives to specialize. At the same time, models might converge in performance levels, while still maintaining distinct capability profiles. Understanding this phenomenon is important because leaderboards may shape not only the models that perform best, but also the direction of model development [8, 4]. Even if models become similarly competitive, differences in their capability profiles determine whether users have access to diverse alternatives or a homogeneous set of models.

In this work, we study how leaderboard competition shapes capability profiles and performance levels by modeling AI evaluation as a multidimensional contest where developers allocate investment across capabilities Figure 1. Our theoretical results characterize when similarities in production functions lead to monoculture and when diminishing returns can lead to convergence in performance. Empirically, we find that some developers maintain differentiated capability profiles over time, alongside an increasingly crowded competitive frontier. Our results reveal that competition can lead to convergence in developer performance while still maintaining an ecosystem of meaningfully different models.

Figure 1. Head-to-head leaderboards compare models, creating incentives for developers to allocate investment across capabilities. How these incentives interact with developers’ production functions can shape both the similarity of capability profiles and convergence in performance. In practice, we find persistent differences in developers’ capability profiles alongside an increasingly crowded performance frontier.

Our paper connects to work on contests theory, monoculture, and LLM leaderboards. Contest theory studies how competitors allocate effort in response to a common reward [34, 29], while algorithmic monoculture studies when shared algorithms lead to similar outcomes [19, 25]. LLM leaderboards combine aspects of both settings: developers compete under a common evaluation, but allocate investment across multiple capabilities. More recently, Chen et al. [4] studied strategic post-training and the resulting equilibria, while Donahue & Raghavan [9] study how aggregation and evaluation affect producer incentives. Unlike Chen et al. [4] and Donahue & Raghavan [9], we explicitly model how production functions shape capability development, which allows us to characterize when common evaluation produces similar capability profiles.

Our contributions include:

  1. We characterize how multidimensional competition under shared evaluation produces similar or differentiated capability profiles. Similar production functions can induce monoculture, while comparative advantage can sustain differentiation and, in some settings, specialization.

  1. We study how competition evolves over time and identify when diminishing returns can lead to convergence in performance. We find that convergence in performance can arise when competitors can quickly catch up in categories where they lag.

  1. We find that the performance frontier becomes increasingly crowded over time, even as developers retain persistent differences in their relative capability profiles. Along with crowding, we find increasingly dispersed model usage across developers.2

Model and Preliminaries

Our model consists of NN model developers (e.g., OpenAI, Anthropic, DeepMind), each of which competes on KK dimensions (e.g., math, coding, science). Developer ii has a production function Fi:R+K→R+KF_i: \mathbb{R}_{+}^{K} \to\mathbb{R}_{+}^{K} and investment xix_i, which produces capabilities θi:=Fi(xi)\theta_i := F_i(x_i). Investment at level xix_i results in a cost c(∥xi∥1)c(\lVert x_i\rVert_1), where cc is a strictly increasing, differentiable, continuous, and unbounded cost function with c(0)=0c(0)=0 (per-developer cost functions cic_i can be folded into the model without changing the equilibria; see Appendix § 15.§ 1). We model each developer as maximizing its expected win rate. We use expected weighted pairwise win rate as a simplified objective for relative competitive performance, rather than assuming that developers literally optimize a particular public leaderboard. This objective directly corresponds to maximizing weighted category-level pairwise win probabilities under Bradley–Terry/Elo-style evaluation (see Appendix § 11). Here, win-rate is weighted according to w≥0,∑kwk=1w \ge0, \sum_k w_k = 1. We define the utility as

ui(x)=1N−1∑i′≠i∑kwkσ((Fi(xi))k−(Fi′(xi′))k)−c(∑kxi,k).(1)u_i(x) = \frac{1}{N-1}\sum_{i'\ne i}\sum_k w_k \sigma\left((F_i(x_i))_k - (F_{i'}(x_{i'}))_k\right) - c\left(\sum_k x_{i,k}\right). \tag*{(1)}

Here, σ\sigma denotes the win-probability according to some pairwise preference model. We assume σ\sigma is strictly increasing, twice differentiable, bounded in [0,1][0,1], σ′(x)>0\sigma'(x)>0, and symmetric so that σ(x)+σ(−x)=1\sigma(x)+\sigma(-x)=1 (e.g., Bradley–Terry). Our model abstracts from the details of developers’ internal R&D processes and instead captures three features of model development: developers care about their capabilities relative to competitors, allocate costly investment across multiple capabilities, and differ in how effectively investment translates into capability improvements. We do not require that developers directly optimize a particular leaderboard; tournament-style evaluation provides a tractable representation of competition based on relative model performance. The model can also be viewed as a multidimensional contest, related to Colonel Blotto games [29], in which investment is transformed into capabilities through heterogeneous production functions FiF_i.

Let x=(x1,…,xN)\mathbf{x}=(x_1,\ldots,x_N) be a Nash equilibrium, and let Hi,i′(x)=cos⁡(θi,θi′)H_{i,i'}(\mathbf{x})=\cos(\theta_i,\theta_{i'}) be the pairwise monoculture for developers ii and i′i', with Hi,i′(x)=1H_{i,i'}(\mathbf{x})=1 indicating that θi\theta_i and θi′\theta_{i'} are parallel. We can view θi\theta_i as consisting of a direction di∗∈S+K−1\mathbf{d}_i^* \in S_+^{K-1} and a magnitude αi∗\alpha_i^*, with only the direction impacting Hi,i′H_{i,i'}. We focus on cosine similarity here to capture capability profiles rather than capability levels; while two developers might differ in their overall level of output, they can still be similar if their relative category outputs are similar [1]. We return to similarity in the overall level of output in § 5. We define overall monoculture as H(x)=1(N2)∑i<i′Hi,i′(x)H(\mathbf{x})=\frac{1}{\binom{N}{2}}\sum_{i<i'}H_{i,i'}(\mathbf{x}), with H=1H=1 indicating that all developers have parallel θi\theta_i. When exactly one capability vector is zero we define Hi,i′(x∗)=0H_{i,i'}(\mathbf{x}^*)=0, and when both are zero we define Hi,i′(x∗)=1H_{i,i'}(\mathbf{x}^*)=1.

Existence of Pure Strategy Equilibrium

We discuss when the competition model exhibits pure-strategy equilibrium, where we focus on pure strategies because they correspond to stable, deterministic investment profiles, which do not require developers to randomize. We first consider the intuition in the N=2N=2, K=1K=1 setting, where we refer to the production function as ff because it is scalar. A sufficient condition for existence is that diminishing returns in production are sufficiently strong relative to the curvature of the evaluation function. That is, when cc is convex and fi′(xi)>0f_i'(x_i)>0, we can write this as

sup⁡zσ′′(z)σ′(z)<−fi′′(xi)fi′(xi)2∀xi.\sup_{z}\frac{\sigma''(z)}{\sigma'(z)}<-\frac{f_i''(x_i)}{f_i'(x_i)^2}\quad\forall x_i.

The following theorem generalizes this condition to arbitrary NN and KK.

Theorem 1 (Equilibrium Existence). Suppose that gi,k(xi)=[Fi(xi)]kg_{i,k}(x_i)=[F_i(x_i)]_k is twice continuously differentiable and cc convex. Let ρσ:=sup⁡z∈Rσ′′(z)σ′(z)\rho_\sigma:=\sup_{z\in\mathbb{R}}\frac{\sigma''(z)}{\sigma'(z)}. There is a pure-strategy Nash equilibrium if

−∇2gi,k(xi)⪰ρσ∇gi,k(xi)∇gi,k(xi)⊤∀xi, ∀i,k.-\nabla^2 g_{i,k}(x_i)\succeq\rho_\sigma\nabla g_{i,k}(x_i)\nabla g_{i,k}(x_i)^\top\qquad\forall x_i,\ \forall i,k.

Equilibrium exists when developers’ production functions have sufficiently strong diminishing returns relative to the curvature of the evaluation function. Since the win-rate payoff is at most 1, any investment with c(∥xi∥1)>1c(\lVert x_i\rVert_1)>1 is dominated by zero investment. That is, in Equation (1), if c(∥xi∥1)>1c(\lVert x_i\rVert_1)>1 then ui(xi)<0u_i(x_i)<0 because 1N−1∑i′≠i∑kwkσ([Fi(xi)]k)≤1\frac{1}{N-1}\sum_{i'\ne i}\sum_k w_k\sigma([F_i(x_i)]_k)\leq1. By unboundedness of cc, we can restrict each developer’s strategy space to a compact convex set Xi={xi≥0:∥xi∥1≤xˉ}X_i=\{x_i\geq0:\lVert x_i\rVert_1\leq\bar{x}\}. Under the theorem’s curvature condition, the Hessian of uiu_i is negative semidefinite, so uiu_i is concave in xix_i. Debreu–Fan–Glicksberg then gives existence (details in Appendix § 15.§ 2). When production functions violate the conditions, pure-strategy equilibrium need not exist (see Appendix § 12).

When Does Monoculture Arise?

Given that a pure-strategy equilibrium exists, we next ask what determines whether developers produce similar capability profiles or specialize in different capabilities. We first give interpretable sufficient conditions for monoculture and specialization, before a general characterization.

Similar Production Functions Can Induce Monoculture

A simple sufficient condition for monoculture is developers having identical production functions.

Lemma 1 (Symmetric Developers). Suppose the conditions of Theorem 1 hold. If Fi=Fi′F_i=F_{i'}, then there exists a pure-strategy Nash equilibrium x∗\mathbf{x}^* such that xi∗=xi′∗x_i^*=x_{i'}^* and Hi,i′(x∗)=1H_{i,i'}(\mathbf{x}^*)=1. Moreover, if Fi=FF_i=F for every developer ii, then there exists a symmetric pure-strategy Nash equilibrium satisfying x1∗=⋯=xN∗x_1^*=\cdots=x_N^*, and at this equilibrium, θ1∗=⋯=θN∗\theta_1^*=\cdots=\theta_N^* with H(x∗)=1H(\mathbf{x}^*)=1.

Restricting attention to profiles with xi=xi′x_i = x_{i'}, the two developers have identical best-response correspondences; Kakutani’s fixed-point theorem then gives the equilibrium (Appendix § 15.§ 3).

References

  1. [1]Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in neural information processing systems, 35:3663–3678, 2022.arxiv.org/abs/2211.13972
  2. [2]Emile Borel. The theory of play and integral equations with skew symmetric kernels. Econometrica: journal of the Econometric Society, pp. 97–100, 1953.DOI
  3. [3]Alex Chan. Benchmark mineability and the financing of ai innovation. Technical report, National Bureau of Economic Research, 2026.DOI
  4. [4]Yatong Chen, Guanhua Zhang, and Moritz Hardt. Leaderboard incentives: Model rankings under strategic post-training. arXiv preprint arXiv:2603.08371, 2026.arxiv.org/abs/2603.08371
  5. [5]Yuxin Chen and Changho Suh. Spectral mle: Top-k rank aggregation from pairwise comparisons. In International conference on machine learning, pp. 371–380. PMLR, 2015.arxiv.org/abs/1504.07218
  6. [6]Wayne Chi, Valerie Chen, Anastासios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar. Copilot arena: A platform for code llm evaluation in the wild. arXiv preprint arXiv:2502.09328, 2025.arxiv.org/abs/2502.09328
  7. [7]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024.arxiv.org/abs/2403.04132
  8. [8]Ricardo Dominguez-Olmedo, Florian Eddie Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. In International Conference on Learning Representations, volume 2025, pp. 68564–68594, 2025.
  9. [9]Kate Donahue and Manish Raghavan. Impacts of aggregation on model diversity and consumer utility. arXiv preprint arXiv:2602.23293, 2026.arxiv.org/abs/2602.23293
  10. [11]Nika Haghtalab, Ariel D Procaccia, Han Shao, Serena Lutong Wang, and Kunhe Yang. Pluralistic leaderboards. arXiv preprint arXiv:2606.02547, 2026.arxiv.org/abs/2606.02547
  11. [12]Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pp. 111–122, 2016.
  12. [13]Chris Hays, Rachel Li, Bailey Flanigan, and Manish Raghavan. Strategic candidacy in generative ai arenas. arXiv preprint arXiv:2603.26891, 2026.arxiv.org/abs/2603.26891
  13. [14]Jenny Huang, Yunyi Shen, Dennis Wei, and Tamara Broderick. Dropping just a handful of preferences can change top large language model rankings. In International Conference on Learning Representations, volume 2026, pp. 90685–90721, 2026.arxiv.org/abs/2508.11847
  14. [15]Meena Jagadeesan, Michael Jordan, Jacob Steinhardt, and Nika Haghtalab. Improved bayes risk can yield reduced social welfare under competition. Advances in Neural Information Processing Systems, 36:66940–66952, 2023.arxiv.org/abs/2306.14670
  15. [16]Nathanael Jo, Nikhil Garg, and Manish Raghavan. The subjectivity of monoculture. arXiv preprint arXiv:2602.24086, 2026.arxiv.org/abs/2602.24086
  16. [17]Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. arXiv preprint arXiv:2506.07962, 2025.arxiv.org/abs/2506.07962
  17. [18]Jon Kleinberg and Manish Raghavan. How do classifiers induce agents to invest effort strategically? ACM Transactions on Economics and Computation (TEAC), 8(4):1–23, 2020.arxiv.org/abs/1807.05307
  18. [19]Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021.DOI
  19. [20]Kai A Konrad and Dan Kovenock. Multi-battle contests. Games and Economic Behavior, 66(1):256–274, 2009.DOI
  20. [21]Marc Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoogchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating agents using social choice theory. arXiv preprint arXiv:2312.03121, 2023.arxiv.org/abs/2312.03121
  21. [22]Lydia T Liu, Nikhil Garg, and Christian Borgs. Strategic ranking. In International Conference on Artificial Intelligence and Statistics, pp. 2489–2518. PMLR, 2022.arxiv.org/abs/2109.08240
  22. [23]Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. The social cost of strategic classification. In Proceedings of the conference on fairness, accountability, and transparency, pp. 230–239, 2019.arxiv.org/abs/1808.08460
  23. [24]Rui Min, Tianyu Pang, Chao Du, Qian Liu, Minhao Cheng, and Min Lin. Improving your model ranking on chatbot arena by vote rigging. arXiv preprint arXiv:2501.17858, 2025.arxiv.org/abs/2501.17858
  24. [25]Kenny Peng and Nikhil Garg. Monoculture in matching markets. Advances in Neural Information Processing Systems, 37:81959–81991, 2024.arxiv.org/abs/2312.09841
  25. [26]Juan Perdomo, Tiana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International conference on machine learning, pp. 7599–7609. PMLR, 2020.arxiv.org/abs/2002.06673
  26. [28]Arun Rajkumar, Suprovat Ghoshal, Lek-Heng Lim, and Shivani Agarwal. Ranking from stochastic pairwise preferences: Recovering condorcet winners and tournament solution sets at the top. In International Conference on Machine Learning, pp. 665–673. PMLR, 2015.
  27. [29]Brian Roberson. The colonel blotto game. Economic Theory, 29(1):1–24, 2006.DOI
  28. [30]Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Lee Boyd-Graber. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4486–4503, 2021.DOI
  29. [31]Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A Smith, et al. The leaderboard illusion. arXiv preprint arXiv:2504.20879, 2025.arxiv.org/abs/2504.20879
  30. [32]Stergios Skaperdas. Contest success functions. Economic theory, 7(2):283–290, 1996.
  31. [33]Gordon Tullock. Efficient rent seeking. In Efficient rent-seeking: Chronicle of an intellectual quagmire, pp. 3–16. Springer, 2001.
  32. [34]Milan Vojnović. Contest theory: Incentive mechanisms and ranking methods. Cambridge University Press, 2015.

Paper details

vlnc:W_yxm2t3tg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents