betaFeedback

WorksComputer Science

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Abstract

Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Kaiser Sun1∗†^{1*\dagger}, Yongchan Kwon2^{2}, Zain Hasan2^{2}, Shang Zhu2^{2}, Kaitlyn Zhou3,2^{3,2}, Federico Bianchi2†^{2\dagger}

1^{1}Johns Hopkins University2^{2}Together AI3^{3}Cornell University

†^{\dagger}Corresponding authors

References

  1. [1]Agarwal, Ahmad, Ai, Altman, Applebaum, Arbus, Arora, Bai, Baker, & Bao. (2025). gpt-oss-120b \& gpt-oss-20b model card. https://arxiv.org/abs/2508.10925
  2. [2]Aggarwal, & Welleck. (2025). L1: Controlling how long a reasoning model thinks with reinforcement learning. https://arxiv.org/abs/2503.04697
  3. [3]Anthropic. (2026a). Claude Sonnet 5 System Card , June 2026 a. https://www.anthropic.com/claude-sonnet-5-system-cardanthropic.com/claude-sonnet-5-system-card
  4. [4]Anthropic . \newblock Effort. \newblock https://platform.claude.com/docs/en/build-with-claude/effort, 2026 b . \newblock Claude API documentation. Accessed September 16, 2026.platform.claude.com/docs/en/build-with-claude/effort
  5. [5]Bai, Huang, Wang, Sun, Mihalcea, Brynjolfsson, Pentland, & Pei. (2026). How do ai agents spend your money? analyzing and predicting token consumption in agentic coding tasks. https://arxiv.org/abs/2604.22750
  6. [6]Chen, Zhang, He, Stoica, Zaharia, & Zou. (2026). The price reversal phenomenon: When cheaper reasoning models cost more. https://arxiv.org/abs/2603.23971
  7. [7]Chen, Xu, Liang, He, Pang, Yu, Song, Liu, Zhou, Zhang, Wang, Tu, Mi, & Yu. (2025). Do NOT think that much for 2+3=? O n the overthinking of long reasoning models. https://proceedings.mlr.press/v267/chen25bx.htmlproceedings.mlr.press/v267/chen25bx.html
  8. [8]Cuadron, Li, Ma, Wang, Wang, Zhuang, Liu, Schroeder, Xia, & Mao. (2025). The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. https://arxiv.org/abs/2502.08235
  9. [9]Darby, & Karni. (1973). Free competition and the optimal amount of fraud. The Journal of law and economics, 16(1), 67–88.
  10. [10]DeepSeek-AI, Xu, Lin, Xue, Wang, Xu, Wu, Zhang, Lin, Dong, Ling, Lu, Zhao, Deng, Hou, Xu, Shao, Ruan, Sun, … Yao. (2026). Deepseek-v4: Towards highly efficient million-token context intelligence. https://arxiv.org/abs/2606.19348
  11. [11]Dekoninck, Jovanović, Gehrunger, Rögnvaldsson, Petrov, Sun, & Vechev. (2026). Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. https://arxiv.org/abs/2605.00674
  12. [12]Deng, Da, Pan, He, Ide, Garg, Lauffer, Park, Pasari, & Rane. (2025). Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?. https://arxiv.org/abs/2509.16941
  13. [13]Dulleck, & Kerschbamer. (2006). On doctors, mechanics, and computer specialists: The economics of credence goods. Journal of Economic literature, 44(1), 5–42.
  14. [14]Erol, El, Suzgun, Yuksekgonul, & Zou. (2026). Cost-of-pass: An economic framework for evaluating language models.
  15. [15]Fan, Vasilevski, Lin, Chen, Chen, Zhong, Zhang, He, & Hassan. (2025). Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints. https://arxiv.org/abs/2509.09853
  16. [16]Gao, Sun, Huang, Van Koevering, Ji, Huang, Shi, Lu, Xiao, & Khashabi. (2026). How to interpret agent behavior. https://arxiv.org/abs/2605.13625
  17. [17]Huang, Lee, Tng, & Ge. (2026). Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks. https://arxiv.org/abs/2607.07946
  18. [18]Jain. (2026). Reward hacking is swamping model intelligence gains. https://cursor.com/blog/reward-hacking-coding-benchmarkscursor.com/blog/reward-hacking-coding-benchmarks
  19. [19]Kapoor, Stroebl, Siegel, Nadgir, & Narayanan. (2024). Ai agents that matter. https://arxiv.org/abs/2407.01502
  20. [20]Kapoor, Stroebl, Kirgis, Nadgir, Siegel, Wei, Xue, Chen, Chen, & Utpala. (2026). Holistic agent leaderboard: The missing infrastructure for ai agent evaluation.
  21. [21]Leroy. (2009). Why is it so hard to do my work? the challenge of attention residue when switching between work tasks. Organizational Behavior and Human Decision Processes, 109(2), 168–181.
  22. [22]Mark, Gudith, & Klocke. (2008). The cost of interrupted work: more speed and stress.
  23. [23]Merrill, Shaw, Carlini, Li, Raj, Bercovich, Shi, Shin, Walshe, & Buchanan. (2026). Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.
  24. [24]Moon. (2026). The price of thinking: Reasoning effort as a model-specific api contract. https://arxiv.org/abs/2608.16956
  25. [25]Muennighoff, Yang, Shi, Li, Fei-Fei, Hajishirzi, Zettlemoyer, Liang, Candès, & Hashimoto. (2025). s1: Simple test-time scaling. https://aclanthology.org/2025.emnlp-main.1025/aclanthology.org/2025.emnlp-main.1025
  26. [26]OpenAI. (2026a). GPT-5.6 System Card. https://deploymentsafety.openai.com/gpt-5-6deploymentsafety.openai.com/gpt-5-6
  27. [27]OpenAI. (2026b). Reasoning models. https://developers.openai.com/api/docs/guides/reasoningdevelopers.openai.com/api/docs/guides/reasoning
  28. [28]Rein, Hou, Stickland, Petty, Pang, Dirani, Michael, & Bowman. (2023). Gpqa: A graduate-level google-proof q\&a benchmark. https://arxiv.org/abs/2311.12022
  29. [29]Robbes, Matricon, Degueule, Hora, & Zacchiroli. (2026). Agentic much? adoption of coding agents on github.
  30. [30]Snell, Lee, Xu, & Kumar. (2025). Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning.
  31. [31]Sun, Wang, Tian, Liu, Shen, He, He, Ye, Wang, & Li. (2025). Coin: Counting the invisible reasoning tokens in commercial opaque llm apis. https://arxiv.org/abs/2505.13778
  32. [32]Team, Bai, Bai, Bao, C, Cai, Cai, Cao, Cao, Chai, Charles, Che, Chen, Chen, Chen, Chen, Chen, Chen, Chen, … Zu. (2026). Kimi k3: Open frontier intelligence. https://arxiv.org/abs/2607.24653
  33. [33]Thinking Machines Lab. (2026). Inkling: Our open-weights model. https://thinkingmachines.ai/news/introducing-inkling/thinkingmachines.ai/news/introducing-inkling
  34. [34]Velasco, Tsirtsis, Okati, & Gomez-Rodriguez. (2025). Is your llm overcharging you? tokenization, transparency, and incentives. https://arxiv.org/abs/2505.21627
  35. [35]Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, & Lv. (2025). Qwen3 technical report. https://arxiv.org/abs/2505.09388
  36. [36]Z.ai. (2026). GLM-5.2 : Built for long-horizon tasks. https://z.ai/blog/glm-5.2z.ai/blog/glm-5.2
  37. [37]Zheng, & Gao. (2024). Fragmented moments, balanced choices: How do people make use of their waiting time?.

Paper details

vlnc:W_qkdz4er4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents