betaFeedback

WorksComputer Science

How Efficient Is Your Agent? Online Predictive Bounds under Budget Constraints

Abstract

We evaluate AI agents in an online setting where tasks arrive sequentially and, before running each task, we construct an upper predictive bound on its time-to-success—the number of interactions until the task is completed successfully. Under a limited budget, some runs are stopped early, so their time-to-success is unobserved (right-censored), and the correctness of the bound may be unknown. We introduce PRedictive Bounds for Online Budget Efficiency (PROBE), a method for constructing calibrated upper predictive bounds while adaptively allocating a limited budget across agent tasks to reduce the anticipated coverage deficit. Our method calibrates the bounds online using inverse-probability-weighted feedback together with a correction designed to minimize coverage deficit while remaining unbiased. We prove that PROBE provides a coverage guarantee for any finite time horizon while satisfying the budget constraint. Experiments on software engineering, retrieval-augmented generation, and instruction following show that PROBE attains a coverage rate closer to the target level with tighter predictive bounds than a calibrated baseline.

Introduction

AI agents rarely complete complex tasks in a single model call [32, 50]. A coding agent, for example, may inspect a repository, modify files, run commands, diagnose errors, and repeatedly revise its approach before producing a correct solution [57, 23]. To effectively evaluate agents, we should therefore account not only for the final outcome, but also for how many interactions were required to reach the event of interest.

We consider an online setting where tasks arrive sequentially as a stream. At time tt, the agent receives a task XtX_t and interacts with its environment until either the task is stopped or successfully completed. Success is determined by an external verifier, such as a unit test, a task-specific correctness checker, or an independent LLM-based evaluator [58]. Only then does the next task Xt+1X_{t+1} arrive. An interaction may correspond to a model call, a tool invocation, an environment action, or an edit-and-test cycle, depending on the application [28, 56].

Following Davidov et al. [ [7], Feldman and Romano [2026], we define TtT_t as the time-to-success—the number of interactions required for the agent to complete task XtX_t. For a software-engineering agent, TtT_t may count the number of edit-and-test iterations required to fix a bug; for a browser agent, it may count the actions required to complete a transaction; for a research or data-analysis agent, it may count the tool calls required to produce a verifiably correct response [60, 57]. Unlike a binary success indicator, TtT_t captures the computational effort required to obtain that success and hence provides a more informative measure of agentic efficiency.

Our goal is to construct a calibrated upper predictive bound (UPB) U^t(Xt)\hat{U}_t(X_t) for the unknown time-to-success TtT_t of an arriving task XtX_t before executing it. We write Ut=U^t(Xt)U_t = \hat{U}_t(X_t) for ease of notation. Formally, for a given horizon TT, at a user-specified target level 1−α1-\alpha, we seek to satisfy the following finite-horizon average conditional coverage requirement:

1T∑t=1TP(Tt≤Ut∣Xt,Dt−1)≥1−α.(1)\frac{1}{T}\sum_{t=1}^{T}\mathbb{P}(T_t \le U_t \mid X_t, \mathcal{D}_{t-1}) \ge1-\alpha. \tag*{(1)}

Here, Dt−1\mathcal{D}_{t-1} denotes the completed observed history before task tt, as defined in § 1.2.

In practice, however, we do not always observe the time-to-success for all tasks as agent interactions are computationally expensive. We therefore assume an average budget of CtaskC_{\mathrm{task}} agentic iterations per task. To satisfy this constraint, we assign a censoring time CtC_t for each task tt, which limits the maximum number of iterations the agent may spend on that task. The censored event time is therefore formulated as: T~t=min⁡{Tt,Ct}\widetilde{T}_t = \min\{T_t, C_t\}. Importantly, the bound UtU_t and the censoring time CtC_t are deliberately distinct: when Ct<UtC_t < U_t, censoring may prevent us from determining whether the UPB UtU_t successfully covered the true time-to-success TtT_t.

For example, consider an organization whose employees use AI agents for tasks such as software engineering. Before running the agent, an employee may want to know how many interactions it will take to complete the task successfully. To address this, the organization can provide calibrated UPBs on the time-to-success, updated continually from the employees’ own interactions as projects, users, and deployed agents change over time. However, if an employee stops before the task is completed, the time-to-success and correctness of the UPB may remain unknown. To recover this information, the organization can continue the stopped trajectory in a controlled environment, such as a sandbox, without affecting the employee’s workflow or production systems. Since continuing every trajectory to completion is expensive, the organization can continue only some of them. Our framework runs on top of existing usage of the user and calibrates the UPBs from the resulting censored observations, providing UPBs with coverage guarantees under a limited budget.

Our problem is related to conformal survival analysis works [4, 20, 40, 39, 6], which study how to construct calibrated predictive bounds from censored observations. However, these methods are offline: they collect a fixed dataset of censored samples prior to calibrating the predictive bounds. In contrast, in our setting, the censoring times and predictive bounds are updated online. For each arriving task, we must determine for how many iterations the agent should execute based on information accumulated from previous tasks and the current task XtX_t. Crucially, this decision affects not only the computational cost of the current task, but also the quality of future predictions: the resulting censored trajectory becomes part of the data used to construct subsequent upper predictive bounds. Thus, prediction and data collection are inherently coupled.

To illustrate the challenge in this setup, we conduct an experiment on the software engineering tasks [30]. Figure 2 shows that ACI of Gibbs and Candes [2021] that is adapted to account for censoring yields overly conservative upper bounds under a limited budget. Censoring can hide prediction errors, making it more challenging to maintain informative bounds from the available feedback. This motivates the novel adaptive strategy we propose, which achieves the target coverage level by coupling budget allocation with an online calibration update that is explicitly designed to maintain tight and informative bounds under censored feedback.

Figure 2. Empirical evaluation of Upper Predictive Bounds on the SWE data. Left: Cumulative coverage rate over time comparing the Uncalibrated baseline, ACI+IPCW (§ 2), and the proposed method against a 90% target coverage rate. Right: Boxplots of the mean UPB across repetitions for the corresponding methods (lower is better). Full experimental details are provided in § 4.

This raises two questions: First, how can we construct valid upper predictive bounds from censored online data? Second, how should we allocate the budget across tasks to derive efficient and informative bounds?

Our Contribution

We introduce PRedictive Bounds for Online Budget Efficiency (PROBE), an online framework for constructing calibrated upper predictive bounds for the time-to-success of tasks under budget constraints (illustrated in Figure 1). Specifically, our main contributions are as follows:

Illustration of PROBE framework

Figure 1. Illustration of PROBE framework. For each arriving task (1), the system predicts the required time-to-success or token count (2). After environmental interaction where the agent executes the provided task (3), a budget is allocated for additional execution for agent evaluation (4). The result evaluation step (5) determines whether the task finished on time, leading to either a censored outcome with history-based correction (5a) or an observed outcome with direct trajectory evaluation (5b). Both outcomes are integrated via a coverage-aware update (6) (Theorem 3) consisting of (6a) online calibration to adjust predictive upper bounds (τt+1\tau_{t+1}) and (6b) a budget strategy to optimize future budget allocation (πt+1\pi_{t+1}).

  • Censoring-aware online calibration: Our method constructs UPBs from censored observations with a provably valid coverage bound for any fixed time horizon (§ 3.1; Appendix § 8.§ 1). We further identify a terminal cancellation condition under which this coverage certificate reaches the nominal threshold (Theorem 3).

  • Coverage-informed budget allocation: We formulate a constrained optimization objective for allocating budget across tasks to obtain a coverage rate closer to the target level. Our allocation aims to reduce future coverage error while satisfying the budget constraint (§ 3.2; Algorithm 4).

Code and instructions for reproducibility are provided in our GitHub

Problem Setup

In our setup, tasks arrive sequentially over rounds t=1,…,Tt = 1,\ldots,T. We denote by Xt∈XX_t \in\mathcal{X} all information available before the tt-th trajectory begins. In our software-engineering application, for example, XtX_t contains the model name and problem specification of the current task, together with information about the corresponding repository.

For each task XtX_t, the agent A\mathcal{A} interacts sequentially with a task-specific environment Φt\Phi_t. We initialize the interaction history as Ht(0)=XtH_t(0) = X_t. At each interaction step k≥1k \ge1, the agent selects an action based on the information observed so far At(k)=A(Ht(k−1))A_t(k) = \mathcal{A}(H_t(k - 1)). The environment then produces an observation in response to this action Ot(k)=Φt(Xt,Ht(k−1),At(k))O_t(k) = \Phi_t(X_t, H_t(k - 1), A_t(k)). A task-specific auditor Jt\mathcal{J}_t determines whether the event of interest has occurred kk: Yt(k)=Jt(Xt,Ht(k))∈{0,1}Y_t(k) = \mathcal{J}_t(X_t, H_t(k)) \in\{0,1\}. If Yt(k)=1Y_t(k) = 1, the event occurred and the trajectory terminates; otherwise, the interaction continues to step k+1k + 1. The history is updated as Ht(k)=(Ht(k−1),At(k),Ot(k),Yt(k))H_t(k) = (H_t(k - 1), A_t(k), O_t(k), Y_t(k)). The corresponding uncensored event time is

Tt=inf⁡{k≥1:Yt(k)=1},(2)T_t = \inf\{k \ge1 : Y_t(k) = 1\}, \tag*{(2)}

with the convention Tt=∞T_t = \infty if the event never occurs. In our setup, the user runs the task up to time RtR_t. After this user run, we choose a censoring time CtC_t beyond this user runtime RtR_t, and the censored time-to-event is thus T~t=min⁡(Tt,Ct)\widetilde{T}_t = \min(T_t, C_t). We denote by Δt=1{Tt≤Ct}\Delta_t = 1\{T_t \le C_t\} the censoring indicator: Δt=1\Delta_t = 1 means that the event time is observed at or before the timeout, whereas Δt=0\Delta_t = 0 certifies only Tt>CtT_t > C_t. The indicator resolves equality at the boundary: T~t=Ct\widetilde{T}_t = C_t with Δt=1\Delta_t = 1 means Tt=CtT_t = C_t, while the same equality with Δt=0\Delta_t = 0 means Tt>CtT_t > C_t. The actually consumed budget for the tt-th task is [T~t−Rt]+[\widetilde{T}_t - R_t]_+. Let

Dt−1={Hs(T~s),Rs,Cs,Δs}s<t\mathcal{D}_{t-1} = \{H_s(\widetilde{T}_s), R_s, C_s, \Delta_s\}_{s<t}

denote the completed observed history. The formal filtration convention is detailed in Appendix § 8.§ 1.1.

The budget constraint therefore limits the expected actual interaction cost to CtaskC_{\mathrm{task}} per task at every prefix:

∑s=1tE[[T~s−Rs]+∣Xs,Ds−1]≤tCtask,∀t.(3)\sum_{s=1}^{t} \mathbb{E}\left[[\widetilde{T}_s-R_s]_+ \mid X_s,\mathcal{D}_{s-1}\right] \le tC_{\mathrm{task}}, \quad\forall t. \tag*{(3)}

In Appendix § 10, we consider an alternative requirement in which the budget is controlled deterministically rather than in expectation.

In addition to the budget constraint (3), we adopt the standard assumption that CC is conditionally independent of TT given X,RX, R:

Assumption 1 (Conditionally independent censoring). *For every round tt, Ct⊥ ⁣ ⁣ ⁣⊥Tt∣Xt,Rt,Dt−1C_t \perp\!\!\!\perp T_t \mid X_t, R_t, \mathcal{D}_{t-1}.

Our budget allocator satisfies this condition by construction. Since we design the allocator, the distribution of Ct∣RtC_t \mid R_t is known, formalized by the conditional survival function:

Gt(u)=P(Ct≥u∣Xt,Rt,Dt−1).(4)G_t(u) = \mathbb{P}(C_t \ge u \mid X_t, R_t, \mathcal{D}_{t-1}). \tag*{(4)}

Our goal is to construct the calibrated UPB Ut=U^t(Xt)U_t = \widehat{U}_t(X_t) for the unknown time-to-event TtT_t of an arriving task XtX_t before executing it. Formally, for a target miscoverage level α∈(0,1)\alpha\in(0,1), this UPB should miscover TtT_t with probability at most α\alpha:

1T∑t=1Tmt=1T∑t=1TP(Tt>Ut∣Xt,Dt−1)≤α.(5)\frac{1}{T}\sum_{t=1}^{T}m_t = \frac{1}{T}\sum_{t=1}^{T}\mathbb{P}(T_t > U_t \mid X_t,\mathcal{D}_{t-1}) \le\alpha. \tag*{(5)}

Above, mt=P(Tt>Ut∣Xt,Dt−1)m_t = \mathbb{P}(T_t > U_t \mid X_t,\mathcal{D}_{t-1}) is the conditional miscoverage probability. Our objective is therefore to construct the predictive bounds UtU_t and allocate censoring times CtC_t online so that both the coverage and budget requirements are satisfied despite observing only right-censored event times.

Background

References

  1. [1]Meshi Bashari, Yonghoon Lee, Roy Maor Lotan, Edgar Dobriban, and Yaniv Romano. General synthetic-powered inference. In *Forty-third International Conference on Machine Learning*, 2026a. URL https://openreview.net/forum?id=sxLncu2Fhx.arxiv.org/abs/2509.20345
  2. [3]Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning to rank dataset for medical information retrieval. In *Advances in Information Retrieval*, pages 716–722. Springer, 2016. doi: 10.1007/978-3-319-30671-1_58.DOI
  3. [4]Emmanuel Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. *Journal of the Royal Statistical Society Series B: Statistical Methodology*, 85(1):24–45, 2023. doi: 10.1093/jrsssb/qkac004.DOI
  4. [5]Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In *2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)*, pages 23–42. IEEE, 2025.arxiv.org/abs/2310.08419
  5. [6]Hen Davidov, Shai Feldman, Gil Shamai, Ron Kimmel, and Yaniv Romano. Conformalized Survival Analysis for General Right-Censored Data. In *International Conference on Learning Representations*, 2025. URL https://openreview.net/forum?id=JQtuCUmAFD.
  6. [7]Hen Davidov, Shai Feldman, Gilad Freidkin, and Yaniv Romano. Calibrated predictive lower bounds on time-to-unsafe-sampling in LLMs. In *The 29th International Conference on Artificial Intelligence and Statistics*, 2026.arxiv.org/abs/2506.13593
  7. [8]Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025.
  8. [9]Bat-Sheva Einbinder, Shai Feldman, Stephen Bates, Anastasios N Angelopoulos, Asaf Gendler, and Yaniv Romano. Label noise robustness of conformal prediction. Journal of Machine Learning Research, 25(328):1–66, 2024.
  9. [10]Valerii Vadimovich Fedorov. Theory of optimal experiments. Elsevier, 2013.
  10. [11]Shai Feldman and Yaniv Romano. Robust conformal prediction using privileged information. Advances in Neural Information Processing Systems, 37:117813–117852, 2024.arxiv.org/abs/2406.05405
  11. [13]Shai Feldman, Stephen Bates, and Yaniv Romano. Improving conditional coverage via orthogonal quantile regression. Advances in neural information processing systems, 34:2060–2071, 2021.arxiv.org/abs/2106.00394
  12. [14]Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in online learning settings. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.arxiv.org/abs/2205.09095
  13. [16]Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP, pages 3356–3369, 2020.arxiv.org/abs/2009.11462
  14. [17]Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660–1672, 2021.arxiv.org/abs/2106.00170
  15. [18]Isaac Gibbs and Emmanuel J Candès. Conformal Inference for Online Prediction with Arbitrary Distribution Shifts. Journal of Machine Learning Research, 25(162):1–36, 2024.
  16. [19]Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, 87(4):1100–1126, 2025.arxiv.org/abs/2305.12616
  17. [20]Yu Gui, Rohan Hore, Zhimei Ren, and Rina Foygel Barber. Conformalized survival analysis with adaptive cut-offs. Biometrika, 111(2):459–477, 2024. doi: 10.1093/biomet/asad076.DOI
  18. [21]Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics, 29(1):96–108, 2011.
  19. [22]Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023.
  20. [23]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024a.arxiv.org/abs/2310.06770
  21. [24]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=VTF8yNQM66.arxiv.org/abs/2310.06770
  22. [25]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning, pages 5753–5763. PMLR, 2021.arxiv.org/abs/2103.05331
  23. [26]Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705, 2023.arxiv.org/abs/2309.02705
  24. [27]Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. Active evaluation acquisition for efficient llm benchmarking. arXiv preprint arXiv:2410.05952, 2024.arxiv.org/abs/2410.05952
  25. [28]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325–74362, 2024.arxiv.org/abs/2401.13178
  26. [29]Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065–61105, 2024.arxiv.org/abs/2312.02119
  27. [30]Nebius. SWE-agent-trajectories. Hugging Face dataset, 2024. URL https://huggingface.co/datasets/nebius/SWE-agent-trajectories.
  28. [31]Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European conference on machine learning, pages 345–356. Springer, 2002.
  29. [32]Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meridith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023.arxiv.org/abs/2304.03442
  30. [33]Coby Penso and Jacob Goldberger. A conformal prediction score that is robust to label noise. arXiv preprint arXiv:2405.02648, 2024.arxiv.org/abs/2405.02648
  31. [34]Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022.arxiv.org/abs/2202.03286
  32. [35]Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. In Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations, pages 431–445, 2023.
  33. [36]Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3982–3992. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410/.DOI
  34. [37]Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023.arxiv.org/abs/2310.03684
  35. [39]Matteo Sesia and Vladimir Svetnik. Doubly Robust Conformalized Survival Analysis with Right-Censored Data. arXiv preprint arXiv:2412.09729, 2024.arxiv.org/abs/2412.09729
  36. [40]Matteo Sesia and Vladimir Svetnik. Conformal survival bands for risk screening under right-censoring. arXiv preprint arXiv:2505.04568, 2025.arxiv.org/abs/2505.04568
  37. [41]Burr Settles. Active learning literature survey. University of Wisconsin, Madison, 52, 07 2010.
  38. [42]Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of machine learning research, 9(3), 2008.arxiv.org/abs/0706.3188
  39. [43]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2021. URL https://arxiv.org/abs/2104.08663.
  40. [44]Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1904.06019
  41. [45]Anastasios A Tsiatis. Semiparametric theory and missing data. Springer, 2006.
  42. [46]Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6):544–557, 2009.DOI
  43. [47]Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A Hale, and Paul Röttger. SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models. arXiv preprint arXiv:2311.08370, 2023.arxiv.org/abs/2311.08370
  44. [48]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World, volume 29. Springer, 2005. doi: 10.1007/b106715.DOI
  45. [49]Ganghua Wang, Zhaorun Chen, Bo Li, and Haifeng Xu. Cer-eval: Certifiable and cost-efficient evaluation framework for llms. arXiv preprint arXiv:2505.03814, 2025.arxiv.org/abs/2505.03814
  46. [50]Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.DOI
  47. [51]Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, pages 5776–5788, 2020.
  48. [52]Yuxin Wang, Dennis Frauen, Jonas Schweisthal, Maresa Schröder, Emil Javurek, and Stefan Feuerriegel. Adaptive experimentation for censored survival outcomes. arXiv preprint arXiv:2605.18459, 2026a.arxiv.org/abs/2605.18459
  49. [53]Yuyao Wang, Alexander W Levis, Shu Yang, and Larry Han. History-aware conformal prediction sets for censored time-to-event outcomes. arXiv preprint arXiv:2605.06581, 2026b.arxiv.org/abs/2605.06581
  50. [54]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in neural information processing systems, 36:80079–80110, 2023.arxiv.org/abs/2307.02483
  51. [55]Skyler Wu, Yash Nair, and Emmanuel J Candès. Efficient evaluation of llm performance with statistical guarantees. arXiv preprint arXiv:2601.20251, 2026.arxiv.org/abs/2601.20251
  52. [56]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024.arxiv.org/abs/2404.07972
  53. [57]John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024.arxiv.org/abs/2405.15793
  54. [58]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.arxiv.org/abs/2306.05685
  55. [59]Jin Peng Zhou, Christian K Belardi, Ruihan Wu, Travis Zhang, Carla P Gomes, Wen Sun, and Kilian Q Weinberger. On speeding up language model evaluation. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2407.06172
  56. [60]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606, 2024.arxiv.org/abs/2307.13854
  57. [61]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.arxiv.org/abs/2307.15043
  58. [62]Tijana Zrnic and Emmanuel Candès. Active Statistical Inference. In International Conference on Machine Learning, 2024.arxiv.org/abs/2403.03208

Paper details

vlnc:W_rumxq5zk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents