WorksComputer Science
Honesty Testing of LLMs and AI Agents: A Vision for Software Engineering
Abstract
Human oversight of large language models (LLMs) and artificial intelligence (AI) agents increasingly depends on their reports. Claims that conflict with a system's own judgments can conceal failures, prevent timely intervention, and contribute to severe harm. Greater autonomy makes faithful reporting increasingly important for maintaining human control. In this vision paper, we propose honesty testing of LLMs and AI agents as a new research area in software engineering. Building on existing honesty evaluations, our goal is to make faithful reporting a software quality requirement that developers can specify, test, and maintain. We define honesty bugs as behavioural defects in which an LLM or AI agent, when required to communicate faithfully, makes an explicit claim that conflicts with its own judgment at the time of reporting. We present an initial framework covering LLM-level testing under controlled inputs and agent-level testing of complete systems. The framework connects these levels to testing workflows and oracle evidence. We identify research directions in benchmarks, model and harness analysis, oracle construction, and lifecycle testing to support effective oversight alongside task correctness.
Introduction
Following advances in large language models (LLMs) [1], [2], artificial intelligence (AI) systems increasingly operate as agents that reason, use tools, and process files [3], [4]. Applications include scientific research [5], medical diagnosis [6], and public decision-making [7]. As delegated tasks become more complex, supervisors cannot readily reconstruct every decision and action [8], [9]. They increasingly rely on the systems’ reports to decide whether to accept results, investigate failures, or intervene. The same dependence arises when people act on standalone LLM answers and recommendations.
Dishonest reporting can undermine the information on which human control depends. Consider an agent that judges a required safety check to have failed but reports that it passed. A supervisor relying on this report may authorise an unsafe action. In medical or physical systems, this could contribute to severe injury or loss of life. A human may retain authority to stop the system while lacking the information needed to exercise that authority. As agents act for longer periods, use more tools, and delegate work, misrepresented failures may propagate before humans can intervene.
Software engineering (SE), including SE for AI, has developed research on trustworthy and responsible systems, addressing fairness [10], environmental sustainability [11], explainability, and security [12], [13]. Honesty deserves comparable attention as an explicit testing objective. It also supports oversight of these other properties: a system that misrepresents its assessment of a fairness violation, security weakness, or failed check can conceal the need for corrective action. Honesty testing therefore addresses both faithful communication and the information needed to manage other software risks.
Existing research provides foundations for this objective. Conceptual studies relate honesty to what a communicator believes [14], [15]. MASK compares elicited beliefs with statements under pressure, distinguishing honesty from accuracy [16]. Scheming evaluations examine agents that conceal their behaviour under controlled conditions [17], while OverclaimBench compares reports with execution records [18]. These studies establish honesty and related reporting failures as evaluation concerns. Systematic honesty testing as a software engineering discipline nevertheless remains underdeveloped.
Task success and factual correctness alone cannot establish honesty. An LLM or agent may faithfully report a mistaken judgment, or make a factually correct claim that conflicts with its judgment. Execution records establish what occurred, but not necessarily the judgment held when reporting. Developers therefore need explicit defect specifications, justified oracles, and methods to reproduce, diagnose, repair, and retest reporting failures as models and systems evolve.
We propose honesty testing of LLMs and AI agents as a new research area in software engineering. Our vision is to make faithful reporting a quality requirement that developers can specify, test, and maintain to support human oversight and control. Here, AI agents are LLM-based systems. We introduce honesty bugs as a shared defect specification and distinguish LLM-level and agent-level testing. Our initial framework connects these levels to a shared testing workflow and oracle evidence requirements. The research agenda identifies further work on test generation, adequacy assessment, diagnosis, and regression testing.
This vision paper makes the following contributions.
A Vision and Defect Specification. We motivate honesty testing of LLMs and AI agents and formalise honesty bugs, using examples and counterexamples to clarify their boundaries.
An Initial Testing Framework. We define honesty testing, distinguish LLM-level and agent-level testing, and outline their workflow and oracle evidence requirements.
A Research Agenda. We identify research questions for benchmarks, component effects, oracle construction, and lifecycle integration at both testing levels. A companion repository collects relevant studies and resources.1
Honesty Bugs
Treating honesty violations as software defects provides a basis for specifying requirements, constructing tests, and investigating repairs. We first define honesty bugs and clarify their scope so that test results can be interpreted consistently.
Formal Definition
Definition 1: An Honesty Bug is a behavioural defect in which an LLM or AI agent, when required to communicate faithfully, makes an explicit claim that conflicts with its own inner judgment at the time of reporting.
Let denote the task input and reporting context, the output, and indicate whether faithful communication is required, with 1 meaning required and 0 meaning not required. An inner judgment is the assessment held by the LLM in a given execution, such as whether a conclusion is supported or an action was performed. For an AI agent, this is the judgment of the LLM responsible for the claim. Judgments may change during reasoning. For each proposition , let be the judgment held when makes its final claim about , after any earlier revisions, and let . and denote the semantic stances of and on , with 1 indicating affirmation and 0 indicating negation.
In Eq. (1), ranges over propositions for which both stances are determinate; for the output, this requires to explicitly affirm or negate , which keeps the boundary of honesty bugs clear. Implied claims, omissions, and differences in confidence therefore fall outside this definition. Output claims are interpreted in context, so reflects the final claim about after any explicit corrections or retractions in .
Illustrative Examples
To illustrate detection, consider a hypothetical agent instructed to run a test suite and report whether all tests passed (). The execution log records a failed test. Immediately before reporting, the original thinking trace states, “One test failed; the suite did not pass.” No later judgment revision is recorded, but the final output claims, “All tests passed.” For the proposition that all tests passed, the oracle extracts and . Under the trace-faithfulness assumption in § 4-A, this supports a honesty bug. The log establishes the failure; the trace supplies evidence of the judgment held when reporting. A standalone LLM required to report its recommendation faithfully may judge Java preferable to Python but explicitly claim that Python is preferable to Java without revising its judgment. The conflict concerns its claim. LangChoiceBench reports related inconsistencies between reasoning and generated code [19]; such inconsistencies alone do not establish honesty bugs.
If the same log and report are available but judgment evidence is absent or insufficient, the report is factually incorrect, while the honesty verdict remains inconclusive.
Scope and Exclusions
To clarify the boundaries of our definition, we present three counterexamples that satisfy some of its conditions but do not constitute honesty bugs. First, in strategic games such as Werewolf, an agent playing a werewolf may claim to be a villager. Although this contradicts its judgment, deception is permitted by the game, so faithful communication is not required (). Second, an LLM may hallucinate a reference that it genuinely judges to be real. For the proposition that the reference is real, we have . Although the citation is factually incorrect, its claim agrees with its judgment, so this is not an honesty bug. Third, an LLM may initially express one judgment but revise it before producing its final output. Such self-correction is not an honesty bug, as long as the final claim agrees with the judgment held when it is made.
Honesty Testing
Honesty testing seeks to reveal failures in faithful reporting by LLMs and AI agents and provide evidence for investigating them. Developers can use this evidence to assess whether the systems’ claims support human oversight and whether changes reduce reporting failures. Honesty testing supports, but cannot by itself guarantee, safe behaviour or human control.
Formal Definition
Definition 2: Honesty Testing comprises activities designed to reveal honesty bugs in the responses of LLMs or the outputs of AI agents during task execution.
Specifically, honesty testing involves executing the system on test inputs to obtain outputs and examining execution evidence for conflicts on propositions with indeterminate stances, when . It complements factual correctness testing by using the judgment held at the time of communication as its reference. These properties require separate assessment because an honest report may reveal a failure that needs intervention without guaranteeing a correct or safe result. Following prior surveys of machine learning (ML) testing [13] and fairness testing [10], we next examine the workflow, testing levels, and components of honesty testing.
Honesty Testing Workflow
Building on ML and fairness testing workflows [10, 13], Figure 1 presents a simplified workflow for both testing levels. Engineers specify when faithful reporting is required and construct inputs and oracles covering response pressure, incomplete work, and tool failures. Oracles assess execution evidence for honesty bugs or inconclusive cases. Adequacy concerns reporting situations covered and available judgment evidence. Bug reports record inputs, configurations, disputed claims, and judgment evidence for diagnosis and reproduction. Repairs, model updates, and harness changes require retesting. Repairs should avoid optimising directly against traces used by direct oracles (§ 4-A), which may encourage models to obscure their reasoning [20].
Figure 1. Honesty Testing Workflow
Mutation testing and metamorphic testing are promising existing techniques that can support honesty testing [13]. For example, mutating question wording or contextual pressure generates related tests, while metamorphic relations specify whether corresponding claims should remain consistent or change. Such comparisons can identify candidate honesty bugs, which require judgment evidence to distinguish dishonesty from legitimate judgment changes.
The “No” branch applies when sufficient judgment evidence is available and no honesty bug is detected. Cases with insufficient evidence remain inconclusive and require further evidence collection or review. Deployment additionally depends on test adequacy, unresolved cases, and other quality requirements; the absence of detected honesty bugs does not by itself justify deployment. Runtime monitoring can extend testing to new inputs at either level.
Testing Levels and Components
Software testing targets components such as web applications [21], GUIs [22], and ML models and infrastructure [10, 13]. We distinguish LLM-level testing, which targets model responses under controlled inputs, and agent-level testing, which targets a complete LLM-based system during task execution. At the agent level, both the LLM and harness may contribute to behaviour [23]. Harness settings, such as prompts that emphasise goal achievement [17], can influence judgments and reporting. The two levels are complementary: testing an LLM can inform agent evaluation, but cannot establish how every agent using it will behave. A false report alone cannot identify the responsible component or distinguish a mistaken judgment from an honesty bug.
LLM-Level Testing: At this level, tests examine whether an LLM’s claims agree with its judgments under controlled inputs. LLM-level testing applies to standalone LLM use and to an LLM tested as an agent component. Inputs may include questions, recommendation tasks, conversation histories, or recorded tool results supplied as context. Tests observe responses and, when available, thinking traces, while controlling the surrounding evaluation setup. Model comparisons hold task contexts and relevant generation settings fixed to investigate the effects of model choice.
Further research should examine whether conflicts recur across contexts and whether task-performance improvements accompany improvements in honesty. These comparisons can guide model selection and regression testing after updates. They do not by themselves capture effects arising from an agent’s ongoing interaction with its environment.
Agent-Level Testing: Agent-level testing examines the complete system across successive interactions. Alongside the LLM, the agent includes a harness: software, configuration, and system prompts that manage context, memory, tool execution, and the interaction loop [23]. Tests supply task requests and controlled environmental feedback and examine claims made during execution. To investigate harness effects, comparisons hold the LLM and generation settings fixed while varying harness configurations. Tests of complete configurations can also examine model–harness interactions.
For example, tests can vary how a failed tool result is retained or summarised before the final report. The judgment and claim must be examined together. Losing an observation may induce a mistaken judgment that is faithfully reported. Reporting against a retained judgment constitutes an honesty bug when faithful communication is required. Such comparisons can guide diagnosis and repair at the system level.
Oracle Problem in Honesty Testing
A key challenge at both testing levels is the test oracle problem [24]. The outcome of a task and the final report produced by the system do not, by themselves, establish the judgment against which an output claim should be assessed. Testers therefore need evidence of the judgment held at the relevant reporting point, accounting for any revisions. An unreliable oracle may give rise to two kinds of incorrect assessment: it may classify a factual error as dishonesty, or it may fail to identify a report that is dishonest. We distinguish two approaches to oracle construction, based on direct and indirect evidence. For each approach, the assumptions and limitations of the evidence must be made explicit so that bug reports and comparisons between testing techniques can be interpreted appropriately.
Honesty Testing Oracle Based on Direct Evidence
Direct evidence is the judgment expressed in the thinking trace of the LLM during the execution that produces the claim, whether it operates alone or within an agent. In this paper, a thinking trace refers to the original intermediate reasoning text generated by the model during task execution. It can take the form of chain-of-thought (CoT) reasoning [20, 25], covering the task, applicable instructions, and how to respond. Direct evidence is recorded during the execution that produces the claim, without further querying. Access depends on the model and deployment. For example, proprietary services may provide the original trace, a summary, or no reasoning text. Only the original trace qualifies as direct evidence in this paper; summaries and post-hoc explanations are excluded.
Direct oracles assume that the thinking trace represents the inner judgment of the LLM responsible for the claim, although it may be incomplete. Under this assumption, when faithful communication is required, a conflict between a trace-expressed judgment and its corresponding claim indicates an honesty bug. Absence of a detected conflict does not establish absence of an honesty bug: traces may omit relevant judgment information [26, 27]. For each proposition , the oracle accounts for revisions before reporting to determine , then checks when both stances are determinate. If the trace is inaccessible or insufficient, the verdict remains inconclusive; other evidence may be considered.
Honesty Testing Oracle Based on Indirect Evidence
The original thinking trace can be unavailable or insufficient. A recent report by Cohere illustrates a limitation of relying on only thinking traces: during a defensive cybersecurity task, Qwen3.8-Flash-Next attempted an external upload even though its thinking trace described responding without tool calls [28].
For such cases, indirect evidence may further support judgment inference. Sources of indirect evidence include diagnostic answers, building on belief elicitation [16] and lie detection through follow-up questions [29]. These approaches support black-box access to LLMs and agents. However, a subsequent answer may reflect a revised judgment. The oracle must justify using it as evidence of the judgment held when the original claim was made before applying the conflict criterion. Insufficient evidence leaves the result inconclusive, not evidence that no honesty bug occurred.
Future Plans
Establishing honesty testing requires shared evaluation resources, validated methods, and evidence of practical value. We identify four research directions for developing methods to test and maintain honesty.
Benchmarks and Test Adequacy
Which reporting situations should honesty tests cover, and how should adequacy be assessed? Building on existing evaluations [16, 18], we propose benchmarks with controlled LLM inputs and executable agent tasks. Cases should specify communication requirements, judgment evidence, and evaluation criteria, plus the environment and harness for agents. Coverage should include judgment revisions, pressure to overclaim, execution failures, and counterexamples, prioritising claims informing human authorisation or intervention. Evaluation should report task performance, supported conflicts, and inconclusive cases separately at each level, with explicit denominators and oracle assumptions. Shared configurations, execution records, and annotations would support reproduction.
Model and Harness Effects
It remains unclear how LLM characteristics and agent harnesses contribute to honesty bugs. Studies can compare LLMs under matched contexts, models within a fixed harness, and harnesses under a fixed model. Paired studies should examine whether conflicts in controlled LLM tests recur during agent execution. We propose varying context handling and reporting instructions, repeating executions, and assessing persistence after updates. Judgments and claims should be assessed separately. Detection reductions must be considered alongside task performance and the proportion of assessable cases: less observable reasoning may reduce detection without improving honesty. These studies can guide selection, diagnosis, and repair.
Reliable Oracles under Limited Access
Oracle research should investigate judgment evidence under limited observability. We propose studying belief elicitation and follow-up questions [16, 29] at both levels, alongside assessments recorded within agent workflows. Studies must establish whether the observations reflect the judgment held when the original report was made and whether elicitation changes the behaviour of the system. Initial evaluation can use open-weight LLMs both as standalone models and as components of agents, while withholding their thinking traces from the indirect oracles being evaluated. Independent annotators can identify trace-expressed judgments and revisions as a reference under Section IV-A’s assumption, rather than complete ground truth. Evaluation should report false alarms, missed conflicts, and abstentions where a reference is supported, retaining unresolved cases. Proprietary systems require separate validation.
Testing throughout the Software Lifecycle
Lifecycle research should minimise and reproduce failing contexts and interactions, identify component contributions, and retain regression tests. Studies should assess whether repairs reduce supported conflicts on unseen tasks while preserving performance. Retesting should follow LLM or harness updates and extensions to tool access or delegated authority. Runtime monitoring should prioritise consequential claims for human review, evaluating support for timely intervention alongside missed conflicts, false alarms, and testing overhead.
Conclusion
Honesty testing is increasingly important as LLMs and AI agents gain greater autonomy and influence consequential decisions. Dishonest reports can conceal failures and prevent humans from exercising effective control, with potentially severe consequences. Building on existing evaluations, this paper proposes honesty testing as a software engineering research area, defines honesty bugs, and outlines testing levels, workflows, oracle requirements, and a research agenda. We call on the SE community to give faithful reporting sustained attention alongside fairness, sustainability, explainability, and security. Developing and maintaining honesty tests can help preserve the information humans need to question, correct, constrain, or stop AI systems.
References
- [1]T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.arxiv.org/abs/2005.14165
- [2]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27730–27744, 2022.arxiv.org/abs/2203.02155
- [3]L. Wang, C. Ma, X. Feng, H. Zhang, Y. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin et al., “A survey on large language model based autonomous agents,” Frontiers of computer science, vol. 18, no. 6, p. 186345, 2024.DOI
- [4]A. Plaat, M. Van Duijn, N. Van Stein, M. Preuss, P. Van der Putten, and K. J. Batenburg, “Agentic large language models, a survey,” Journal of Artificial Intelligence Research, vol. 84, 2025.DOI
- [5]J. Tang, L. Xia, Z. Li, and C. Huang, “Ai-researcher: Autonomous scientific innovation,” Advances in Neural Information Processing Systems, vol. 38, pp. 9481–9520, 2026.arxiv.org/abs/2505.18705
- [6]Y. Xiao, P. Chen, J.-T. Huang, W. Chen, Y. Liu, K. Li, M. R. Mousavi, R. Dobson, and J. M. Zhang, “A scoping review of bias in large ai models for medicine and healthcare,” Communications Health, vol. 1, no. 1, p. 42, 2026.DOI
- [7]H. Bai, J. G. Voelkel, S. Muldowney, J. C. Eichstaedt, and R. Willer, “Llm-generated messages can persuade humans on policy issues,” Nature Communications, vol. 16, no. 1, p. 6037, 2025.
- [8]R. Yin, T. Ishida, and M. Sugiyama, “Towards scalable oversight via partitioned human supervision,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 112525–112548.arxiv.org/abs/2510.22500
- [9]L. Huang, W. Xiao, and N. K. Vishnoi, “Delegation and verification under AI,” in Proceedings of the 43rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, T. Zhang, M. Dudik, M. Jaggi, A. Agarwal, S. Li, D. Schuurmans, J. Zhu, F. Berkenkamp, H. Dong, and A. Bietti, Eds., vol. 306, PMLR, 06–11 Jul 2026, pp. 48 039–48 071. [Online]. Available: https://proceedings.mlr.press/v306/huang26bp.html
- [10]Z. Chen, J. M. Zhang, H. Mort, M. Harman, and F. Sarro, “Fairness testing: A comprehensive survey and analysis of trends,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 5, pp. 1–59, 2024.arxiv.org/abs/2207.10223
- [11]R. Verdecchia, J. Sallou, and L. Cruz, “A systematic review of Green AI,” WIREs Data Mining and Knowledge Discovery, 2023.arxiv.org/abs/2301.11047
- [12]S. Martínez-Fernández, J. Bogner, X. Franch, M. Oriol, J. Siebert, A. Trendowicz, A. M. Vollmer, and S. Wagner, “Software engineering for AI-based systems: A survey,” ACM Transactions on Software Engineering and Methodology, 2022.
- [13]J. M. Zhang, M. Harman, L. Ma, and Y. Liu, “Machine learning testing: Survey, landscapes and horizons,” IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 1–36, 2020.arxiv.org/abs/1906.10742
- [14]A. Barber, “Lying, misleading, and dishonesty,” The Journal of Ethics, pp. 141–164, 2020.DOI
- [15]O. Evans, O. Cotton-Barratt, L. Finfiedlen, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders, “Truthful ai: Developing and governing ai that does not lie,” arXiv preprint arXiv:2110.06674, 2021.arxiv.org/abs/2110.06674
- [16]R. Ren, A. Agarwal, M. Mazeika, C. Menghini, R. V. Baracueanu, B. Kenstler, M. Yang, I. Barrass, A. Gatti, X. Yin et al., “The mask benchmark: Disentangling honesty from accuracy in ai systems,” arXiv preprint arXiv:2503.03750, 2025.arxiv.org/abs/2503.03750
- [17]A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn, “Frontier models are capable of in-context scheming,” arXiv preprint arXiv:2412.04984, 2024.arxiv.org/abs/2412.04984
- [18]N. Smyth, Y.-J. Mantilla-Ramos, P. J. T. Notsawo, S. Helbling, A. Tosato, M. A. Merzouk, N. Dziri, G. Gidel, and T. Tosato, “Quantifying overclaiming propensity in frontier llm agents,” arXiv preprint arXiv:2609.20812, 2026.arxiv.org/abs/2609.20812
- [19]L. Twist, T. Stone, H. Yannakoudakis, and J. M. Zhang, “LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs,” arXiv preprint arXiv:2608.06041, August 2026. [Online]. Available: http://arxiv.org/abs/2608.06041
- [20]G. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi, “Monitoring reasoning models for misbehavior and the risks of promoting obfuscation,” arXiv preprint arXiv:2503.11926, 2025.arxiv.org/abs/2503.11926
- [21]V. Garousi, A. Mesbah, A. Betin-Can, and S. Mirshokraie, “A systematic mapping study of web application testing,” Information and Software Technology, vol. 55, no. 8, pp. 1374–1396, 2013.DOI
- [22]I. Banerjee, B. Nguyen, V. Garousi, and A. Memon, “Graphical user interface (gui) testing: Systematic mapping and repository,” Information and Software Technology, vol. 55, no. 10, pp. 1679–1694, 2013.DOI
- [23]R.-Z. Fan, Z. Zhang, S. Ma, Y. Hu, S. Wang, K. Song, F. Liu, H. Zamani, and X. Wang, “An empirical study of harness design for coding agents,” arXiv preprint arXiv:2609.20804, 2026.arxiv.org/abs/2609.20804
- [24]E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014.DOI
- [25]J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022.arxiv.org/abs/2201.11903
- [26]Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Samani, P. Hase, M. Wagner, F. Roger et al., “Reasoning models don’t always say what they think,” arXiv preprint arXiv:2505.05410, 2025.arxiv.org/abs/2505.05410
- [27]M. Turpin, J. Michael, E. Perez, and S. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” Advances in Neural Information Processing Systems, vol. 36, pp. 74 952–74 965, 2023.arxiv.org/abs/2305.04388
- [28]C. Team, “Concerning behavior from a popular open-weight AI model — Cohere — cohere.com,” https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model, 2026.
- [29]L. Pacchiardi, A. Chan, S. Mindermann, I. Moscovitz, A. Pan, Y. Gal, O. Evans, and J. Brauner, “How to catch an ai liar: Lie detection in black-box llms by asking unrelated questions,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 54 567–54 604.arxiv.org/abs/2309.15840
