WorksComputer Science
The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering
Abstract
Context: Large Language Models (LLMs) are increasingly embedded in empirical software engineering (ESE) workflows, supporting tasks such as classification, coding, extraction, synthesis, and evaluation. Their probabilistic and opaque behavior, however, introduces methodological concerns that may affect empirical rigor. Aims: This paper investigates threats to validity introduced by LLM-assisted empirical workflows and examines how LLM interactions influence empirical studies. Method: We conducted a secondary analysis of 50 primary studies comprising 69 LLM-assisted research tasks identified through a systematic literature review. We examined the extent to which LLM-related threats were reported, how these threats were characterized, and how they related to established validity dimensions. Results: LLM-related threats remain substantially underreported. We identified recurring threats including classification errors, prompt-dependent interpretation, workflow instability, evaluator bias, downstream propagation of outputs, and model dependence. These threats can affect multiple validity dimensions depending on where and how LLM outputs enter the workflow. Conclusion: We introduce the concept of hidden methodological layer as a conceptual lens for understanding how LLMs influence empirical software engineering workflows. We argue that rigorous LLM-assisted ESE requires making prompts, model configurations, interaction protocols, validation procedures, and downstream reuse of LLM-generated outputs visible as part of the research method.
CCS Concepts: • General and reference → Surveys and overviews.
Additional Key Words and Phrases: Large language model, LLM, empirical software engineering, threats to validity
Introduction
Empirical Software Engineering (ESE) relies on carefully designed methodological processes to produce valid and reliable knowledge about software systems and practices [17, 22, 30]. This reliance on methodological rigor makes the explicit control and reporting of research decisions central to ESE, particularly those concerning data collection, construct operationalization, analysis procedures, and interpretation of results [8, 14]. Together, these elements form the backbone of empirical rigor, enabling researchers to assess validity, replicate studies, and build cumulative knowledge [2].
Given ESE’s dependence on the transparency and control of these methodological decisions, the increasing adoption of Large Language Models (LLMs) in software engineering introduces a fundamental shift in how empirical research is conducted [13]. Unlike traditional tools, which are typically deterministic and explicitly configured, LLMs act as adaptive, general-purpose agents capable of interpreting instructions, generating outputs, and making implicit decisions.
Authors’ Contact Information: Victória Gomes, Virginia Commonwealth University, Richmond, USA, [email protected]; Fabio Palomba, University of Salerno, Fisciano, Italy, [email protected]; Robert Feldt, Chalmers University of Technology, Gothenburg, Sweden, [email protected]; Thomas Zimmermann, University of California, Oakland, USA, [email protected]; David Lo, Singapore Management University, Singapore, Singapore, [email protected]; Rodrigo Spinola, Virginia Commonwealth University, Richmond, USA, [email protected]. during task execution [4, 5, 28]. As a result, they are no longer merely supporting tools, but active components in the research process, influencing how data is processed, interpreted, and transformed [32].
This shift changes the methodological conditions under which empirical research is conducted. When researchers rely on LLMs to perform tasks such as classification, coding, or synthesis, part of the research logic becomes embedded in interactions with the model, including prompts, parameter settings, and iterative refinements [26]. These interactions may introduce assumptions, biases, and stochastic behaviors that mediate how empirical evidence is constructed, interpreted, and reported [23, 29]. Importantly, while these interactions provide some level of control, the internal behavior of LLMs remains largely opaque and only partially controllable, limiting the researcher’s ability to fully understand or predict how outputs are produced [4, 29]. Consequently, part of the methodological process becomes effectively hidden, operating beneath the observable workflow [7].
While this hidden layer may streamline and scale empirical activities, it also raises critical concerns. If key methodological decisions are implicit, difficult to control, or inconsistently reported, then traditional assumptions about validity, reproducibility, and transparency may no longer hold. Understanding this shift is essential for ensuring that the integration of LLMs strengthens rather than undermines empirical rigor.
Problem Statement
Current research on LLM-assisted empirical software engineering has primarily focused on the capabilities and applications of these models, examining how they can support specific tasks or improve efficiency [13]. However, much less attention has been given to the methodological implications of their use, particularly regarding how LLMs alter the structure and properties of empirical workflows [23, 27].
In this paper, we argue that this gap reflects a deeper methodological issue. Although LLMs may be used as analytical tools, their integration into empirical workflows can become methodologically consequential when model-mediated outputs participate in the construction, transformation, interpretation, or evaluation of empirical evidence. We refer to the resulting set of partially visible methodological influences as the hidden methodological layer. This layer is “hidden” not because the use of the LLM itself is concealed, but because part of its methodological influence is mediated through opaque, probabilistic, and input-sensitive model behavior, as well as through prompts, configurations, and interaction patterns that may be only partially inspectable or insufficiently reported. As a result, it becomes difficult not only to assess how results were produced and reproduce findings, but also to determine whether the methodological risks introduced by LLM use were explicitly recognized and addressed by researchers. Empirical studies may therefore be subject to validity concerns that arise not solely from explicit study design decisions, but also from how LLM-mediated processes operate within the broader empirical workflow, affecting established study-specific validity dimensions in subtle and sometimes difficult-to-observe ways
In practice, this means that constructs may be implicitly redefined through prompts, causal relationships may be confounded by stochastic model behavior, findings may depend on specific models or configurations, and conclusions may vary across repeated executions of the same workflow [9, 28]. Despite their importance, these issues have not yet been systematically conceptualized or analyzed.
Objectives
The goal of this paper is to characterize and conceptualize the hidden methodological layer introduced by LLMs in empirical software engineering, through the analysis of how threats to validity emerging from LLM-assisted workflows are reported, manifested, and related to established validity dimensions. To operationalize this goal, we investigate three complementary aspects: (i) the extent to which such threats are explicitly reported in the literature, (ii) the types of threats reported in LLM-assisted empirical workflows, and (iii) how these threats relate to classical validity dimensions in empirical software engineering. These objectives are organized around the following overarching research question:
Overarching Research Question
How does the hidden methodological layer manifest in LLM-assisted empirical software engineering?
The specific research questions derived from this overarching question are presented in § 3, together with the methodological procedures adopted to investigate them.
Study Overview
This study builds upon a systematic literature review on LLM-assisted empirical software engineering, which analyzed 50 primary studies published between 2020 and 2025 across leading software engineering venues [11]. The original SLR focused on characterizing the ESE research tasks supported by LLMs, the phases of the empirical research process in which they are applied, how they are integrated into empirical workflows, the benefits and limitations reported in the literature, and the extent to which studies report information required for reproducible LLM usage.
In contrast, the present work adopts a complementary perspective by performing a secondary analysis of the same corpus to investigate the methodological implications of using LLMs as part of empirical research tasks. Specifically, we focus on examining (i) whether threats to validity introduced by LLM-assisted research tasks are explicitly reported, (ii) what kinds of validity threats are reported as arising from the integration of LLMs into empirical workflows, and (iii) how these threats relate to classical validity dimensions in empirical software engineering.
To support these analyses, we introduce a conceptual model that represents LLM-assisted empirical workflows as a multi-layered structure. The model distinguishes between the observable elements of empirical processes, the interactions with LLMs, and the underlying mechanisms through which these interactions influence research outcomes, as well as their implications for threats to validity. This layered perspective provides a structured lens to systematically reason about where and how validity issues may arise.
Contributions
This paper makes the following contributions:
A conceptualization of the hidden methodological layer introduced by LLMs in empirical software engineering workflows, together with a conceptual model describing the multi-layered structure of LLM-assisted empirical processes;
An empirical investigation of how LLM-related threats to validity are currently reported and manifested across different forms of LLM involvement in empirical workflows;
A taxonomy of reported LLM-related threats to validity, including their mapping to established study-specific validity dimensions, providing a structured framework for reasoning about methodological risks in LLM-assisted empirical software engineering.
Paper Organization
The remainder of this paper is organized as follows. § 2 presents the background and introduces key concepts related to empirical software engineering and LLM-assisted workflows. § 3 describes the research method, including the data source and the analysis procedure adopted in this study. § 4 presents the results, structured around the research questions. § 5 discusses the findings and introduces the proposed conceptual model, synthesizing how LLM-assisted workflows give rise to validity concerns. § 6 outlines the threats to validity of this study. § 7 concludes the paper and outlines directions for future research.
Background
In this section, we first revisit established perspectives on validity in empirical studies, highlighting the assumptions underlying traditional frameworks. We then characterize key properties of LLMs that are particularly relevant from a methodological perspective, providing the foundation for understanding how these systems influence empirical processes and contribute to validity concerns.
Validity in Empirical Software Engineering
Empirical software engineering relies on well-established frameworks to assess the rigor and credibility of research findings [17, 25, 31]. Among the most widely adopted are the validity dimensions articulated by Wohlin et al. [30]: construct validity, internal validity, external validity, and conclusion validity. These dimensions provide a structured lens for reasoning about whether a study accurately captures the intended concepts, establishes credible causal relationships, generalizes beyond the studied context, and supports statistically sound conclusions [2, 22].
Although these dimensions are central to empirical software engineering, validity concerns are not operationalized in the same way across all empirical methods. Different study types involve different sources of evidence, procedures, and inferential goals, and therefore emphasize different kinds of validity concerns. For example, experiments and case studies commonly rely on classical validity dimensions such as construct, internal, external, and conclusion validity, whereas secondary studies emphasize risks related to search, selection, data extraction, and synthesis. Survey-based research, in turn, foregrounds threats associated with sampling, non-response, instrument design, and response behavior. Table 1 summarizes these validity dimensions and threat categories across empirical software engineering studies.
| Study Type | Validity dimension | Description |
| Study type | Threat | Description |
| Experiments, Case Studies, and MSR [30] | Construct Validity | Extent to which the operationalization of variables accurately reflects the intended theoretical constructs. Threats arise from mismatches between concepts and their measurements. |
| Experiments, Case Studies, and MSR [30] | Internal Validity | Degree to which observed effects can be attributed to the variables under study rather than confounding factors. Threats include biases and uncontrolled variables. |
| Experiments, Case Studies, and MSR [30] | External Validity | Extent to which findings generalize beyond the study context. Threats occur when results do not hold across different populations, settings, or conditions. |
| Experiments, Case Studies, and MSR [30] | Conclusion Validity | Reliability of conclusions drawn from the data, especially regarding statistical relationships. Threats include low statistical power and improper analysis methods. |
| SLRs and Mapping Studies [16] | Search Bias | Incomplete or inadequate search strategies that fail to identify all relevant studies, potentially leading to skewed evidence. |
| SLRs and Mapping Studies [16] | Selection Bias | Incorrect inclusion or exclusion of studies due to ambiguous or inconsistently applied selection criteria. |
| SLRs and Mapping Studies [16] | Data Extraction Bias | Errors or inconsistencies in extracting data from primary studies, which may affect synthesis results. |
| SLRs and Mapping Studies [16] | Synthesis Bias | Inappropriate aggregation or interpretation of results across studies, potentially leading to misleading conclusions. |
| Surveys [18] | Sampling Bias | The selected sample is not representative of the target population, limiting generalizability. |
| Surveys [18] | Non-response Bias | Differences between respondents and non-respondents that may distort results. |
| Surveys [18] | Instrument Bias | Poorly designed survey instruments (e.g., ambiguous or leading questions) that affect the accuracy of responses. |
| Surveys [18] | Response Bias | Systematic tendencies of respondents to answer inaccurately or untruthfully (e.g., social desirability bias). |
Table 1. Threats to validity across empirical software engineering study types.
These established perspectives have been fundamental for understanding how validity threats arise from study design choices, data collection procedures, measurement decisions, and analysis methods [8, 14]. However, they largely assume that empirical work is conducted either by human researchers or through tools whose behavior is deterministic and explicitly configured. This assumption becomes less sufficient when modern AI-based systems are integrated into research workflows, because such systems may participate in data transformation, interpretation, classification, synthesis, or evaluation in ways that are adaptive, probabilistic, and only partially transparent [4, 24].
Methodologically Relevant Properties of LLMs
The methodological impact of LLMs on empirical research is closely tied to a set of properties that distinguish them from traditional deterministic tools. These properties shape how LLMs influence empirical workflows and contribute to validity concerns in LLM-assisted studies. Prior research on LLMs and neural text generation has extensively discussed these characteristics [3, 4, 12, 15, 19, 20, 21, 33]:
Opacity. The internal reasoning processes of LLMs are not fully observable or explainable. While models can generate outputs and, in some cases, provide explanations, the underlying decision-making process remains largely inaccessible, making it difficult to inspect or justify intermediate analytical steps.
Probabilistic Outputs. LLMs operate as probabilistic systems, meaning that identical inputs may yield different outputs across executions. This non-determinism introduces variability that can affect the stability, reproducibility, and comparability of empirical results across runs and settings.
Input Sensitivity. Model outputs are highly sensitive to prompt formulation. Small changes in wording, structure, or context may lead to substantially different responses, effectively turning prompt design into a critical and often implicit methodological decision.
Hidden Configuration. The behavior of LLMs is influenced by factors such as model version, parameter settings (e.g., temperature), and system-level prompts. These elements are frequently underreported, yet they can significantly affect outputs and, consequently, research findings.
Autonomous Influence. LLMs can directly shape research outcomes by generating labels, classifications, or interpretations that may be incorporated into the analysis without systematic human validation, particularly in large-scale or automated workflows.
Training Data Dependence. LLM outputs reflect patterns learned from large and heterogeneous training corpora. As a result, they may encode biases, outdated information, or domain-specific gaps that can influence the interpretation and generation of empirical data.
Contextual Variability. Outputs may depend on broader interaction context, including prior prompts or conversation history. This makes results sensitive not only to the immediate input but also to the sequence of interactions leading to it.
Output Plausibility. LLMs are optimized to produce fluent and contextually plausible outputs, which may appear correct even when they are factually inaccurate or unsupported by evidence. This tendency can make errors difficult to detect, particularly in tasks involving classification, summarization, or synthesis.
These properties expose a fundamental limitation of existing validity frameworks: they do not explicitly account for the methodological implications of LLM-assisted processes. While prior work provides a solid foundation for reasoning about threats arising from study design and data collection, it offers limited guidance on how to account for variability, bias, and hidden decision points introduced by LLMs. As a result, important aspects of the empirical process may remain implicit, hindering transparency, reproducibility, and critical assessment of results. Addressing this gap requires a systematic examination of how LLMs are integrated into empirical workflows and how their properties shape research outcomes. In the following section, we describe the methodological approach adopted in this study to investigate these issues and to characterize threats to validity introduced by LLM-assisted research tasks.
Research Method
This section describes the methodological approach adopted in this study. Building on a previously conducted systematic literature review [11], we extend the original study by introducing additional data collection and analysis procedures to address the new research questions investigated in this paper. While the underlying review protocol remains unchanged, the present work focuses on extracting and analyzing information related to the threats to validity that arise in LLM-assisted empirical software engineering. The following subsections detail the reuse of the base SLR protocol, and the specific procedures employed for data collection and analysis.
Research Questions
To operationalize the overarching research question introduced in § 1, we investigate three complementary aspects of LLM-related threats to validity: the extent to which they are explicitly reported, the types of threats reported, and how these threats relate to established validity dimensions. These aspects are captured by the research questions presented in Table 2.
| Id | Research question |
| RQ1 | To what extent are threats to validity introduced by LLM-assisted research tasks explicitly reported in empirical software engineering workflows? |
| RQ2 | What threats to validity introduced by LLM-assisted research tasks are reported in empirical software engineering workflows? |
| RQ3 | How do reported threats to validity introduced by LLM-assisted research tasks relate to established validity dimensions in empirical software engineering? |
Table 2. Specific research questions addressed in the study.
The first aspect concerns the visibility of LLM-related threats in the literature. As LLMs are increasingly used to support or automate empirical research activities, their integration into research workflows introduces methodological risks that may not always be explicitly acknowledged. Motivated by this concern, RQ1 investigates the extent to which threats to validity introduced by LLM-assisted research tasks are explicitly reported, and whether this reporting varies according to the role assigned to the LLM within the empirical workflow. This focus on reporting is important because the absence of explicit discussion can make it difficult to assess how LLM use may have influenced the research process, limiting transparency, reproducibility, and the critical evaluation of LLM-assisted workflows.
While RQ1 examines whether LLM-related threats are made explicit, it does not explain what kinds of methodological risks are being reported when such threats are acknowledged. RQ2 addresses this next step by focusing on the nature of the reported threats. Characterizing these threats is necessary because LLM use can affect empirical workflows through different mechanisms, including classification errors, hallucinations, prompt sensitivity, workflow instability, evaluator bias, and dependence on specific models or configurations. This question therefore aims to identify and synthesize recurring forms of threats to validity associated with LLM-assisted research tasks, supporting the development of a taxonomy of reported methodological risks in LLM-assisted empirical workflows.
Finally, RQ3 examines how the identified threats intersect with established validity dimensions in empirical software engineering. This is important because traditional validity frameworks were developed primarily around human-driven procedures or deterministic tools, whereas LLMs introduce probabilistic, opaque, and prompt-sensitive mechanisms into empirical workflows. As a result, LLM-related threats may cut across conventional validity categories rather than aligning with a single dimension. RQ3 therefore investigates how reported threats associated with LLM-assisted research tasks are distributed across classical validity dimensions, clarifying how LLM use affects the assessment of empirical rigor.
Base SLR Protocol
This study builds on a previously conducted SLR on the use of LLMs in empirical software engineering [11]. While the original review characterized how LLMs are used in empirical software engineering, including their benefits and limitations, the present study extends that work by investigating their methodological implications, particularly the threats to validity introduced by LLM-assisted research.
The original SLR examined peer-reviewed studies published between 2020 and 2025. Because the use of LLMs to support empirical research is often described within methodological sections rather than titles or abstracts, the review adopted a curated venue-based search strategy instead of relying solely on database search strings. The search covered 12 leading software engineering conferences and journals (ICSE, FSE, EASE, MSR, ESEM, CAIN, ESE, IST, JSS, TSE, and TOSEM), followed by full-text keyword searches using terms such as Generative AI, LLM, ChatGPT, Copilot, Gemini, and LLaMA. Candidate studies were then screened using predefined inclusion and exclusion criteria to identify papers in which LLMs supported or were empirically investigated within empirical software engineering research workflows. This process resulted in a final corpus of 50 primary studies.
In this study, we reuse the same corpus of primary studies and conduct an additional round of data extraction and analysis guided by the new research questions introduced in this paper. The complete SLR protocol, including the search strategy, venue selection, study selection process, and inclusion/exclusion criteria, is described in detail in our previous work [11]. By retaining the same study corpus and methodological foundations, we ensure consistency with the original review while extending the analysis to investigate threats to validity associated with LLM-assisted empirical research. An overview of the primary studies included in the analysis is presented in § 4.1.
Data Extraction
We collected one additional data item from each study: threats to validity related to LLM use. For this item, we extracted verbatim excerpts from the papers to preserve the authors’ original wording and support traceability throughout the analysis. We considered only threats explicitly reported by the study authors as related to LLM use, rather than threats inferred by us from the study design or the presence of LLM-assisted tasks. As a result, our analysis reflects how LLM-related threats are currently reported and framed in the literature.
Importantly, the data extraction was conducted at the level of the LLM-assisted empirical research task, rather than at the paper level. In our previous SLR, we had already identified and coded the empirical research tasks supported by LLMs in each study. Building on that analysis, for the present study, we preserved the association between extracted threats and the corresponding LLM-assisted tasks as explicitly reported by the authors. Since studies typically described the specific activity in which a validity concern emerged, this enabled us to maintain the connection between reported threats and the particular use of LLMs within the empirical workflow.
This task-based perspective was particularly important because some studies used LLMs to support multiple empirical research tasks, while only acknowledging threats to validity for a subset of those uses. If the analysis had been conducted only at the paper level, such studies would be treated as having reported LLM-related validity threats overall, potentially obscuring the fact that other LLM-assisted tasks within the same study were not discussed from a validity perspective. Analyzing threats at the task level therefore allowed us to obtain a more precise view of how studies acknowledge the validity implications of LLM use across different forms of involvement in empirical research workflows.
Data Analysis
The data analysis procedures were organized according to the three specific research questions. Figure 1 illustrates the overall methodological workflow adopted in this study, highlighting both the stages inherited from our previous SLR and the analyses newly conducted in the present work. The figure also shows how each research question draws upon specific process stages, making explicit the provenance of the data and analytical procedures used to address RQ1, RQ2, and RQ3.
Figure 1. Overview of the study design and analytical procedure.
To answer RQ1 - To what extent are threats to validity introduced by LLM-assisted research tasks explicitly reported in empirical software engineering workflows?, we examined how frequently threats to validity introduced by LLM-assisted research tasks were explicitly reported across the analyzed corpus.
For each LLM-assisted task, we recorded whether the corresponding paper explicitly reported at least one threat to validity associated with LLM use. We then organized these results according to the taxonomy of LLM involvement established in our SLR: augmentation, automation, decision support, and evaluation [11]. In this taxonomy, augmentation refers to scenarios in which LLMs support or accelerate human activities while the researcher remains primarily responsible for the task; automation refers to workflows in which the LLM independently performs substantial parts of the empirical task with limited human intervention; decision support captures cases in which the LLM provides recommendations, suggestions, or analytical insights that inform human decision making; and evaluation refers to situations in which the LLM acts as an assessor or judge, for example by evaluating quality, correctness, similarity, or relevance. Organizing the analysis according to these categories allowed us to investigate whether the explicit reporting of LLM-related validity threats varied depending on the role assigned to the LLM within the empirical workflow.
To answer RQ2 - What threats to validity introduced by LLM-assisted research tasks are reported in empirical software engineering workflows?, we analyzed the reported threats through thematic analysis, following established guidelines for thematic synthesis in software engineering research [6]. The analysis was conducted only for the subset of tasks in which at least one LLM-related validity threat was explicitly reported. We adopted an inductive coding process in which the extracted verbatim excerpts were first examined and assigned descriptive codes representing the specific validity concerns identified by the study authors. Conceptually related codes were then grouped into lower-level themes representing recurring forms of methodological risk introduced by LLM use. Finally, these lower-level themes were further abstracted into higher-level themes to characterize broader patterns in how threats to validity introduced by LLM-assisted research tasks are manifested and discussed in the literature.
The coding and synthesis process was conducted iteratively by one researcher and reviewed by a second researcher. We coded 33 extracted threat excerpts in total, treating each excerpt as a distinct threat occurrence. Because a single LLM-assisted task could be associated with more than one reported threat, multiple coded excerpts could be linked to the same task. The first researcher generated the initial codes, interpretations, and candidate themes, which were then reviewed and discussed with the second researcher across two refinement rounds. During these rounds, the researchers revisited the coded excerpts to improve conceptual consistency, reduce overlaps between categories, and ensure that each theme remained aligned with the methodological implications reported in the primary studies. Disagreements or ambiguous cases were resolved through discussion until consensus was reached. The resulting taxonomy therefore reflects iterative abstraction, peer review of coding decisions, and consensus-based refinement.
To answer RQ3 - How do reported threats to validity introduced by LLM-assisted research tasks relate to established validity dimensions in empirical software engineering?, we mapped each identified threat to its corresponding validity dimension. This mapping was conducted using the validity taxonomy appropriate for the type of empirical study, as presented in Table 1. Since the study type of each paper had already been extracted during our previous SLR, we used this information to determine the corresponding validity dimension before performing the classification. Each identified threat was then examined in terms of its underlying methodological implication and assigned to the validity dimension whose definition most closely reflected the nature of the concern being reported. For example, threats related to unreliable outputs were mapped to dimensions associated with construct validity, whereas concerns involving limited generalizability or contextual dependency were mapped to external validity dimensions. This procedure enabled us to systematically analyze how LLM-related validity threats manifest across different forms of empirical software engineering research and how they align with study-specific validity concerns.
The mapping process was conducted collaboratively, by one researcher and one reviewer, based on the definitions associated with each validity framework. Because some threats to validity introduced by LLM-assisted research tasks affected multiple aspects of empirical rigor simultaneously, a single threat could be associated with more than one validity dimension when conceptually justified by the reported methodological implications. The mapping decisions were iteratively reviewed and refined to improve consistency across the analyzed studies.
Analysis of the Results
This section presents the results of the study, organized according to the specific research questions introduced in § 3. Together, these analyses provide complementary perspectives on how the hidden methodological layer manifests in LLM-assisted empirical software engineering. We first present an overview of selected primary studies. Next, we examine the extent to which LLM-related validity threats are explicitly acknowledged in the literature (RQ1), providing a view of the visibility of these concerns across different forms of LLM involvement in empirical workflows. We then characterize the reported threats through the development of a taxonomy of recurring methodological risks (RQ2). Finally, we analyze how these threats relate to established validity dimensions in empirical software engineering (RQ3), providing a structured view of their implications for empirical rigor and research credibility.
To support transparency and traceability, the replication package for this study [10] provides the data used in these analyses, including the extracted threat excerpts, coding decisions, threat categories, and validity-dimension mappings.
Overview of the Primary Studies
The analyzed corpus consisted of 50 primary studies published between 2020 and 2025, comprising 69 LLM-assisted empirical software engineering research tasks. When analyzed by task, the corpus covered multiple empirical research methodologies, including mining software repositories (34 tasks), controlled experiments and case studies (21 tasks), secondary studies such as systematic literature reviews and systematic mapping studies (11 tasks), and survey-based research (3 tasks). Across the corpus, LLMs were predominantly applied to structured and automation-oriented activities, particularly classification (17 tasks), screening (14 tasks), evaluation (11 tasks), synthetic data generation (8 tasks), content transformation (5 tasks), pattern analysis (5 tasks), and information extraction (5 tasks).
The analyzed studies also revealed a strong concentration around proprietary conversational LLMs. Overall, the 50 papers reported 62 individual model usages, since several studies employed more than one model. ChatGPT alone accounted for 38 usages, followed by Llama (8), Claude (4), and DeepSeek (4). Only two studies did not explicitly report the adopted LLM model. In terms of publication trends, the analyzed corpus further indicates that LLM-assisted empirical software engineering remains an emerging methodological area, with the large majority of selected studies appearing after 2023, following the widespread adoption of ChatGPT and related conversational LLMs.
The diversity of empirical methods, research tasks, and forms of LLM involvement represented in the analyzed corpus provided a broad empirical basis for investigating the reporting, characterization, and validity implications of threats introduced by LLM-assisted research tasks across empirical software engineering workflows.
RQ1: Reporting of LLM-related threats to validity
Among the 69 LLM-assisted research tasks analyzed, 30 tasks (43.5%) explicitly reported at least one LLM-related threat to validity. In contrast, the remaining 39 tasks (56.5%) did not explicitly acknowledge or discuss potential validity concerns introduced by LLM use.
To further examine this reporting pattern, we analyzed whether the explicit acknowledgment of LLM-related threats varied according to the role assigned to the model in the empirical workflow. This distinction is important because the methodological implications of LLM use may depend on how the model is integrated into the research process. For example, risks associated with automated classification or screening may differ from those arising in assistive settings where researchers retain stronger oversight over interpretation and decision-making. Therefore, we organized the analyzed tasks according to the taxonomy of LLM involvement established in our previous work: augmentation, automation, decision support, and evaluation [11]. Figure 2 presents, for each involvement type, the number of tasks that explicitly reported at least one LLM-related threat to validity.

Figure 2. Distribution of LLM-assisted tasks with and without explicitly reported LLM-related threats to validity, grouped by nature of LLM involvement.
The results indicate that explicit reporting of LLM-related validity threats differs according to how the LLM is positioned within the empirical workflow. This difference is particularly evident in tasks categorized as automation, where the LLM performs substantial parts of the empirical task with limited human intervention. Among these tasks, 22 of 42 (52.4%) explicitly reported at least one LLM-related threat to validity, making automation the category with the highest threat-reporting rate. This suggests that studies involving more autonomous forms of LLM use are more likely to acknowledge explicit methodological concerns.
At the same time, however, the automation category also contained 20 tasks in which no LLM-related threat to validity was explicitly discussed. This result is particularly noteworthy because automation-oriented workflows are those in which LLM usage may introduce stronger methodological risks. When an LLM performs substantial parts of an empirical task with limited human intervention, the absence of explicit threat reporting becomes especially concerning, as it may obscure whether researchers assessed the reliability of generated outputs, controlled for hallucinations or misclassifications, evaluated prompt sensitivity, or validated the automated procedure itself. Therefore, the automation category stands out not only as the form of involvement most frequently associated with reported threats, but also as a particularly sensitive context in which missing threat reporting and management may have greater methodological consequences.
In contrast, augmentation tasks showed the lowest reporting rate, with only 3 of 13 tasks (23.1%) acknowledging LLM-related threats. This may indicate that, when LLMs are used mainly to support or accelerate human work, authors are less likely to frame their use as introducing distinct validity concerns. In augmentation settings, the presence of human oversight may lead authors to perceive the LLM as less methodologically consequential. Nevertheless, this interpretation requires caution. Human involvement does not eliminate LLM-related risks; it may only change how they appear. For example, an LLM-generated suggestion, summary, or intermediate output may still influence how researchers interpret data, prioritize evidence, or make analytical decisions, even when the final judgment remains human.
Tasks categorized as evaluation showed an intermediate pattern: 5 of 13 tasks (38.5%) had at least one LLM-related threat reported in the corresponding paper, while 8 did not. This role deserves particular attention because LLMs used for evaluation are not merely producing intermediate artifacts; they are participating in judgment. When LLMs are used as evaluators, judges, or assessors, they can influence how quality, correctness, relevance, similarity, or performance are measured. Therefore, even though the reporting rate for evaluation was lower than automation, the methodological stakes remain high. In these cases, threats such as evaluator bias, inconsistency, misalignment with human judgment, and inaccurate assessment may directly affect the study’s findings.
The only task categorized as decision support did not report any LLM-related threat to validity. Consistent with the interpretation above, this should be understood as a reporting outcome rather than as evidence that decision-support uses are methodologically unproblematic. Conceptually, decision support may still introduce validity concerns because LLM-generated recommendations, rankings, explanations, or interpretations can shape human decisions even when the final choice remains with the researcher. Because this category contains only one task, however, the result does not support substantive comparison with the other forms of LLM involvement.
The high number of tasks without explicit threat reporting reveals an important transparency gap in LLM-assisted empirical software engineering research. When studies incorporate LLMs into research tasks but do not discuss their validity implications, readers are left with limited insight into how the use of these models may have shaped the research process. This absence of reporting does not indicate that LLM-related threats were absent; rather, it means that the methodological account does not provide enough information to assess whether such risks were considered, monitored, or controlled. As a result, readers cannot easily determine how LLM outputs were checked, how errors or biases may have affected the analysis, or whether safeguards were applied to address risks such as hallucinations, misclassification, prompt sensitivity, unstable outputs, or evaluator bias. This underreporting weakens the methodological traceability of LLM-assisted workflows and limits the extent to which their findings can be assessed, reproduced, or reused.
RQ2: LLM-related threats to validity
The previous analysis showed that many LLM-assisted tasks did not explicitly report LLM-related threats to validity. For the tasks in which threats were reported, however, the authors often reported more than one validity concern associated with LLM use, meaning that the number of identified threats exceeds the number of tasks in which threats were reported.
To characterize these concerns, we synthesized a taxonomy of LLM-related threats to validity. In this taxonomy, lower-level themes represent specific validity threats, while the higher-level themes group related threats according to broader forms of methodological risk introduced by LLM use. Figure 3 provides a visual overview of the taxonomy and the distribution of coded threats across its different levels, while Table 3 defines each threat and clarifies its methodological meaning.

Figure 3. Taxonomy of reported LLM-related threats to validity, where numbers indicate coded threat occurrences extracted from verbatim threat excerpts.
| Higher-level theme | Description | Lower-level theme | Description | Paper IDs |
| LLM-generated artifact unreliability | LLM-generated or LLM-rewritten artifacts may be inaccurate, unrealistic, incomplete, or may distort the intended meaning of the original material. | A46, A671, A926, A1004 | ||
| Threats to empirical evidence construction | Threats in which LLM use may compromise the construction, transformation, classification, screening, or extraction of empirical evidence used in the study. | LLM classification and labeling errors | LLMs may assign incorrect labels, categories, or classifications, including errors caused by hallucination, subjective interpretation, or limitations of automated classification. | A46, A64, A96, A154, A180, A447, A806, A936, A966 |
| Threats to empirical evidence construction | Threats in which LLM use may compromise the construction, transformation, classification, screening, or extraction of empirical evidence used in the study. | LLM screening and filtering errors | LLMs may incorrectly include, exclude, miss, or filter relevant items during screening or selection procedures. | A125, A180, A806, A1455 |
| Threats to empirical evidence construction | Threats in which LLM use may compromise the construction, transformation, classification, screening, or extraction of empirical evidence used in the study. | LLM extraction errors | LLMs may extract incorrect, incomplete, or imprecise information from empirical sources, affecting the quality of the evidence base. | A1315 |
| Threats from LLM-based evaluation and measurement | Threats that arise when LLMs are used to evaluate, judge, score, or measure empirical artifacts, potentially affecting the validity of the resulting assessment. | LLM evaluation bias | LLM-based judgments may introduce evaluator bias, misalignment with human judgment, or inaccuracies when the model is used as an assessor. | A46, A767, A779 |
| Threats from prompt and workflow contingency | Threats caused by the dependence of LLM outputs on prompt design, iterative workflows, automation loops, or other contingent execution conditions. | LLM workflow instability | LLM-assisted workflows may produce unstable or unreliable outcomes when outputs depend on iterative refinement, feedback loops, or automated processing steps. | A64, A806 |
| Threats from prompt and workflow contingency | Threats caused by the dependence of LLM outputs on prompt design, iterative workflows, automation loops, or other contingent execution conditions. | Prompt-dependent interpretation | LLM outputs may vary according to prompt wording, prompt templates, or prompt design choices, affecting the consistency and interpretation of results. | A776, A925, A926 |
| Threats from model contingency and domain fit | Threats related to the dependence of results on the selected model, model version, or the degree of alignment between the model and the empirical task domain. | Model dependence and domain mismatch | Findings may be affected by model version instability or by a mismatch between the selected LLM and the domain, context, or task being analyzed. | A925 |
| Threats from downstream propagation of LLM outputs | Threats in which errors or distortions introduced by LLMs in earlier stages propagate into later stages of analysis, synthesis, or reporting. | Downstream distortion of empirical results | LLM-generated summaries, classifications, or extractions may distort empirical findings and propagate these distortions into subsequent analyses or conclusions. | A1106 |
Table 3. Taxonomy of LLM-related threats to validity.
The identified threats were distributed across five higher-level themes. Most threats were concentrated in threats to empirical evidence construction, which accounted for 19 coded occurrences. This concentration indicates that LLM-related validity threats were more likely to be reported when the model was used to produce, transform, classify, filter, or extract the empirical evidence used in later analyses.
Within this higher-level theme, the most frequent lower-level theme was LLM classification and labeling errors, with 10 occurrences. Classification and labeling activities were a major source of concern, likely because they often serve as intermediate analytical steps that influence subsequent synthesis, comparison, and interpretation. As a result, errors introduced during these stages may propagate through the empirical workflow and affect the validity of later findings and conclusions. This concern is visible in reports where authors explicitly connected LLM-based classification to possible misclassification, unpredictability, and reduced robustness:
“A key threat in our study is the reliance on LLMs for issue classification. Although we performed validation on a labeled subset of data, LLMs can still make misclassifications or provide unpredictable results, which could affect the robustness of our conclusions.” [A180]
The remaining lower-level themes within threats to empirical evidence construction show that this concern extends beyond incorrect labels. LLM screening and filtering errors and LLM-generated artifact unreliability appeared 4 times each, while LLM extraction errors appeared once. Together, these themes indicate that LLMs may compromise empirical evidence in different but related ways: by excluding relevant material, retaining irrelevant material, producing artifacts that inaccurately represent the original data, or extracting incomplete or imprecise information from empirical sources. In this sense, the central risk is not simply that an LLM may produce an incorrect answer, but that its output may reshape what counts as evidence in the study.
A different type of concern appears when the validity risk is tied less to the empirical artifact itself and more to the conditions under which the LLM output is produced. This is captured by threats from prompt and workflow contingency, which accounted for 5 coded threat occurrences. This theme reflects the dependence of LLM-assisted results on prompt design, execution conditions, iterative workflows, and automation loops. Within this theme, prompt-dependent interpretation appeared in 3 coded occurrences, showing that prompts were sometimes recognized as part of the research instrument rather than as minor implementation details. In these cases, prompt design could shape the results produced by the model, as acknowledged in the following excerpt:
“We notice that changing the way we phrase the GPT-4 prompt gives us different results. This shows that it is not always easy to figure out the best way to phrase the prompts.”[A926]
This concern becomes more pronounced when LLMs are embedded in repeated or chained workflows. The lower-level theme LLM workflow instability, with 2 occurrences, captures risks that emerge when outputs depend on feedback loops, refinement cycles, or automated processing steps. In these cases, validity threats may arise not only from a single inaccurate response, but also from the behavior of the workflow as a whole, including error accumulation, unstable outputs across iterations, or failure of refinement loops to converge. This suggests that some LLM-related threats are workflow-level threats rather than merely output-level threats.
The taxonomy also distinguishes cases in which LLMs were used not to construct evidence, but to evaluate or measure empirical artifacts. These cases are captured by threats from LLM-based evaluation and measurement, which accounted for 4 coded threat occurrences, all grouped under LLM evaluation bias. This theme is methodologically important because the LLM acts as an assessor rather than merely as a support tool. When models are used to evaluate quality, judge relevance, classify correctness, or compare artifacts, their biases, inconsistencies, or misalignments with human judgment may directly influence the study’s reported results. Among the studies coded under this theme, the following excerpt illustrates how authors framed LLM-based evaluation as a possible source of bias or inaccuracy:
“Our evaluation of assertion message quality (LLM-Eval), structural classification, and test comments relied on GPT-4, potentially introducing evaluator bias or inaccuracies from automated classification.”[A46]
Other threats become visible only when considering how LLM outputs move through the broader empirical workflow. Threats from downstream propagation of LLM outputs, with 3 coded occurrences, captures cases in which LLM-generated or LLM-mediated outputs are reused as inputs to later stages of analysis, synthesis, or reporting. The corresponding lower-level theme, downstream distortion of empirical results, highlights that errors introduced by LLMs may not remain local to the task in which they first appear. Instead, they may propagate into derived metrics, synthesized findings, or final conclusions. Therefore, the validity impact of LLM use depends not only on whether an individual output is correct, but also on how that output is subsequently reused. This propagation risk becomes explicit when authors connect LLM-generated classifications or summaries to later quantitative or interpretive results, as illustrated by the following excerpt: “The use of ChatGPT in the initial classification and summarization of the scraped data introduces a potential threat to validity, particularly in the quantitative correlations presented in relation to RQ3.”[A1106]
Finally, the taxonomy includes a more structural source of validity concern: threats from model contingency and domain fit, which accounted for 2 coded threat occurrences. These threats, grouped under model dependence and domain mismatch, concern the dependence of results on the selected model, model version, training data, or degree of alignment between the model and the domain. This theme points to a different kind of fragility: even when a workflow is carefully designed, findings obtained with one LLM may not necessarily generalize to another model, remain stable after model updates, or transfer to tasks for which the model has limited domain fit. Model contingency was reported as a reproducibility concern when authors noted that results obtained with a specific proprietary model may change as the model is updated:
“A key limitation of our classification approach involves the challenges of reproducibility with OpenAI’s proprietary GPT models, and large language models (LLMs) in general (Biderman et al. 2024; Spirling 2023). Despite the use of the most up-to-date LLM available at the time we conducted this study, updates to GPT models by OpenAI could alter our experimental outcomes, which could complicate consistent comparisons across studies (Biderman et al. 2024).”[A925]
Importantly, model contingency relevance depends on the role played by the LLM and the extent to which the resulting evidence depends on model-specific behavior. For example, when an LLM is used only to produce labels that are independently validated against a reliable reference, variation across models or versions may have limited impact on the study findings. In contrast, model contingency becomes more consequential when LLM outputs are used without comprehensive validation, when the model participates in the operationalization or interpretation of constructs, or when its outputs are propagated into subsequent analyses. Thus, the relevant concern is not simply that different models may produce different outputs, but whether such model-dependent variation can materially alter the empirical evidence or conclusions of the study.
Viewed as a whole, the taxonomy shows that reported LLM-related threats to validity are concentrated mainly around the construction of empirical evidence, especially when LLMs classify, screen, extract, or generate the material on which later analysis depends. However, the reported threats are not limited to inaccurate outputs. They also involve prompt sensitivity, workflow instability, evaluator bias, downstream propagation of errors, and dependence on specific models or domain fit. This suggests that LLMs introduce a methodological layer that is often only partially visible in empirical reports: model choices, prompts, interaction protocols, and reuse of LLM outputs may shape the research process even when they are not treated as methodological decisions. The central concern, therefore, is not only whether an LLM output is correct, but whether the study reports how LLM-assisted steps entered the empirical workflow, how their risks were assessed, and how their effects on the findings were controlled.
RQ3: Mapping LLM-related threats to established validity dimensions
To answer RQ3, we analyzed how the threats to validity introduced by LLM-assisted research tasks identified in RQ2 manifested across the established validity dimensions associated with each study type, following the frameworks presented in Table 1. Figure 4 presents the distribution of reported threats across validity dimensions. Table 4 further complements this analysis by providing the rationale for each mapped relationship, explaining why each LLM-related threat affects specific classical validity dimensions across primary and secondary studies.
Figure 4. Mapping of reported LLM-related threat occurrences to classical validity dimensions.
| Threat Category | Mapped Validity Dimension | Mapping Rationale |
| 1.1 LLM classification and labeling errors | Construct validity | Classification or labeling errors can assign empirical observations to categories that do not represent the intended construct, weakening the alignment between the phenomenon under study and its operationalized representation. |
| 1.2 LLM screening and filtering errors | Internal validity | LLM-driven screening or filtering processes can introduce uncontrolled inclusion and exclusion decisions into the empirical procedure, altering the set of artifacts or data items analyzed. |
| 1.2 LLM screening and filtering errors | Selection bias | LLM-driven screening can systematically exclude relevant studies or retain irrelevant ones, biasing the evidence base used for review. |
| 1.3 LLM-generated artifact unreliability | Construct validity | Generated artifacts may fail to instantiate the construct they are intended to represent, producing empirical material that only partially or inaccurately reflects the target phenomenon. |
| 1.3 LLM-generated artifact unreliability | Internal validity | Variation in the quality or reliability of generated artifacts can become an uncontrolled procedural factor that influences the results independently of the variables under study. |
| 1.4 LLM extraction errors | Construct validity | Extraction errors can misrepresent the empirical attributes being measured or analyzed, weakening the correspondence between source material and analytical representation. |
| 2.1 Prompt-dependent interpretation | Construct validity | Prompt wording can shift how the LLM interprets constructs, leading to inconsistent operationalization across prompts, runs, or studies. |
| 2.1 Prompt-dependent interpretation | Data Extraction Bias | When prompts guide extraction from source materials, prompt wording can influence which information is selected, omitted, or emphasized, producing incomplete or distorted extracted data. |
| 2.2 LLM workflow instability | Internal validity | Iterative or automated workflows may introduce inconsistent intermediate decisions that are difficult to monitor or control. |
| 3.1 LLM evaluation bias | Internal validity | When LLMs act as evaluators, their judgments can introduce systematic bias or inconsistency into the analytical procedure, affecting outcomes independently of the studied phenomenon. |
| 3.1 LLM evaluation bias | Conclusion validity | Biased LLM evaluations can distort comparisons, rankings, or assessments that support the study’s claims, leading to conclusions that are not fully supported by the underlying evidence. |
| 4.1 Downstream distortion of empirical results | Conclusion validity | Errors in LLM-generated intermediate outputs can propagate into derived metrics, interpretations, or claims, reducing the reliability of the conclusions drawn from the analysis. |
| 5.1 Model dependence and domain mismatch | Construct validity | When the model is poorly aligned with the empirical context or domain terminology, it may interpret key concepts through model-specific approximations rather than the intended research constructs. |
| 5.1 Model dependence and domain mismatch | External validity | Findings obtained with a specific model, version, configuration, or domain context may not transfer to other models, datasets, empirical settings, or future model releases. |
Table 4. Rationale for mapping LLM-related threat categories to the validity dimensions used for each study type.
The mapping in Figure 4 shows an uneven distribution of LLM-related threats across classical validity dimensions. Construct validity accounts for the largest share of mapped relationships, followed by internal and conclusion validity, whereas external validity and the secondary-study dimensions of search and synthesis bias are much less visible. This pattern should not be interpreted as evidence that LLM-assisted workflows mainly threaten construct validity in an absolute sense. Rather, it indicates that the threats explicitly reported in the analyzed studies are concentrated around more visible forms of representational error, such as misclassification, unreliable generated artifacts, and inaccurate extraction.
The strongest pattern concerns the role of LLMs in empirical evidence construction. LLM classification and labeling errors were mapped to construct validity in 10 occurrences and reported by nine primary studies. This indicates that, when LLMs are used to assign labels, categories, or classifications, they do not merely accelerate an existing analytical step; they participate in the operationalization of the constructs under study. In these cases, a classification error is not only a local mistake in a dataset. It may alter how the phenomenon is represented, especially when LLM-generated labels are later used as variables, categories, evidence units, or inputs to subsequent analyses. The concentration under construct validity therefore points to a broader methodological concern: LLMs can become implicit instruments of measurement, even when they are described as tools for automation or assistance.
The same concern helps explain why prompt-dependent interpretation, LLM extraction errors, and model dependence and domain mismatch were also mapped to construct validity. In traditional empirical workflows, construct validity is usually threatened by problems in instruments, coding schemes, measurement procedures, or operational definitions. In LLM-assisted workflows, these elements may be partially embedded in prompts, model behavior, and interaction protocols. As a result, the operational meaning of a construct may be shaped not only by researchers’ explicit design decisions, but also by how the model interprets instructions, resolves ambiguity, and maps textual input to output categories.
Internal and conclusion validity reveal a different aspect of the problem. While construct-validity mappings concern whether empirical objects are represented correctly, internal-validity mappings point to procedural instability inside the research workflow. Threats such as LLM screening and filtering errors, LLM-generated artifact unreliability, LLM workflow instability, and LLM evaluation bias show that LLMs may introduce uncontrolled variation into the empirical procedure. This variation may remain hidden behind apparently standardized workflows, since two studies can report the same type of LLM-assisted task while differing in prompt wording, model version, configuration settings, sampling behavior, interaction sequence, or iterative refinement strategy.
Conclusion-validity mappings show how these risks can propagate into final claims. LLM evaluation bias and downstream distortion of empirical results illustrate that LLM-mediated decisions do not necessarily remain confined to the stage in which they first appear. A biased judgment, inaccurate classification, unreliable generated artifact, or distorted intermediate summary may later influence rankings, comparisons, derived metrics, or reported interpretations. For this reason, LLM-related threats are not always well captured as isolated validity problems. They may operate as chains of methodological dependency, where a problem introduced during representation, extraction, generation, or evaluation later affects the credibility of the study’s conclusions.
External validity was rarely reported, and only model dependence and domain mismatch was mapped to this dimension. This low frequency is notable because many LLM-assisted studies rely on specific models, versions, prompts, and configurations whose behavior may not generalize across tools, domains, datasets, or future model releases. The limited reporting of external-validity threats may therefore reflect a blind spot in the current literature rather than a lower level of methodological risk. Existing studies appear to focus more on whether an LLM-assisted procedure works in a local setting than on whether the findings would remain stable under different models, model versions, domains, or execution conditions.
The secondary-study dimensions show a sharper reporting gap. Reported threats were mapped only to selection bias and data extraction bias, while no explicit threats were mapped to search bias or synthesis bias. This does not imply that LLM use cannot affect search or synthesis in secondary studies. Rather, within the analyzed corpus, explicit LLM-related threats were attached only to tasks connected to study screening, study filtering, and data extraction, even though the secondary-study tasks also included activities closer to study identification, topic identification, relevance assessment, and evidence retrieval. This suggests that reporting is more explicit around discrete review steps where LLM outputs can be inspected against inclusion criteria or source documents, whereas threats associated with how LLMs may shape the discovery, prioritization, or organization of evidence remain less documented.
A related reporting gap appears in survey studies. Although three survey studies in the corpus employed LLM-assisted tasks, none explicitly reported LLM-related threats to validity. This absence is relevant because the survey studies used LLMs for tasks such as persona-based generation of survey responses, thematic analysis of textual survey results, and rewriting research artifacts according to a predefined template. The first two uses are especially important because they can influence how participant data are generated, represented, interpreted, or transformed into evidence. The lack of reported threats therefore suggests that survey-based empirical software engineering may not yet have a well-developed vocabulary for identifying how LLM assistance can affect response validity, qualitative interpretation, and the transformation of survey-related materials into empirical findings.
Overall, the mapping shows that classical validity dimensions provide a useful structure for locating LLM-related threats, but it also exposes the limits of treating those dimensions as isolated categories.
Discussion
The results of this study provide complementary perspectives on how the hidden methodological layer manifests in LLM-assisted empirical software engineering. While the previous section examined the visibility, nature, and validity impact of reported threats to validity introduced by LLM-assisted research tasks, the findings collectively suggest a broader methodological shift introduced by the integration of LLMs into empirical workflows. In this section, we synthesize these results through the lens of the proposed conceptual model, discussing how LLM-assisted workflows impact traditional assumptions regarding methodological transparency, controllability, reproducibility, and validity in empirical software engineering research. We further discuss the cross-dimensional nature of threats to validity introduced by LLM-assisted research tasks and the implications of these findings for future empirical software engineering practices and methodological standards.
The Hidden Methodological Layer
The findings of this study suggest that LLM-assisted ESE cannot be fully understood by looking only at the visible sequence of empirical activities reported in a study. When LLMs are used to classify artifacts, screen studies, extract information, generate or transform empirical material, evaluate outputs, or synthesize findings, part of the empirical procedure is delegated to a model-mediated process. In these cases, the LLM is not merely an external productivity tool; it becomes part of the methodological apparatus through which evidence is constructed, interpreted, and reused.
We refer to this phenomenon as the hidden methodological layer. This layer captures the model- and prompt-mediated conditions through which LLMs influence empirical work, including prompt design, model configuration, interaction history, context construction, iterative refinement, automation decisions, validation procedures, and dependencies on specific model versions or capabilities. The layer is “hidden” not because these elements are necessarily inaccessible, but because their methodological influence is often indirect, only partially inspectable, or insufficiently reported as part of the research method. A prompt, for example, may encode assumptions about a construct; a model may interpret domain-specific terminology in ways not anticipated by researchers; and an iterative interaction may progressively reshape how empirical evidence is represented. Thus, methodological decisions are no longer located only in explicit protocols, instruments, or coding schemes, but are distributed across the interaction between researchers, models, and generated outputs.
Figure 5 illustrates this layered structure. Rather than representing LLM-assisted empirical research as a linear sequence of tasks, the model shows how visible empirical workflows are supported by deeper interactional and methodological conditions. The hidden methodological layer makes the model operational through prompts, configurations, outputs, iterations, and reuse of generated artifacts. Moreover, it also captures the less visible conditions that shape model behavior, such as implicit assumptions, opacity, stochasticity, context dependence, and model limitations. The outer layer represents the validity impact: LLM mediation may become visible through cross-dimensional, cascading, context-sensitive, underreported, or cumulative effects on validity.

Figure 5. Conceptual model of the hidden methodological layer in LLM-assisted empirical workflows.
This model helps clarify why LLM-related validity threats are difficult to localize. In conventional empirical workflows, threats to validity are often associated with identifiable methodological components, such as sampling decisions, measurement instruments, coding procedures, or analysis techniques. In LLM-assisted workflows, however, validity-relevant decisions may be embedded in prompts, model responses, interaction sequences, or downstream uses of generated outputs. As a result, a threat may not appear as a single visible methodological flaw. It may emerge from the interaction between the task assigned to the LLM, the model’s behavior, the researchers’ interpretation of its output, and the way that output is incorporated into later stages of the study.
This shift also changes how output errors should be interpreted. The methodological concern is not limited to whether a specific LLM response is correct or incorrect. An incorrect label, summary, extraction, or evaluation becomes more consequential when it enters the evidence pipeline and is reused as a basis for further analysis. In such cases, the LLM output is not simply an isolated artifact; it becomes part of the empirical infrastructure of the study. This helps explain why several threats identified in our taxonomy involve not only visible output errors, but also prompt-dependent interpretation, workflow instability, evaluator bias, model dependence, and downstream distortion of empirical results. The hidden methodological layer also helps interpret the underreporting observed in RQ1. If LLM-mediated decisions are treated as implementation details rather than methodological decisions, their validity implications may remain absent from the threats-to-validity discussion. This absence is problematic because it limits readers’ ability to assess how LLM outputs were produced, whether they were validated, how sensitive they were to prompts or model configurations, and how they affected subsequent analyses. In this sense, underreporting is not merely a documentation problem; it obscures part of the empirical process through which evidence is generated and transformed.
At the same time, the hidden methodological layer should not be interpreted as evidence that LLM-assisted workflows are inherently invalid or unreliable. Rather, it indicates that traditional assumptions about transparency, determinism, and controllability become less stable when LLMs participate in empirical tasks. Established validity concepts remain applicable, but the mechanisms through which validity threats arise may become more dynamic, interaction-dependent, and difficult to inspect. This is why LLM-assisted workflows require reporting and validation practices that make model-mediated methodological conditions visible enough to be assessed.
The central implication is that LLM use should be reported as part of the research method, not only as a tool choice. Making the hidden methodological layer explicit requires documenting prompts, model configurations, interaction protocols and validation procedures. More importantly, it requires treating these elements as validity-relevant methodological decisions. Doing so can help researchers identify hidden assumptions, evaluate robustness, trace methodological dependencies, and develop reporting standards appropriate for LLM-assisted empirical software engineering.
Validity Implications of the Hidden Methodological Layer
The hidden methodological layer reconfigures how validity threats emerge, become visible, and propagate in LLM-assisted empirical studies. Rather than replacing established validity concerns, LLM use changes the mechanisms through which those concerns materialize. A construct-validity concern, for example, may arise not only from a poorly defined coding scheme, but also from the way a prompt leads the model to interpret a category. An internal-validity concern may arise not only from uncontrolled study conditions, but also from variation introduced by model configuration, stochastic generation, or iterative prompting. A conclusion-validity concern may arise when LLM-generated classifications, summaries, or evaluations are reused as inputs to later analyses and eventually shape the claims drawn from the study.
This shift means that validity assessment cannot focus only on observable workflow steps or final outputs. Researchers and reviewers also need to examine the LLM interaction conditions through which outputs are produced: how prompts were designed, how models were selected and configured, how outputs were validated, how many iterations were performed, and how generated artifacts were reused downstream. In this sense, prompts can function as methodological instruments, model settings as methodological variables, and interaction patterns as part of the empirical procedure.
A central implication of this view is that LLM-related threats may be cross-dimensional. A single threat can affect different validity dimensions depending on where it occurs in the workflow and how its outputs are used. For example, prompt-dependent interpretation may affect construct validity when it changes how a concept is operationalized, but it may also affect internal validity when different prompts produce inconsistent analytical behavior. Similarly, an unreliable classification or extraction may begin as a representational problem but later affect conclusion validity if it is used to produce metrics, syntheses, comparisons, or interpretations.
LLM-related threats may also be cascading. Errors or distortions introduced early in the workflow can accumulate as LLM-generated artifacts are reused, refined, summarized, or incorporated into later stages of analysis. This is particularly important because small intermediate errors may appear harmless when inspected locally, but become more consequential when they shape subsequent analytical decisions. The validity impact of LLM-assisted workflows therefore depends not only on the quality of individual outputs, but also on the role those outputs play within the broader evidence pipeline.
These characteristics help explain the reporting gaps observed in this study. Threats that appear as visible output-level errors, such as misclassification, inaccurate extraction, or unreliable generation, are easier to identify and report. Threats embedded in interaction histories, prompt assumptions, model dependence, configuration choices, or downstream reuse are less visible and may therefore remain underreported. This suggests that reporting practices need to move beyond describing that an LLM was used and toward documenting how LLM-mediated decisions entered the empirical workflow.
Implications for Empirical Software Engineering Research
The findings of this study have implications for how LLM-assisted empirical software engineering should be designed, reported, evaluated, and reviewed. As LLMs become increasingly integrated into empirical workflows, methodological rigor cannot rely solely on assumptions inherited from human-driven or deterministic procedures. Instead, researchers need to explicitly account for the hidden methodological mechanisms introduced through LLM interactions and their potential impact on empirical validity.
A first implication concerns the reporting of LLM-assisted empirical workflows. The results of RQ1 showed that many studies employing LLMs did not explicitly report threats to validity associated with their use, even in workflows involving substantial automation or evaluation activities. This suggests that existing reporting practices may not yet adequately capture the methodological implications of LLM integration. To improve transparency and reproducibility, future studies need to document not only the empirical procedures and tasks performed with LLM assistance, but also the model- and interaction-specific conditions under which these tasks were carried out. Such information may include prompt engineering strategies, model versions, parameter settings, interaction protocols, validation procedures, human oversight mechanisms, and the extent to which LLM-generated outputs were reused in downstream analyses.
A second implication concerns reproducibility and methodological stability. Traditional empirical workflows often assume that analytical procedures can be reproduced from explicit methodological descriptions. However, the probabilistic and prompt-sensitive nature of LLMs challenges this assumption. Identical prompts may produce different outputs across executions, while small variations in prompt wording, model configuration, or interaction history may substantially alter empirical outcomes. As a result, reproducibility in LLM-assisted empirical software engineering may require additional methodological controls, such as prompt versioning, repeated executions, sensitivity analyses, configuration tracking, and explicit documentation of model dependencies.
The study also has implications for how empirical results involving LLMs should be evaluated and interpreted. Our taxonomy of threats to validity introduced by LLM-assisted research tasks revealed that methodological risks frequently emerge during the construction, transformation, classification, extraction, and evaluation of empirical evidence. Consequently, researchers and reviewers may need to critically examine not only the final outputs generated by LLM-assisted workflows, but also the processes through which these outputs were produced and validated. In particular, studies using LLMs for classification, synthesis, or evaluation activities may benefit from systematic human validation, cross-model comparisons, consistency checks, and robustness analyses designed to assess the stability and reliability of generated outputs.
Another implication concerns methodological literacy. Researchers using LLMs in empirical software engineering need to understand not only the empirical method they are applying, but also the operational properties of the models they incorporate into that method. This includes recognizing prompts as methodological instruments, model configurations as sources of variation, stochastic outputs as potential threats to stability, and generated artifacts as evidence that may require validation before reuse.
Finally, the findings suggest the need for methodological guidelines and evaluation standards tailored to LLM-assisted empirical research like the recent guidelines proposed by Baltes et al. [1]. Existing validity frameworks remain highly relevant, but they provide limited guidance on how to document and control risks introduced by probabilistic, opaque, and rapidly evolving AI systems. Future guidance should therefore address prompt reporting, model transparency, workflow traceability, output validation, robustness evaluation, and the management of cascading validity effects across empirical workflows.
The broader implication is that LLM integration in empirical software engineering is not only a tooling evolution. It is also a methodological challenge. Ensuring empirical rigor in LLM-assisted workflows depends on making hidden methodological mechanisms visible enough to be inspected, reported, validated, and controlled throughout the research process.
Threats to Validity
As this study builds on a previously conducted SLR [11], its validity is influenced by both the original review protocol and the additional data collection and analysis performed in this work. To mitigate potential threats, we relied on an established protocol for study selection and ensured consistency by reusing the same corpus of primary studies.
However, the present study introduces additional analysis procedures and interpretive abstractions that create new validity considerations beyond those discussed in the original SLR. In particular, this work synthesizes and interprets threats to validity introduced by LLM-assisted research tasks, analyzes their relationships with classical validity dimensions, and proposes the concept of a hidden methodological layer. These activities required additional qualitative interpretation and conceptual abstraction, which may introduce researcher bias in the categorization, naming, and mapping of threats.
To mitigate these risks, the analysis was conducted iteratively through collaborative discussions among the authors, including repeated refinement of threat definitions, mappings, and conceptual relationships. The proposed taxonomy and mappings were grounded in explicit evidence extracted from the primary studies and continuously revisited to preserve consistency between the empirical observations and the resulting abstractions. Nevertheless, alternative interpretations of some threats, relationships, or categorizations may still be possible, particularly because many primary studies report LLM-related limitations with varying levels of methodological detail.
Another threat concerns the interpretation of cross-dimensional relationships among validity threats. While this work argues that LLM-related threats may propagate across multiple validity dimensions, these relationships were derived through analytical synthesis rather than controlled empirical validation. Consequently, the proposed propagation and interaction effects should be interpreted as theoretically grounded conceptual relationships supported by the analyzed evidence, rather than empirically demonstrated causal mechanisms.
Finally, the study is limited by the current maturity of the literature on LLM-assisted empirical software engineering. Many analyzed studies provide incomplete reporting regarding prompts, configurations, validation procedures, or interaction protocols, which constrains the visibility of the hidden methodological layer itself. As a result, some LLM-related threats may remain underreported or insufficiently documented in the analyzed corpus.
Conclusion
The increasing adoption of LLMs in empirical software engineering introduces methodological challenges that extend beyond those traditionally associated with deterministic tools and human-driven workflows. In this paper, we argued that LLM-assisted empirical workflows introduce a hidden methodological layer in which important methodological influences become partially embedded within prompts, model configurations, interaction patterns, and probabilistic model behaviors.
To investigate this phenomenon, we analyzed 50 primary studies involving 69 LLM-assisted empirical research tasks. Our results showed that LLM-related validity threats remain substantially underreported in the current literature, despite the growing use of LLMs in empirical workflows. We also developed a taxonomy of recurring threats to validity introduced by LLM-assisted research tasks, including classification and labeling errors, prompt-dependent interpretation, workflow instability, evaluator bias, downstream propagation of outputs, and model dependence. Furthermore, our analysis revealed that these threats span multiple validity dimensions, with some threats propagating across different stages of the empirical workflow.
Building on these findings, we introduced the concept of the hidden methodological layer as a conceptual lens for understanding how LLMs influence empirical software engineering research. Rather than replacing traditional validity concerns, this perspective highlights how LLM-assisted workflows reshape the mechanisms through which validity threats emerge, propagate, and interact throughout empirical studies. As LLMs and AI take on not just more research tasks but a broader variety of them, the hidden methodological layers they create will expand in both size and complexity.
More broadly, our findings suggest that the increasing integration of LLMs into empirical software engineering may require the development of LLM-aware methodological practices capable of addressing probabilistic behavior, hidden interaction mechanisms, workflow-level propagation effects, and evolving model ecosystems. A direction for future work is to empirically investigate the extent to which the identified threats affect study outcomes. Replication studies of existing LLM-assisted empirical research could systematically address or mitigate these threats and compare the resulting findings with those originally reported. Such studies could help determine whether stronger methodological controls lead not only to more robust and reproducible results, but also to different or more insightful empirical findings. Understanding when and how these hidden methodological influences materially affect empirical findings represents an important next step toward more rigorous, transparent, and trustworthy LLM-assisted empirical software engineering.
References
References
- [1]Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Cafalato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treude, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Cristina Martinez Montes, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, and Stefan Wagner. 2026. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. arXiv:2508.15503 [cs.SE]. https://arxiv.org/abs/2508.15503
- [2]V.R. Basili, F. Shull, and F. Lanubile. 1999. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4 (1999), 456–473. doi:10.1109/32.799939DOI
- [3]Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. doi:10.1145/3442188.3445922DOI
- [4]Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosse lut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kudtipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilsorhan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhua Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs.LG] https://arxiv.org/abs/2108.07258
- [5]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages.arxiv.org/abs/2005.14165
- [6]Daniela S. Cruzes and Tore Dyba. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement (ESEM ’11). IEEE Computer Society, USA, 275–284. doi:10.1109/ESEM.2011.36DOI
- [7]Matheus De Morais Leça, Lucas Valença, Reydne Santos, and Ronnie De Souza Santos. 2025. Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering. In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) (Ottawa, ON, Canada). IEEE Press, 36–43. doi:10.1109/WSESE66602.2025.00013DOI
- [8]Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. 2008. Selecting Empirical Methods for Software Engineering Research. In Guide to Advanced Empirical Software Engineering, Forrest Shull, Janice Singer, and Dag I. K. Sjøberg (Eds.). Springer London, London, 285–311. doi:10.1007/978-1-84800-044-5_11DOI
- [9]Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. Proceedings of the National Academy of Sciences 120, 30 (2023). doi:10.1073/pnas.2305016120DOI
- [10]Victoria Gomes, Fabio Palomba, Robert Feldt, Rodrigo Spinola, and David Lo. 2026. The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering. https://github.com/secentervcu/LLM-Assisted-ESE-Threats-To-Validity-SLRarxiv.org/abs/2611.20033
- [11]Victoria Gomes, Delaney Selb, Fabio Palomba, Rodrigo Spinola, and David Lo. 2026. LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda. arXiv:2604.26192 [cs.SE] https://arxiv.org/abs/2604.26192
- [12]Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (New Orleans, Louisiana, USA) (AAAI’18/IAAI’18/EAAI’18). AAAI Press, Article 392, 8 pages.arxiv.org/abs/1709.06560
- [13]Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 220 (Dec. 2024), 79 pages. doi:10.1145/3695988DOI
- [14]Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting Experiments in Software Engineering. In Guide to Advanced Empirical Software Engineering, Forrest Shull, Janice Singer, and Dag I. K. Sjöberg (Eds.). Springer London, London, 201–228. doi:10.1007/978-1-84800-044-5_8DOI
- [15]Ziwei Ji, Yajun Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Jie Yin Bang, Andrea Zeng, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (March 2023), 38 pages. doi:10.1145/3571730
- [16]Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report EBSE-2007-01. Keele University and Durham University. https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf
- [17]B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (2002), 721–734. doi:10.1109/TSE.2002.1027796DOI
- [18]Johan Linåker, Sardar Muhammad Sulaman, Rafael Maiani de Mello, and Martin Höst. 2015. Guidelines for Conducting Surveys in Software Engineering. Department of Computer Science, Lund University.
- [19]Ninareh Mehrabi, Fred Morstatter, Napisat Sakaki, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. ACM Comput. Surv. 54, 6, Article 115 (July 2021), 35 pages. doi:10.1145/3457607DOI
- [20]OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Fidsman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heevoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanu, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayarvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774
- [21]Laria Reynolds and Martin K. McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 314, 7 pages. doi:10.1145/3411763.3451760DOI
- [22]Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Engg. 14, 2 (April 2009), 131–164. doi:10.1007/s10664-008-9102-8
- [23]June Sallou, Thomas Durrieux, and Annibale Panichella. 2024. Breaking the Silence: the Threats of Using LLMs in Software Engineering. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (Lisbon, Portugal) (ICSE-NIER’24). Association for Computing Machinery, New York, NY, USA, 102–106. doi:10.1145/3639476.3639764DOI
- [24]Ben Shneiderman. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504. doi:10.1080/10447318.2020.1741118
- [25]Forrest Shull, Janice Singer, and Dag I. K. Sjøberg. 2008. Guide to Advanced Empirical Software Engineering. Springer. doi:10.1007/978-1-84800-044-5DOI
- [26]Maria Couto Teixeira, Marisa Tschopp, and Anna Jobin. 2025. Generative Artificial Intelligence in Qualitative Research Methods: Between Hype and Risks? arXiv:2511.08461 [cs.CY] https://arxiv.org/abs/2511.08461
- [27]Stefan Wagner, Marvin Muñoz Barón, Davide Falessi, and Sebastian Baltes. 2025. Towards Evaluation Guidelines for Empirical Studies Involving LLMs. In 2025 IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) (Ottawa, ON, Canada). IEEE Press, 24–27. doi:10.1109/WSESE66602.2025.00011DOI
- [28]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 180, 14 pages.arxiv.org/abs/2201.11903
- [29]Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Ballé, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Jason Gabriel. 2021. Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs.CL] https://arxiv.org/abs/2112.04359
- [30]Claes Wohlin, Per Runeson, Martin Hst, Magnus C. Ohlsson, Björn Regnell, and Anders Wessl. 2024. Experimentation in Software Engineering. Springer Publishing Company, Incorporated. doi:10.1007/978-3-662-69306-3DOI
- [31]Li Zhang, Jia-Hao Tian, Jing Jiang, Yi-Jun Liu, Meng-Yuan Pu, and Tao Yue. 2018. Empirical Research in Software Engineering-A Literature Survey. Journal of Computer Science and Technology 33, 5 (2018), 876–899. doi:10.1007/s11390-018-1864-xDOI
- [32]Yanbo Zhang, Sumeer A. Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil. 2025. Exploring the role of large language models in the scientific method: from hypothesis to discovery. npj Artificial Intelligence 1, 1 (2025), 14. doi:10.1038/s44387-025-00019-5DOI
- [33]Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12697–12706. https://proceedings.mlpress/v139/zhao21c.htmlarxiv.org/abs/2102.09690

