WorksComputer Science
Hidden in Plain Sight: an Empirical Study about The Instruction Stacks of Coding Agents
Abstract
Coding agents send their models more than the user's prompt: they add instructions, tool definitions, and environmental context that form an instruction stack. This stack guides execution and changes across releases, yet remains largely unexposed to users and insufficiently characterized in research. We present a longitudinal, cross-agent study of the instruction stacks captured in the first model requests of 1,886 releases of Claude Code, Codex, Gemini CLI, Qwen Code, and OpenCode. Under controlled conditions, we analyze their composition and evolution and trace changes in the four open-source agents to documented motivations. Our findings show why examining the system prompt or reporting the model alone provides an incomplete account of agent inputs. Tool definitions dominate three agents' latest requests, while occasional concentrated revisions alter instructions governing tool use, attribution, and coding style. Such revisions can complicate how researchers identify agent-authored contributions and interpret generated code. We also found that only 48% of examined change units directly explain the motivation for stack revisions. Across agents, shared capabilities emerge despite increasingly distinct wording. These findings expose an evolving, incompletely documented layer of agent design. The findings motivate documenting captured instruction stacks alongside agent releases, models, and user prompts, and preserving the rationale behind their revisions.
Introduction
Coding agents such as Claude Code [2], Codex [3], and Gemini CLI [4] extend LLM-based software development from generating code suggestions to executing multistep tasks. They inspect repositories, edit files, and run commands on a user’s behalf. However, the user’s request is only one part of the input that guides this work. Before calling a Large Language Model (LLM), an agent supplements the request with instructions, tool definitions, and context. For example, SWE-agent supplies instructions, documentation, and demonstrations of the commands the model may execute [5]. Figure 1 illustrates this additional material in Claude Code alongside the user’s prompt; the request contains agent-authored instructions, definitions of available tools, and facts collected at runtime, such as the current date and working directory.

Figure 1. The first request that Claude Code 2.1.243 sent to its model in our capture, abridged. The user typed one sentence, which is 0.03% of the request. Most of the rest is the instruction stack that the agent adds: instruction text, in the system field and in messages, and tool definitions. Colored tags show how our annotation labels the instruction text (Table II). Underlined values come from the environment where the agent is running: the working directory, the kernel version of the host, and the date.
We call this agent-supplied material the instruction stack. It comprises instruction text—behavioral directives, contextual facts, and structural text supplied through top-level fields or messages—and tool definitions, including descriptions and argument schemas. The system prompt is therefore only one component of the stack. Our analysis examines the stack in the first request to the main model under a fixed experimental configuration, excluding the user prompt and API configuration parameters. We extracted the instruction stack from widely adopted agents in a fresh, sandboxed environment, allowing us to isolate the agent’s contribution to the stack. This stack guides how the model interacts with user-provided code, requests permission, and uses tools. Although parts may be available in source code, the assembled stack is generally not exposed in the user interface and can change between releases.
These changes in the instruction stack matter because instructions can influence model behavior and resource use. Rewording or reformatting prompts can shift accuracy [6], [7], while increasing instruction count or input length can reduce instruction-following reliability [8, 9, 10]. Even changing a tool’s description can alter its invocation frequency [11]. More broadly, a study of 35 Qwen Code releases using a fixed model found that token expenditure per task increased by approximately 70% [12]. Although that finding concerns the harness as a whole, it illustrates why model identity alone cannot characterize an agent’s behavior or cost.
The instruction stack also provides the context in which developer-written guidance operates. Files such as AGENTS.md and CLAUDE.md supply repository-specific instructions [13], and recent studies examine their content and evolution [14, 15, 16, 17]. Evidence of their effectiveness is mixed [18], [19], with agent-specific behavior proposed as one explanation [20]. These files may repeat or contradict vendor instructions, which can themselves conflict [21]. Understanding the surrounding stack is therefore necessary to characterize the conditions under which such guidance is evaluated. More generally, researchers who report only the model and user prompt leave an evolving input to agent behavior unspecified.
Prior work provides only a partial view of this input. Prompt-mining studies examine strings and templates embedded in application source code [22, 23, 24], whereas coding agents assemble requests at runtime from multiple instruction sources, tool definitions, and environmental information. Source inspection alone may therefore miss what reaches the model and is unavailable for closed-source agents such as Claude Code. Harness studies describe implementations at one or two points in time [13], [25]; the closest longitudinal study evaluates one agent’s releases for cost and quality [12]. To our knowledge, no prior study systematically captures and compares the assembled instruction stacks of multiple production coding agents across their release histories.
These observations motivate a longitudinal, cross-agent analysis. Following release histories reveals which directives persist, change, or disappear and, where development records provide evidence, why maintainers revise them. Comparing agents distinguishes shared practices from agent-specific choices about tool use, verification, autonomy, and user interaction. Together, these analyses characterize the evolution of an often unobserved instruction stack and help characterize the design space in which these agents operate.
We study Claude Code, Codex, Gemini CLI, Qwen Code, and OpenCode. We attempted to run all 1,911 stable npm releases published before 25 August 2026, each in a fresh sandbox with the same user prompt and a fixed configuration per agent. A local server intercepted the first request to the main model for 1,886 releases. We focus on this request because it exposes the initial instruction stack before responses from the main model and subsequent tool results cause execution paths to diverge. Later requests combine instructions with task-dependent histories, making differences harder to attribute to release changes. The first request thus provides a controlled starting point for tracing release-level changes, while excluding instructions loaded later or the full context of interactive sessions. We normalized four API protocols into a common representation, segmented the instruction text into sentences, and classified each segment using a functional codebook. For the four open-source agents, we also traced changes to development records to identify documented motivations. We answer four research questions:
RQ1. How are agents’ instruction stacks structured?
RQ2. How do instruction stacks evolve across releases, and what is added or removed?
RQ3. What documented motivations underlie changes to instruction stacks?
RQ4. What content is shared across agents, what is agent-specific, and how do these patterns evolve?
This paper makes four contributions:
A capture infrastructure that records initial model requests from isolated agent releases and normalizes four API protocols.
A longitudinal dataset of 1,886 captured requests from five production coding agents, with capture provenance.
A functional characterization of instruction stacks that distinguishes behavioral directives, contextual information, and structural text.
A longitudinal, cross-agent analysis of stack changes, shared practices, and agent-specific choices, including documented motivations for the four open-source agents.
The rest of the paper is structured as follows. Section 2 defines the terms we use and discusses related work. Section 4 describes how we built the dataset and how we answer each question, and Section 5 reports the results. Section 6 discusses implications and threats to validity, and Section 8 concludes.
Background
A coding agent is a program that pairs an LLM with tools for reading, editing, and running code [5, 26, 27, 28]. It works in a loop of turns. In each turn, it sends a request to a model provider, the company that serves the model’s API, and the model answers with text or with calls to tools. The agent then runs the tools, appends their results to the conversation, and starts the next turn. Recent work calls this program around the model the agent’s harness [12, 25]. Some agents also invoke helper models, for example, to route a task or to write a title for the session. We distinguish these from the main model, which handles the user’s task, and analyze the first request sent to that model.
Like every other request, this first request is a JSON object whose shape depends on the provider’s API protocol. The agents in this study use four protocols: Anthropic Messages [29], Gemini Generate Content [30], OpenAI Responses [31], and OpenAI Chat Completions [32]. The field names differ, yet the four protocols carry text in the same three kinds of places. First, three have a top-level instruction field, system, systemInstruction, or instructions; Chat Completions has none. The content of this field is commonly called the system prompt [33].
Second, all four protocols carry an ordered list of messages, in a field called messages, contents, or input. Each message has a role that says whom it speaks for. System and developer messages carry instructions from the application, user messages carry what the user typed, and assistant messages are the model’s own earlier replies [34], [35]; Gemini calls this last role model. Third, all four protocols carry a list of tool definitions. A tool definition gives a tool’s name, a description in natural language, and a JSON Schema for its arguments. Some protocols also offer built-in tools that a request declares by type only, such as web search.
Beyond tools, recent agents can extend themselves at run time. A skill is a packaged set of instructions that the agent loads when a task needs it [36], and subagents, separate model sessions to which work can be delegated. Agents announce both to the model as named lists with descriptions, and each agent chooses where to put these lists. For instance, in Figure 1, Claude Code lists its skills in a system message that follows the user prompt. Consequently, an agent’s text is spread across several fields and messages, and its layout differs from one agent to another.
We use the following terms to describe these structures consistently, illustrated in the request in Figure 1.
The request is the complete JSON body that the agent sends to the provider.
The user prompt is the text supplied by the user. Our captures use the fixed prompt I AM THE USER PROMPT.
A role says whom a piece of text speaks for. Besides the four message roles, system, developer, user, and assistant, we treat the top-level instruction field as its own role, separate from system messages. The request in Figure 1 uses three roles: the top-level instruction, user, and system. Tool definitions have no role, because their text sits inside structured definitions. We analyze them separately and do not segment them.
The instruction stack comprises instruction text and tool definitions. The instruction text is the text of all roles except the user prompt, including the text that the agent places in user messages. The tool definitions are the structured list of tools. Hence, the system prompt is only one part of the instruction text. The remaining fields of a request, such as the model name, generation settings, and JSON syntax, are not part of the stack. Despite its name, the instruction text does more than instruct: it also states facts and marks the structure of the conversation.
Related work
Researchers have studied prompts as software artifacts, examining their content, evolution, and maintenance. PromptSet extracted over 61,000 prompts from open-source Python programs [22], while Tafreshipour et al. found that only 21.9% of prompt-changing commits mention the change [23]. Other studies classify prompt components [24], [27], investigate prompt management and technical debt [38], [39], and propose engineering support for prompts as programs [40, 41, 42]. These efforts, however, treat prompts as discrete, user-authored artifacts and do not examine the layered, agent-assembled instruction stacks that govern autonomous coding agents at runtime.
As coding agents take on multi-step development tasks, attention has turned to the harnesses that construct model inputs and manage execution. Harness design—tool–result presentation, planning strategies, and context management—demonstrably affects agent performance independently of the underlying model [5], [26], [43, 44, 45]. Architectural studies characterize harness implementations in the wild [13], [46]. Barbaste et al. compare two snapshots of eleven harnesses and report a shift in behavioral policy from prompt prose to structured configuration [25]. Ben Sghaier et al. analyze 35 consecutive Qwen Code releases and find rising token expenditure without commensurate improvement in resolve rates [12]. These studies examine harness architecture and outcomes, whereas we investigate the composition and evolution of the instruction stacks that harnesses send to models.
While the harness implements request construction, tool orchestration, and execution logic, the system prompt provides model-facing behavioral instructions. Research examines these instructions alongside additional guidance supplied by developers. Prompt-extraction studies show that hidden system prompts can often be recovered [47, 48, 49, 50], including from coding agents [51]. Neumann et al. identify seven topics in 1,309 system prompts [33], while Mason analyzes conflicting instructions in snapshots from Claude Code, Codex, and Gemini CLI [21]. Studies of developer-written guidance examine CLAUDE.md and AGENTS.md files [14], [15], [52], Cursor rules [16], [53], skills [54], [55], and other configuration mechanisms [56]. They report limited attention to security and performance [14], infrequent instruction removal [17], and mixed evidence of effectiveness [18]–[20]. These studies illuminate system-prompt content and developer customization, but do not trace the surrounding agent-supplied stack across release histories.
Our contribution combines direct request capture with longitudinal analysis of 1,886 releases across five coding agents. Examining instruction text and tool definitions together reveals what reaches the model under our capture conditions, how it changes, and which practices agents share. For the four open-source agents, we also trace revisions to documented motivations. We capture the first request to the main model to compare initial stacks before model responses and subsequent tool results introduce divergent execution histories. This scope provides a common basis for release comparisons while excluding instructions loaded later. The resulting evidence exposes a source of variation that model identifiers and user prompts alone cannot describe, helping researchers account for release-specific directives when constructing datasets and interpreting agent-generated code.
Methodology
We first capture the request that each release of five agents sends to its model (§ 4.1). Then, each research question builds on the results of the one before it (Sections 4.2 to 4.5). Figure 2 gives an overview of the study.

Figure 2. Overview of the methodology. Each step builds on the one before it, and brackets mark the steps that answer each research question.
Dataset construction
Agent selection. We selected widely adopted coding agents that were actively maintained during the study, provided downloadable version histories through a package manager, and supported non-interactive command-line execution. Our sample comprises five agents/harnesses from different vendors: Claude Code (Anthropic) [2], Codex CLI (OpenAI) [3], Gemini CLI (Google) [4], Qwen Code (Alibaba) [57], and OpenCode (Anomaly) [58]. All except Claude Code are open source. These agents also appear in recent coding-agent studies [12], [56], [59, 60, 61].
Release census. We enumerated releases from each agent’s official npm package. Releases were eligible if published before 25 August 2026, versioned as MAJOR.MINOR.PATCH, and packaged with the agent’s executable. This yielded 1,911 eligible releases shown in Table I.
Controlled capture. We ran each release once in a fresh linux/amd64 container with fixed runtime versions, a new home directory, and an empty working directory. A runner installed the exact release, redirected its provider endpoint to a local HTTP server, and submitted the fixed prompt I AM THE USER PROMPT. This marker identified the user prompt within captured requests. We retained the first request to the main model containing the marker.
Three agents required additional handling because they may first send the prompt to a helper model and, as a result, helper-model requests could also contain the marker. For example, OpenCode and some Claude Code releases ask a model for a session title. Our implementation classified these requests as session-title requests, main-model requests, or unknown. It returned the fixed response title “Controlled capture session” and rejected unknown requests. The server stored the selected request’s complete JSON body and hashes of both the body and the bytes received over HTTP.
We retained default settings except for endpoint redirection and model selection, where the agent has none by default. Claude Code and Gemini CLI used their default models; Codex used its default o4-mini through release 0.19.0 and gpt-5 thereafter. OpenCode used gpt-5, and Qwen Code used qwen3-coder-plus. If a release fails to send its request, we report it in § 5.
Request mapping. To analyze every captured request with one procedure, we map each request to the common terms of § 2. The mapping renames the protocol fields listed there. Thus, system, systemInstruction, and instructions become the top-level instruction, messages, contents, and input become the ordered messages, and each protocol’s tool list becomes the tool definitions. Gemini’s model role also becomes assistant. We derived these correspondences from each protocol’s API reference [29, 30, 31, 32]. The mapping copies every value unchanged, so it never edits, merges, or reorders text. Moreover, it recognized every role, content type, and tool definition in 1,886 captured requests.
RQ1: How are agents’ instruction stacks structured?
To answer RQ1, we characterize instruction stacks along three dimensions: Placement, the distribution of instruction text across roles (§ 2); tools, the declared tool definitions; and function, the purpose of the instruction text. We first count the number of characters inside the requests while grouping by role (e.g., system messages, developer messages). Tools are also an explicit field in the requests these agents make, so we extract tool definitions from their structured fields. However, determining each component’s function in the prompt requires reading the text. Hence, we divide the text into segments and code each one following a codebook.
Segmentation. We use sentence-level segments because sentences typically express individual directives or facts. We first divide each role’s text into structural blocks, including headings, list items, and paragraphs, then split paragraphs and list items into sentences. Headings and line breaks remain attached to neighboring segments, ensuring that each character belongs to exactly one segment. This procedure yielded 349,404 segments.
Codebook. We built the twelve-code codebook from 25 releases, five per agent. Because agents add and drop parts of their stacks over time, we chose each agent’s first and last releases and the releases at the quartiles of its release order. This purposive sample favors diversity over representativeness [62] and is small enough to read in full. We grouped the text of these releases into codes by function, and two authors calibrated the codebook by discussing examples. The codes fall into three families: directive text specifies behavior, information text states facts, and scaffolding text marks conversational structure. Table II details the codebook.
To apply the codebook, we drew a random sample (with a fixed seed), stratified by agent and sized for a margin of error of 10 percentage points at 90% confidence [62], [63]. Prior studies that validated automated output against human judgment used the same parameters [64], [65]. The sample contains 66 requests comprising 12,862 segments. Two authors coded it independently and reached a Cohen’s [66] of 0.75 before adjudication, above our minimum threshold of 0.7 [67]. A third author then resolved their disagreements to produce the reference labels. Our supplementary materials hold the full codebook instructions with 25 worked examples [1].
| Agent | npm package | API protocol | Releases | Captures | Period (MM/YY) |
| Claude Code | @anthropic-ai/claude-code | Anthropic Messages | 494 | 493 | 02/25–08/26 |
| Codex | @openai/codex | OpenAI Responses | 175 | 174 | 04/25–08/26 |
| Gemini CLI | @google/gemini-cli | Gemini Generate Content | 192 | 192 | 06/25–08/26 |
| OpenCode | opencode-ai | OpenAI Responses | 930 | 907 | 06/25–08/26 |
| Qwen Code | @qwen-code/qwen-code | OpenAI Chat Completions | 120 | 120 | 07/25–08/26 |
| Total | 1,911 | 1,886 |
Table 1. The five agents and their release census. Period: month and year of the first and last eligible release.
| Code | The segment states or requires. . . |
| Scaffolding family: structure of the conversation | |
| user input | the fixed user prompt, not instruction text |
| scaffolding | bookkeeping, such as the agent’s version |
| Directive family: what the model must do, avoid, prefer, or may do | |
| context management | how to weigh content in the conversation |
| memory | how to use memory across sessions |
| execution policy | permissions and hard-to-reverse operations |
| tool guidance | when and how to invoke a tool or subagent |
| communication guidance | what the user reads, and in which form |
| engineering guidance | how to analyze, change, and verify code |
| role and scope | which tasks to do, and how autonomously |
| Information family: what the model is told | |
| capability catalog | named lists of skills, subagents, or tools |
| identity | the agent’s name, product, maker, or model |
| context facts | facts on the environment or the session |
| unresolved | a function that the text does not reveal |
Table 2. The RQ1 codebook, with twelve codes in three families and unresolved, in short. The supplementary materials hold the full definitions.
LLM annotation. We used GLM 5.3 Flash{1} to assign one code to each segment. Its input included the codebook, the target segment, and up to 15 preceding and following segments for context. A JSON schema constrained responses to one valid code. Before annotating the full dataset, we validated the LLM against the adjudicated reference produced through independent annotation by two human reviewers. The LLM matched the reference on 93.4% of segments (), with every code achieving . These results met the standard threshold used in practice [67], hence, we use the LLM for annotating the full dataset.
RQ2: How do instruction stacks evolve across releases, and what is added or removed?
To answer RQ2, we compare consecutive eligible releases in npm publication order. Each pair forms a transition, identified by its later release, for example, transition 2.1.69 goes from 2.1.68 to 2.1.69. A transition is measurable when both requests were captured, yielding 1,867 measurable transitions out of 1,906. We track changes to segments coded in RQ1 and tool definitions and compare each agent’s first and latest release.
Edit size. To measure how much a transition changes the request, we represent each request using its top-level instruction text, messages, and tool definitions, with structured content serialized as canonical JSON. For a transition between strings and , the edit size is their character-level Levenshtein distance [68]. The relative edit size, , normalizes this distance by the longer string’s length. Summing edit sizes across measurable transitions yields an agent’s cumulative character edits.
Segment changes. Following approaches for tracking source-code lines across versions [69], [70], we diff the segments of the two requests, matching segments with the same role and text, ignoring whitespace. A segment that is only present in the later request is added, and one only present in the earlier request is removed. Yet, this diff would report a moved or edited segment as one removal and one addition, so we pair them in two cases. First, we pair removals and additions with identical text as moves, including across roles. Second, a removed and an added segment at the same place, between the same two matched segments of a role, form one modified segment if their relative edit size is at most 50%, a threshold also used by git diff [71]; otherwise, it remains a removal and an addition. We match tools by name, classifying them as added or removed when present in only one request, and modified when their definitions differ.
Masking. An exact comparison of the textual contents of the requests would also count values that change while the instructions do not, namely run-dependent values, such as the current date shown in Figure 1, and release stamps, which state the agent’s version. Hence, before computing changes, we analyzed these repeating patterns from the requests and replaced five such values with placeholders: (1) the current date, (2) the temporary directory of the run, (3) the folder where the agent is installed, (4) the host’s kernel version, and (5) Claude Code’s release version, which is included in its requests. We then report edit size with and without this masking.
RQ3: What documented motivations underlie changes to instruction stacks?
The changes of RQ2 show what developers edit, but not the motivation behind the change. Since developers often explain an edit in a commit message, we trace each change to the commits that made it and codify the motivations that developers wrote in the commit message. Tracing needs the source history, so RQ3 covers the four open-source agents, with the exception of Claude Code, which is closed-source.
Tracing. We attempt to link each captured release to the commit that its npm package records. Of the four agents’ 1,393 captured releases, 1,345 could be linked to a commit, so RQ3 traces only transitions whose two releases are both linked. A change of a transition comes from the commits that only one of its two releases contains. Still, these commits often do not contain the exact changed text in the instruction stack. For example, a commit can edit a single line of code to send a model another prompt file, which replaces the model’s instructions without editing either file. Likewise, the text can come from a dependency, and a feature flag or a default setting can decide whether it is sent at all. Line-based techniques such as SZZ [72], which follow the changed lines back to the commits that last edited them, would thus miss these commits. Instead, we used an LLM-based agent to explore the code that builds the request and, for each change, name the commits that made it. This tracing agent is mini-swe-agent [5] with Gemini 4 Argon². We chose mini-swe-agent for its minimal, model-agnostic design. It has also been widely used in recent studies which found it competitive with vendor harnesses such as Claude Code and Codex [73, 74, 75]. The agent works in an offline sandbox with the two requests, their changes, and the Git history up to both releases, and a program checks that every commit it names is between the two releases.
Units. Developers also explain these commits not only in their messages but also in the pull requests that merged them [76, 77]. Hence, we group the traced commits by pull request into units, and a commit that no pull request merged forms a unit of its own. The evidence of a unit is then its commit messages, its pull request’s title, description, and review threads, and its linked issues.
Initial codebook. To analyze the reasons behind these units, two reviewers built the codebook in tbl:3 from a pilot of 20 of the 282 transitions, drawn at random, five for each of the four agents whose repositories are public. Claude Code was
| Code | Meaning |
| Does the evidence say why the stack changed? | |
| yes | it gives a reason for the stack change itself, such as unclear text, a misbehavior, or the needs of a model |
| indirectly | it gives a reason only for the code change that the stack change comes with, such as a new feature or a refactoring |
| no | it gives no reason |
| Motivation behind the change... | |
| model behavior | fixes a misbehavior seen in use or in evals, or steers the model toward the behavior that developers want |
| context and cost | saves tokens, turns, or money, keeps the context or the tool list small, or keeps the prompt cache valid |
| safety and security | protects the user, their system, or their data, such as against prompt injection or destructive actions |
| new capability | supports a new or extended feature, tool, skill, mode, or integration of the agent |
| model adaptation | adapts the request to a specific model or provider, such as a new model or a provider’s API |
| inconsistency with the code | makes the text true to how the agent, its tools, or its environment work, as the evidence says that it was false or outdated |
| maintenance | comes with work that keeps the program in shape without a new capability, such as a refactoring, a bug fix outside the model’s behavior, or a build change |
| text quality | fixes typos, grammar, or formatting, or makes the text clearer or more consistent, without a misbehavior or a false text |
| other | has a stated reason that fits no category above |
Table III. The motivation codebook. For each unit, the coder answers whether the evidence says why the stack changed and records every motivation that it states, each with a quote from the evidence.
excluded because its repository is private, so its commit history cannot be traced. The pilot served only to build the codebook. As in RQ1, two reviewers then independently coded a separate sample, and a judge adjudicated their disagreements.
For each unit, a coder records two things. First, whether the evidence says why the stack changed: yes if it gives a reason for the stack change itself, indirectly if it gives a reason only for the code change that the stack change comes with, such as a new feature, and no if it gives no reason. Second, the motivations that the evidence states.
Coding a unit. We coded each unit once with Gemini 4 Argon, using the same settings as the tracing agent. Its input lists the unit’s changes, followed by the evidence: commit messages, the pull request’s title and description, linked issues, and review threads. The model answers whether the evidence says why the stack changed, names the first source that states the reason, and quotes every stated motivation. A program then checks each quote against the evidence.
Validation. Before applying the tracing and coding pipeline to the full dataset, we validated both steps on 55 non-pilot transitions drawn at random from the 282 (margin of error of 10 percentage points at 90% confidence). Two reviewers independently assessed the traced change–unit links and coded the motivations without seeing the LLM’s codes. Of the 798 traced links, 795 were correct (99.6%), and the agent identified at least one correct unit for all but two sampled changes. For motivation coding, every category reached between reviewers and between the LLM and the reference, above our minimum threshold of 0.7 [67]. These results support applying the pipeline to the full dataset and reporting every category separately.
RQ4: What content is shared across agents, what is agent-specific, and how do these patterns evolve?
We compare findings from RQ1 through RQ3 across agents using each agent’s latest captured release at each month-end. This aligns observations despite different release frequencies. We track cross-agent variation to assess whether the captured stacks become more similar over time.
Shared text. For each ordered pair of agents (, ), we calculate the proportion of segments in whose text also occurs as a segment in . As a sensitivity analysis, we also count near-duplicates. To detect them, we use an edit-similarity threshold of 0.8 [78]: a segment of also counts as shared when its closest segment in has a relative edit size (§ 4.3) of at most 20%. This threshold tolerates light edits, such as a product name replaced in an otherwise copied sentence. Shared text thus measures overlap in wording, complementing the functional comparisons below.
Shared tools and functions. We group tools by function, such as file editing or subagent delegation, to compare capabilities despite different names. We record the first captured appearance of each tool category and code identified in RQ1 in each agent, identifying shared and agent-specific functions and their emergence over time. We also compare the stated motivations identified in RQ3 across the four open-source agents.
Results
We captured requests from 1,886 of 1,911 eligible releases (Table I). The remaining 25 failed before sending a request: 15 OpenCode releases could not initialize the model provider under our configuration, and seven releases across three agents contained broken packages or binaries [1].
RQ1: Structure
Directive text dominates, primarily addressing communication, engineering, and tool use. Across each agent’s releases, the median directive share ranges from 89% to 96% of instruction text (Figure 3, left). Together, communication guidance, engineering guidance, and tool guidance account for median shares of 73% to 78%. Communication guidance exceeds engineering guidance in four agents and reaches 40% in OpenCode. By comparison, execution policy, covering permissions and risky operations, accounts for 2% to 6%. These proportions describe textual allocation, not the importance or effectiveness of the instructions.
Figure 3. Composition of the instruction text (LLM codes), as each code’s share of its characters: the median over the agent’s captured releases (left) and the share in its latest release (right), on one shading scale. Gray cells contain no code text. Bold rows give the family’s share. In the latest release, the codes of a family add up to the family row; the medians are computed row by row and need not add up. Unresolved text is left out, since its median and latest shares round to 0.0% in every agent. The bottom row, below the horizontal line, is instead the share of the request’s characters taken by tool definitions, which are not coded.
Tool definitions dominate three agents’ latest captured requests. They take 77.4% of Claude Code’s request, 72.3% of Qwen Code’s, and 63.2% of OpenCode’s, compared with
40.5% of Codex’s and 18.9% of Gemini CLI’s (Figure 3, bottom row, right). In Claude Code’s request, for instance, the 20 tool definitions account for 57,346 of 74,078 characters (Figure 1). Natural-language descriptions constitute most tool-definition content in the first three agents and also convey behavioral instructions. For example, Claude Code’s Workflow tool has a 19,290-character description, longer than its entire instruction text, which limits invocation to explicit user requests for multi-agent orchestration. The captured request includes the description despite our prompt not requesting this capability. Overall, Claude Code’s tool descriptions are 2.6 times as long as its instruction text, while its system prompt takes only 9,517 characters, 13% of the request (Figure 1). Hence, system-prompt analysis alone misses substantial instructional content.
The user prompt accounts for less than 0.1% of each agent’s latest captured request. The latest requests (i.e., requests from the latest versions of the agents) range from 29,387 characters in OpenCode to 124,873 in Qwen Code, so our 21-character test prompt (Figure 1) is 0.02% to 0.07% of them. Thus, in our experiment, almost all content in these requests is supplied by the agent.
Message roles do not reliably identify authorship. Besides the system prompt, the agent writes messages, each with a role that says whom it speaks for, such as the user or the model (§ 2). Although agents place most instruction text in top-level instruction fields or system/developer messages, Gemini CLI and Qwen Code also populate user-role messages. These messages contain 60% of Gemini CLI’s context facts and 48% of Qwen Code’s; they account for 14% of Qwen Code’s latest instruction text. Both agents also sometimes insert the assistant message “Got it. Thanks for the context!”, although the model did not generate it. Consequently, a study or tool that reads user messages by looking at messages with the user role might credit the agent’s text to the user.
Observation 1 Tool definitions and directive text dominate the captured instruction stacks, and the total stack size can exceed +120, 000 characters, excluding the user prompt.
Evolution
Request size grew in four agents but shrank in OpenCode. Between the earliest and latest captured releases, request sizes increased by factors of 7.8 in Codex, 4.5 in Qwen Code, 2.3 in Claude Code, and 1.4 in Gemini CLI; OpenCode’s decreased by almost a quarter (Figure 4, top). Tool definitions drove the largest component growth in Claude Code and Qwen Code, increasing from 21,005 to 57,346 and from 9,317 to 90,317 characters, respectively.
Figure 4. Evolution of each agent’s request over npm publication time. Top: characters of instruction text and, stacked on it, of tool definitions in each captured release, which together make up at least 93% of the request; missing captures break the areas. Bottom: relative edit size of each measurable transition, the edit distance between the textual contents of the two requests as a percentage of the longer one. Each bar is one transition, and triangles mark transitions whose request names a different model. Labels name the transition with the largest relative edit size. In Codex, the vertical line marks the switch from the TypeScript to the Rust implementation.
Changes are infrequent and come in bursts. After masking run-dependent values and release stamps (see § 4.3), 544 of 1,867 transitions (29%) change a segment or tool (Figure 5). Among changing transitions, median relative edit sizes range from 0.6% to 2.0% across agents (Figure 4, bottom). Only 5 to 10 transitions account for half of each agent’s cumulative character edits. These changes mostly add or remove whole segments: the transitions added 3,108 segments and removed 2,751, but modified only 537. As a result, the agents rewrote far more text than their growth shows, adding 306,176 characters of segments and removing 263,685, for a net gain of only 42,491.
Figure 5. Measurable transitions of each agent by what they change. Blue: the transition changes a segment or a tool once run-dependent values and release stamps are masked. Gray: it edits the request but changes no segment or tool. Light gray: the two requests are identical. Labels give the share of the blue and gray parts.
Large transitions reconfigure tool availability. Claude Code 2.1.69 replaced 18 tool definitions with a single ToolSearch tool for on-demand discovery, reducing its request from 77,601 to 20,380 characters. Release 2.1.70 restored the definitions a day later. Qwen Code 0.21.2 instead added 65,282 characters of tool definitions, approximately twice the length of its instruction text.
Attribution and commenting directives vary across releases. In 492 of 493 Claude Code captures, the Bash description instructs the model to append a Co-Authored-By: Claude trailer to commit messages, the marker AIDev uses to identify Claude Code pull requests [28]. OpenCode includes a similar directive in 312 of 907 captures, removing it in release 0.4.20 in August 2025. No captured stack from the other three agents specifies an attribution marker. Claude Code’s comment policy also changed five times. Its strictest form, “DO NOT ADD ANY COMMENTS unless asked”, appears in 186 of 493 captures, spanning releases 0.2.49 to 1.0.128. From release 2.1.154, the policy instead asks the model to “match its comment density”. Such revisions change the directives governing attribution and coding style without necessarily changing the user’s task.
Observation 2 Instruction stacks evolve through occasional concentrated revisions, with additions and removals revealing substantially more change than net size alone captures.
Motivations
Most units document a reason, but fewer than half explain the stack change itself. The changes come from 496 units (Section 4.4), namely 440 pull requests and 56 commits outside any pull request. Of these, 444 (90%) state a reason (Figure 6, top). Yet only 236 (48%) give a reason for the stack change itself, whereas 208 (42%) explain only the accompanying code change, such as a new feature. Moreover, units without a pull request explain the least: 21 of the 56 commit-only units (38%) state no reason, against 31 of the 440 pull-request units (7%).
Figure 6. Units and their motivations (LLM codes). Top: share of each agent’s units by whether their evidence says why the stack changed (Table III). Bottom: number of units that state each motivation, among the 444 units that answer yes or indirectly, split by that answer. A unit can state several motivations, so the bars add up to 545.
Stated motivations center on model behavior, capabilities, and maintenance. Among the 444 units stating a reason, 150 (34%) cite model behavior, 127 (29%) new capability, and 102 (23%) maintenance (Figure 6, bottom). Yet these motivations differ in how directly the units explain them. Nearly all units citing model behavior (146 of 150) say directly why the stack changed; in fact, they make up 62% of the 236 units that do, as in Gemini CLI pull request #17608: “Fix issue where Gemini CLI can make changes when simply asked a question”. Conversely, most units citing new capability (114 of 127) and maintenance (83 of 102) explain only the accompanying code change. Hence, developers explain the stack change itself mainly when they steer the model, and otherwise the stack changes along with the surrounding software.
Safety is rarely an explicit motivation, including for execution-policy changes. Only 13 units cite safety and security. Execution policy text, which governs permissions and hard-to-reverse operations, did not shrink between any agent’s earliest and latest captured releases and grew by 3,400 characters in Qwen Code. Of the 24 units that modify this text and state a reason, two cite safety, and 13 cite model behavior. Safety is therefore infrequently documented as a motivation, even when changes concern potentially risky operations.
Observation 3 Fewer than half of change units explain the stack revision itself; direct explanations primarily concern model behavior, while safety is rarely an explicit motivation.
Commonality
Shared wording is limited and transient. Six of ten agent pairs never share an identical segment of at least 30 characters (Figure 7, top). Overlap in the remaining pairs reflects adaptation or reuse. Qwen Code originated from Gemini CLI and later incorporated Claude Code’s todo-list and memory guidance, accounting for 31% of Qwen Code’s segments in April 2026. In OpenCode, 87% of segments matched Claude Code in June 2025, while up to 90% matched Codex between September and December 2025. These overlaps subsequently disappeared as the agents revised their stacks, including changes in the source agents. By August 2026, at least 95% of each agent’s segments appeared in no other agent. Wording therefore became largely agent-specific despite earlier reuse.

Figure 7. Commonality over time. Each column is one month and compares the agents’ latest captured releases at its end. Top: for the four pairs of agents that ever share a segment, is the share of ’s segments that sends verbatim, where sent the text first. Segments shorter than 30 characters, the user prompt, and its delimiters are left out. Bottom: share of directive text in the instruction text, across the four agents and Claude Code’s line, from July 2025, when all five agents were released.
Capability catalogs and subagent delegation became common across all five agents. Between the earliest and latest captures, capability catalog text grew in every agent by up to 5,600 characters. Early catalogs listed user-invoked slash commands such as /help; skill or subagent catalogs appeared between December 2025 (Codex 0.76.0 and Gemini CLI 0.22.0) and June 2026 (Qwen Code 0.18.0). In Claude Code, these catalogs account for 37% of the latest instruction text. Subagent delegation appeared in Claude Code’s first capture and in the other four agents between June 2025 and March 2026. These capabilities became shared features despite limited overlap in wording.
Agents share functional categories, while directive shares converge in four agents. Eight of the eleven RQ1 codes (Table II) appear in every agent’s latest stack, generally from early releases onward. Agents differ in their allocation of text across these functions and their inclusion of less-common categories (Figure 3). The directive shares of the four open-source agents narrowed from 88% to 97% in July 2025 to 84% to 87% in August 2026 (Figure 7, bottom). Claude Code diverged: its directive share fell from 85% in April 2026 to 54% following release 2.1.154 (§ 5.2). Thus, four agents became more similar in directive share despite increasingly distinct wording, while Claude Code moved away from this pattern (Figure 7).
Observation 4 Agents share core functions and capabilities despite using increasingly distinct wording.
Discussion
Instruction stacks need maintenance rationale, not just version histories. Substantial additions and removals reveal ongoing revision, yet only 48% of change units explain the change itself. Without this rationale, maintainers may struggle to distinguish enduring requirements from workarounds for earlier models or failures. Retaining obsolete instructions or removing necessary ones could accumulate instruction-related technical debt. Work treating prompts as programs [40], [41] could address this risk by linking revisions to their intended behavior, applicable models, and regression evaluations.
Shared practices suggest emerging engineering conventions. Capability catalogs and subagent delegation became common across all five agents despite increasingly distinct wording. These patterns suggest recurring approaches to exposing capabilities and coordinating work, rather than convergence on a common prompt. They offer a basis for reusable tooling and comparable evaluations, although shared functions alone establish neither interoperability nor effectiveness.
The agent release should be part of the experimental configuration. A coding agent interprets the user’s task alongside substantial agent-supplied content, including tool descriptions and messages beyond the system prompt. Because this content changes across releases, model identity and the user prompt alone do not fully specify experimental conditions. Differences attributed to model capabilities may also reflect changes in harness-supplied instructions. Our study exposes this source of variation without establishing its causal effects on performance or cost.
Reporting recommendations for LLM-based studies [79]–[81] should therefore extend to the exact agent release, model configuration, repository guidance, and enabled capabilities. Where feasible, researchers should also archive captured instruction stacks with their capture conditions. Release-level information matters for repository mining as well: attribution directives can affect which contributions marker-based searches retrieve [28], while commenting policies offer an alternative explanation for differences in generated code. Accounting for agent releases is thus necessary even when the researcher-written prompt and selected model remain unchanged.
Threats to validity
Construct validity. RQ3 captures developers’ stated motivations, which may omit or misrepresent their actual reasons [77], [82]. We examine linked pull requests, reviews, and issues alongside commit messages and do not infer motivations from code changes alone. Our findings therefore describe stated motivations, not verified intent.
Internal validity. Human judgment and LLM-assisted annotation may introduce labeling errors. We followed established evaluation practices, including independent labeling, adjudication, and validation against human references using predefined acceptance criteria [83], [84]. We also restricted commit tracing to eligible candidates and checked supporting quotations. These steps reduce annotation errors but cannot eliminate shared biases or guarantee accuracy beyond the validation samples.
External validity. We study five command-line agents under a controlled configuration, capturing only the first request to the main model in an empty working directory without user interaction. This supports release-level comparison but does not capture variation from user customization, alternative model choices, or later interactions. Other agents and configurations may produce different stacks, and shared content across the five agents does not establish broader generality. Our findings cover captured releases published before 25 August 2026 and may not extend to subsequent releases.
Conclusion
We analyzed the initial instruction stacks of 1,886 releases across five coding agents. The study shows that agent releases revise model-facing instructions beyond the system prompt, often without documenting the reasons, while shared capabilities emerge despite increasingly distinct wording. These changes introduce a source of variation that model identifiers and user prompts alone cannot capture. Researchers should therefore document the agent release and captured stack when reporting experiments, and maintainers should preserve the rationale for instruction changes.
Our analysis characterizes these changes; however, establishing their effects requires controlled experiments. Future work should vary captured stacks while holding models, tasks, and execution conditions fixed to measure changes in correctness, tool use, latency, and cost. Ablations could test whether particular instructions improve outcomes or introduce redundancy and conflicts. Capturing later turns would extend this analysis to repository guidance, dynamically loaded capabilities, and execution history. Our replication package is available at [1].
References
- [1]Anonymous authors, “Replication package of this study,” anonymous replication package. [Online]. Available: https://anonymous.4open.science/r/InstructionStacksStudyReplication-C58B/
- [2]Anthropic, “Claude Code overview: Claude Code documentation,” https://code.claude.com/docs/en/overview, accessed 19 October 2026.
- [3]OpenAI, “Codex CLI: GitHub repository,” https://github.com/openai/codex, accessed 19 October 2026.
- [4]Google, “Gemini CLI: GitHub repository,” https://github.com/google-gemini/gemini-cli, accessed 19 October 2026.
- [5]J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024.arxiv.org/abs/2405.15793
- [6]M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting,” in International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://arxiv.org/abs/2310.11324
- [7]M. Mizrahi, G. Kaplan, D. Malkin et al., “State of what art? A call for multi-prompt LLM evaluation,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 933–949, 2024.
- [8]D. Jaroslawicz, B. Whiting, P. Shah, and K. Maamari, “How many instructions can LLMs follow at once?” arXiv:2507.11538, 2025, preprint.arxiv.org/abs/2507.11538
- [9]M. Levy, A. Jacoby, and Y. Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” in Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 15 339–15 353.arxiv.org/abs/2402.14848
- [10]Y. Du, M. Tian, S. Ronanki et al., “Context length alone hurts LLM performance despite perfect retrieval,” in Findings of the Association for Computational Linguistics: EMNLP, 2025, pp. 23 281–23 298.arxiv.org/abs/2510.05381
- [11]K. Faghih, W. Wang, Y. Cheng et al., “Tool preferences in agentic LLMs are unreliable,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025, pp. 20 954–20 969.arxiv.org/abs/2505.18135
- [12]O. Ben Sghaier, H. Li, B. Adams, and A. E. Hassan, “Don’t blame the large language model: How agent harness evolution shapes coding agent quality,” arXiv:2607.03691, 2026, preprint.arxiv.org/abs/2607.03691
- [13]J. Liu, X. Zhao, X. Shang, and Z. Shen, “Dive into Claude Code: The design space of today’s and future AI agent systems,” arXiv:2604.14228, 2026, preprint.arxiv.org/abs/2604.14228
- [14]W. Chatlatanagulchai, H. Li, Y. Kashiwa et al., “Agent READMEs: An empirical study of context files for agentic coding,” ACM Transactions on Software Engineering and Methodology, 2026.arxiv.org/abs/2511.12884
- [15]S. Mohsenimofidi, M. Galster, C. Treude, and S. Baltes, “Context engineering for AI agents in open-source software,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2026, pp. 194–198.arxiv.org/abs/2510.21413
- [16]S. Jiang and D. Nam, “Beyond the prompt: An empirical study of Cursor rules,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2026, pp. 397–409.DOI
- [17]K. Chakrabarti, “Why does CLAUDE.md keep growing? Catastrophic remembering in agentic coding,” arXiv:2608.11095, 2026, preprint.arxiv.org/abs/2608.11095
- [18]T. Gloaguen, N. Mündler, M. N. Müller et al., “Evaluating AGENTS.md: Are repository-level context files helpful for coding agents?” in ICLR Workshop MemAgents, 2026. [Online]. Available: https://openreview.net/forum?id=pLi3A8bscparxiv.org/abs/2602.11988
- [19]J. L. Lulla, S. Mohsenimofidi, M. Galster et al., “On the impact of AGENTS.md files on the efficiency of AI coding agents,” in JAWS Workshop at ICSE, 2026. [Online]. Available: https://arxiv.org/abs/2601.20404
- [20]P. Khatri, “Do context files help coding agents? A two-agent ablation study on real repositories,” arXiv:2607.27250, 2026, preprint.arxiv.org/abs/2607.27250
- [21]T. Mason, “Arbiter: Detecting interference in LLM agent system prompts,” arXiv:2603.08993, 2026, preprint.arxiv.org/abs/2603.08993
- [22]K. Pister, D. J. Paul, I. Joshi, and P. Brophy, “PromptSet: A programmer’s prompting dataset,” in International Workshop on Large Language Models for Code (LLM4Code), 2024, pp. 62–69.DOI
- [23]M. Tafreshipour, A. Imani, E. Huang et al., “Prompting in the wild: An empirical study of prompt evolution in software repositories,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2025, pp. 686–698.arxiv.org/abs/2412.17298
- [24]Y. Mao, J. He, and C. Chen, “From prompts to templates: A systematic prompt template analysis for real-world LLMapps,” in Companion Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE Companion), 2025, pp. 75–86.
- [25]P. Barbaste, T. Darrigol, G. Vu, and T. Wiltberger, “Harness engineering: Anatomy, architecture, and evolution of coding agents. A source-code study of eleven systems,” arXiv:2609.00006, 2026, preprint.arxiv.org/abs/2609.00006
- [26]X. Wang, B. Li, Y. Song et al., “OpenHands: An open platform for AI software developers as generalist agents,” in International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://arxiv.org/abs/2407.16741
- [27]J. Liu, K. Wang, Y. Chen et al., “Large language model-based agents for software engineering: A survey,” ACM Transactions on Software Engineering and Methodology, 2026.arxiv.org/abs/2409.02977
- [28]H. Li, H. Zhang, and A. E. Hassan, “The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,” arXiv:2507.15003, 2025, preprint.arxiv.org/abs/2507.15003
- [29]Anthropic, “Create a message: Claude API reference,” https://platform.claude.com/docs/en/api/messages/create, accessed 23 September 2026.
- [30]Google, “Generating content: Gemini API reference,” https://ai.google.dev/api/generate-content, accessed 23 September 2026.
- [31]OpenAI, “Migrate to the Responses API,” https://developers.openai.com/api/docs/guides/migrate-to-responses, accessed 23 September 2026.
- [32]OpenAI, “Create chat completion: OpenAI API reference,” https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create, accessed 13 October 2026.
- [33]A. Neumann, Y. Pi, and J. Singh, “Who controls the conversation? User perspectives on generative AI (LLM) system prompts,” in Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), 2026.DOI
- [34]OpenAI, “Text generation: OpenAI API guide,” https://developers.openai.com/api/docs/guides/text, accessed 13 October 2026.
- [35]Anthropic, “Using the Messages API: Claude API guide,” https://platform.claude.com/docs/en/build-with-claude/working-with-messages, accessed 13 October 2026.
- [36]S. Wang, J. He, Y. Zhao et al., “MalSkills: Detecting malicious skills in the agentic supply chain via neuro-symbolic reasoning,” in Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), 2026, pp. 562–574.
- [38]H. Li, H. Masri, F. R. Cogo et al., “Understanding prompt management in GitHub repositories: A call for best practices,” IEEE Software, vol. 43, no. 2, pp. 85–93, 2026.DOI
- [39]A. Aljohani and H. Do, “PromptDebt: A comprehensive study of technical debt across LLM projects,” in International Conference on Evaluation and Assessment in Software Engineering (EASE), 2025, pp. 371–382.DOI
- [40]J. T. Liang, M. Lin, N. Rao, and B. A. Myers, “Prompts are programs too! Understanding how developers build software containing prompts,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1591–1614, 2025.arxiv.org/abs/2409.12447
- [41]Z. Chen, C. Wang, W. Sun et al., “Promptware engineering: Software engineering for prompt-enabled systems,” ACM Transactions on Software Engineering and Methodology, 2026.arxiv.org/abs/2503.02400
- [42]H. Villamizar, J. Fischbach, A. Korn et al., “Prompts as software engineering artifacts: A research agenda and preliminary findings,” in Product-Focused Software Process Improvement (PROFES), 2025, pp. 470–478.
- [43]C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Demystifying LLM-based software engineering agents,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, 2025.DOI
- [44]S. Lewis, “Same model, different harness: Different coding-agent results,” arXiv:2608.26218, 2026, preprint.arxiv.org/abs/2608.26218
- [45]R.-Z. Fan, Z. Zhang, S. Ma et al., “An empirical study of harness design for coding agents,” arXiv:2609.20804, 2026, preprint.arxiv.org/abs/2609.20804
- [46]R. Rombaut, “Inside the scaffold: A source-code taxonomy of coding agent architectures,” arXiv:2604.03515, 2026, preprint.
- [47]Y. Zhang, N. Carlini, and D. Ippolito, “Effective prompt extraction from language models,” in Conference on Language Modeling (COLM), 2024. [Online]. Available: https://openreview.net/forum?id=0o95CVdNuzarxiv.org/abs/2307.06865
- [48]B. Hui, H. Yuan, N. Gong et al., “PLeak: Prompt leaking attacks against large language model applications,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), 2024, pp. 3600–3614.arxiv.org/abs/2405.06823
- [49]J. Yu, Y. Wu, D. Shu et al., “Assessing prompt injection risks in 200+ custom GPTs,” in ICLR Workshop on Secure and Trustworthy Large Language Models, 2024. [Online]. Available: https://arxiv.org/abs/2311.11538
- [50]Z. Sha and Y. Zhang, “Prompt stealing attacks against large language models,” arXiv:2402.12959, 2024, preprint.arxiv.org/abs/2402.12959
- [51]Y. Xie, M. Luo, Z. Liu et al., “Red-teaming coding agents from a tool-invocation perspective: An empirical security assessment,” arXiv:2509.05755, 2025, preprint.arxiv.org/abs/2509.05755
- [52]W. Chatlanagulchai, K. Thonglek, B. Reid et al., “On the use of agentic coding manifests: An empirical study of Claude Code,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2025, pp. 543–551.
- [53]S. Sun, J. Akhoundali, A. Kudriavtseva et al., “A study of cursor rules files in GitHub open source projects,” arXiv:2608.10622, 2026, preprint.arxiv.org/abs/2608.10622
- [54]D. B. Hong, A. Imani, and I. Ahmed, “From anatomy to smells: An empirical study of SKILL.md in agent skills,” arXiv:2607.01456, 2026, preprint.arxiv.org/abs/2607.01456
- [55]H. Gao, J. L. Lulla, H. Y. Lin et al., “From registry to repository: How AI agent skills are written, adapted, and maintained,” arXiv:2607.00911, 2026, preprint.arxiv.org/abs/2607.00911
- [56]M. Galster, S. Mohseninofidi, J. L. Lulla et al., “Configuring agentic AI coding tools: An exploratory study,” in ACM International Conference on AI-Powered Software (AIware), 2026, pp. 11–20.DOI
- [57]Qwen Team, “Qwen Code: GitHub repository,” https://github.com/QwenLM/qwen-code, accessed 19 October 2026.
- [58]Anomaly, “OpenCode: GitHub repository,” https://github.com/anomalyco/opencode, accessed 19 October 2026.
- [59]B. P. Vangala, A. Gehani, and T. Malik, “AI-generated code is not reproducible (Yet): An empirical study of execution reliability in LLM-based coding agents,” in Proceedings of the ACM Conference on Reproducibility and Replicability (ACM REP), 2026, pp. 33–46.DOI
- [60]R. Zhang, W. Dai, H. V. Pham et al., “Engineering pitfalls in AI coding tools: An empirical study of bugs in Claude Code, Codex, and Gemini CLI,” in Companion Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE Companion), 2026, pp. 393–403.
- [61]N. Nashid, D. Ding, K. Gallaba et al., “Beyond accuracy: Behavioral dynamics of agentic multi-hunk repair,” arXiv:2511.10112, 2025, accepted in ACM Transactions on Software Engineering and Methodology.
- [62]S. Baltes and P. Ralph, “Sampling in software engineering research: A critical review and guidelines,” Empirical Software Engineering, vol. 27, no. 4, 2022.arxiv.org/abs/2002.07764
- [63]W. G. Cochran, Sampling Techniques, 3rd ed. John Wiley & Sons, 1977.
- [64]T. H. M. Le, X. Du, and M. A. Babar, “Are latent vulnerabilities hidden gems for software vulnerability prediction? An empirical study,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2024, pp. 716–727.arxiv.org/abs/2401.11105
- [65]S. W. Thomas, B. Adams, A. E. Hassan, and D. Blostein, “Studying software evolution using topic models,” Science of Computer Programming, vol. 80, pp. 457–479, 2014.
- [66]J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960.DOI
- [67]J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977.pubmed.ncbi.nlm.nih.gov/843571
- [68]V. I. Levenshtein, “Binary codes capable of correcting deletions, insertions, and reversals,” Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710, 1966.
- [69]T. Zimmermann, S. Kim, A. Zeller, and E. J. Whitehead, “Mining version archives for co-changed lines,” in Proceedings of the International Workshop on Mining Software Repositories (MSR), 2006, pp. 72–75.DOI
- [70]S. P. Reiss, “Tracking source locations,” in International Conference on Software Engineering (ICSE), 2008, pp. 11–20.DOI
- [71]Git Project, “git-diff documentation,” https://git-scm.com/docs/git-diff, accessed 22 October 2026.
- [72]J. Śliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?” ACM sigsoft software engineering notes, vol. 30, no. 4, pp. 1–5, 2005.
- [73]W. Huang, C. Lee, L. Tng, and S. Ge, “DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks,” arXiv:2607.07946, 2026, preprint.arxiv.org/abs/2607.07946
- [74]J. Williams, D. Farris, J. Farris, and J. Jiao, “SWE-Serve: Benchmarking agentic engineering for production inference serving,” arXiv:2609.26777, 2026, preprint.arxiv.org/abs/2609.26777
- [75]M. Raghavendra, S. Dan, M. Romero Calvo et al., “SWE Atlas: Benchmarking coding agents beyond issue resolution,” arXiv:2605.08366, 2026, preprint.arxiv.org/abs/2605.08366
- [76]J. Pantiuchina, F. Zampetti, S. Scalabrino et al., “Why developers refactor source code: A mining-based study,” ACM Transactions on Software Engineering and Methodology, vol. 29, no. 4, 2020.DOI
- [77]M. Paixão, A. Uchôa, A. C. Bibiano et al., “Behind the intents: An in-depth empirical study on software refactoring in modern code review,” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2020.
- [78]K. Lee, D. Ippolito, A. Nystrom et al., “Deduplicating training data makes language models better,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022, pp. 8424–8445.arxiv.org/abs/2107.06499
- [79]J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using LLMs in software engineering,” in International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2024, pp. 102–106.DOI
- [81]A. Korn, L. Zaruchas, C. Arora et al., “Reporting LLM prompting in automated software engineering: A guideline based on current practices and expectations,” arXiv:2601.01954, 2026, preprint, to appear at FORGE 2026.arxiv.org/abs/2601.01954
- [82]G. Cai, R. Li, P. Liang et al., “Rule taxonomy and evolution in AI IDEs: A mining and survey study,” arXiv:2606.12231, 2026, preprint.arxiv.org/abs/2606.12231
- [83]P. Ralph, N. bin Ali, S. Baltes et al., “Empirical standards for software engineering research,” arXiv:2010.03525, 2021, version 0.2.0.arxiv.org/abs/2010.03525
- [84]T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can LLMs replace manual annotation of software engineering artifacts?” in Proceedings of the International Conference on Mining Software Repositories (MSR), 2025, pp. 526–538.arxiv.org/abs/2408.05534



