WorksComputer Science
Context Matters: Improving LLM-Based Code Summarization with Hierarchical Information
Abstract
Source code documentation plays an important role in software quality activities, such as code review, program comprehension, maintenance, and bug fixing. However, it is often fragmented across individual code elements and focuses on local implementation details, providing little information about their surrounding context. This limits both developers' understanding of software systems and the effectiveness of Large Language Models (LLMs), whose reasoning is constrained by finite context windows. Enriching documentation with hierarchical context has the potential to improve software comprehension and support downstream, quality-related tasks for both developers and LLM-based tools. In this paper, we present an in silico study that showcases the impact of augmenting context on the generation of code-level documentation, using a variety of LLMs. By varying the function, class, and module hierarchy visible to the LLMs, we assess whether this variation produces a significant difference in the resulting code documentation compared with the existing documentation, which serves as a proxy for developer intent. Our results show that Documentation-Aware prompts with class or module-level context consistently yield summaries that are closest to the original documentation. Adding surrounding documentation yields statistically significant improvements over the baseline across models, with effect sizes ranging from negligible (DeepSeek) through weak (GPT-4o-mini) to medium (Gemini). We restrict the study to this immediate hierarchical context; call-graph traversal and cross-module dependency context fall outside our scope and are left to future work. These findings provide a clear path to improve automated documentation tools and enhance LLMs' ability to reason about codebases for downstream tasks, such as code retrieval.
Introduction
During software evolution, developers spend over half their working time on program comprehension [24], yet few software projects adequately comment their code [21]. In this context, source code documentation plays an important role in quality-related activities, including program comprehension, code review, and bug fixing. Furthermore, it has become of utmost relevance with the widespread adoption of generative artificial intelligence in the development process, where automated code generation needs to be verified and validated frequently, and likely reduces code ownership [15].
When such documentation exists, it is often mismatched with the source code, missing, or outdated [25]. Code summarization aims to close this gap: given a method’s signature and body, Sridhara et al. [21] define the task as generating natural language text that "summarizes the overall actions of [the method] accurately, adequately, and concisely," producing descriptive comments that convey a method’s intent through its major algorithmic actions to aid program comprehension and software maintenance.
The field has progressed from information retrieval and heuristic methods to abstract syntax tree embeddings [2], specialized transformer architectures such as CodeBERT [6], and hierarchical representations [28]. Large Language Models (LLMs), trained on vast amounts of code and natural language data and viewable as learning a mapping function from source code to a natural language description , have surpassed these approaches on standard benchmarks [19, 22], and developers now use LLM-based summarization in their workflows through products such as GitHub Copilot. Choosing how much surrounding context to provide is therefore a concrete design decision for anyone building documentation tools.
Current evaluations of LLM-based code summarization focus on whether models produce high-scoring summaries according to metrics such as BLEU [17], METEOR [11], ROUGE [12], and human or LLM-based judgment. They do not measure how the presence of existing documentation and varying levels of structural context influence whether the generated output approximates developer-written references, which remain the standard ground truth in code summarization evaluation [25].
Ahmed et al. [1] augment prompts with semantic information at the function level but do not investigate broader hierarchical context. Makharev and Ivanov [14] explore class and repository-level context but rely on retrieval-augmented generation and few-shot learning, which conflates the effects of intrinsic code structure with externally retrieved information, preventing attribution of gains to either source. Without controlled evidence on how hierarchical scope affects summary quality, tool designers face an uninformed trade-off: including too much context wastes inference budget, while including too little risks producing summaries that miss the neighboring methods, shared attributes, and module-level definitions the target function depends on. There is no empirical basis to guide this decision.
By measuring how closely generated summaries match developer-written references at each context level, we identify which layers of context improve quality and which add cost without benefit. We investigate whether expanding hierarchical context from function-only to class or module level improves similarity to developer-written references (RQ1), whether preserving surrounding documentation adds value beyond code structure alone (RQ2), and whether these effects vary across three models, GPT-4o-mini, Gemini-3-Flash, and DeepSeek v3.2 (RQ3); the Research Method section states these questions in full.
We expand prior work by exploring a multi-layered context that includes the function itself, the class, and the module1 to which it belongs, to produce hierarchically informed code documentation. We focus on the immediate hierarchical context (class and module definitions), which exposes the neighboring methods, shared attributes, and module-level definitions the target uses, rather than inter-procedural call graphs, cross-module data flow, or external retrieval-augmented generation (RAG) methods. By isolating the hierarchical scope, we establish a baseline for how well LLMs utilize the intrinsic structure of the codebase before introducing the complexity of dependency-aware context and external retrieval systems, which we leave to future work.
This distinction matters for tool designers: if hierarchical code structure alone drives quality improvements, tools can simplify the input pipeline by omitting surrounding documentation. The remainder of this paper is organized as follows: § 2 positions the study in relation to related work, § 3 describes the experimental method, and § 4 presents the results.
Background And Related Work
Recent advances in code summarization have explored context augmentation and prompting strategies. We organize prior work along these two dimensions, paying attention to how each study treats pre-existing documentation in the surrounding scope.
Context Augmentation For Code Summarization
The most directly related line of work investigates how additional context beyond the bare function body improves generated summaries. Ahmed et al. [1] augmented LLM prompts with syntactic elements extracted via program analysis (tagged identifiers, data-flow graphs, and repository metadata) and reported significant metric improvements over baselines. Their augmentation targets lexical and structural representations at the function level, but they do not investigate the effect of supplying the enclosing class or module definitions, nor do they examine whether pre-existing documentation in the surrounding scope influences the generated summary.
Lomshakov et al. [13] introduced ProConSuL, a framework that constructs project-level context for C/C++ code summarization by building a call graph and recursively summarizing callees. Their pipeline combines supervised fine-tuning with ORPO preference alignment and evaluates on 25 test functions. ProConSuL gathers
context from other files via call-graph traversal, a program-analysis-based approach distinct from similarity-based retrieval. Their work does not examine the effect of intra-file hierarchical context or vary the presence of surrounding documentation.
Makharev and Ivanov [14] examined code summarization beyond the function level, evaluating LLMs with class-level and repository-level context on Python benchmarks (1,200 functions). At the class level, they provide the enclosing class body directly to the model; at the repository level, they apply RAG-retrieved code chunks. Their results show that few-shot learning and RAG-retrieved context enhance LLM performance. At the class level, the input representation overlaps: both approaches supply the enclosing class code.
Two gaps remain. Makharev and Ivanov do not distinguish between settings in which the surrounding documentation (docstrings of neighboring methods, the class-level docstring) is present versus stripped. They also do not extend the hierarchy to module-level scope (the full file), and their function-level and repository-level experiments rely on few-shot prompting and RAG, respectively.
Prompting Strategies, Evaluation, And The Role Of Function Names
A second line of work asks how prompting techniques and evaluation methods, rather than input content, affect summary quality. Sun et al. [22] conducted a systematic study of LLM-based code summarization, comparing five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert). They found that advanced prompting strategies do not consistently outperform simple zero-shot prompting, and that GPT-4-based evaluation aligns most closely with human judgment. Their study varies the prompting strategy while holding the input context constant at the function level. If zero-shot prompting already captures most gains from prompt engineering, the remaining lever for practitioners is the scope and composition of the input context itself.
Crupi et al. [5] investigated the reliability of the LLM-as-a-Judge approach for code generation and summarization. When comparing judgments from multiple LLMs against human assessments, they found that GPT-4-turbo achieves moderate-to-substantial agreement with humans on summary content adequacy, whereas smaller models perform poorly as judges.
Haldar and Hockenmaier [7] found that LLMs rely heavily on function names rather than on code structure when generating summaries, and that metrics such as BLEU and BERTScore provide largely redundant assessments. If models lean on function names as a shortcut, richer surrounding context may reduce that reliance, but no existing study tests this hypothesis directly.
Positioning Of This Work
The studies above leave unanswered whether improvements from the broader context are driven by additional code structure or by documentation already present in that structure. All prior work supplies context with documentation intact and does not test what happens when it is stripped. Makharev and Ivanov [14] test class-level context but do not vary the presence of documentation, and do not extend to the module scope. Our study addresses this gap by crossing context granularity (Function, Class, Module) with documentation mode, Documentation-Aware (DA) vs. Documentation-Independent (DI), as independent factors across multiple LLMs and 6,932 methods from 12 mature Python repositories.
Research Method
The goal of this study is to investigate how the presence of existing documentation and varying levels of context affect LLMs’ ability to generate documentation that approximates developer intent.
Research Questions
RQ1: Does expanding hierarchical context from function-only to class or module level improve the similarity of LLM-generated documentation to developer-written references?
RQ2: Does preserving surrounding documentation add value beyond code structure alone?
RQ3: Do these effects vary across models?
Study Design
While human-written references often lack the stylistic completeness of LLM outputs, they remain the source of ground truth for semantic alignment. Therefore, while also including -gram metric analysis, we use BERTScore to measure semantic alignment, and LLM-as-a-Judge to evaluate informational completeness. Additionally, this study examines whether retaining the original reference documentation affects the metrics and keywords used in the generated summaries.
Following guidelines for empirical studies utilizing LLMs [3] and empirical standards for data science experiments [20], we use an incomplete factorial design with five experimental conditions, controlled by two independent factors. The first factor, context granularity, controls the hierarchical scope of code visible to the model (Function, Class, or Module). The second factor, documentation mode, controls whether existing documentation in the surrounding scope is retained or removed. With this approach, context granularity determines how much code structure the model sees, while documentation mode determines whether the docstrings within that structure are preserved.
We define Documentation-Aware (DA) as an input where the target function’s docstring is removed, but documentation in the surrounding scope (class or module) is preserved. Documentation-Independent (DI) involves stripping all documentation from the input entirely.
At the function-only level, no surrounding scope is provided to the model. Therefore, the distinction between Documentation-Aware (DA) and Documentation-Independent (DI) is not applicable. We treat the "Function-only" condition as the baseline representing standard code summarization without hierarchical context. This yields five experimental scenarios: the Function-only baseline, plus Class and Module contexts under both DI and DA conditions, illustrated in Figure 1.

Figure 1. The five experimental conditions applied to lower_items from requests/structures.py. Ellipses denote truncated code. The target function’s docstring is removed in all conditions.
These conditions map directly to our research questions. RQ1 is addressed by comparing the Function-only baseline against Class and Module conditions, isolating the effect of hierarchical scope. RQ2 is addressed by comparing DI against DA at each context level, isolating the effect of surrounding documentation. RQ3 is addressed by conducting these comparisons separately for each of the three LLMs.
To illustrate the five conditions, Figure 1 shows an abbreviated module from the requests library. Colors indicate which scope includes each region; underlined docstrings are preserved in DA conditions but stripped in DI.
Dataset
We sourced 12 Python open-source repositories from the SWE-Bench database [9], chosen for their maturity and active developer communities. Only artifacts documented at all three hierarchical levels (function, class, and module) were retained, ensuring consistency in metrics derived from context augmentation across experimental conditions. Table 1 lists the repositories and their pinned commits, along with descriptive statistics per repository.
To ensure a consistent basis for comparison across experimental conditions, we filtered the dataset, excluding modules that were not originally documented at all three hierarchical levels. To simplify the extraction process, we excluded nested class definitions. After filtering, the dataset consisted of 6,932 methods across the 12 repositories. We excluded 4 samples from metric evaluation because their reference docstrings were empty or consisted solely of numeric values, yielding 6,928 samples for the evaluation reported in Section 4.2. Across the dataset, modules contain a median of two classes (Q1=1, Q3=4), indicating that class-level and module-level contexts differ in scope. Figure 2 shows the architecture of the data preparation pipeline. The median module size is 517 LoC (IQR: 239–993), and the median reference docstring length is 94 characters (IQR: 47–273).

Figure 2. Experimental sample pipeline procedure
| Repository | SHA Commit | Modules | Classes | Functions | Docstrings | After Filter |
| astropy | d16bfe05 | 227 | 978 | 2,328 | 3,533 | 1,205 |
| django | e7fd69d0 | 153 | 606 | 2,153 | 2,912 | 480 |
| flask | d8c37f43 | 1 | 42 | 154 | 197 | 10 |
| matplotlib | b7d05919 | 104 | 543 | 2,398 | 3,045 | 1,798 |
| pylint | 182cc539 | 66 | 162 | 577 | 805 | 343 |
| pytest | 6995257c | 13 | 19 | 286 | 318 | 1 |
| requests | 110048f9 | 20 | 65 | 164 | 249 | 77 |
| scikit-learn | b90661d6 | 169 | 385 | 864 | 1,418 | 750 |
| seaborn | 94621cef | 16 | 69 | 263 | 348 | 78 |
| sphinx | 7bdc11e8 | 169 | 395 | 465 | 1,029 | 380 |
| sympy | 8dcb12a6 | 306 | 938 | 3,018 | 4,262 | 1,794 |
| xarray | 863e4906 | 22 | 92 | 405 | 519 | 16 |
| Total | 1,266 | 4,294 | 13,075 | 18,635 | 6,932 |
Table 1. Studied repositories: pinned commits and per-repository descriptive statistics. Modules, Classes, and Functions columns count only those with docstrings. "After Filter" is the count of functions documented at all three hierarchical levels.
Experimental Procedure Setup
To generate code summaries, we used the models listed in Table 2, setting the temperature to 0 and top_p to 1. We employed low-cost and general-purpose API models from three different providers, including DeepSeek v3.2, which has open weights. The generation prompt does not enforce a specific docstring style, allowing each model to produce documentation in whatever format it deems appropriate for the function. The full generation prompt is in the replication package.
| Provider | Model ID | Knowledge Cutoff |
| OpenAI | gpt-4o-mini-2024-07-18 | Oct 2023 |
| gemini-3-flash-preview | Jan 2025 | |
| DeepSeek | Deepseek v3.2 | Mar 2025 |
Table 2. Large Language Models used in the experiment
Two input conditions were prepared for each artifact: (1) with existing contextual documentation retained, but removing the target documentation, and (2) with all documentation removed. In both cases, the same source code context was supplied to the LLM, ensuring comparability across conditions. The experiment ran in a WSL2 environment on Windows using Python 3.11. Generation invoked the OpenAI, Gemini, and DeepSeek APIs, whereas the LLM-as-a-Judge evaluation used the Claude API from a model family different from those of the generators to avoid self-preference bias (§ 3.5). BERTScore was computed using an NVIDIA RTX 4090 24GB with the google/mt5-small model as the underlying encoder (this model is used solely for the BERTScore computation). No truncation was required, as the longest (prompt + context) input was 48,568 tokens, as counted by tiktoken using the cl100k_base encoding.
Evaluation
N-gram based metrics. An -gram is a contiguous sequence of tokens that captures ordering and co-occurrence between consecutive words. BLEU [17] computes -gram precision between candidate and reference with a brevity penalty, ROUGE [12] measures the ROUGE-1 F1 score of unigram overlap between candidate and reference, and METEOR [11] combines unigram precision and recall with stemming and synonym matching.
Similarity-based metrics. While -gram metrics capture syntactic overlap, they often fail to recognize semantic equivalence when the same idea is expressed with different wording. BERTScore [26] addresses this by encoding candidate and reference texts into dense vector representations and computing cosine similarity in the embedding space. We compute BERTScore using the google/mt5-small encoder. We selected this encoder for two reasons: it shows a reasonable correlation with human judgments (0.64 on the WMT16 To-English task, as reported in the BERTScore model-selection sheet), and it supports a large input token limit, which helps accommodate longer docstrings without truncation.
LLM-as-a-Judge. Zheng et al. [27] proposed the LLM-as-a-Judge framework to address the scalability constraints of human annotation and the semantic limitations of -gram metrics. Among the evaluation protocols they establish, pairwise comparison has been shown to yield more stable and distinguishable results by grounding the evaluation in relative preference rather than absolute scoring.
Once the summaries are generated, we compute BLEU, ROUGE, METEOR, and BERTScore using the Evaluate library [23] and compare them against the original reference documentation. Before computing the metrics, we extract the docstring from each generated output and remove Markdown code fences, triple quotes, indentation, and surrounding whitespace from the generated and reference docstrings; each metric then uses its default tokenization. Statistical testing follows the procedure described in § 3.6.
To mitigate the limitations of -gram metrics and their potential bias toward verbose summaries, we complemented our evaluation with an LLM-as-a-Judge assessment. In this setup, a separate LLM acts as an automated evaluator: it receives a function’s source code, its surrounding context, and three candidate comments, then produces a strict ranking from best to worst. The three candidates are the reference docstring extracted from the repository, the DI-generated comment, and the DA-generated comment.
For each evaluation instance, the mapping between candidates and the labels A, B, C is determined by a random permutation, drawn once per function and context level to mitigate position bias [27]. The judge is prompted to return a JSON object containing a ranked list such as ["B","A","C"] and a short rationale. We extract the ranking from the response and convert it to numerical ranks (1=best, 3=worst) for each candidate. The prompt enforces strict ordering with no ties. It instructs the judge to treat the function body as primary evidence and the surrounding context only as secondary support for disambiguating intent, penalizing claims not supported by the code (the full judge prompt is in the replication package).
We adopted three evaluation criteria from prior work on LLM-based code summarization assessment: content adequacy, conciseness, and fluency and understandability. Crupi et al. [5] validated these criteria against human judgments for code summarization and code generation tasks, finding that GPT-4-based rankings under these criteria achieved high agreement with expert annotators. Sun et al. [22] independently confirmed that GPT-4 evaluation aligned most closely with human assessment among several automated metrics they tested. Adequacy measures whether the comment captures what the function does, conciseness penalizes unnecessary detail, and fluency assesses whether a developer can read the comment without difficulty. Together, they cover complementary dimensions of comment quality.
Self-preference bias refers to the tendency of an LLM judge to systematically favor outputs produced by its own model family over outputs of comparable quality from other models [4]. To mitigate this risk, we use claude-sonnet-4-6 as the judge, a model from a different provider and family than all three generators in our study. The judge was queried with a temperature of 0 to maximize reproducibility. We sampled 1000 instances for evaluation, stratified by reference docstring length into quartiles (250 instances per quartile) using a fixed random seed.
Human calibration of the judge. We calibrated the judge on a stratified sample of 20 tasks covering all six model and context-level instances, ranked blind by one author under the same criteria as the judge, with a second blinded author as an inter-rater baseline. The judge agreed with each author on the DA versus DI comparison underlying RQ2 in 65–70% of tasks (Cohen’s kappa 0.30–0.40) and matched the raters’ consensus choice in 11 of the 15 tasks on which they aligned (73%). Over the full rankings (60 rank assignments), the judge agreed with each rater (Krippendorff ordinal alpha 0.33 and 0.41) more than the raters agreed with each other (alpha 0.11), so the judge ranks candidate summaries at least as consistently as the raters themselves. This supports relying on the judge for the aggregate DA versus DI comparisons. The rating sheets are in the replication package.
Statistical Tests
We first test for normality using the D’Agostino-Pearson test. If normality is rejected, we apply a non-parametric Kruskal-Wallis test to assess whether each metric exhibits the same distribution across context levels and documentation scenarios, at a significance level of 0.05. Post-hoc pairwise comparisons use the Dunn-Bonferroni test, and we report effect sizes using Pearson’s , benchmarked with Cohen’s guidelines (r = .10 weak, .30 medium, .50 strong) following Kampenes et al. [10]. The statistics were calculated using IBM SPSS Statistics software [8].
Results
Overview
Table 3 summarizes the generated description lengths across all five experimental conditions. The median values range from 586 to 674 characters. Descriptions produced with broader context tend to be longer, with Module DA producing the highest median (674 characters).
| Condition | Median | Q1 | Q3 | IQR |
| Function-only (Baseline) | 586 | 334 | 960 | 626 |
| Class DI | 641 | 365 | 1009 | 644 |
| Class DA | 622 | 334 | 988 | 654 |
| Module DI | 662 | 367 | 1042 | 675 |
| Module DA | 674 | 362 | 1046 | 684 |
Table 3. Generated description length (characters) across experimental conditions, pooled over all models.
The median generated docstring length varies by model: Gemini produces the shortest outputs (median 312 characters at baseline), GPT-4o-mini produces mid-length outputs (median 586), and DeepSeek produces the longest (median 1029). The pooled median (586 characters) is roughly six times the median reference docstring (94 characters).
We report the metrics from comparing generated docstrings against the original references. Table 4 summarizes the pooled statistics (mean std) for each metric across all scenarios. Per-model results are available in the replication package; the pooled table highlights the overall effect across models.
| Metric | Function-only (Baseline) | Class DI | Class DA | Module DI | Module DA |
| BERTScore | 0.5566 ± 0.1157 | 0.5683 ± 0.1288 | 0.6182 ± 0.1669 | 0.5745 ± 0.1353 | 0.6124 ± 0.1651 |
| BLEU | 0.0611 ± 0.1393 | 0.0810 ± 0.1681 | 0.1738 ± 0.2771 | 0.0926 ± 0.1848 | 0.1664 ± 0.2696 |
| ROUGE | 0.2482 ± 0.1897 | 0.2649 ± 0.2111 | 0.3400 ± 0.2758 | 0.2731 ± 0.2211 | 0.3286 ± 0.2714 |
| METEOR | 0.2831 ± 0.1814 | 0.3058 ± 0.1993 | 0.4028 ± 0.2631 | 0.3176 ± 0.2098 | 0.3955 ± 0.2592 |
Table 4. Overall pooled performance across all generated instances (mean ± std) by context level and condition.
Across pooled results, adding hierarchical context improves scores relative to the function-only baseline. Documentation-Aware (DA) consistently outperforms Documentation-Independent (DI) at both class and module levels. The largest gains are observed for DA with class context, which achieves the highest pooled values across all four metrics. Class DA and Module DA produce similar scores for Gemini and GPT-4o-mini, where differences do not reach statistical significance. For DeepSeek, Class DA significantly exceeds Module DA on BERTScore, ROUGE, and METEOR.
Statistical Analysis
Normality is rejected for all metric distributions across all models and scenarios (D’Agostino-Pearson, ), so we apply the non-parametric tests described in § 3.6. Kruskal–Wallis tests showed significant differences across the five experimental conditions for every metric, as in Table 5. We follow up with a post-hoc Dunn-Bonferroni test to identify which of the groups had significant deviations, by computing the effect size statistic and then benchmark it using Cohen’s guidelines of Pearson’s (weak effect), (medium effect), and (strong effect), in line with existing literature in software engineering [10].
| Model | Metric | |||
| deepseek v3.2 | BERTScore | 226.354 | 0.006 | <.001 |
| deepseek v3.2 | BLEU | 677.710 | 0.019 | <.001 |
| deepseek v3.2 | ROUGE | 215.258 | 0.006 | <.001 |
| deepseek v3.2 | METEOR | 443.108 | 0.013 | <.001 |
| gemini-3-flash-preview | BERTScore | 2501.200 | 0.072 | <.001 |
| gemini-3-flash-preview | BLEU | 2587.016 | 0.075 | <.001 |
| gemini-3-flash-preview | ROUGE | 2174.908 | 0.063 | <.001 |
| gemini-3-flash-preview | METEOR | 2785.378 | 0.080 | <.001 |
| gpt-4o-mini | BERTScore | 428.818 | 0.012 | <.001 |
| gpt-4o-mini | BLEU | 1597.195 | 0.046 | <.001 |
| gpt-4o-mini | ROUGE | 197.029 | 0.006 | <.001 |
| gpt-4o-mini | METEOR | 1371.514 | 0.039 | <.001 |
Table 5. Kruskal–Wallis tests across experimental groups, reported by model. Effect size computed as with groups and per model.
Table 6 shows the effect sizes for DI metrics, per level of context and model. For DeepSeek and GPT-4o-mini, all effect sizes are negligible (). Gemini stands out: at the module level, Baseline vs. DI reaches weak effects (–), indicating that code structure alone produces measurable gains for this model. At the class level, Gemini DI effects remain negligible but approach the weak threshold (–).
| Comparison | Metric | Effect | |
| DeepSeek v3.2 Baseline vs. DI Class | BERTScore | 0.005 | Neg. |
| DeepSeek v3.2 Baseline vs. DI Class | BLEU | 0.041 | Neg. |
| DeepSeek v3.2 Baseline vs. DI Class | ROUGE | 0.012 | Neg. |
| DeepSeek v3.2 Baseline vs. DI Class | METEOR | 0.018 | Neg. |
| Baseline vs. DI Module | BERTScore | 0.008 | Neg. |
| Baseline vs. DI Module | BLEU | 0.058 | Neg. |
| Baseline vs. DI Module | ROUGE | 0.019 | Neg. |
| Baseline vs. DI Module | METEOR | 0.018 | Neg. |
| Gemini-3-Flash Baseline vs. DI Class | BERTScore | 0.097 | Neg. |
| Gemini-3-Flash Baseline vs. DI Class | BLEU | 0.098 | Neg. |
| Gemini-3-Flash Baseline vs. DI Class | ROUGE | 0.090 | Neg. |
| Gemini-3-Flash Baseline vs. DI Class | METEOR | 0.099 | Neg. |
| Baseline vs. DI Module | BERTScore | 0.145 | Weak |
| Baseline vs. DI Module | BLEU | 0.146 | Weak |
| Baseline vs. DI Module | ROUGE | 0.132 | Weak |
| Baseline vs. DI Module | METEOR | 0.150 | Weak |
| GPT-4o-Mini Baseline vs. DI Class | BERTScore | 0.008 | Neg. |
| GPT-4o-Mini Baseline vs. DI Class | BLEU | 0.050 | Neg. |
| GPT-4o-Mini Baseline vs. DI Class | ROUGE | 0.001 | Neg. |
| GPT-4o-Mini Baseline vs. DI Class | METEOR | 0.032 | Neg. |
| Baseline vs. DI Module | BERTScore | 0.013 | Neg. |
| Baseline vs. DI Module | BLEU | 0.068 | Neg. |
| Baseline vs. DI Module | ROUGE | 0.001 | Neg. |
| Baseline vs. DI Module | METEOR | 0.046 | Neg. |
Table 6. Function-only (Baseline) vs. Documentation-Independent (DI) effect sizes (, with ). Most contrasts are statistically significant (Dunn–Bonferroni adjusted ). All effect sizes are negligible () for DeepSeek and GPT-4o-mini, Gemini Module DI reaches the weak threshold. Effect labels: Neg. < .10, Weak .10–.30, Mod. .
Table 7 shows the effect sizes for DA per level of context. All Baseline vs. DA contrasts are statistically significant (). The magnitude varies by model. Gemini produces medium effect sizes across all four metrics (–), the largest observed in this study. GPT-4o-mini shows mostly weak effects (–), strongest on BLEU and METEOR. DeepSeek effects are mostly weak (–), with only BLEU and METEOR reaching the weak threshold at both context levels.
| Comparison | Metric | Effect | |
| DeepSeek v3.2 Baseline vs. DA Class | BERTScore | 0.100 | Weak |
| DeepSeek v3.2 Baseline vs. DA Class | BLEU | 0.178 | Weak |
| DeepSeek v3.2 Baseline vs. DA Class | ROUGE | 0.086 | Neg. |
| DeepSeek v3.2 Baseline vs. DA Class | METEOR | 0.142 | Weak |
| Baseline vs. DA Module | BERTScore | 0.069 | Neg. |
| Baseline vs. DA Module | BLEU | 0.162 | Weak |
| Baseline vs. DA Module | ROUGE | 0.042 | Neg. |
| Baseline vs. DA Module | METEOR | 0.106 | Weak |
| Gemini-3-Flash Baseline vs. DA Class | BERTScore | 0.345 | Mod. |
| Gemini-3-Flash Baseline vs. DA Class | BLEU | 0.347 | Mod. |
| Gemini-3-Flash Baseline vs. DA Class | ROUGE | 0.322 | Mod. |
| Gemini-3-Flash Baseline vs. DA Class | METEOR | 0.356 | Mod. |
| Baseline vs. DA Module | BERTScore | 0.330 | Mod. |
| Baseline vs. DA Module | BLEU | 0.339 | Mod. |
| Baseline vs. DA Module | ROUGE | 0.306 | Mod. |
| Baseline vs. DA Module | METEOR | 0.354 | Mod. |
| GPT-4o-Mini Baseline vs. DA Class | BERTScore | 0.126 | Weak |
| GPT-4o-Mini Baseline vs. DA Class | BLEU | 0.252 | Weak |
| GPT-4o-Mini Baseline vs. DA Class | ROUGE | 0.085 | Neg. |
| GPT-4o-Mini Baseline vs. DA Class | METEOR | 0.230 | Weak |
| Baseline vs. DA Module | BERTScore | 0.114 | Weak |
| Baseline vs. DA Module | BLEU | 0.255 | Weak |
| Baseline vs. DA Module | ROUGE | 0.066 | Neg. |
| Baseline vs. DA Module | METEOR | 0.224 | Weak |
Table 7. Function-only (Baseline) vs. Documentation-Aware (DA) effect sizes (, with ). All contrasts are statistically significant (). Effect labels: Neg. < .10, Weak .10–.30, Mod. .
Table 8 shows the effect sizes for comparisons between DI and DA at each level of context. All DI vs. DA contrasts are statistically significant (). Gemini shows the largest DI-to-DA gains, with class-level effects reaching – across all metrics. GPT-4o-mini shows weak effects on BLEU () and METEOR (), with negligible effects on ROUGE. DeepSeek effects are negligible to weak (–). These results confirm that the benefit of preserving documentation is model-dependent, with Gemini deriving the largest gains.
| Comparison | Metric | Effect | |
| DeepSeek v3.2 | |||
| DI Class vs. DA Class | BERTScore | 0.095 | Neg. |
| DI Class vs. DA Class | BLEU | 0.137 | Weak |
| DI Class vs. DA Class | ROUGE | 0.098 | Neg. |
| DI Class vs. DA Class | METEOR | 0.124 | Weak |
| DI Module vs. DA Module | BERTScore | 0.061 | Neg. |
| DI Module vs. DA Module | BLEU | 0.105 | Weak |
| DI Module vs. DA Module | ROUGE | 0.061 | Neg. |
| DI Module vs. DA Module | METEOR | 0.088 | Neg. |
| Gemini-3-Flash | |||
| DI Class vs. DA Class | BERTScore | 0.248 | Weak |
| DI Class vs. DA Class | BLEU | 0.249 | Weak |
| DI Class vs. DA Class | ROUGE | 0.232 | Weak |
| DI Class vs. DA Class | METEOR | 0.257 | Weak |
| DI Module vs. DA Module | BERTScore | 0.185 | Weak |
| DI Module vs. DA Module | BLEU | 0.193 | Weak |
| DI Module vs. DA Module | ROUGE | 0.175 | Weak |
| DI Module vs. DA Module | METEOR | 0.204 | Weak |
| GPT-4o-Mini | |||
| DI Class vs. DA Class | BERTScore | 0.118 | Weak |
| DI Class vs. DA Class | BLEU | 0.202 | Weak |
| DI Class vs. DA Class | ROUGE | 0.086 | Neg. |
| DI Class vs. DA Class | METEOR | 0.198 | Weak |
| DI Module vs. DA Module | BERTScore | 0.101 | Weak |
| DI Module vs. DA Module | BLEU | 0.187 | Weak |
| DI Module vs. DA Module | ROUGE | 0.067 | Neg. |
| DI Module vs. DA Module | METEOR | 0.178 | Weak |
Table 8. Documentation-Independent vs. Documentation-Aware (DI vs. DA) effect sizes for matched context levels (, with ). All shown DI vs. DA contrasts are statistically significant (). Effect labels: Neg. , Weak –, Mod. .
Returning to RQ1, expanding the hierarchical context under DA prompts produces statistically significant improvements across all four metrics and all three models. Under DI prompts, effect sizes remain negligible for DeepSeek and GPT-4o-mini () but reach weak levels for Gemini at module scope (–). Under DA prompts, the effect is model-dependent: Gemini produces medium effect sizes across all four metrics (–), GPT-4o-mini produces mostly weak effects (–), and DeepSeek produces mostly weak effects (–). Broader context increases similarity to developer-written references, with the practical magnitude ranging from mostly weak (DeepSeek) to medium (Gemini).
For RQ2, Documentation-Aware outperforms Documentation-Independent prompts at matched context levels, but the magnitude is model-dependent. Gemini shows the largest DI-to-DA effect sizes (–), GPT-4o-mini shows weak effects on BLEU and METEOR (–), and DeepSeek shows negligible to weak effects (–). The LLM-as-a-Judge evaluation confirms the direction for every model: DA is significantly preferred over DI at all model and context-level combinations (–, ), with the judge effect sizes mostly weak even where the metric effects are medium. Preserving surrounding documentation provides a consistent benefit on both similarity metrics and judge preference.
LLM-as-a-Judge Analysis
Table 9 reports win rates from the Claude Sonnet 4.6 judge. The developer-written reference achieves the highest win rate for GPT-4o-mini (51.5–53.4%) and DeepSeek (42.4–47.1%), while Gemini references win under 10% of their comparisons. Between the two generated candidates, DA beats DI consistently, confirmed by Wilcoxon signed-rank tests for every model and context level. For GPT-4o-mini, DA beats DI in 56.8% of class comparisons (, ) and 56.1% of module comparisons (, ). For DeepSeek, DA beats DI in 57.3% of class (, ) and 53.6% of module comparisons (, ). For Gemini, DA beats DI in 52.0% of class (, ) and 53.6% of module comparisons (, ). Figure 4 shows one complete evaluation.
| Model | Context | Ref | DA | DI | ||
| DeepSeek v3.2 | Class | 42.4 | 31.0 | 26.6 | <.0001 | 0.159 |
| DeepSeek v3.2 | Module | 47.1 | 27.5 | 25.4 | .0162 | 0.077 |
| gemini-3-flash-preview | Class | 8.1 | 47.3 | 44.6 | .0004 | 0.113 |
| gemini-3-flash-preview | Module | 9.6 | 47.6 | 42.8 | .0006 | 0.109 |
| gpt-4o-mini | Class | 51.5 | 28.3 | 20.2 | <.0001 | 0.134 |
| gpt-4o-mini | Module | 53.4 | 27.0 | 19.6 | .0006 | 0.109 |
Table 9. Win rates (%) and Wilcoxon signed-rank test results by judged model and context (– valid rankings per row, Claude Sonnet 4.6 judge). Highlighted .

Figure 4. A complete judge evaluation (DeepSeek candidates, module context). The DA summary states the fallback role of the property, a fact evidenced only by the text property elsewhere in the module. The DI summary compensates with speculation the judge penalizes as unsupported. Candidate texts are abbreviated with [...] and available in full in the replication package.
We also find a statistically significant association between the length of the reference docstring and the reference win rate: as reference docstrings get longer, they are more likely to be ranked first. Figure 3 shows histograms of reference docstring lengths for cases where the reference is ranked first (green) versus ranked second/third (red). Table 10 reports the confirming Mann–Whitney tests with length summary statistics. The judge does not simply prefer longer text. Generated candidates are about six times longer than references. However, the judge prefers references more as they get longer. This is the opposite of what would happen if the judge liked long text. The judge’s reasons focus on information, not length. For example, the judge ranked a 526-character reference first because it “correctly notes that the function does nothing” when both label positions are equally free. Both generated candidates missed this edge case. Longer references win because they contain more of the information the judge verifies against the code.

Figure 3. Reference vs LLM-generated comment wins by comment size
| Model | Context | Ref Rank1 (median±std) | ||
| GPT-4o-mini | Class | , 157 ± 365 | <.0001 | −0.284 |
| GPT-4o-mini | Module | , 157 ± 357 | <.0001 | −0.288 |
| gemini-3-flash-preview | Class | , 316 ± 508 | <.0001 | −0.207 |
| gemini-3-flash-preview | Module | , 257 ± 464 | <.0001 | −0.194 |
| DeepSeek v3.2 | Class | , 162 ± 354 | <.0001 | −0.250 |
| DeepSeek v3.2 | Module | , 166 ± 364 | <.0001 | −0.281 |
Table 10. Mann–Whitney U test for reference docstring length when reference is ranked first, by judged model and context. Values are reported as , median±std. Highlighted .
For RQ3, all three models produce higher similarity scores with a broader context, but the magnitude varies substantially. Gemini shows the strongest response to hierarchical context, with medium effect sizes for Baseline vs. DA (–). GPT-4o-mini produces mostly weak effects (–), and DeepSeek produces mostly weak effects (–). The judge evaluation confirms
Function (requests, Response.apparent_encoding) @property def apparent_encoding(self): return chardet.detect(self.content)[’encoding’]
Reference (ranked 2nd): "The apparent encoding, provided by the chardet library"
DA, module (ranked 1st): "The apparent encoding, guessed by using the chardet library. The encoding is detected by analyzing the response content (bytes). This is used as a fallback when the ’encoding’ attribute is None. Returns: str or None [...]"
DI, module (ranked 3rd): "The apparent encoding of the response content, as detected by chardet. [...] It is particularly useful when the server does not specify an encoding in the Content-Type header [...] For large responses, consider alternative approaches to avoid loading the entire content into memory. [...]"
Judge rationale: the DA summary "accurately captures the core behavior [...] the return type (str or None), and the fallback usage seen in [the] text property, without adding unsupported claims". The DI summary "is penalized for verbosity and for including unsupported details not evidenced by the Function". The reference "is minimal but correct and concise".
the DA preference for all three models (–), and the models differ sharply in how their outputs fare against the reference: GPT-4o-mini and DeepSeek references win 42–53% of comparisons while Gemini references win under 10%. Model choice moderates both the size of the context benefit and the degree to which surrounding documentation contributes. The effect is model-dependent in magnitude, ranging from mostly weak (DeepSeek and GPT-4o-mini) to medium (Gemini).
The effects of DA extend to repositories with a lower representation in the sample (astropy, matplotlib, and sympy account for 69.2% of the rows). On BERTScore, the DA median exceeds the DI median in 12 of 12 repositories for GPT-4o-mini and DeepSeek, and in 11 of 12 for Gemini (only xarray reverses, by 0.0011). Macro-averaging the per-repository DA–DI differences, which weights each repository equally, preserves the advantage for all three models (BERTScore +0.011 for GPT-4o-mini, +0.064 for Gemini, +0.017 for DeepSeek). The macro-average over the nine non-dominant repositories alone remains positive (+0.010, +0.049, +0.017). The replication package contains DA and DI results per repository.
Discussion
Comparison With The Literature
Our results align with and quantify the observations of [14]: DA prompts consistently outperform DI prompts on similarity metrics, class-level DA context yields the highest pooled values, and the magnitude of improvement is mixed, suggesting that models differ in their ability to exploit hierarchical context and surrounding documentation. Our findings also offer a new perspective on the work of [7], who reported that LLMs often rely more on function names than the actual code structure. We observed that, with a sufficiently broad view, such as that provided by module-level context, the model can leverage the surrounding code structure and existing documentation to generate a summary closer to the developer’s intent. This effect is strongest for Gemini, which shows the largest sensitivity to hierarchical context among the three models tested.
Implications For Research And Practice
-gram metrics such as BLEU and ROUGE are sensitive to length differences between generated and reference text. By not enforcing a specific docstring style, we reduce this mismatch and allow the metrics to better reflect semantic alignment. We further triangulate with BERTScore and an LLM-as-a-Judge evaluation, following Crupi et al. [5], who found lower correlation between human judgment and -gram metrics for code summarization.
The judge evaluation shows DA significantly preferred over DI for all three models, but with mostly weak effect sizes (–), even for Gemini, whose metric effects are medium (–). This gap between metric and judge effects calls for explanation. Copying does not provide one, since Gemini’s DA outputs show the lowest lexical overlap with the surrounding documentation of the three models (Jaccard 0.046 DA vs. 0.042 DI). A more consistent explanation is stylistic alignment. Gemini adopts the documentation conventions of the surrounding scope, which raises similarity to references written under those same conventions by more than it raises the informational quality the judge rewards. Output style points the same way. Gemini produces the shortest summaries (median 312 characters at baseline), leaving the most headroom to absorb reference-like phrasing from context, while DeepSeek’s verbose outputs (median 1,029) dilute overlap with any single source, consistent with its mostly weak effects.
Figure 4 illustrates what surrounding documentation contributes and what the judge rewards. The DA summary states the fallback role of the property, a fact evidenced only by the text property elsewhere in the module, and ranks first. The DI summary, generated without documentation cues, compensates with speculation about protocol headers and memory usage that the judge penalizes as unsupported, and the terse reference ranks second. The example matches the aggregate rationale statistics. The judge cites accuracy or correctness in 97% of rationales, penalizes verbose or redundant summaries in 54%, and praises detail or comprehensiveness in under 2%. The DA advantage appears across the repositories and across both short and long summaries.
We also observe a weak but consistent dependence on reference comment length. As references become longer, they are increasingly ranked first by the judge, so improvements from context augmentation are most visible when the existing reference is under-specified rather than already comprehensive. For practitioners and tool developers, prompts should include broader hierarchical context and can benefit from retaining nearby documentation. Adding context increases average prompt size from 277 tokens for the Function-only baseline to 4,506 for Class DA and 13,233 for Module DA.
Risks And Mitigation
Generated docstrings can be plausible but incorrect (wrong parameter meanings, side effects, exceptions, or pre- and post-conditions). If adopted without review, such documentation may increase defects, maintenance costs, or security vulnerabilities by encouraging incorrect API usage. We therefore encourage human-in-the-loop review: treat generated docstrings as drafts requiring developer validation, especially for public APIs, security-sensitive code, or code with complex side effects.
Threats To Validity
Drawing on Maxwell’s worldview perspective [16] and Petersen et al. [18], which include positivist aspects, we detail the key threats to validity identified in our study and the mitigation strategies considered.
Theoretical Validity
The metrics used in this study serve the key purpose of indicating developer intent. Although these metrics are widely used in the literature, -gram metrics such as BLEU are sensitive to length differences between generated and reference docstrings. To mitigate this threat, we complement -gram metrics with BERTScore for semantic similarity and an LLM-as-a-Judge evaluation for holistic quality assessment, supported by findings from Crupi et al. [5].
A related concern is that the DA advantage reflects summary length rather than context use. Length correlates negatively with every metric (Spearman rho between -0.10 and -0.34), so longer summaries do not score higher, and DA still exceeds DI within every length quartile. The most concise model, Gemini, also shows the largest effect, so the result is not due to any single length-sensitive measure.
Because the SWE-Bench snapshot predates every model’s knowledge cutoff, the entire sample may have been seen during pre-training. Our design controls for this. Each of the 6,932 functions appears in all five conditions, so memorized content contributes equally to every condition and cannot produce the condition differences our conclusions point to. Three observations argue against recall as the driver. Generated summaries are roughly six times longer than their references rather than reproductions of them. DA summaries are barely more lexically similar to the surrounding documentation than DI summaries. The DA advantage persists in the repositories least represented in our sample (Section 4.3). A direct test would contrast functions committed after the models’ knowledge cutoffs with pre-cutoff functions, which requires code newer than the pinned SWE-Bench snapshots. We leave it to future work.
Under DA, a model might paraphrase neighboring docstrings instead of reasoning over the code. On 400 sampled class-level targets per model, DA summaries are only marginally more similar to the surrounding documentation than DI summaries (Jaccard 0.065 vs. 0.058; DA higher in 59% of pairs), and the shared tokens are mostly generic API vocabulary, so the DA improvement does not come from copying the surrounding documentation.
All judge-based results are based on a single judge model. A judge outside the generators’ families avoids intra-family self-preference but has its own preferences. We mitigated this risk by running a blinded 20-task calibration (Section 3.5) which shows the judge agreeing with a human rater at the level the raters agree with each other, supporting the aggregate comparisons.
External Generalizability
The 12 repositories are Python projects, so our findings may not generalize to languages with different documentation conventions. Within the sample, three repositories (astropy, matplotlib, and sympy) contribute 69.2% of the filtered functions. The repository-balanced analysis in Section 4.3 shows the DA advantage does not depend on this imbalance, but generalization beyond these domains remains untested. Our claims are likewise scoped to the three evaluated model families. The negligible-to-medium spread in effect sizes across them cautions against extrapolating to other families, including smaller open-weight and code-specialized models.
Descriptive Validity
Future retraining of the LLM with a current knowledge cutoff may affect experimental outcomes, as earlier revisions are not accessible via the current API. As an inherent consequence of working with closed LLMs offered as a service, this limitation undermines the reproducibility of results over time. To mitigate this threat, we employed the open-source model provider Deepseek in our analysis and noted the knowledge cutoff dates for each model, allowing for long-term replicability of results.
Conclusion
Generated docstrings in both Documentation-Independent (DI) and Documentation-Aware (DA) scenarios are longer than their references, and pooled BERTScore, BLEU, METEOR, and ROUGE values peak under class-level DA context. Kruskal–Wallis tests confirm that metric distributions differ across conditions for all three models, and Dunn–Bonferroni post-hoc tests show that most pairwise contrasts are statistically significant. The effect size varies by model, with Gemini showing medium effects for Baseline vs. DA (r = 0.31–0.36), GPT-4o-mini mostly weak effects (r = 0.07–0.26), and DeepSeek mostly weak effects (r = 0.04–0.18). The benefit of hierarchical context and surrounding documentation is consistent in direction but model-dependent in magnitude. Gemini produced summaries closest to the developer-written references, achieving the highest mean scores on all four metrics in every condition.
We triangulated our findings with a Claude Sonnet 4.6 LLM-as-a-Judge ranking on 1000 stratified samples per model using adequacy, conciseness, and fluency criteria. DA outputs are significantly preferred over DI for all three models, with mostly weak effect sizes. Longer developer-written references are more likely to be ranked first, suggesting that the headroom for LLM improvements depends on the completeness of the existing documentation.
Several directions remain for future work. The repositories used originate from SWE-bench [9] and are publicly available, so the evaluated models may have encountered them during pre-training. Comparing performance on functions present in training corpora against those absent, for example functions committed after March 2025, the most recent knowledge cutoff among the evaluated models, would separate contextual reasoning from memorization. This requires sampling commits newer than the pinned SWE-Bench snapshots. Repeating the study with smaller open-weight models (7B and 13B parameters) would clarify whether context sensitivity generalizes across model sizes [22]. Extending the hierarchy to repository-wide context [13, 14] would test whether the modest effect sizes observed at the module level grow when cross-module dependencies enter the prompt. Applying the study to languages beyond Python would test generalizability across different documentation conventions and structural norms.
Artifact Availability
Code, prompts and data supporting this study are openly available at: https://doi.org/10.6084/m9.figshare.31046290.v2, under the MIT (code and prompts) and CC BY 4.0 (data) licenses.
References
- [1]Toufique Ahmed, Kunal Suresh Pai, Prekumnar Devanbu, and Earl T. Barr. 2024. Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). In Proceedings of the 46th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, Article 220, 13 pages. doi:10.1145/3597503.3639183DOI
- [2]Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. code2vec: learning distributed representations of code. Proc. ACM Program. Lang. 3, POPL, Article 40 (Jan. 2019), 29 pages. doi:10.1145/3290353DOI
- [3]Sebastian Baltes, Florian Angerer, Chetan Arora, Marvin Muñoz Barón, Chun-yang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucì, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Rignard van Tonder, Christoph Treude, and Stefan Wagner. 2025. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. arXiv:2508.15503 [cs.SE] https://arxiv.org/abs/2508.15503
- [4]Zhi-Yuan Chen, Hao Wang, Xinyu Zhang, Ennui Hu, and Yankai Lin. 2025. Beyond the Surface: Measuring Self-Preference in LLM Judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tammy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 1653–1672. doi:10.18653/v1/2025.emnlp-main.86
- [5]Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-a-Judge for Code Generation and Summarization. IEEE Transactions on Software Engineering 51, 8 (2025), 2329–2345. doi:10.1109/TSE.2025.3586082DOI
- [6]Zhangyin Feng, Dayu Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 1536–1547. doi:10.18653/v1/2020.findings-emnlp.139
- [7]Rajrashi Haldar and Julia Hockenmaier. 2024. Analyzing the Performance of Large Language Models on Code Summarization. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 995–1008. https://aclanthology.org/2024.lrec-main.89/
- [8]IBM Corp. 2025. IBM SPSS Statistics. https://www.ibm.com/products/spss-statistics
- [9]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770 [cs.CL] https://arxiv.org/abs/2310.06770
- [10]Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg. 2007. A systematic review of effect size in software engineering experiments. Information and Software Technology 49, 11-12 (2007), 1073–1086.
- [11]Alon Lavie and Abhaya Agarwal. 2007. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (Prague, Czech Republic) (StatMT ’07). Association for Computational Linguistics, USA, 228–231.DOI
- [12]Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/
- [13]Vadim Lomashkov, Andrey Podivilov, Sergey Savin, Oleg Baryshnikov, Alena Lisevych, and Sergey I Nikolenko. 2024. ProConSuL: Project Context for Code Summarization with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics, 866–880. doi:10.18653/v1/2024.emnlp-industry.65
- [14]Vladimir Makharev and Vladimir Ivanov. 2025. Code Summarization Beyond Function Level. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). IEEE Computer Society, Los Alamitos, CA, USA, 153–160. doi:10.1109/LLM4Code66737.2025.00024DOI
- [15]Alberto Martin-Lopez, Rosalia Tufano, Emanuela Guglielmi, Ana B Sánchez, Ana Lourdes Sanz, Simone Scalabrino, Rocco Oliveto, Sergio Segura, and Gabriele Bavota. 2026. More Code, Less Understanding? On the Impact of AI Assistants on Developers’ Productivity and Code Ownership. IEEE Transactions on Software Engineering (2026).
- [16]J. A. Maxwell. 1992. Understanding and validity in qualitative research. Harvard Educational Review 62, 3 (1992), 279–301.
- [17]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL ’02). Association for Computational Linguistics, USA, 311–318. doi:10.3115/1073083.1073135DOI
- [18]Kai Petersen and Cigdem Gencel. 2013. Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research. In 2013 Joint Conference of the 23rd International Workshop on Software Measurement (IWSM) and the Eighth International Conference on Software Process and Product Measurement (Mensura). IEEE Computer Society, USA, 81–89. Kai Petersen: School of Computing, Blekinge Institute of Technology, Karlskrona, Sweden; Cigdem Gencel: Faculty of Computer Science, Free University of Bolzano/Bozen, Italy.DOI
- [19]Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyai Yang. 2023. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houdai Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 1339–1384. doi:10.18653/v1/2023.emnlp-main.85DOI
- [20]Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, Breno Bernard Nicolau de França, Carlo Alberto Furia, Greg Gay, Nicolas Gold, Daniel Grazotin, Pinja He, Rashina Hoda, Natalia Juristo, Barbara Kitchenham, Valentina Lenarduzzi, Jorge Martínez, Jorge Melegati, Daniel Mendez, Tim Menzies, Jefferson Moller, Dietmar Pfahl, Romain Robbes, Daniel Russo, Nyyti Saarimäki, Federica Sarro, Janet Siegmund, Diomidis Spinellis, Miroslaw Staron, Klaas Stol, Margaret-Anne Storey, Davide Taibi, Damian Tamburri, Marco Torchiano, Christoph Treude, Burak Turhan, Xiaofeng Wang, and Sira Vegas. 2020. Empirical Standards for Software Engineering Research. arXiv:2010.03525 [cs.SE] https://arxiv.org/abs/2010.03525
- [21]Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K. Vijay-Shanker. 2010. Towards automatically generating summary comments for Java methods. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering (Antwerp, Belgium) (ASE ’10). Association for Computing Machinery, New York, NY, USA, 43–52. doi:10.1145/1858996.1859006
- [22]Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2025. Source Code Summarization in the Era of Large Language Models. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE ’25). IEEE Press, USA, 1882–1894. doi:10.1109/ICSE55347.2025.00034
- [23]Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. 2022. Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Wanxiang Che and Ekaterina Shutova (Eds.). Association for Computational Linguistics, Abu Dhabi, UAE, 128–136. doi:10.18653/v1/2022.emnlp-demos.13DOI
- [24]Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2017. Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering PP (07 2017), 1–1. doi:10.1109/TSE.2017.2734091DOI
- [25]Chunyuan Zhang, Junchao Wang, Qinglei Zhou, Ting Xu, Ke Tang, Hairen Gui, and Fundong Liu. 2022. A Survey of Automatic Source Code Summarization. Symmetry 14, 3 (2022). doi:10.3390/sym14030471
- [26]Tianyi Zhang*, Varsha Kishore*, Felix Wu*, K!ian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFdrarxiv.org/abs/1904.09675
- [27]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2020, 29 pages.arxiv.org/abs/2306.05685
- [28]Ziyi Zhou, Huiqun Yu, Guisheng Fan, Zijie Huang, and Xingguang Yang. 2022. Summarizing source code with hierarchical code representation. Information and Software Technology 143 (2022), 106761. doi:10.1016/j.infsof.2021.106761DOI