betaFeedback

WorksComputer Science

Think Before You Code: A Benchmark for Testing LLMs on Software Engineering Deliberation

Abstract

Think Before You Code (TBYC) is a benchmark pipeline for studying whether code-focused language models can reason about software issues in ways that resemble human engineering deliberation. The benchmark curates GitHub issue discussions into structured records, extracts human discussion artifacts such as problems, solutions, constraints, rationales, and trade-offs, and then prompts models to generate issue-level "thoughts" using retrieved code context and few-shot examples. It converts model responses back into the same artifact space and evaluates them against human annotations using complementary metrics for artifact type detection, metadata matching, tag overlap, and semantic summary similarity. Beyond scoring, this study includes visualization, leaderboard, and baseline tools that make it possible to compare models, prompting strategies, and extraction methods across repositories and issues. Think Before You Code is an end-to-end evaluation environment for measuring how closely LLM software reasoning aligns with human strategic judgment.

CODE FOR THE PROJECT CAN BE FOUND HERE – ashman17/ThinkBeforeYouCode

Introduction

This project addresses the use case of evaluating whether code LLMs can do more than generating plausible code and instead reason through software engineering problems the way human developers do. In real development, strong performance depends on understanding bug reports, surfacing constraints, weighing trade-offs, asking clarifying questions, using code context, and proposing grounded next steps. Think Before You Code (TBYC) turns that broader reasoning process into something measurable by building a pipeline from GitHub issue discussions to structured human reasoning artifacts, then comparing model behavior against those artifacts. The intended downstream users are researchers and practitioners who want to benchmark coding models not just on final answers, but on the quality of their intermediate engineering judgment.

The main research questions are (modified from the proposal):

  1. Can code LLMs reconstruct the kinds of reasoning humans express in issue discussions?

  2. How much does retrieved code context help compared with no-context, BM25-only, or few-shot baselines?

  3. Which parts of human reasoning are easiest or hardest for models to match, such as identifying artifact types, preserving metadata, assigning tags, or producing semantically aligned summaries?

This research project studies these questions by extracting human artifacts from discussions, generating model “issue thoughts,” re-extracting comparable artifacts from model outputs, and scoring agreement with a suite of metrics and visualizations.

Users interact with the application primarily through the CLI pipeline. They can fetch and normalize GitHub issues, extract human discussion artifacts, retrieve relevant code chunks, run different reasoning-generation settings, derive comparable artifacts from model outputs, compute metrics, and generate graphs, leaderboards, and an HTML viewer. In that sense, the application behaves like an end-to-end research workbench rather than a consumer-facing product: the user supplies repositories and models, and the system produces structured benchmark outputs for analysis and comparison.

The project engages with course material in several direct ways. It connects to code-writing assistants by focusing on software reasoning rather than raw code completion, and it uses the science of prompting through structured prompts for both artifact extraction and issue-thought generation. It draws on learning representations and embeddings, retrieval, and retrieval-augmented generation through CodeBERT embeddings, BM25, FAISS indexing, and hybrid code-context retrieval. It also engages LLMs for evaluation by using model outputs as analyzable artifacts and scoring them against human annotations, while raising questions related to deployment and harms - a coding assistant that sounds confident but misses human constraints or trade-offs can still be unreliable in practice.

Literature Review

There is not yet a benchmark that directly evaluates whether LLMs can reason through real GitHub issues in terms of design choices, feasibility, prioritization, and trade-offs, but several related efforts study code generation, repository-level reasoning, and software engineering performance.

Code Generation

SWE-Bench (Jimenez et al. [ [9]) evaluates models on real GitHub issues paired with gold patches and test suites, measuring whether models can resolve issues through execution-grounded patch generation. It is highly realistic, but focuses on patch correctness rather than the quality of engineering deliberation. LiveCodeBench (Jain et al. [ [7]) provides a contamination-resistant and continuously updated benchmark for code generation, self-repair, and execution on recent programming tasks. Its emphasis is on robustness and iterative problem solving, not human-like reasoning about software decisions. HumanEval and Codex (Chen et al. [ [2]) established execution-based code evaluation through hidden unit tests and introduced pass@kk as a standard metric for code synthesis. The benchmark measures whether generated code works, but not whether the reasoning behind it reflects human engineering judgment. The survey in Jiang et al. [ [8] reviews major techniques in code LLMs, including instruction tuning, retrieval augmentation, tool use, and RLHF, while also summarizing common evaluation strategies. It highlights that most code benchmarks prioritize functional correctness over higher-level reasoning such as prioritization, feasibility, and trade-off analysis.

Code Reasoning

RepoCoder (Zhang et al. [ [13]) introduces iterative retrieval-augmented generation for repository-level code completion and shows that cross-file context improves performance. Its evaluation centers on completion quality rather than reasoning over issue discussions. CrossCodeEval (Ding et al. [ [4]) studies cross-file code completion across multiple programming languages, requiring models to reason over inter-file dependencies. It captures repository structure well, but still evaluates completion accuracy rather than deliberative engineering behavior. CRUXEval (Gu et al. [ [6]) focuses on reasoning about existing code by testing models on execution behavior and program semantics. This is closer to code understanding, but it does not evaluate reasoning about collaborative software discussions. CodeT ((Chen et al. [2022]) improves code generation by jointly generating code and tests, then selecting candidates through verification. It shows the value of reasoning with validation, though the target remains final solution correctness. CodeXGLUE (Lu et al. [ [10]) is a broad benchmark suite covering code understanding and generation tasks such as completion, translation, summarization, and defect detection. It demonstrates the breadth of code intelligence tasks, but not issue-level engineering deliberation. SWE-agent (Xia et al. [ [12]) studies agentic systems that interact with repositories and tools to resolve GitHub issues. It moves closer to realistic software workflows, but still evaluates end-to-end task completion more than human-like intermediate reasoning.

Industry Efforts

OpenAI Codex (OpenAI [ [11]) helped popularize natural-language-to-code generation and demonstrated the practical value of large code models. Its framing is centered on synthesis capability and productivity rather than explicit benchmarking of engineering judgment. GitHub Copilot (GitHub [ [5]) brings code completion and chat-based coding assistance into everyday developer workflows. It is an important practical system, but public evaluation focuses more on usefulness and productivity than on reasoning quality. Devin (Cognition [2024]) represents the shift toward autonomous software agents that can plan, code, debug, and iterate across tools. However, public descriptions emphasize task completion and autonomy claims rather than alignment with human software reasoning.

Research Gap

Across these works, the dominant evaluation targets are code correctness, repository-aware completion, and end-to-end software task performance. What remains underexplored is whether models can reason like human engineers before implementation by surfacing trade-offs, feasibility limits, blockers, uncertainty, and prioritization. This is an important capability because LLMs should not only be able to “reproduce” the known solutions but be able to “produce” these solutions. This will help LLMs solve unsolved problems.

The benchmark addresses this gap by shifting evaluation from whether code works to whether the model’s reasoning aligns with human engineering deliberation in real GitHub issue discussions. This makes it complementary to existing code-generation and software-engineering benchmarks rather than a replacement for them.

Method

The method evaluates whether code-oriented LLMs can reason about real software engineering issues in ways that align with human discussion, rather than only producing correct code. The full pipeline is organized into six (Figure 1): (1) issue and repository collection, (2) gold artifact extraction from human discussions, (3) retrieval-augmented context construction, (4) LLM inference for issue-level reasoning, (5) artifact extraction from LLM outputs, and (6) evaluation of alignment between gold and model-generated artifacts. The overall design converts unstructured issue discussions into a structured representation of engineering reasoning, then measures how closely LLM-generated reasoning matches that representation.

Phase-by-phase breakdown of the Think Before You Code benchmark

Figure 1. Phase-by-phase breakdown of the Think Before You Code benchmark.

Phase 1: Repository and Issue Curation

The pipeline begins by selecting public GitHub repositories with active maintainers, reputational significance, and sufficiently rich issue discussions. To maximize the value of human deliberation signal, issues with dense commentary, diversity in issue types, and closed status are prioritized so that each discussion is complete. Recent issues are also emphasized to reduce the risk of benchmark contamination from model pretraining data.

For each repository, the GitHub GraphQL API is used to collect issue-centered snapshots. Each snapshot includes the issue title and body, comment thread, labels, state, author metadata, and timeline events. These raw payloads are stored for auditability and future reprocessing. The raw snapshots are then normalized into curated issue records containing the input vector (title and body), the deliberation thread, file references mentioned in the discussion, taxonomic metadata, timeline events, and actor typology. The current curated dataset in the repository contains seven repositories and ≈1000\approx1000 issues in total as shown in Table 1. This construction yields a multi-repository benchmark over real issue threads rather than synthetic problem statements.

RepositoryCategoryIssuesWhy It’s Important
BitcoinDistributed Systems120Deep discussions on consensus, security, decentralization, and protocol-level trade-offs often with rigorous reasoning and long-term design implications
KubernetesSystems81Rich discussions on scalability, reliability, and distributed system trade-offs
RustProg. Language92Extremely structured design debates with clear alternatives and rationales
PyTorchML Framework76Frequent discussions on performance vs usability and API evolution
BazelDeveloper Tools97Deep reasoning about dependency management and build performance
FastAPIWeb Framework57Clean, well-structured discussions led by maintainers—great for annotation
Hugging FaceML / LLM43Rapid iteration with strong focus on API design and usability trade-offs
PostgreSQLDatabase75High-quality discussions on correctness, indexing, and query optimization
ReactUI Framework129Focus on API ergonomics and backward compatibility decisions
RedisDatabase89Clear trade-offs between latency, memory usage, and data structures

Table 1. Target repositories along with number of issues parsed for each for curating the dataset.

Phase 2: Gold Artifact Extraction from Human Discussion

The second phase converts human issue discussions into structured reasoning artifacts. The goal is to encode the latent engineering content of a discussion in a machine-comparable schema. An artifact schema is defined with four required fields: type, summary, tags, and metadata. The current implementation supports 31 artifact types, including problem_statement, proposed_solution, trade_off_argument, rationale, constraint, question, task_assignment, status_update, blocking_issue, and dependency. Each type has a fixed metadata schema, such as problem_kind, affected_area, and user_or_system_impact for problem statements, or options_compared, comparison_axes, and preferred_option for trade-off arguments. Refer to the Appendix (Table 6) for complete list with descriptions.

Tags provide finer-grained labels that capture bug classes, platform details, infrastructure signals, and software domains. The current schema contains 81 allowed tags (Appendix-§ 9 for complete list), including labels such as regression, flaky_test, runtime_error, performance_issue, compatibility_issue, linux, macos, docker, frontend, backend, rpc, and fix_proposed. Each artifact also includes a short summary that must state the subject, event, and key technical detail from the underlying comment.

To extract these artifacts from human discussion, a structured extraction pipeline is run over individual comments. The extraction prompt (Appendix-A) instructs the model to return strict JSON only, choose exactly one of the allowed artifact types, provide one or more supported tags from the controlled vocabulary, and fill in the fixed metadata keys for that type without null values or extra fields. The prompt is added to the Appendix for reference. The implementation uses Pydantic schemas together with structured decoding through Outlines/Ollama to enforce well-formed JSON output. A significant limitation of this method is that the artifacts can have errors due to the LLM annotation design. There is no automatic method that can eliminate the need of an LLM. A smaller human verified dataset will be an ideal solution and is part of the future works. For now, to combat the problem of LLM bias, artifacts from multiple open-source models (Gemma3, Gemma4, Llama) are combined to increase diversity and recall while reducing bias. These open-source models do not compete in the the final experiments and leaderboard.

Phase 3: Retrieval-Augmented Context Construction

The third phase builds repository context for issue-conditioned reasoning. The core question in this phase is what information a model should see when asked to reason like a human engineer. The benchmark intentionally provides the issue body and retrieved code context, but not the original human comments, since the model’s task is to independently reconstruct human-like deliberation rather than paraphrase the discussion.

To supply code context, the pipeline first resolves the repository snapshot corresponding to the latest default-branch commit before the earliest issue timestamp in the curated set. In the repository snapshot, non-source files are pruned and source files are split into code chunks, ideally at the symbol level and otherwise in fixed-size windows. Each issue query is formed from the issue title and body.

Overview of artifact extraction pipeline

Figure 2. Overview of artifact extraction pipeline. The input is batched into chunks and passed through multiple open-source models (Gemma and Llama family) to produce the extractions which undergo a schema matching to yield the final extracted artifacts.

Retrieval uses a hybrid sparse–dense architecture (ablation in results). For sparse retrieval, each chunk is indexed with BM25. Given issue query qq and chunk dd, the Okapi BM25 score is:

sbm25(d,q)=∑t∈qIDF⁡(t)⋅f(t,d)(k1+1)f(t,d)+k1(1−b+b∣d∣avgdl⁡)s_{\mathrm{bm25}}(d,q) = \sum_{t \in q} \operatorname{IDF}(t) \cdot\frac{f(t,d)(k_1+1)}{f(t,d)+k_1\left(1-b+b\frac{\lvert d\rvert}{\operatorname{avgdl}}\right)}

This component is especially useful for exact matches on identifiers, file names, APIs, and issue-specific terminology. For dense retrieval, both queries and code chunks are encoded with a CodeBERT-family encoder. After mean pooling and L2 normalization, chunk relevance is scored by cosine similarity:

sdense(d,q)=⟨e^(d),e^(q)⟩=cos⁡(e^(d),e^(q)).s_{\mathrm{dense}}(d,q) = \langle\hat{e}(d),\hat{e}(q)\rangle= \cos\left(\hat{e}(d),\hat{e}(q)\right).

This branch helps when lexical overlap is weak but semantic intent is similar. The BM25 and dense rankings are fused using Reciprocal Rank Fusion (RRF):

RRF⁡(d)=∑r∈{bm25,dense}1k+rank⁡r(d).\operatorname{RRF}(d) = \sum_{r \in\{\mathrm{bm25},\mathrm{dense}\}} \frac{1}{k+\operatorname{rank}_{r}(d)}.

The implementation uses k=60k = 60 by default. This hybrid strategy combines lexical precision, semantic generalization, and rank-level robustness. The top retrieved chunks are then formatted as prompt context in the form [file:start–end (symbol)] followed by chunk text.

Phase 4: LLM Inference for Issue Thoughts

In the fourth phase, modern LLMs are prompted to generate issue-level reasoning traces. The prompt asks the model to “read the issue and think like a human engineer,” then produce at least 20 short comments, one per line, each formatted as [Topic] comment (full prompt in Appendix-A). The topic inventory is fixed and mirrors the artifact schema categories, including Problem Statement, Proposed Solution, Trade-off Argument, Rationale, Constraint, Question, Suggestion, Task Assignment, Status Update, and Dependency. Conflicting or partially unrelated comments are explicitly allowed so that the model can surface multiple lines of thought rather than converge on a single polished answer.

The prompt consists of the issue title and body plus the retrieved code context from Phase 3. In additional baseline settings, the system can omit code context entirely or include few-shot examples derived from repository-matched extractions. Thus, the repository supports three main inference settings: the original RAG condition, a few-shot condition, and a no-context condition.

The evaluation artifacts cover nine models: gemini-2.5-flash, gemini-3.1-flash, gpt-5.4, gpt-5, gpt-4.1-mini, claude-opus-4.6, claude-sonnet-4.6, qwen2.5:7b-instruct, deepseek-r1:8b. This mix includes both open-weight and API-based frontier models, allowing comparison across provider families and capability tiers.

Phase 5: Artifact Extraction from LLM Responses

The fifth phase maps raw LLM thoughts back into the same structured space used for gold artifacts. Each model response is split into comment-like units, and the artifact extraction system is re-run over those units using the same artifact types, tag vocabulary, metadata fields, and structured decoding rules as in the gold pipeline. This makes the representation on the human side and the model side directly comparable.

This symmetry is important. Instead of comparing free-form model outputs to free-form human discussion, structured artifacts are compared to structured artifacts. The result is a two-stage benchmark: first generate issue reasoning, then evaluate the extracted reasoning content. This design separates the quality of deliberation from the surface style of generation.

Dataset

The benchmark uses real GitHub issue discussions as its primary data source. The main dataset curation criteria are:

  • issues with dense commentary, so there is enough deliberative signal to recover,

  • diversity in issue types, to avoid overfitting to a narrow class of bugs or requests,

  • well-maintained and reputable repositories, so the discussions reflect realistic engineering practice,

  • closed issues, so the discussion thread is sufficiently complete,

  • recent issues, to reduce the likelihood of leakage from model pretraining corpora.

In addition to the processed issue records, the repository includes raw issue snapshots, code snapshot, extraction outputs, retrieval indexes, model responses, derived artifact files, metrics, baseline comparisons, and visualization manifests. This makes the benchmark reproducible end-to-end.

SymbolDescription
HiH_iSet of human artifact types present in issue ii
LiL_iSet of LLM artifact types present in issue ii
Mi=Hi∩LiM_i = H_i \cap L_iSet of matching artifact types in issue ii
Fi,tH,Fi,tLF^H_{i,t}, F^L_{i,t}Human and LLM metadata field-name sets for issue ii, type tt
Vi,t,fH=(v1,…,vm)V^H_{i,t,f} = (v_1,\ldots,v_m)Human metadata values for issue ii, type tt, field ff
Vi,t,fL=(u1,…,un)V^L_{i,t,f} = (u_1,\ldots,u_n)LLM metadata values for issue ii, type tt, field ff
Gi,tH,Gi,tLG^H_{i,t}, G^L_{i,t}Human and LLM tag sets for issue ii, type tt
Si,tH=(s1,…,sa)S^H_{i,t} = (s_1,\ldots,s_a)Human summary strings for issue ii, type tt
Si,tL=(r1,…,rb)S^L_{i,t} = (r_1,\ldots,r_b)LLM summary strings for issue ii, type tt
combine⁡(x1,…,xk)\operatorname{combine}(x_1,\ldots,x_k)Newline-joined concatenation of non-empty summary strings
ratio⁡(x,y)\operatorname{ratio}(x,y)x/yx/y if y>0y > 0, 11 if y=x=0y = x = 0, and 00 otherwise
harm⁡(p,r)\operatorname{harm}(p,r)00 if p+r=0p+r=0, 2prp+r\frac{2pr}{p+r} otherwise

Table 2. Notations for Metrics and Evaluation

Evaluation Scheme

The evaluation compares gold human artifacts and LLM-generated artifacts along four complementary dimensions: type matching, metadata matching, tag matching, and semantic summary matching. Together, these metrics assess not only whether the model identified the right reasoning categories, but also whether it preserved the fine-grained content and meaning of human discussion.

Type Matching

Type matching measures whether the model predicts the correct artifact types for each issue. Let HiH_i be the set of human artifact types for issue ii and LiL_i the set of LLM artifact types. Then:

TPi=∣Hi∩Li∣,FPi=∣Li∖Hi∣,FNi=∣Hi∖Li∣.\mathrm{TP}_i = \lvert H_i \cap L_i \rvert,\quad\mathrm{FP}_i = \lvert L_i \setminus H_i \rvert,\quad\mathrm{FN}_i = \lvert H_i \setminus L_i \rvert.
Precisioni=ratio⁡(TPi,∣Li∣),Recalli=ratio⁡(TPi,∣Hi∣),\mathrm{Precision}_i = \operatorname{ratio}(\mathrm{TP}_i,\lvert L_i\rvert),\quad\mathrm{Recall}_i = \operatorname{ratio}(\mathrm{TP}_i,\lvert H_i\rvert),
F1i=harm⁡(Precisioni,Recalli),Jaccardi=ratio⁡(TPi,∣Hi∪Li∣).\mathrm{F1}_i = \operatorname{harm}(\mathrm{Precision}_i,\mathrm{Recall}_i),\quad\mathrm{Jaccard}_i = \operatorname{ratio}(\mathrm{TP}_i,\lvert H_i \cup L_i\rvert).

These scores measure whether the model identifies the same kinds of reasoning present in the human discussion.

Tag Matching

Tag matching evaluates whether the detailed labels attached to correctly matched types agree. For issue ii and matched type tt, let Gi,tHG^H_{i,t} and Gi,tLG^L_{i,t} denote the human and LLM tag sets. Then:

Precisioni,t=ratio⁡(∣Gi,tH∩Gi,tL∣,∣Gi,tL∣)Recalli,t=ratio⁡(∣Gi,tH∩Gi,tL∣,∣Gi,tH∣),\mathrm{Precision}_{i,t} = \operatorname{ratio}(\lvert G^H_{i,t} \cap G^L_{i,t}\rvert,\lvert G^L_{i,t}\rvert)\quad\mathrm{Recall}_{i,t} = \operatorname{ratio}(\lvert G^H_{i,t} \cap G^L_{i,t}\rvert,\lvert G^H_{i,t}\rvert),
F1i,t=harm⁡(Precisioni,t,Recalli,t),Jaccardi,t=ratio⁡(∣Gi,tH∩Gi,tL∣,∣Gi,tH∪Gi,tL∣).\mathrm{F1}_{i,t} = \operatorname{harm}(\mathrm{Precision}_{i,t},\mathrm{Recall}_{i,t}),\quad\mathrm{Jaccard}_{i,t} = \operatorname{ratio}(\lvert G^H_{i,t} \cap G^L_{i,t}\rvert,\lvert G^H_{i,t} \cup G^L_{i,t}\rvert).

This captures whether the model matches the detailed characterization of an artifact once it has identified the right artifact type.

Metadata Matching

Metadata matching compares the field values attached to each artifact type and field. Since metadata values are free-form strings after normalization, exact equality would be too brittle. Both hard and soft matching scores are therefore computed.

For a given issue ii, type tt, and field ff, let Vi,t,fHV_{i,t,f}^{H} and Vi,t,fLV_{i,t,f}^{L} be the human and LLM metadata values. Pairwise phrase similarities are computed, one-to-one matches are formed greedily, and a hard true positive is counted when the similarity exceeds threshold τ\tau (default 0.82). Hard metrics are:

Precision⁡i,t,f=ratio⁡(TP⁡i,t,f,∣Vi,t,fL∣),Recall⁡i,t,f=ratio⁡(TP⁡i,t,f,∣Vi,t,fH∣),\operatorname{Precision}_{i,t,f} = \operatorname{ratio}(\operatorname{TP}_{i,t,f}, \lvert V_{i,t,f}^{L} \rvert), \qquad\operatorname{Recall}_{i,t,f} = \operatorname{ratio}(\operatorname{TP}_{i,t,f}, \lvert V_{i,t,f}^{H} \rvert),
F1⁡i,t,f=harm⁡(Precision⁡i,t,f,Recall⁡i,t,f),Jaccard⁡i,t,f=ratio⁡(TP⁡i,t,f,TP⁡i,t,f+FP⁡i,t,f+FN⁡i,t,f).\operatorname{F1}_{i,t,f} = \operatorname{harm}(\operatorname{Precision}_{i,t,f},\operatorname{Recall}_{i,t,f}), \qquad\operatorname{Jaccard}_{i,t,f} = \operatorname{ratio}(\operatorname{TP}_{i,t,f}, \operatorname{TP}_{i,t,f} + \operatorname{FP}_{i,t,f} + \operatorname{FN}_{i,t,f}).

Soft metrics are also computed by assigning each human value and each LLM value its best similarity match on the opposite side. If Σi,t,fL\Sigma_{i,t,f}^{L} and Σi,t,fH\Sigma_{i,t,f}^{H} denote the summed best-match similarities, then:

SoftPrecision⁡i,t,f={1∣Vi,t,fL∣=0Σi,t,fL/∣Vi,t,fL∣otherwise\operatorname{SoftPrecision}_{i,t,f} = \begin{cases} 1 & \lvert V_{i,t,f}^{L} \rvert= 0 \\ \Sigma_{i,t,f}^{L}/\lvert V_{i,t,f}^{L} \rvert& \text{otherwise} \end{cases}
SoftRecall⁡i,t,f={1∣Vi,t,fH∣=0Σi,t,fH/∣Vi,t,fH∣otherwise\operatorname{SoftRecall}_{i,t,f} = \begin{cases} 1 & \lvert V_{i,t,f}^{H} \rvert= 0 \\ \Sigma_{i,t,f}^{H}/\lvert V_{i,t,f}^{H} \rvert& \text{otherwise} \end{cases}
SoftF1⁡i,t,f=harm⁡(SoftPrecision⁡i,t,f,SoftRecall⁡i,t,f).\operatorname{SoftF1}_{i,t,f} = \operatorname{harm}(\operatorname{SoftPrecision}_{i,t,f},\operatorname{SoftRecall}_{i,t,f}).

The implementation supports multiple similarity backends, including token F1, sequence ratio, token Jaccard, character 3-gram Jaccard, token containment, and a max-over-backends ensemble. Refer to Appendix-§ 10 for more details.

Semantic Summary Matching

Summary matching evaluates whether the semantic content of human and LLM artifact summaries aligns for matched types. For each issue ii and matched type tt, all human summaries of that type are concatenated into Sˉi,tH\bar{S}_{i,t}^{H} and all LLM summaries into Sˉi,tL\bar{S}_{i,t}^{L}. These aggregated summaries are scored using three complementary semantic similarity measures.

First, CodeBERT cosine similarity is computed from mean-pooled encoder embeddings:

CodeBERTCosine⁡(x,y)=ϕ(x)⋅ϕ(y)∥ϕ(x)∥2∥ϕ(y)∥2.\operatorname{CodeBERTCosine}(x,y) = \frac{\phi(x) \cdot\phi(y)}{\lVert\phi(x) \rVert_{2}\lVert\phi(y) \rVert_{2}.}

Second, BERTScore is computed by contextual token alignment:

BERTScorePrecision⁡(x,y)=1n∑j=1nmax⁡kAjk,BERTScoreRecall⁡(x,y)=1m∑k=1mmax⁡jAjk,\operatorname{BERTScorePrecision}(x,y) = \frac{1}{n}\sum_{j=1}^{n}\max_{k} A_{jk}, \qquad\operatorname{BERTScoreRecall}(x,y) = \frac{1}{m}\sum_{k=1}^{m}\max_{j} A_{jk},
BERTScoreF1⁡(x,y)=harm⁡(BERTScorePrecision⁡(x,y),BERTScoreRecall⁡(x,y)).\operatorname{BERTScoreF1}(x,y) = \operatorname{harm}(\operatorname{BERTScorePrecision}(x,y),\operatorname{BERTScoreRecall}(x,y)).

Third, a BLEURT score β(x,y)\beta(x,y) is computed, optionally with raw, clipped, or sigmoid-normalized post-processing. Refer to Appendix for equations and details of how BLEURT is calculated. These three measures together capture embedding-level similarity, token-level semantic coverage, and learned text quality. Refer to Appendix-§ 11 for details on BLUERT calculation.

Coverage-Aware Evaluation

A key challenge is that semantic summary comparison is only defined for matched artifact types. A model might achieve strong semantic similarity on a small number of matched types while missing many human reasoning categories entirely. To account for this, issue-level coverage metrics are computed:

CovPrecision⁡i=ratio⁡(∣Hi∩Li∣,∣Li∣),CovRecall⁡i=ratio⁡(∣Hi∩Li∣,∣Hi∣),\operatorname{CovPrecision}_{i} = \operatorname{ratio}(\lvert H_{i} \cap L_{i}\rvert, \lvert L_{i}\rvert), \qquad\operatorname{CovRecall}_{i} = \operatorname{ratio}(\lvert H_{i} \cap L_{i}\rvert, \lvert H_{i}\rvert),
CovF1⁡i=harm⁡(CovPrecision⁡i,CovRecall⁡i).\operatorname{CovF1}_{i} = \operatorname{harm}(\operatorname{CovPrecision}_{i}, \operatorname{CovRecall}_{i}).

Summary quality is then penalized by coverage recall: qipen=PenaltyApply⁡(qˉi,CovRecall⁡i)q_{i}^{\mathrm{pen}} = \operatorname{PenaltyApply}(\bar{q}_{i}, \operatorname{CovRecall}_{i}).

This prevents overestimating models that write high-quality summaries for only a narrow subset of the human reasoning space.

Experimental Results

This section presents the empirical evaluation of the nine state-of-the-art models using the benchmark. The primary objective of these experiments is to determine whether current large language models (LLMs) can replicate the nuanced deliberation found in professional software engineering discussions. The results indicate that while models demonstrate a baseline ability to identify technical problems, they remain significantly limited in their capacity for deep engineering reasoning and structural deliberation.

Taxonomy and Reasoning Type Discrepancies

An analysis of per-type F1 scores (Figure 3) reveals “design choice blindness” within current models. While models are relatively proficient at identifying problem_statement and proposed_solution artifacts, performance plummeted when encountering trade_off_argument and design_decision categories (Figure 4). This discrepancy highlights that LLMs are currently optimized for execution-based tasks—identifying what is broken and how to fix it—rather than the deliberative reasoning required to assess feasibility or prioritize competing implementation strategies. In discussing the correct “types” of topics (Figure 5), models are somewhat similar in their behavior but remain far from perfect in capturing the breadth of human deliberation.

Scores for the 9 models across the 7 metrics

Figure 3. Scores for the 9 models across the 7 metrics. Metadata F1 scores are below 0.2 which suggests that the models do not know the correct ideas under a topic and use generic terms. Tag and Type F1 scores are also lower in general. The models show considerably higher scores on summary. This suggests that models often respond in generic terms based on their training data and are not able to narrow down to the exact topics to think about.

Per-Type type_f1 by Model bar chart

Figure 4. Figure shows the Type F1 scores. The models were asked to discuss about one of the 32 types (see Appendix for complete list). The granular results show that the state of the art models are significantly poor when thinking about trade-offs, data structures, and design choices. This confirms the hypothesis that most code LLMs work well because they reproduce previously seen code and not due to a superior (and human-like) reason ability.

Box plots showing per-issue type, metadata, tag, and summary BERTScore F1 values across models

Figure 5. Box Plot for F1 scores in Type, Metadata and Tag matching for Kubernetes repository. Other repos show similar box plots. The exceptionally low metadata scores indicate that the current state of the art models are not good at discussing the right sub-topics within the type of issue. Type and Tag F1 scores are similar across models but still exhibit a scope for improvement (F1 score below 0.4). Acrtoss summary metrics, OpenAI, Qwen, and Deepseek models show a low standard deviation showing a consistent reasoning capability across different issues.

Structural Deliberation and the Metadata Gap

A critical failure point identified in the evaluation is the extraction of structured metadata. As shown in the per-issue metadata F1 results (Figure 5), scores are universally low, often failing to cross the 0.20 threshold even for soft-matching metrics. This indicates that while a model might correctly identify a trade_off_argument, it struggles to extract the specific options_compared or the preferred_option with the same precision as a human developer. This structural conclusion gap represents the largest scope for future improvement, as it requires the model to not only recognize a discussion but to derive actionable, structured conclusions from it.

Semantic Alignment and Stability

An ensemble of semantic similarity measures was employed to evaluate the alignment of model “thoughts” with human summaries. The findings confirm that BLEURT serves as a significantly harder metric than BERTScore or CodeBERT. The former’s learned sensitivity to text quality and technical logic frequently penalized models that, according to token-alignment metrics like BERTScore, appeared to be performing well. Furthermore, newer iterations of OpenAI and DeepSeek models demonstrated higher stability (Figure 5), characterized by lower standard deviation in summary metrics across all repositories, suggesting a maturing capability in consistent text generation.

Global Leaderboard and Reasoning Capability

The overall performance across all repositories and issues is summarized in the leaderboard presented in Table 3. GPT-5.4 achieved the highest overall score of 0.436, followed by DeepSeek-R1 8B at 0.422 and GPT-5 at 0.411. The most striking observation is the consistently low scores across all LLMs; even the top-performing models failed to exceed a score of 0.45. This suggests a significant gap between the capabilities of frontier models in code generation and their ability to reason through the complex design choices, constraints, and trade-offs inherent in real-world software engineering.

ModelMeta F1Meta Soft F1Tag F1Type F1BLEURTBERTScore F1CodeBERTOverall
GPT-5.40.05 (0.01)0.14 (0.01)0.31 (0.05)0.33 (0.00)0.39 (0.00)0.86 (0.00)0.97 (0.00)0.436 (0.008)
GPT-50.05 (0.00)0.14 (0.00)0.23 (0.00)0.34 (0.00)0.37 (0.00)0.82 (0.00)0.92 (0.00)0.411 (0.000)
GPT-4.1 Mini0.05 (0.00)0.14 (0.00)0.33 (0.06)0.34 (0.02)0.29 (0.03)0.63 (0.05)0.70 (0.05)0.353 (0.030)
Gemini 3.1 Flash0.05 (0.01)0.15 (0.01)0.31 (0.03)0.36 (0.01)0.33 (0.00)0.72 (0.00)0.81 (0.00)0.391 (0.002)
Gemini 2.5 Flash0.06 (0.00)0.17 (0.00)0.36 (0.08)0.37 (0.03)0.32 (0.03)0.68 (0.05)0.76 (0.05)0.386 (0.034)
Claude Opus 4.60.06 (0.00)0.17 (0.00)0.35 (0.02)0.35 (0.04)0.31 (0.05)0.67 (0.09)0.76 (0.10)0.383 (0.038)
Claude Sonnet 4.60.05 (0.02)0.15 (0.04)0.35 (0.05)0.30 (0.06)0.24 (0.10)0.52 (0.22)0.58 (0.25)0.312 (0.091)
DeepSeek-R1 8B0.05 (0.01)0.13 (0.00)0.32 (0.06)0.33 (0.01)0.39 (0.01)0.83 (0.02)0.92 (0.02)0.422 (0.000)
Qwen2.5 7B0.04 (0.01)0.12 (0.01)0.31 (0.07)0.30 (0.04)0.35 (0.08)0.73 (0.16)0.81 (0.18)0.381 (0.078)

Table 3. Global metrics and overall leaderboard score. Overall Score is calculated by averaging over all metrics (equal weight) and normalizing the score out of 100. Standard Deviation indicated in parenthesis.

Baseline and Ablation Analysis

To quantify the impact of the various components in the reasoning pipeline, an extensive ablation study was performed across four primary configurations: the proposed hybrid RAG pipeline (Orig.), a Few-Shot setting, a No Context (NoCtx) setting, and a sparse retrieval baseline (BM25 Only). The results, detailed in Table 4, Table 5 and Figure 6, underscore that while specific prompting or retrieval strategies can shift metrics, current LLMs still exhibit a fundamental struggle with deep repository-level reasoning.

Baseline Comparisons and Ablation Study

Figure 6. Baseline Comparisons and Ablation Study: The figure shows all metrics on the Kubernetes and Bitcoin issues. Adding few-shots to the prompt in Phase 5 leads to slightly better tagging. However, the additional cost of providing few shots within the prompt outweighs the benefit. Excluding the code context from the prompt significantly reduces performance across metrics. Proving code context using only BM25 as the RAG component leads to better tagging and type matching, which comes at the cost of a poor summary. Because of these the ‘Original’ system uses BM25 + Dense Retrieval System (combined with Reciprocal Rank Fusion) without few-shots to keep the system effective and cost efficient.

ModelMetadata Soft F1Tag F1Type F1
Orig.FewNoCtxBM25Orig.FewNoCtxBM25Orig.FewNoCtxBM25
GPT-5.40.140.160.050.160.310.330.290.400.330.310.110.45
GPT-50.140.150.050.160.230.330.240.380.340.310.110.45
GPT-4.1 Mini0.140.150.050.150.330.330.300.440.340.330.130.46
Gemini 3.1 Flash0.150.150.050.170.310.260.280.460.360.320.110.46
Gemini 2.5 Flash0.170.160.060.190.360.420.340.500.370.330.110.48
Claude Opus 4.60.170.160.060.190.350.390.360.440.350.310.110.48
Claude Sonnet 4.60.150.160.060.160.350.380.290.460.300.310.110.38
DeepSeek-R1 8B0.130.140.040.160.320.320.310.460.330.310.110.40
Qwen2.5 7B0.120.150.040.150.310.430.270.400.300.210.110.34

Table 4. Baseline comparison (Metadata, Tag, Type). Few (Few Shot prompt), NoCtx (No code context in the prompt), BM25 (only BM25 RAG, no dense retrieval), Orig. (BM25+ Dense Retrieval code context without few shot prompts).

ModelBLEURTBERTScore F1CodeBERT
Orig.FewNoCtxBM25Orig.FewNoCtxBM25Orig.FewNoCtxBM25
GPT-5.40.390.400.100.290.860.820.220.680.970.940.240.75
GPT-50.370.390.080.300.820.900.180.650.920.920.200.70
GPT-4.1 Mini0.290.290.090.260.630.690.210.420.700.710.230.46
Gemini 3.1 Flash0.330.380.080.270.720.780.170.550.810.800.190.67
Gemini 2.5 Flash0.320.370.070.240.680.660.150.460.760.780.170.44
Claude Opus 4.60.310.350.070.230.670.660.160.470.760.780.180.57
Claude Sonnet 4.60.240.270.080.150.520.660.170.330.580.580.190.39
DeepSeek-R1 8B0.390.400.090.250.830.810.200.640.920.930.230.77
Qwen2.5 7B0.350.390.100.280.730.800.220.520.810.820.240.64

Table 5. Baseline comparison - Summary (BLEURT, BERTScore, CodeBERT). Few (Few Shot prompt), NoCtx (No code context in the prompt), BM25 (only BM25 RAG, no dense retrieval), Orig. (BM25+ Dense Retrieval code context without few shot prompts).

  • Contextual and Retrieval Ablations: The catastrophic performance of the No Context baseline confirms that engineering reasoning is highly dependent on grounded technical facts. In this setting, Type F1 scores collapsed to a baseline average of 0.11 across all models, and semantic similarity measures like BERTScore fell into the 0.15–0.22 range. While the BM25 Only baseline occasionally yielded higher Tag F1 and Type F1 scores—reaching 0.50 for Gemini 2.5 Flash—it frequently struggled with semantic alignment compared to the hybrid dense-sparse architecture. This suggests that while keyword matching can help identify the ”topic” of a discussion, it is insufficient for replicating the actual logic and deliberative content found in human engineering threads.

  • Impact of Few-Shot Prompting: The inclusion of repository-matched few-shot examples generally provided a performance boost in structured identification tasks. For instance, Gemini 2.5 Flash saw its Tag F1 increase from 0.36 in the Orig. setting to 0.42 with few-shot guidance. However, this gain did not consistently translate to semantic summary quality, as metrics like BLEURT remained relatively stagnant or showed only marginal improvements. This indicates that while models can be steered toward the correct taxonomic labels through in-context learning, the underlying ”thought” generation—specifically regarding trade_off_arguments and design_decisions—remains limited by the model’s intrinsic reasoning capabilities.

  • Heuristic vs. Neural Extraction: The extraction pipeline was further evaluated against a Regex-based heuristic baseline. As illustrated in the global average tag frequency analysis (Figure 7, Appendix-§ 9), the Regex approach is significantly more ”noisy,” frequently over-tagging generic technical terms without regard for context. For example, mentions of ”test” were universally tagged as unit_test artifacts, leading to inflated recall but extremely poor precision. In contrast, the neural extraction system employed in the benchmark demonstrates a more nuanced understanding of how specific technical artifacts function within a deliberation, though it still faces a significant challenge in capturing the complex logic required for accurate metadata extraction.

Tags extracted from GitHub comments and LLM responses

Figure 7. Tags extracted from GitHub comments (left) and LLM responses (right). While the distribution is similar, there are subtle differences as well. For example, the human comments tend to have more discussion around bugs whereas LLMs treat these as edge cases because they are conditioned to view errors as “missing test cases” rather than actual design issues.

Conclusion

To summarize, the following has been achieved by this study:

  1. Refined Artifact Extraction System: This goal has been fully operationalized through a concrete schema and a validated runtime pipeline. The extraction system defines a comprehensive artifact space, requiring models to adhere to fixed metadata fields and a constrained tag vocabulary. The implementation utilizes Pydantic for output validation and records extraction failures on a per-comment basis to ensure data integrity. To date, the system has successfully curated a large dataset as described above.

  1. Prompting Methods Implemented and Tested: A full prompting loop was implemented that integrates hybrid retrieval (BM25 + CodeBERT with reciprocal-rank fusion) and structured deliberation traces. The “structured deliberation” is operationalized through a thought prompt that mandates topic-labeled engineering comments. Furthermore, the system supports few-shot prompting by dynamically loading prior human extraction examples into the inference context. The

custom CLI facilitates the original RAG, few-shot, no-context, and heuristic baselines, allowing for the comprehensive ablation studies presented in Section 5.

  1. Evaluation Framework: The framework computes alignment across four dimensions: type matching, metadata matching, tag matching, and semantic summary similarity. The orchestration code supports batch execution of metrics and generates a combined leaderboard using rank-fusion. The metrics such as ”decision agreement” are currently studied thoroughly and the visualization suite effectively translates these raw outputs into comparative graphs and tables.

  1. Comprehensive Benchmark Results and Analysis: The project has delivered a complete set of benchmark outputs for nine frontier model families. These results include a global leaderboard where GPT-5.4 currently ranks as the top performer. To facilitate qualitative analysis, I developed a side-by-side viewer that assembles raw GitHub issues, human ”gold” extractions, and model-derived artifacts within a unified interface. This enables researchers to perform deep-dive audits of model reasoning quality across multiple repositories.

References

Prompt Templates

Following is the prompt given to gemma3:12b, gemma4:e4b, llama3.1:8b to extract the artifacts from the human comments in the GitHub issues and from the thought responses of the SoTA LLMs:

<System Prompt>

Extract discussion artifacts from one GitHub comment.
Return strict JSON only: {"artifacts": [...]}.

Rules:
1) type: required, one of [<allowed_artifact_types>].
2) summary: required, specific key takeaway (not generic). Include
   - subject (component/file/feature)
   - event (bug/change/question/suggestion)
   - key technical detail (error/behavior/number/condition/cause if known)
3) tags: required, at least 1 tag, use as many as needed, unique, lowercase snake_case,
   only from [<allowed_tags>].
   Be generous with relevant tags: include all tags clearly supported by the comment.
   For each candidate tag, explicitly check if evidence in the comment supports it; add
   it when supported, skip when not supported.
   Prefer broader correct coverage over minimal tagging, but do not add speculative tags.
4) metadata: required; include exactly the fixed keys for the chosen type, never use
   null, no extra keys.
5) Use only the provided comment text; no outside knowledge.
6) If no valid artifact exists, return {"artifacts": []}.

<User Prompt>

Repo: <repository>
Issue: <issue_number>
Comment JSON:
{"author":"<comment_author>","link":"<comment_link>","text":"<comment_text>"}

Metadata keys by type (use exact keys; never null; use ’unknown’ when needed):
- <artifact_type_1>: <metadata_key_1>, <metadata_key_2>, ...
- <artifact_type_2>: <metadata_key_1>, <metadata_key_2>, ...
- ...

Following is the prompt given to the LLMs for thinking like human engineers to produce their artifacts:

You are given a GitHub issue. Think like a human engineer reading the issue
for the first time.

Instructions:
- Produce at least 20 short comments
- Each comment must be on its own line
- Each line must follow the format: [Topic] comment
- Comments may be conflicting, uncertain, speculative, or partially redundant
- Use only the following topics: <allowed_thought_topics>

Issue:
- Number: <issue_number>
- Title: <issue_title>
- Body: <issue_body>

FEW SHOT EXAMPLES:
<optional_few_shot_examples>

CODE SNIPPETS:
<file_1.py>
<retrieved_code_context>
....
<file_n.py>
<retrieved_code_context>

Dataset Details

Allowed tag categories:

bug, regression, flaky_test, ci_failure, build_failure, runtime_error, compile_error, assertion_failure, performance_issue, memory_issue, security_issue,
data_corruption, incorrect_behavior, edge_case, compatibility_issue, race_condition, non_determinism, timestamp_issue, floating_point_error, off_by_one,
null_pointer, uninitialized_state, state_sync_issue, logic_bug, api_misuse, config_issue, dependency_issue, version_mismatch, platform_specific,
overflow_underflow, unit_test, integration_test, e2e_test, functional_test, test_framework_issue, test_flakiness, test_timeout, test_infra_issue,
networking, filesystem, database, api, cli, ui, backend, frontend, distributed_system, p2p, consensus, mempool, rpc, serialization, crypto, validation,
scheduler, training, inference, data_pipeline, tokenization, linux, macos, windows, arm, x86, docker, ci_environment, gpu, low_memory, build_time,
startup, runtime, shutdown, deployment, crash, hang, timeout, incorrect_output, missing_output, extra_output, log_error, performance_degradation,
workaround, fix_proposed, fix_merged, refactor, test_adjustment, relax_assertion.
Annotation TypeDescription
Problem StatementDescription of a bug, feature request, or user need.
Proposed SolutionSuggested fix, implementation, or approach.
Alternative SolutionCompeting or different approach to solve the same problem.
Design DecisionFinal or tentative choice among alternatives.
Trade-off ArgumentComparison between options highlighting pros and cons.
RationaleExplanation or justification for a decision.
ConstraintLimitation or requirement such as performance or compatibility.
AssumptionStated or implicit premise about the system or inputs.
Implementation DetailLow-level description of how something should be built.
Code SnippetConcrete code example or patch.
Algorithm / ApproachHigh-level method or technique.
API DesignDiscussion of interfaces, endpoints, or function signatures.
Data Structure ChoiceSelection of data structures and their implications.
Configuration ChoiceDecisions about parameters or settings.
Benchmark ResultMeasured performance or experimental result.
Performance ClaimAssertion about speed, memory, or efficiency.
Test CaseExample used to validate correctness.
Bug Reproduction StepsSteps to reproduce an issue.
Edge CaseRare or boundary scenario that needs handling.
Empirical EvidenceData or experiments supporting a claim.
QuestionInquiry asking for clarification or information.
Answer / ClarificationResponse resolving a question.
AgreementExpression of alignment or approval.
DisagreementExpression of conflict or opposing view.
SuggestionRecommendation for improvement.
CritiqueEvaluation or review of a proposal.
Task AssignmentDelegation of work to a contributor.
Status UpdateProgress or current state of work.
Priority DiscussionConversation about importance or urgency.
Blocking IssueIssue preventing further progress.
DependencyRelationship to another issue, module, or system.

Table 6. Entity classes used for Extraction.

Artifact TypeMetadata Fields
problem_statementproblem_kind, affected_area, user_or_system_impact
proposed_solutionsolution_kind, target_area, expected_benefit
alternative_solutionalternative_to, differentiator, comparison_focus
design_decisiondecision_status, chosen_option, rejected_options
trade_off_argumentoptions_compared, comparison_axes, preferred_option
rationalesupports_artifact_type, reasoning_kind, impact_scope
constraintconstraint_type, scope, severity
assumptionassumption_scope, confidence, failure_mode
implementation_detailcomponent, lifecycle_stage, implementation_action
code_snippetlanguage, snippet_kind, intended_use
algorithm_approachapproach_name, target_problem, scale_or_complexity_notes
api_designinterface_kind, api_surface, compatibility_impact
data_structure_choicedata_structure, workload, motivation
configuration_choiceparameter, selected_value, config_scope
benchmark_resultmetric, baseline, candidate, environment
performance_claimmetric_focus, claim_direction, workload
test_casetest_level, coverage_target, expected_outcome
bug_reproduction_stepsenvironment, prerequisites, determinism
edge_casetrigger, impact, rarity
empirical_evidenceevidence_type, sample_source, reliability
questionquestion_kind, target_area, urgency
answer_clarificationtarget_question_topic, certainty, resolution_status
agreementtarget, strength, justification
disagreementtarget, contention_point, severity
suggestiontarget_area, expected_improvement, urgency
critiquetarget, critique_dimension, severity
task_assignmentassignee, deliverable, due_context
status_updatestatus_kind, progress_state, blocking
priority_discussionpriority_level, compared_against, scope
blocking_issueblocker_type, blocking_target, unblock_condition
dependencydependency_type, dependency_target, relationship

Table 7. Metadata fields required for each artifact type in the discussion-artifact extraction schema.

Similarity Metrics Used

For normalized phrases aa and bb, let T(a)T(a) be the set of whitespace-delimited tokens, and let Cn(a)C_n(a) be the set of character nn-grams of the padded string “ aa ”.

Token F1: Let o=∣T(a)∩T(b)∣o = \lvert T(a) \cap T(b) \rvert. Then:

tokPrec⁡(a,b)=ratio⁡(o,∣T(b)∣),tokRec⁡(a,b)=ratio⁡(o,∣T(a)∣),\operatorname{tokPrec}(a,b) = \operatorname{ratio}(o,\lvert T(b) \rvert), \qquad\operatorname{tokRec}(a,b) = \operatorname{ratio}(o,\lvert T(a) \rvert),
tokenF1⁡(a,b)=harm⁡(tokPrec⁡(a,b),tokRec⁡(a,b)).\operatorname{tokenF1}(a,b) = \operatorname{harm}(\operatorname{tokPrec}(a,b),\operatorname{tokRec}(a,b)).

Measures overlap between unique token sets.

Sequence Ratio: sequenceRatio⁡(a,b)=SequenceMatcher⁡(a,b).ratio⁡()\operatorname{sequenceRatio}(a,b)=\operatorname{SequenceMatcher}(a,b).\operatorname{ratio}().

Measures surface-form similarity of the normalized strings.

Token Jaccard: tokenJaccard⁡(a,b)=ratio⁡(∣T(a)∩T(b)∣,∣T(a)∪T(b)∣)\operatorname{tokenJaccard}(a,b)=\operatorname{ratio}(\lvert T(a)\cap T(b)\rvert,\lvert T(a)\cup T(b)\rvert).

Measures set overlap between token vocabularies.

Character 3-gram Jaccard: char3Jaccard⁡(a,b)=ratio⁡(∣C3(a)∩C3(b)∣,∣C3(a)∪C3(b)∣)\operatorname{char3Jaccard}(a,b)=\operatorname{ratio}(\lvert C_3(a)\cap C_3(b)\rvert,\lvert C_3(a)\cup C_3(b)\rvert).

Measures fuzzy overlap at the subword character level.

Token Containment: cont⁡(a,b)=ratio⁡(∣T(a)∩T(b)∣,min⁡(∣T(a)∣,∣T(b)∣))\operatorname{cont}(a,b)=\operatorname{ratio}(\lvert T(a)\cap T(b)\rvert,\min(\lvert T(a)\rvert,\lvert T(b)\rvert)).

Measures whether the shorter token set is mostly contained in the longer one.

Max-All Ensemble.

maxAll⁡(a,b)=max⁡{tokenF1⁡(a,b),sequenceRatio⁡(a,b),tokenJaccard⁡(a,b),char3Jaccard⁡(a,b),cont⁡(a,b)}.\operatorname{maxAll}(a,b)=\max\{\operatorname{tokenF1}(a,b),\operatorname{sequenceRatio}(a,b),\operatorname{tokenJaccard}(a,b),\operatorname{char3Jaccard}(a,b),\operatorname{cont}(a,b)\}.

Measures best-case phrase agreement across several lightweight similarity functions.

Greedy One-to-One Matching:

For one field ff of one type tt in one issue ii, compute all pairwise similarities:

σjk=sim⁡(vj,uk),vj∈Vi,t,fH, uk∈Vi,t,fL.\sigma_{jk}=\operatorname{sim}(v_j,u_k),\quad v_j\in V^{H}_{i,t,f},\ u_k\in V^{L}_{i,t,f}.

Sort all candidate pairs by descending σjk\sigma_{jk} and greedily keep a pair only if neither value has been used before. This produces a one-to-one set of greedy pairs PP.

BLEURT Calculation

Let βraw(x,y)\beta_{\mathrm{raw}}(x,y) be the scalar output of the configured BLEURT sequence-classification model. The reported score depends on post-processing mode.

Raw mode: β(x,y)=βraw(x,y)\beta(x,y)=\beta_{\mathrm{raw}}(x,y).

Clip mode: For clipping floor cmin⁡c_{\min}:

β(x,y)={max⁡(cmin⁡,βraw(x,y))cmin⁡≠∅βraw(x,y)otherwise.\beta(x,y)= \begin{cases} \max(c_{\min},\beta_{\mathrm{raw}}(x,y)) & c_{\min}\ne\varnothing\\ \beta_{\mathrm{raw}}(x,y) & \text{otherwise.} \end{cases}

Sigmoid mode: For temperature T>0T>0 and bias bb:

z=βraw(x,y)−bT,z←min⁡(60,max⁡(−60,z)),β(x,y)=11+e−zz=\frac{\beta_{\mathrm{raw}}(x,y)-b}{T},\qquad z\leftarrow\min(60,\max(-60,z)),\qquad\beta(x,y)=\frac{1}{1+e^{-z}}

The implementation returns 1 for two empty strings and 0 if exactly one side is empty. Off these, the sigmoid mode was deemed to be the best and used for all the experiments.

Per-issue scatter plot comparing summary F1 and metadata F1

Figure 8. Correlation comparison between metadata summary metrics. The OpenAI models (blue) concentrate towards the right - showing superior summary but varying metadata accuracy across runs. No correlation appears among other model families.

Visualizer

Leaderboard for TBYC showing model rankings and scores

Figure 9. Leaderboard for TBYC - An interactive leader board to see the ranking and scores of models across various metrics and a global scoring schema. The global score is calculated as an average overall metrics.

Interactive explorer interface for pulling up a repository and issue from the dataset

Figure 10. Interactive explorer for pulling up a repo and issue from the dataset. Users can view the issue, hummand comments, links to GitHub issues and the human and LLM artifacts across a wide range of models.

References

  1. [2]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  2. [4]Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36:46701–46723, 2023.arxiv.org/abs/2310.11248
  3. [5]GitHub. Github copilot documentation. https://docs.github.com/en/copilot, 2024. Accessed: 2026-04-28.
  4. [6]Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024.arxiv.org/abs/2401.03065
  5. [7]Naman Jain, Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 58791–58831, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/94074dd5a072d28ff75a76dabed43767-Paper-Conference.pdf.arxiv.org/abs/2403.07974
  6. [8]Juyong Jiang, Fan Wang, Jiási Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 2024.DOI
  7. [9]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.arxiv.org/abs/2310.06770
  8. [10]Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021.arxiv.org/abs/2102.04664
  9. [11]OpenAI. Introducing codex. https://openai.com/index/introducing-codex/, 2021. Accessed: 2026-04-28.
  10. [12]Cong Xia, Yuheng Deng, Alexander Dunn, Lingming Zhang, et al. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793, 2024.
  11. [13]Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.151. URL https://aclanthology.org/2023.emnlp-main.151/.DOI

Paper details

vlnc:W_xq93ssfn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents