Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks

Introduction

Motivation: What Counts as Maintenance Completion?

The rapid evolution of Large Language Models (LLMs) and autonomous scaffolding frameworks has transformed automated program repair into repository-scale software engineering. Benchmarks such as SWE-bench [1], SWE-bench Pro [2], SWE-PolyBench [3], and SWE-EVO [4] evaluate coding agents on tasks derived from software issues, repository changes, or evolution histories. Although coding-agent benchmarks provide quantitative performance measurements, their leaderboard scores do not directly establish readiness for enterprise software maintenance.

A fundamental question precedes test adequacy: what combination of conditions defines completion of a maintenance task? In enterprise services, a single change can entail temporally ordered recalculation, preservation of prior data, and consistent downstream state. These are not necessarily independent checklist items: one operation can trigger another, and the task is complete only when the applicable conditions hold together. We call this multi-condition fulfillment, distinguishing the number of conditions from their relationships. Apache Fineract backdating requirements provide concrete instances of such linked obligations; whether comparable structures are visible in benchmark task descriptions is an empirical question.

The distinction matters for benchmark interpretation. A task may mention several conditions without stating how they interact, while a compact issue may specify a sequence of dependent operations. Evaluation tests are one downstream way to check those conditions, not the definition of the conditions themselves. We therefore first compare requirement structures across corpora, and only then inspect selected benchmark tests as illustrative consequences for evaluation design.

Limitations of Surface Complexity Metrics

Existing benchmark frameworks predominantly evaluate task difficulty through surface syntactic and navigational metrics: (1) Syntactic Churn (LOC added/deleted, file counts); (2) Repository Topography (search space breadth, AST exploration radius, directory depth [5]); and (3) Generic Labels ("Bug Fix", "Feature").

We examine whether surface task descriptors obscure differences in fulfillment-condition structures. Some maintenance activities ($A$) impose dependent requirements ($B$), as illustrated by backdated financial operations:

The analytical question is whether tasks with similar surface profiles describe comparable sets and relationships of fulfillment conditions; evaluation coverage is a subsequent implication, not the primary outcome measured here.

Research Questions

This investigation is guided by two fundamental research questions:

• RQ1 (Semantic Heterogeneity & Attribute Diversity): To what extent do conventional benchmark task categories and surface churn metrics conceal semantic heterogeneity, and how does the diversity of required maintenance attributes ($|B|$) compare between enterprise maintenance and AI coding benchmarks?

• RQ2 (Multi-Condition Fulfillment Structure): (a) Corpus-Level Co-occurrence (RQ2a): How do observed patterns of jointly specified conditions ($\phi$) differ between the studied enterprise and benchmark corpora? (b) Directional Dependencies (RQ2b): What source-grounded action–condition and condition–condition relationships characterize selected enterprise maintenance requirements, and what do selected benchmark cases illustrate about representing these obligations in evaluation?

Core Contributions

We present a mixed-methods empirical study grounded in verified raw data across three distinct analytical levels:

1. Unmasking Semantic Heterogeneity (Level 1 / E1): In 1,177 controlled task pairs matched on repository native types and code churn (714 SWE-bench Pro pairs, 463 PolyBench Java pairs, 581 unique tasks), matching on surface task descriptors does not guarantee similar fulfillment-condition sets (mean $d_J = 0.8138$, median $1.0000$, $\sigma= 0.2745$), with 57.4% of pairs (676 pairs) exhibiting completely disjoint criteria ($d_J = 1.0$).

2. Attribute Diversity Paradox (Level 2 / E2 Macro): Across 944 AI tasks (SWE-bench Pro $N = 731$, PolyBench $N = 165$, SWE-EVO $N = 48$), SWE-bench Pro exhibits higher attribute diversity (mean $|B| = 2.47$) than enterprise core banking (Fineract 1.78, Welch’s $t = 15.78$, $p < 10^{-50}$). However, linear co-occurrence between temporal backdating ($B_4$) and numerical recalculation ($B_2$) in AI benchmarks shows near-zero correlation ($\phi= -0.013$, $p = 0.825$, 5 tasks) vs. enterprise banking ($\phi= +0.430$, $p < 10^{-90}$, 254 issues), showing that attribute breadth alone does not characterize joint-condition structure.

3. Directional Fulfillment Contracts (Level 3 / E2 Micro): We trace 16 source-grounded Fineract requirement relationships (19 quotes) to illustrate how operations such as backdating, recalculation, and state preservation form joint completion obligations. Four selected benchmark cases illustrate possible alignment boundaries between explicit conditions and inspected assertions; they do not establish a population-level failure rate.

4. Reporting Implications: We distinguish attribute breadth, statistical co-occurrence, and source-grounded directionality as non-interchangeable properties of maintenance tasks, with secondary implications for specification-aligned benchmark checks.

Related Work & Research Positioning

Evolution of AI Coding Benchmarks

Automated code generation benchmarks have evolved rapidly from isolated synthetic function completion (e.g., HumanEval [6], MBPP) to repository-level software maintenance. SWE-bench [1] pioneered the extraction of historical pull requests from Python repositories, packaging them with reproducible execution containers. Subsequent benchmarks extended this paradigm: (1) SWE-PolyBench [3] supports repository-level evaluation across Java, JavaScript, TypeScript, and Python; (2) SWE-bench Pro [2] targets long-horizon, multi-file software engineering tasks across public, held-out, and commercial splits; and (3) SWE-EVO [4] introduces longitudinal software evolution across sequential commits to evaluate regression avoidance. Across all these benchmarks, candidate patches are evaluated by executing automated test harnesses to verify whether target unit tests transition from failing to passing (fail_to_pass) without inducing regressions in pre-existing tests (pass_to_pass).

Characterizing Benchmark Complexity & Audit Studies

Several empirical frameworks have investigated the demands of repository-level coding tasks. Syntactic & Navigational Complexity (SNC) [5] profiles Spread (file distribution entropy), Novelty (the proportion of added versus deleted code), and Centrality (dependency graph importance), examining task demands through code-change and repository structure. In contrast, UTBoost [7] identifies insufficient issue-specific benchmark tests that can allow incorrect candidate patches to pass, and augments tests to improve patch discrimination. These studies examine repository-level demands and patch evaluation, respectively, rather than the specified structure of multi-condition maintenance completion.

The Distinct Dimension: Multi-Condition Fulfillment Structure. Test-adequacy studies such as UTBoost address whether existing tests discriminate correct and incorrect patches. We instead examine what a maintenance task requires: which completion conditions are specified, which appear together, and which are directionally dependent. We do not claim that test coverage has been unexplored, nor that our descriptive corpus comparisons prove a missing condition in any entire benchmark harness.

Research Positioning: From Task Size to Joint Fulfillment Conditions

As summarized in Table 1, repository-complexity studies analyze properties of code changes and navigation demands, while test-audit work examines patch evaluation. Our unit of analysis is the maintenance fulfillment contract: developer activities ($A$), specified conditions ($B$), and, where supported by source text, relationships among those conditions. We distinguish three complementary observations: (1) condition breadth within an issue ($|B|$); (2) corpus-level co-occurrence ($\phi$), which does not establish direction; and (3) explicit or source-grounded directional obligations linking actions to downstream states. Selected oracle inspections are a secondary application of this characterization.

Analytical Dimension	SNC Audit [5]	UTBoost [7]	This Work
Primary Problem	Repository topography ambiguity and task demand categorization	Incomplete tests and false pass judgments	Multi-condition structure of maintenance completion
Unit of Analysis	Repository graph metrics (AST distance, spread, novelty)	Tests and candidate patches in SWE-bench tasks	Requirement fulfillment invariants ($A \to\Delta S \to B$)
Requirement Perspective	Surface issue labels (Bug, Feature, Enhancement)	Task-specific tests and patch evaluation	Ontological taxonomy (ISO 14764 / 25010) with verified quotes
Oracle Evaluation Scope	Accepts existing benchmark test harnesses as ground truth	Augments benchmark tests to flag incorrect passes	Secondary case-level assertion inspection
Empirical Scope	SWE-bench Lite/Verified (Python repository graphs)	SWE-bench Python repos (LLM-augmented test suites)	Analyzed AI tasks ($N = 944$) vs. Fineract issues ($N = 2{,}925$)

The principal research question is therefore not simply whether a test catches a buggy patch. It is: Which fulfillment conditions co-occur across the analyzed corpora, and what source-grounded dependencies characterize completion in selected enterprise maintenance requirements?

Empirical Foundations & Methodology

Reference Corpora

Our empirical foundation comprises two extensive software maintenance corpora:

• Enterprise Reference Corpus (Apache Fineract): Apache Fineract is an open-source core banking platform maintained by the Apache Software Foundation [8], providing financial service infrastructure (loan scheduling, multi-currency ledgers, portfolio management). We analyze a collected corpus of 2,925 maintenance issues from its issue tracker, comprising 3,811 granular requirement units, including 2,393 issues identified as resolved in the source metadata. Fineract serves as a domain-specific enterprise reference corpus in which maintenance issues can impose financial calculations, audit requirements, and state-consistency constraints; its banking domain limits direct generalization to other systems.

• AI Coding Benchmark Corpus (944 Analyzed Tasks): We analyze 944 tasks from three benchmark sources: (1) SWE-bench Pro public V1: 731 tasks (77.44%) from its original public split; (2) SWE-PolyBench Java: 165 tasks (17.48%) from its Java split; and (3) SWE-EVO: 48 tasks (5.08%) included in the analyzed corpus. These counts define the scope of our study rather than the complete populations of all benchmark releases. Across these 944 tasks, 3,446 itemized criteria assignments were systematically annotated with exact text quotes and code diff references.

The Immutable 18-Criteria Maintenance Taxonomy: Provenance and Grounding

To support a transparent and consistently applied cross-domain comparison, we establish an immutable 18-criteria maintenance taxonomy in three operational stages: (1) Standard-Based Concept Derivation: Development Activities ($A_1$–$A_6$) were adapted from the maintenance categories of ISO/IEC/IEEE 14764 [9] (Corrective, Adaptive, Perfective, Preventive), while Fulfillment Conditions ($B_1$–$B_{12}$) were adapted from the quality characteristics of ISO/IEC 25010 (SQuaRE) [10] formalized as state invariants; (2) Empirical Concretization & Codebook Freezing: Through inductive thematic analysis across 2,925 Apache Fineract maintenance issues (3,811 requirement units), standard categories were operationalized into concrete criteria and formally frozen into an immutable codebook (framework-snapshot.json); and (3) Uniform Cross-Corpus Application: This frozen taxonomy was applied without alteration across the 944 analyzed AI benchmark tasks, maintaining consistent operational definitions across the selected corpora. The proposed 18-criteria taxonomy is an empirically operationalized classification informed by ISO/IEC/IEEE 14764 and ISO/IEC 25010, rather than a direct reproduction of either standard. The six activity categories and twelve fulfillment conditions were developed through inductive analysis of Apache Fineract maintenance requirements, with their operational definitions and classification boundaries documented in the accompanying codebook.

Ontological Separation: Activities (A) vs. Conditions (B)

Conventional benchmarks collapse maintenance tasks into 1-dimensional categorical labels ("Bug Fix", "Feature"). However, in enterprise service maintenance, the developer’s maintenance activity (Activity, $A$) is ontologically distinct from the multi-step system invariants (Conditions, $B$) required to declare the task complete: $A \xrightarrow{\Delta S} B_{\text{temporal}} \xrightarrow{\text{triggers}} B_{\text{numeric}} \models B_{\text{invariant}}$. The taxonomy establishes 6 Development Activities ($A_1$–$A_6$: Product Operation, Structure/Refactoring, Testing/QA, Execution/Operations, Deployment/Migration, Process/Docs) and 12 Fulfillment Conditions ($B_1$–$B_{12}$: Functional Result, Numerical Precision, Data Consistency, Temporal Semantics, State Transition, Legacy Preservation, Concurrency Boundaries, Authorization, Performance, Service Continuity, Interface Compatibility, User Interaction), quantified across corpora in Table 2.

Code	Codebook Criterion Name	AI All ($N = 944$)	Pro ($N = 731$)	Poly ($N = 165$)	EVO ($N = 48$)
A1	Product Operation & Business Rules	426 (45.13%)	300 (41.04%)	84 (50.91%)	42 (87.50%)
A2	Structure & Implementation Method	572 (60.59%)	505 (69.08%)	50 (30.30%)	17 (35.42%)
A3	Testing & Quality Assurance	94 (9.96%)	70 (9.58%)	10 (6.06%)	14 (29.17%)
A4	Execution & Operations	122 (12.92%)	93 (12.72%)	20 (12.12%)	9 (18.75%)
A5	Deployment & Environment Migration	60 (6.36%)	39 (5.34%)	9 (5.45%)	12 (25.00%)
A6	Development Process & Docs	53 (5.61%)	25 (3.42%)	17 (10.30%)	11 (22.92%)
B1	Functional Result & Input Condition	315 (33.37%)	276 (37.76%)	25 (15.15%)	14 (29.17%)
B2	Numerical & Calculation Precision	65 (6.89%)	49 (6.70%)	11 (6.67%)	5 (10.42%)
B3	Data Relationship & Consistency	214 (22.67%)	184 (25.17%)	25 (15.15%)	5 (10.42%)
B4	Temporal Semantics & Ordering	86 (9.11%)	77 (10.53%)	7 (4.24%)	2 (4.17%)
B5	State Transition & Business Workflow	131 (13.88%)	109 (14.91%)	20 (12.12%)	2 (4.17%)
B6	Preservation of Existing Data & Behavior	368 (38.98%)	336 (45.96%)	17 (10.30%)	15 (31.25%)
B7	Concurrency & Transaction Boundaries	42 (4.45%)	35 (4.79%)	4 (2.42%)	3 (6.25%)
B8	Authorization & Data Protection	154 (16.31%)	130 (17.78%)	16 (9.70%)	8 (16.67%)
B9	Performance & Resource Utilization	83 (8.79%)	60 (8.21%)	13 (7.88%)	10 (20.83%)
B10	Service Continuity & Failure Handling	87 (9.22%)	72 (9.85%)	8 (4.85%)	7 (14.58%)
B11	Interface & Environmental Compatibility	391 (41.42%)	311 (42.54%)	51 (30.91%)	29 (60.42%)
B12	User Interaction Semantics	183 (19.39%)	169 (23.12%)	4 (2.42%)	10 (20.83%)

Annotation Procedure, Semantic Validity, and Boundary Resolution

To ensure criteria assignments reflect authentic operational semantics rather than keyword matching, we enforced strict annotation protocols: (1) Operational Rules: $A$ labels required concrete code diff intent ($A_1$ functional logic, $A_2$ structural refactoring), while $B$ conditions required explicit specification mandates for state invariants or quality constraints; (2) Boundary Adjudication: (a) $B_1$ vs. $B_2$: input validation and boolean returns were coded as $B_1$, whereas financial calculations, rounding, or multi-threaded counters were coded as $B_2$; (b) $A_1$ vs. $A_2$: functional behavior modifications were classified as $A_1$, whereas internal refactoring preserving behavior was classified as $A_2$; (c) $B_3$ vs. $B_6$: multi-entity referential integrity was coded as $B_3$, whereas legacy data retention and non-regression guarantees were coded as $B_6$; (3) Batch Protocol & Verification: Semantic coding across the 944 tasks was conducted by the primary investigator via an iterative batch protocol binding all 3,446 criteria assignments to verbatim textual quotes and rationales. An automated pipeline (validate_*.py) programmatically verified contiguous task sequences, file hashes, exact string offsets in raw JSON, and codebook schema compliance, ensuring quotation fidelity rather than dual-coder consensus (documented in § 6); and (4) Procedural Consistency: Applying this immutable codebook across both corpora ensures procedural consistency while preserving domain characteristics.

Three-Level Analytical Protocol: Distinct Dimensions of Construct Validity

To separate the breadth of completion conditions from their relationships, we use three complementary lenses (Figure 1): (1) Condition Breadth ($|B|$) counts distinct tagged fulfillment criteria in a task, without asserting that they interact; (2) Corpus Co-occurrence ($\phi$) measures pairwise association across issue specifications, without identifying direction or causality; and (3) Source-Grounded Directional Dependencies traces action–condition or condition–condition obligations when supported by requirement text. The figure combines findings from FINERACT-753 and FINERACT-2338 as an illustrative synthesis, not a verbatim single-issue specification. Subsequent inspection of selected benchmark assertions is an optional evaluation-facing application of these three lenses, not a fourth measurement of task complexity.

E1: Unmasking Semantic Heterogeneity Beyond Surface Metrics

Unmasking Semantic Heterogeneity Beyond Surface Churn

Conventional benchmark leaderboards and dataset summaries commonly categorize maintenance tasks under broad native issue classifications (e.g., "Bug Fix") alongside summary metrics of code churn and file counts. To empirically evaluate whether tasks sharing comparable surface characteristics exhibit comparable requirement structures, we constructed a controlled matched-pair analysis across the 944 benchmark tasks.

We identified pairs of tasks $(t_i,t_j)$ that satisfied three equivalence constraints:

1. Identical native repository issue classification.

2. Identical modified file count $(|Files(t_i)| = |Files(t_j)|)$.

3. Code churn: $|LOC_i - LOC_j| \le\max\{3, 0.30\max(LOC_i, LOC_j)\}$.

For each matched pair, we computed the Jaccard distance over their active fulfillment condition sets $(B(t_i), B(t_j))$:

We emphasize that this matched-pair construction controls for surface task similarity (repository issue classification, file modification count, and line churn magnitude), but does not imply or guarantee identical cognitive problem-solving difficulty. Rather, it isolates whether tasks with matched issue classifications and file counts and churn within tolerance share common fulfillment criteria. Across $N = 1,177$ surface-matched pairs (714 SWE-bench Pro pairs and 463 PolyBench Java pairs, spanning 581 unique tasks), the mean Jaccard distance is $\bar{d}_J = 0.8138$ (median 1.0000, sample standard deviation $\sigma= 0.2745$). Crucially, 57.4% of all controlled pairs (676 pairs) exhibit $d_J = 1.0000$, meaning their underlying fulfillment conditions are completely disjoint despite matching issue classifications and modified-file counts, with code churn within the specified tolerance.

A representative example is observed in internetarchive/openlibrary: OpenLibrary #0143 (29 LOC, 1 file, tagged Bug Fix) modifies author-edition entity relationships requiring relational state consistency $(A_1 \to B_3, B_5)$, whereas OpenLibrary #0233 (27 LOC, 1 file, tagged Bug Fix) corrects string sanitization and character encoding compatibility $(A_1 \to B_1, B_6, B_{11})$. Although both tasks have comparable surface characteristics (1 modified file each, 27 and 29 LOC, native ’Bug Fix’ tag), their underlying semantic fulfillment conditions are completely disjoint $(d_J = 1.0000)$, demonstrating that surface metadata fails to reflect whether a task demands relational database integrity (#0143) or character encoding transformations (#0233).

Stratified Benchmark Family Profile (N=944)

Rather than aggregating AI benchmarks into a monolithic entity, we analyze empirical distributions stratified across the three constituent benchmark families in Table 2 (gate3_stratified_family_metrics.csv).

Multi-Attribute Co-presence Complexity ($|B|$)

Table 3 reports the distribution of $|B|$ (gate3_coupling_complexity.csv).

Metric / Bin	Fineract ($N = 2,925$)	AI All ($N = 944$)	Pro ($N = 731$)	Poly ($N = 165$)	EVO ($N = 48$)
Mean $|B|$	1.78 (1.7798)	2.24 (2.2447)	2.47 (2.4733)	1.22 (1.2182)	2.29 (2.2917)
Median $|B|$	2.0	2.0	2.0	1.0	2.0
$|B| = 0$	331 (11.32%)	3 (0.32%)	0 (0.00%)	3 (1.82%)	0 (0.00%)
$|B| = 1$	872 (29.81%)	248 (26.27%)	109 (14.91%)	126 (76.36%)	13 (27.08%)
$|B| = 2$	1,097 (37.50%)	357 (37.82%)	304 (41.59%)	33 (20.00%)	20 (41.67%)
$|B| = 3$	451 (15.42%)	231 (24.47%)	219 (29.96%)	3 (1.82%)	9 (18.75%)
$|B| = 4$	112 (3.83%)	79 (8.37%)	77 (10.53%)	0 (0.00%)	2 (4.17%)
$|B| = 5$	37 (1.26%)	15 (1.59%)	12 (1.64%)	0 (0.00%)	3 (6.25%)
$|B| = 6$	21 (0.72%)	6 (0.64%)	6 (0.82%)	0 (0.00%)	0 (0.00%)
$|B| = 7$	4 (0.14%)	3 (0.32%)	2 (0.27%)	0 (0.00%)	1 (2.08%)
$|B| = 8$	0 (0.00%)	2 (0.21%)	2 (0.27%)	0 (0.00%)	0 (0.00%)
Subtotal ($|B| \ge2$)	1,722 (58.87%)	693 (73.41%)	622 (85.09%)	36 (21.82%)	35 (72.92%)

Attribute Breadth Is Not Joint Fulfillment Structure

In the analyzed corpora, SWE-bench Pro has a higher mean number of tagged fulfillment conditions ($|B| = 2.47$) than Apache Fineract (1.78, Welch’s $t = 15.78$, $p < 10^{-50}$). Thus the contrast cannot be reduced to a claim that benchmark tasks contain fewer conditions. But $|B|$ measures breadth only: two tasks with the same count may differ in which conditions must hold jointly and whether one condition triggers another. This result does not by itself show that enterprise issues are universally deeper or more difficult.

This motivates the next question: how do specified conditions combine? Section 5 first examines corpus-level pairwise association and then traces source-grounded directional relationships in enterprise issues. Selected benchmark cases subsequently illustrate why the distinction can matter for evaluation, without treating test-assertion coverage as our principal empirical outcome.

E2: Joint Fulfillment Patterns and Directional Dependencies

Macro Statistical Co-occurrence Divergence (phi)

To measure how often pairs of fulfillment conditions are specified together, we compute Pearson’s $\phi$ across the studied corpora (Table 4; gate3_phi_correlation_matrix.csv). This is a measure of association, not evidence that one condition triggers another. The near-zero $\phi$ for $B_4$ and $B_2$ in the pooled AI tasks ($-0.013$, $p = 0.825$) indicates little observed pairwise association, not proof of independence or the absence of dependent tasks.

Pair	Condition Pair (non-directional)	Fineract ($N = 2,925$) Co-occur / $\phi$	AI Bench ($N = 944$) Co-occur / $\phi$ ($p$)
$B_4 \leftrightarrow B_2$	Temporal / Numerical	254 / +0.430	5 / −0.013 (0.825)
$A_1 \leftrightarrow B_4$	Business Rule / Temporal	471 / +0.309	42 / +0.024 (0.496)
$A_1 \leftrightarrow B_2$	Business Rule / Numerical	440 / +0.343	34 / +0.039 (0.246)
$A_1 \leftrightarrow B_3$	Business Rule / Consistency	538 / +0.329	115 / +0.094 (0.005)
$B_4 \leftrightarrow B_3$	Temporal / Consistency	138 / +0.069	22 / +0.022 (0.501)
$B_4 \leftrightarrow B_6$	Temporal / Preservation	84 / +0.068	28 / −0.042 (0.246)
$B_2 \leftrightarrow B_3$	Numerical / Integrity	189 / +0.226	14 / −0.007 (1.000)
$B_2 \leftrightarrow B_6$	Numerical / Preservation	64 / +0.036	19 / −0.054 (0.113)
$B_3 \leftrightarrow B_6$	Integrity / Preservation	89 / +0.053	63 / −0.106 (0.001)
$B_7 \leftrightarrow B_6$	Isolation / Preservation	8 / +0.005	14 / −0.025 (0.519)
$B_7 \leftrightarrow B_3$	Isolation / Integrity	20 / +0.040	7 / −0.031 (0.451)

Conditional and higher-order co-specification. To make the joint-condition contrast interpretable, we calculate the proportion of $B_4$-tagged issues that also specify $B_2$. In Fineract, this is 254/520 (48.85%); in the pooled AI benchmark corpus, it is 5/86 (5.81%). The descriptive conditional risk ratio is 8.40 (95% CI: 3.57–19.76; two-sided Fisher’s exact test, $p = 3.14 \times10^{-16}$). An exploratory higher-order check finds $B_4$ and $B_2$ jointly specified with at least one of $B_3$ (data consistency) or $B_6$ (preservation) in 108/2,925 Fineract issues versus 1/944 benchmark tasks. These counts are computed from the frozen criterion annotations (gate5_multi_condition_summary.csv); they measure co-specification, not directed dependency rates or causal depth. Because the corpora represent different software domains, the contrast cannot be attributed to benchmark design alone.

To examine directionality, Section 5.2 separately inspects source-grounded requirements.

Source-Grounded Directional Fulfillment in Enterprise Maintenance

To characterize what a joint completion obligation can look like, we examine 16 source-grounded business requirement relationships in Apache Fineract (Table 5; gate2_dependencies.csv). The FINERACT-2338 examples connect backdating to repayment recalculation and preservation constraints; FINERACT-2836 specifies transaction boundaries for savings-charge processing. Supporting quotations were checked against source text (gate2_raw_source_snippets.json). These purposively documented relationships illustrate directionality; they are not a frequency estimate or evidence that all Fineract issues exhibit this structure.

ID	Issue	Classification	Structural Chain	Verbatim Jira Excerpt
Causal Cascade Focus: Multi-Tier Loan Modification (FINERACT-2338)
DEP-01	FINERACT-2338	Explicit Dependency	$A_1 \to B_4$	“Backdate can be done on already paid Installments as well”
DEP-02	FINERACT-2338	Explicit Dependency	$B_4 \to B_2$	“Repayment schedule will be recalculated with New EMI and Interest from the Interest applied schedule date”
DEP-03	FINERACT-2338	Explicit Dependency	$B_2 \to B_6$	“No of Installments will remain the same, only EMI and Interest would get affected”
DEP-04	FINERACT-2338	Explicit Dependency	$B_4 \to B_3$	“standard accounting rules and not charge-off accounting rules”
Cross-Entity Invariant Preservation & Transaction Isolation
DEP-05	FINERACT-753	Explicit Dependency	$A_1 \to B_4, B_2$	“now reverse the deposit of 500”
DEP-07	FINERACT-2836	Explicit Dependency	$B_7 \to B_6$	“Each due savings-charge installment is processed in an independent transaction.”

Illustrative Evaluation Implications in Four Benchmark Tasks

The preceding results concern specified conditions and their relationships, not agent success or general test quality. As a secondary illustration, we inspected task text, source diffs, and selected test assertions for four benchmark tasks (Table 6; data_and_tables/ai_case_sources/). These cases show why a multi-condition specification can call for more than a localized check, and also show instances of appropriate alignment. They are not representative samples and do not establish full-harness behavior.

Axis / Task	Specification Mandate ($R_{\mathrm{spec}}$)	Evaluation Test Oracle ($R_{\mathrm{eval}}$)	Observed Scope	Construct Alignment or Gap Analysis
Axis 1 (Direct) NodeBB #0030	Explicit ($A_{5} \rightarrow B_{6}$): DB migration script (chat_allow_list.js, 44 LOC) migrating legacy restrictChat records.	Leaf Unit Assertion: Asserts canMessageUser() on mock objects in memory (test/messaging.js, L103–121).	In-memory leaf	Construct Underrepresentation: Direct test patch asserts leaf permissions; DB migration script unasserted in analyzed test patch.
Axis 2 (Scope) Element-Web #0139	Explicit badge update; inferred cache persistence: Re-calculate unread badges; IndexedDB cache synchronization is inferred rather than explicitly mandated.	Mocked State: Asserts helper return values under synthetic event mocks (test/Unread-test.ts).	In-memory mock	Mocked Logic vs. Persistent Cache: In-memory transitions verified; asynchronous IndexedDB cache persistence outside evaluation scope.
Axis 3 (Hierarchy) RocketMQ #0819	Explicit ($B_{7}, B_{2}$): Thread-safe counter insertion (StatsItemSet, 20kL) under JVM thread contention.	Thread Stress Test: 100 worker threads executing 10,000 tasks (StatsItemSetTest.java).	JVM thread contention	Single JVM Concurrency Met: Checks concurrent counter accumulation within a single JVM; distributed transaction boundaries are not mandated by the task.
Axis 4 (Aligned) Navidrome #0495	Explicit ($A_{4}, B_{6}, B_{10}$): Online SQLite DB backup, destructive DDL recovery, and retention pruning.	Integrated Engine Test: Executes backup, runs destructive DDL, restores DB, asserts pruning (db/backup_test.go).	Persistent DB state	Aligned Invariant Testing: Positive case in which the inspected tests directly address the explicitly required persistent-state operations.

We establish a unified 3-Question Comparative Protocol applied systematically across all four axes: (1) Specification Mandate ($R_{\mathrm{spec}}$): operational state transformations and invariants required by the raw task description and DoD; (2) Oracle Assertion Scope ($R_{\mathrm{eval}}$): specific conditions directly asserted by test code in the benchmark’s evaluation patch; and (3) **Construct

Alignment or Structural Gap: structural correspondence or divergence between mandate and assertions. To distinguish task demands from implementation details, requirement conditions are categorized into three provenance tiers: (a) Explicit Requirement (directly stated in prompt or interface); (b) Developer-Implemented Obligation (concrete implementation in reference patch); and (c) Inferred Requirement (deduced from architectural context but unstated in prompt).

Comparative Case Analysis across the Three Questions and Provenance Tiers

Applying the protocol illustrates different alignments between explicit task obligations and the inspected assertions; it does not quantify the prevalence of evaluation gaps:

Axis 1 (NodeBB #0030: Construct Underrepresentation). Mandate ($R_{spec}$): The issue explicitly mandates an upgrade path in src/upgrades/4.3.0/chat_allow_list.js to migrate legacy restrictChat=1 records (Explicit Requirement), implemented via a 44-line batch migration script (batch.processSortedSet, 500 records/chunk) in the developer patch (+383/-50 lines across 12 files). Oracle ($R_{eval}$): Static inspection of the analyzed test patch (test/messaging.js:L103-121) confirms it creates mock users in memory with pre-populated chatAllowList arrays, asserting Messaging.canMessageUser() with zero assertions exercising src/upgrades/. Structural Gap: Static comparison confirms developer migration omission in an ablated patch modifying 11 files, coupled with complete migration coverage absence in analyzed test assertions. Within this verified static inspection scope, assertions evaluate localized leaf permissions while leaving the database migration unasserted.

Axis 2 (Element-Web #0139: Mocked Logic vs. Inferred Cache Persistence). Mandate ($R_{spec}$): The issue mandates recomputing unread badges when receipts arrive across thread timelines, implemented in src/Unread.ts (Explicit Requirement and Developer Obligation). Persistent IndexedDB cache synchronization across network streams is an Inferred Requirement. Oracle ($R_{eval}$): Test code in test/Unread-test.ts asserts in-memory helper returns under synthetic mocked events (mocked(haveRendererForEvent)). Structural Gap: While in-memory state transitions for the Explicit Requirement are validated, persistent IndexedDB synchronization and asynchronous event reordering remain unasserted, reflecting the boundary between unit mock verification and inferred persistent client state.

Axis 3 (RocketMQ #0819: JVM Contention vs. Transaction Boundaries). *Mandate

($R_{\mathrm{spec}}$): The task mandates thread-safe counter accumulation under concurrent thread contention in StatsItemSet (Explicit Requirement), resolved using putIfAbsent (Developer Obligation); distributed transactions were neither requested nor implied. Oracle ($R_{\mathrm{eval}}$): Test code in StatsItemSetTest.java spawns 100 worker threads executing 10,000 tasks, asserting atomic accumulation of 20,000L. Structural Alignment: The inspected tests exercise concurrent accumulation within a single JVM, in line with the Explicit Requirement.

Axis 4 (Navidrome #0495: Aligned Invariant Testing Counterexample). Mandate ($R_{\mathrm{spec}}$): The task explicitly mandates online SQLite database backup, destructive DDL recovery, and backup retention pruning (Explicit Requirement), implemented across cmd/backup.go and db/backup.go (Developer Obligation). Oracle ($R_{\mathrm{eval}}$): Test code in db/backup_test.go creates real SQLite backups, executes destructive DDL (DELETE FROM sqlite_master), verifies schema restoration, and asserts retention pruning policies. Structural Alignment: The inspected tests provide direct evidence of alignment with the explicitly specified SQLite backup, schema restoration, and retention requirements.

Threats to Validity

Internal & Construct Validity

A primary threat to internal and construct validity is potential subjective bias in applying the 18-criteria taxonomy. Semantic coding across 944 AI benchmark tasks was conducted by a single investigator against frozen codebook definitions rather than a multi-researcher adjudication committee. While all 3,446 criteria assignments were validated through automated scripts (validate_*.py), programmatic offset checking guarantees textual quotation fidelity and source hash integrity rather than multi-coder consensus; we make no claim of dual-coder agreement statistics. We mitigated potential subjective drift through four procedural controls: (1) freezing the 18-criteria operational definitions and boundary rules prior to corpus-wide annotation (framework-snapshot.json), anchoring criteria in domain-neutral standards (ISO/IEC/IEEE 14764, ISO/IEC 25010); (2) requiring an explicit verbatim textual quote from the task specification for every criterion assignment; (3) providing all evidence quotes, code diff references, and validation scripts in our replication package for independent audit; and (4) verifying all 16 Fineract dependency chains via automated byte-level offset matching against raw Jira JSON archives (gate2_raw_source_snippets.json). Additionally, to address potential matched-pair clustering in E1 across identical repositories, we confirmed that semantic heterogeneity persists across cross-repository pairs regardless of repository origin.

External Validity & Scope Boundaries

Evaluating 944 benchmark tasks across three established benchmarks spanning multiple languages (Python, Java, TypeScript, Go), we explicitly delimit our empirical scope: (1) Empirical Scope vs. Universal Generalization: Findings reflect quantitative replicability and static verification across the analyzed corpora, rather than a universal claim that all AI benchmarks lack causal dependencies; well-designed harnesses can evaluate persistent state operations (Navidrome #0495); (2) Static Oracle Analysis vs. Dynamic Solve Rates: We examine test oracle construct representation via diffs and assertions, without claiming dynamic execution pass rates for unexecuted models; (3) Domain Confounding: Fineract is a banking system with temporally dependent ledger requirements, while the benchmark corpus spans different projects. Using a shared ISO-informed taxonomy makes labels comparable but does not control for domain composition or demonstrate benchmark selection bias. This also applies to the conditional and exploratory higher-order co-specification rates, whose sparse benchmark cells limit precision; and (4) Dependency and Case Selection: The 16 source-grounded relationships and four oracle case studies illustrate possible structures, not their prevalence in the populations. We do not establish that enterprise maintenance universally requires more dependencies, measure production failure rates, or assess a complete execution harness from an inspected test patch.

Discussion & Benchmark Design Implications

Re-evaluating the Benchmark Complexity Paradigm

Our findings demonstrate that software maintenance complexity cannot be adequately characterized solely by surface metadata such as lines of code churn or file navigation graphs. In E1, 1,177 pairs matched on native issue classification and modified-file count, with code churn within the prespecified tolerance, exhibited mean fulfillment-set Jaccard distance 0.8138; 57.4% (676 pairs) had completely disjoint condition sets ($d_J = 1.0000$). This indicates that matching on native issue classification, exact modified-file counts, and code churn within the specified tolerance does not ensure semantic similarity in fulfillment conditions.

Furthermore, SWE-bench Pro has higher mean attribute breadth ($|B| = 2.47$) than Apache Fineract (1.78) in the studied corpora. Thus the number of tagged fulfillment conditions alone cannot characterize maintenance requirement structure; their co-specification patterns and source-grounded dependencies must also be considered. The Fineract examples show how a backdated operation can impose linked recalculation and preservation obligations. The co-occurrence contrast and the additional conditional rates ($P(B_2 \mid B_4)$: 48.85% in Fineract versus 5.81% in the pooled AI corpus) are consistent with different patterns of jointly specified conditions. They do not measure causal dependency strength, and domain composition prevents attributing this contrast to benchmark design alone. Selected oracle inspections illustrate implications for checking explicit obligations rather than forming the principal empirical claim.

Implications for Describing and Evaluating Maintenance Tasks

The empirical results motivate three reporting and design considerations, rather than a claim that any one test architecture is universally superior:

1. Report fulfillment-condition profiles alongside surface size: Task descriptions and benchmark reports should distinguish native issue labels, file counts, and churn from the types and breadth ($|B|$) of required completion conditions. In E1, superficially matched tasks frequently have different condition sets.

2. Represent linked conditions when the specification warrants them: Beyond listing attributes, descriptions should preserve source-grounded dependencies, such as a backdated change that triggers

financial recalculation while retaining historical consistency. The observed $\phi$ differences motivate closer analysis but do not themselves prove directional dependencies or broader causal mechanisms.

3. Align checks with explicitly required multi-condition completion: If a task specifies a migration, transaction boundary, or state-preservation obligation, evaluation design should consider checking that obligation jointly with the local functional change. NodeBB and Navidrome illustrate contrasting scopes in inspected patches. This is a downstream implication of requirement analysis, not evidence that benchmark oracles generally fail. When possible, benchmark reports should make these verification boundaries transparent.

Conclusion

Using 944 analyzed coding-benchmark tasks and 2,925 Apache Fineract issues, we examined how maintenance completion conditions differ in breadth, co-occurrence, and source-grounded directionality.

RQ1 (Semantic Heterogeneity and Breadth): Among 1,177 pairs matched on issue type, modified-file count, and churn tolerance, mean fulfillment-set Jaccard distance was 0.8138, with 57.4% disjoint sets. SWE-bench Pro also had greater mean attribute breadth ($|B| = 2.47$) than Fineract (1.78): an explanation based on fewer benchmark attributes is therefore insufficient for these corpora.

RQ2a (Co-occurrence): Temporal and numerical conditions exhibited positive association in Fineract ($B_4/B_2$, $\phi= +0.430$, 254 issues) but near-zero association in the pooled benchmark corpus ($\phi= -0.013$, $p = 0.825$, 5 tasks). Among items specifying $B_4$, $B_2$ also appears in 254/520 Fineract issues (48.85%) versus 5/86 benchmark tasks (5.81%). These domain-confounded co-specification contrasts do not establish directional dependency rates or prove missing dependencies across AI benchmarks.

RQ2b (Directional Fulfillment): Sixteen source-grounded Fineract relationships illustrate how maintenance completion may depend on multiple linked obligations, including backdating, recalculation, and preservation constraints. Four selected benchmark cases show that the inspected evaluation assertions address explicit obligations with varying scope; these cases do not establish a general oracle-failure rate.

Together, our results establish that matching surface task characteristics does not ensure similar fulfillment conditions and that condition counts alone do not characterize observed joint-specification patterns. Maintenance benchmarks should therefore distinguish condition breadth, statistical association, and source-grounded directional obligations. Test-assertion alignment remains a downstream application rather than the principal empirical claim.

Data Availability

Analysis datasets, selected source-grounded evidence, and targeted verification scripts are publicly available in the replication package: public GitHub repository. Some upstream source materials and complete benchmark execution environments are not redistributed.