WorksComputer Science
WAR-CRAFT: Refining Web API Recommendations through Requirement Auditing
Abstract
Web API recommenders return a set of APIs for a mashup request but do not check it against provider documentation, the main public evidence of an API's capabilities. The set can therefore contain unsupported or redundant APIs and miss requested capabilities. Filling every apparent gap does not fix this, because a gap may already be served, may not need its own API, or may have no documented provider. We study post-recommendation refinement, which revises a frozen recommender's output using only the request, the catalog, and provider documentation. We propose WAR-CRAFT, which treats refinement as three decisions. An LLM removes unsupported or subsumed APIs by judging candidates jointly, then audits the remaining set against a requirement checklist derived from the request. For each uncovered requirement, a fine-tuned catalog-constrained Resolver proposes a catalog API or abstains, and an independent documentation check admits only supported proposals. On ProgrammableWeb with 7 heterogeneous upstream recommenders, WAR-CRAFT raises mean SetF1 from 0.558 to 0.644, above the strongest of 5 refinement baselines (62.5%), and roughly halves functional redundancy (0.127 to 0.065) without changing mean functional coverage. Gains range from +2.2 points on generative recommenders to +18.2 on fixed-cutoff rankers. At matched set sizes, WAR-CRAFT still outperforms truncating the upstream output, so the gain is not explained by set-size calibration alone. Our code is available at https://github.com/ZsXu-enrico/WAR-CRAFT.
Introduction
Web platforms expose data and functionality through public Web APIs, and Web applications compose these APIs instead of reimplementing each function [1, 23]. Such an application, commonly called a mashup [11], usually has several functional requirements served by different providers, so selecting its APIs is a set-level decision. Two properties of the Web API ecosystem make this decision hard. Providers are independent, so an API’s implementation is opaque and its documentation is the main public evidence of what it can do. Providers also overlap in function, while each added API brings its own documentation, credentials, and usage terms, so an irrelevant or duplicated API costs integration effort without adding functionality. LLM-based agents that select Web services or tools for a task face the same set-level decision [19, 26].
Web API recommenders estimate the relevance of each API to a mashup description, using content-based [24, 38], collaborative-filtering [11, 16], hybrid [10, 22], or LLM-based methods [18, 39, 43]. While they differ in how they score APIs, they share one output form: a ranked list cut at a fixed size or a generated set, neither checked against provider documentation as a set. Two gaps remain between recommending APIs and establishing their suitability for an application. 1) Unverified set-level suitability. Selecting relevant APIs does not by itself establish whether each API has a documented contribution, whether contributions overlap unnecessarily, or whether required functions remain uncovered. Under DOEM’s functional labels [28], only 53.25% of the functions provided by the initial sets of seven upstream recommenders in our evaluation are required by the mashup on average, and 12.65% of their functional provision is duplicated. 2) Missing or inconsistent API-level justification. Many existing recommenders return API names or rankings without explaining how each API serves the request. Even WAR-R1, which generates rationales, shows mismatches in our evaluation: 8.56% of recommended API instances lack rationales, while 7.53% of API instances mentioned in its rationales are absent from the corresponding recommendation sets. It also shows rationales that misstate an API’s functionality [39]. This motivates pairing each selected API with a request-specific justification checked against the request and API documentation.
Both gaps can be addressed after recommendation by checking the returned set against the request and API documentation without access to the recommender’s internals. Doing so, however, does not mean adding an API for every requirement that appears uncovered. A requirement may already be served by a retained API, may not warrant a separate API, or may have no documented provider in the catalog. In our experiments, always adding the top retrieved API for each uncovered requirement lowers SetF1 below filtering alone (0.581 versus 0.591), and the Resolver abstains on 69.7% of requirement-level decisions. Whether to add is therefore a decision in its own right.
We therefore study post-recommendation refinement of Web API sets (Figure 1): given a request, an API catalog, and the output of an existing recommender, revise the set by removing unsupported or redundant APIs and selectively adding APIs for unmet requirements, without accessing or retraining the recommender. The refinement is evidence-constrained: each decision to keep, remove, or add an API must be justified by the request and API documentation.
Figure 1. Post-recommendation refinement for a mashup request. The unsupported API B is removed, and the uncovered map-display and email-delivery requirements are filled with catalog APIs C and D after verification against their documentation. Each selected API receives a functional role describing its contribution.
We propose WAR-CRAFT, a recommender-agnostic framework that treats refinement as three decisions: delete, detect gaps, and add or abstain. A Role Writer jointly evaluates the initial candidates, removes unsupported or fully subsumed APIs, and assigns functional roles to retained APIs. A Joint Requirement Auditor constructs a requirement checklist and assesses its coverage in a single step, grounding requirements in the request and coverage in API documentation. For each uncovered requirement, a fine-tuned, catalog-constrained Resolver proposes an eligible API or abstains. A Commit Verifier checks the proposed capabilities against API documentation and writes a functional role for each accepted addition. A structured audit trace records the decisions and their supporting evidence. We evaluate WAR-CRAFT on ProgrammableWeb with seven heterogeneous upstream recommenders using the same refinement configuration. It improves SetF1 and CovF1 for all seven sources and reduces average functional redundancy. It also outperforms truncating the upstream output to its own set size or to the recorded set size, so the gains are not explained by set size alone, and it exceeds the strongest independent refinement baseline by 1.92 points. LLM-based evaluation, supplemented by human validation, further indicates that the generated roles are relevant to mashup requirements, grounded in API documentation, and clearly expressed. Our contributions are:
We formulate post-recommendation refinement of Web API sets and show that whether to add an API is a decision in its own right: always filling uncovered requirements lowers SetF1 below filtering alone.
We design WAR-CRAFT around a Resolver that, given the current set, decides for each uncovered requirement whether to add a catalog API or abstain, with documentation-grounded filtering, auditing, verification, and functional roles.
On seven upstream recommenders, WAR-CRAFT improves SetF1 and CovF1 for every source and outperforms set-size truncation, five refinement baselines, and a Toolken++-style abstaining retriever; human audits support these results.
Related Work
Web API Recommendation
Web API recommendation includes content-based, collaborative-filtering (CF), hybrid, and LLM-based approaches. Content-based methods match mashup and API descriptions through compatibility-aware text mining [24] or hierarchical BERT-based matching [38]. CF methods exploit historical mashup–API interactions [17]. Hybrid methods combine interactions with side information through multi-view hypergraphs [10] or causality-inspired context adjustment [22]. Complementary work studies API set and bundle recommendation, considering joint API selection, compositional matching, and diversity within recommended sets [11, 12, 41]. STAR sequentially recommends services while updating residual requirements [29]. Recent LLM-based methods generate recommendations or enhance representations [25, 31, 43]; WAR-R1 generates API-level rationales [39], and MARS uses multi-agent reasoning [18]. These methods primarily produce API recommendations; WAR-CRAFT instead audits and revises their frozen outputs against mashup requirements and API documentation, without modifying the upstream models.
Tool Retrieval and Selection
Tool retrieval identifies relevant APIs for LLM agents through learned retrievers [26], reranking [44], and document expansion or intent-based retrieval [4, 19]. ToolkenGPT learns tool-token embeddings [13], while ToolGen uses dedicated tokens for tool retrieval and calling [34]. Toolken+ incorporates tool documentation into reranking and introduces a learned reject option [40]. COLT models tool collaboration to improve retrieval completeness [27], while AnyTool and MCP-Zero support tool discovery during task solving [5, 6]. PTR retrieves initial tool bundles, maps their functional coverage, removes tools that serve no requirement, and adds tools for unmet requirements [7]. WAR-CRAFT incorporates token-based selection and abstention into evidence-constrained refinement of frozen Web API recommendations for mashup development. Its joint audit identifies which requested capabilities require external services and checks their coverage by the retained APIs. Proposed additions undergo a separate check of their documented support for the requirements that motivated them.
LLM-Based Refinement and Verification
LLMs have been used for zero-shot ranking [14] and reranking supplied candidates, as in LLM4Rerank [8]. Other methods leverage LLM-derived preference and item-relation knowledge to improve recommendations [35, 42]. Beyond recommendation, LLMs can revise their outputs through feedback and reflection [20, 30]. RARR further demonstrates evidence-based revision of unsupported claims [9]. For fine-grained verification, FActScore and RefChecker check factual support at the level of atomic facts and claim triplets, respectively [15, 21]. Beyond reranking a fixed candidate list, WAR-CRAFT can remove existing APIs and select additions from the catalog. Each proposed addition is accepted only if its documentation supports the corresponding requirement.
Problem Definition
Let be a mashup described by a natural-language request , and let denote the Web API catalog. Each API has a catalog description . A frozen recommender maps the mashup request and catalog to an initial API set
We study post-recommendation refinement. The refiner is given only , with kept in its upstream output order; it neither modifies nor accesses its parameters, scores, or intermediate representations. It returns
Here, is the refined API set. For each , we represent its functional role by , a concise natural-language description of one observable contribution to the mashup supported by and . The audit trace records refinement decisions together with their supporting evidence and rationales.
Let and denote the deleted and added API sets, respectively. The refined set is
The task imposes three constraints. First, every retained or added API must have a documented capability that can support a function requested by . Second, a supported initial API may be removed as redundant only when its documented mashup-relevant contribution is fully subsumed by another retained initial API, without losing a distinction required by the mashup. Third, additions must target request-grounded capabilities not already covered by the retained set. Consequently, the output cardinality can change through deletion and coverage-driven addition, rather than being constrained by a fixed cutoff.
WAR-CRAFT
Figure 2 presents WAR-CRAFT, a recommender-agnostic framework for refining the output of a frozen Web API recommender. The Role Writer filters into and assigns functional roles. The Joint Requirement Auditor jointly constructs a checklist from and assesses coverage, yielding missing-item records . For each missing query, the Resolver selects an eligible catalog API or abstains; deduplicated API proposals form . The Commit Verifier checks their documented support for the originating queries and assigns functional roles to accepted additions, which are then merged with to form . If the Auditor returns COMPLETE, the Resolver and Commit Verifier are skipped and . The framework returns (Section 3); only the Resolver is fine-tuned.
Figure 2. Overview of WAR-CRAFT. The framework refines API recommendations through role writing and joint requirement auditing, invoking the Resolver and Commit Verifier only when requirements remain uncovered.
Set-Aware API Filtering and Role Writing
To filter unsupported and functionally redundant APIs, the Role Writer jointly evaluates all candidates in using the mashup request and their full catalog descriptions in a single LLM call, allowing redundancy to be judged relative to other candidates. Canonical API names are withheld to focus evaluation on documented capabilities. Each KEEP decision includes a concise, verb-led functional role , while each DELETE decision includes a reason code (UNSUPPORTED or REDUNDANT) and a brief rationale.
The Role Writer first removes candidates whose documented capabilities do not support any observable function requested by the mashup. A supported candidate is removed as redundant only when another retained API fully subsumes its documented mashup-relevant contribution without losing distinctions required by the request, such as the data source or communication channel. Partial overlap is insufficient: in Best Dining, Google Maps and Yelp Fusion both support restaurant search but additionally provide map visualization and restaurant ratings and reviews, respectively. Among interchangeable candidates, the Role Writer retains the API whose documented capabilities most directly match the request.
Candidates assigned KEEP form the retained set
where contains the candidates assigned DELETE. The retained set, which may be empty, is passed to the Joint Requirement Auditor for coverage assessment. Deleted candidates cannot be added later.
Joint Requirement and Coverage Audit
The Joint Requirement Auditor jointly identifies external-service requirements and checks their coverage by the retained APIs. In one LLM call, it constructs an ordered checklist from and the catalog descriptions and functional roles of the APIs in . Retained APIs use arbitrary identifiers such as A1, with canonical names and recommendation ranks withheld. Inspired by candidate-based checklist generation for response evaluation [33], the Auditor uses retained API functions to guide decomposition and clarify external-service boundaries. Each requirement must be justified by , whether or not the retained APIs support it. Coverage judgments rely on API descriptions; roles provide context but cannot establish undocumented capabilities. The checklist covers external-service capabilities explicitly requested or unambiguously required by the stated workflow, excluding optional enrichments and generic infrastructure unless requested. Each record contains a distinct, atomic, verb-led requirement and a verbatim span from stating the need or the workflow step that requires it.
A requirement is marked COVERED only if at least one retained API description directly supports its entire atomic capability in the mashup context. The Auditor records all such APIs, each with a short verbatim evidence span from its description. Otherwise, the requirement is marked MISSING and receives one query $q_i^{(j)} reformulating it for catalog resolution. Requirements and queries are provider-neutral and contain 5–30 words; queries are not used as final functional roles. An API may support multiple requirements.
Let index the MISSING records in . In checklist order, the corresponding missing-item sequence is
The Auditor returns COMPLETE when ; otherwise, it returns RESOLVE_BATCH with all items in , without a fixed cardinality cap. Before resolution, a deterministic validator checks the output schema, sequential identifiers, evidence-substring constraints, query lengths, and consistency between requirement statuses and the overall decision. The full checklist and its supporting evidence are recorded in , while is passed to the Resolver.
Catalog-Constrained Resolution
To constrain API proposals to the catalog while allowing abstention, the Resolver represents each documented API by a dedicated action token , such as <API_google_maps>. For each missing requirement, it selects an eligible API token or <NO_ADD> in the same action space. It is trained to abstain when a separate API is unnecessary or no eligible catalog API supports the queried capability in the current recommendation state.
Let denote the API action tokens, and let contain the tokens of all initial candidates, retained or deleted. For each , the Resolver input is
The input encodes and as the initial API tokens, each labeled KEEP or DELETE according to the Role Writer’s decision. All tokens in are excluded from the active action space
At the first decoding step, let denote the next-token logit assigned to action by the Resolver. We mask all tokens outside and select the highest-scoring eligible action:
All missing queries are resolved in one batch against the same fixed and . We discard abstentions and deduplicate API identities to form the candidate addition set
Each candidate retains its originating requirement–query associations for Commit Verification.
Resolver supervision. Training examples combine reviewed requirement–API pairs from training mashups with short functional anchors for documented catalog APIs; inputs follow Eq. (6). When multiple APIs independently satisfy the same atomic requirement, their action tokens are valid positives. Requirements needing complementary contributions from multiple APIs are split into atomic requirements. For an example , let be the active action space and its set of valid positive actions. We minimize the multi-positive loss
The loss maximizes total probability mass on valid actions and reduces to cross-entropy over for a single positive.
Commit Verification
The Resolver ensures catalog eligibility but neither reads API descriptions at decision time nor generates functional roles. A separate Commit Verifier therefore checks each proposal against its catalog description before addition and writes roles for accepted APIs. For each distinct , it receives , , and all originating queries in one LLM call, with API names withheld. Queries specify the capabilities to check; only serves as evidence of support.
The Verifier assesses each query separately, assigning SUPPORTED if it judges that directly supports the entire atomic capability in the context of , and UNSUPPORTED otherwise. Supported judgments include evidence from ; unsupported ones include a brief rationale based on . We assign ACCEPT to an API if at least one originating query is SUPPORTED, and REJECT otherwise. Acceptance does not imply support for every originating query. The accepted addition set is
For each accepted API, the Verifier writes using and to describe one atomic contribution from a supported query–API match. When multiple queries are supported, one representative contribution is selected for the role, while all query-level judgments are retained in . The final recommendation is , with exactly one functional role for each API.
Deterministic controller. A deterministic controller applies the masking, deduplication, and acceptance rules above and commits each accepted API once. It assembles from the structured outputs of all stages, including filtering decisions and deletion reasons, requirement checklist and coverage evidence, the resolution decisions and verification outcomes. The controller has no learned parameters and makes no semantic relevance judgments.
Experiments
We evaluate WAR-CRAFT through five research questions:
RQ1 Does WAR-CRAFT improve recommendation quality across heterogeneous upstream recommenders, and can changes in set size alone explain the gains? (§ 5.2)
RQ2 How does WAR-CRAFT compare with independent refinement methods and controlled variants? (§ 5.3)
RQ3 How do set-aware filtering, coverage-driven addition, and commit verification affect recommendation quality? (§ 5.4)
RQ4 Are the generated functional roles aligned with the request, grounded in API documentation, and clearly expressed? (§ 5.5)
RQ5 How does initial set size affect refinement? (§ 5.6)
Experimental Setup
Dataset and split. We use the ProgrammableWeb corpus, the standard benchmark for Web API recommendation [36, 37, 38], with 8,217 mashups and 1,647 APIs. A mashup uses 2.09 recorded APIs on average, and mashup and API descriptions average 17.0 and 44.3 words. We randomly split the mashups into 4,930 training, 1,644 validation, and 1,643 test instances (3:1:1), using the same test set for all comparisons.
Upstream recommenders. We refine the frozen outputs of seven upstream recommenders. The ranking sources are the hierarchical BERT-based recommender WARBERT [38], cross-view hypergraph recommender CVH-REC [10], multi-agent recommender MARS [18], and Tool-Embed-4B, a semantic retriever fine-tuned from Tool-Embed-4B [19] on our training split. The generative sources are LLMAR [25], WAR-R1 [39], and zero-shot GPT-5.6 Sol. For ranking sources, the initial set contains the top-2 APIs, close to the corpus average of 2.09; § 5.6 varies this cutoff. For generative sources, we retain the generated sets. At inference, WAR-CRAFT uses only the predicted APIs and their order, mashup requests, and catalog descriptions.
Refinement baselines. All baselines use the same frozen upstream predictions (Appendix § 9.§ 2). Independent refinement uses none of our components. Tuned- keeps the first upstream APIs. MMR-Cut [2] and DPP-Cut [3] select subsets for relevance and diversity from top-10 ranked lists or generated sets, with tuned size caps or gain thresholds. LLM-Select uses GPT-5.6 Sol to select from ranking sources’ top-10 APIs using the request and API descriptions, following LLM-based reranking [8, 32]. Retrieve-and-Select (R&S) applies the same selector to the initial set plus ten new catalog APIs retrieved for the request. PTR-Refine adapts PTR [7] to the initial set, omitting bundle acquisition. It retains APIs matched to requirements and selects additions for unmatched requirements through multi-view retrieval voting. Controlled variants. Direct-Edit replaces the Role Writer and Auditor with one editing call; Self-Check adds a call to review its draft. Separated extracts requirements from the request alone and assesses coverage in a separate call. Ret-Abstain replaces only the Resolver with a Tool-Embed-4B retriever trained on the same supervision, scoring APIs alongside a learned no-add candidate analogous to Toolken+’s reject option [40]. Other retrieval and embeddings use Tool-Embed-4B. Independent baselines are tuned per source on validation SetF1 where applicable; WAR-CRAFT and its variants use fixed settings across sources.
Resolver supervision. We construct supervision from recommendations on the original mashup training split using CVH-REC and zero-shot GPT-5.6 Sol. To broaden supervision beyond the evaluated upstreams, we also include an auxiliary Qwen3-4B-Instruct-2507 recommender trained following LLMAR [25]. Only the first two also serve as evaluated upstreams. CVH-REC and the Qwen recommender use three-fold out-of-fold predictions: each mashup is predicted by a model trained on the other two folds. The Role Writer and Joint Requirement Auditor process these recommendations. GPT-5.6 Sol reviews the generated queries in their recommendation contexts, followed by author verification. After review and deduplication, the retained supervision comprises 3,434 API-action states (310 with multiple valid targets) and 3,946 <NO_ADD> states. We add 436 catalog anchors, each pairing a standalone capability query rewritten from an API description with its API action token, yielding 7,816 states. All anchors remain in training; 10% of the reviewed recommendation states are held out for Resolver validation, separate from the mashup validation split (Appendix § 9.§ 1).
Implementation details. The Role Writer, Joint Requirement Auditor, and Commit Verifier use GPT-5.6 Sol with reasoning effort set to none, temperature 0. Their structured outputs are checked against predefined schemas. We average recommendation metrics over three runs of each GPT-based refinement method on the same initial sets. The Resolver is fine-tuned from Llama-3.2-3B-Instruct with LoRA; API action-token embeddings are initialized from API names and descriptions. We select the checkpoint with the highest state-level Hit@1 on held-out Resolver states and use it for all seven upstreams. A hit matches any valid target action.
Metrics. Our primary metrics measure agreement between and the recorded API set : SetPrecision (SetP), SetRecall (SetR), and SetF1, computed per mashup and macro-averaged; we also report mean output size. NDCG and MAP assess recovery and ranking over the complete ordered output without a fixed cutoff. NDCG uses the ideal DCG of the full recorded set, and average precision is normalized by . Retained APIs keep their upstream order, followed by accepted additions in audited requirement order.
To assess functional alignment beyond recorded API membership, we use DOEM’s 53-label taxonomy [28], allowing functional matches through APIs outside . Coverage measures the fraction of required labels provided, Efficiency the fraction of provided labels required, and Redundancy the share of repeated label occurrences. CovF1 is the harmonic mean of Coverage and Efficiency. These metrics are computed per mashup and macro-averaged.
Functional roles are scored by a Qwen3-235B-A22B-Instruct-2507 judge, from a different model family than GPT-5.6, to reduce self-preference bias. The judge sees only the mashup description, the API description, and the role, without the system identity, and rates mashup alignment, API grounding, and role clarity [39] from 0 to 2; the Functional Role Score (FRS) is their normalized sum. Definitions and protocols are in Appendices A.3 and C.1.
Overall Effectiveness (RQ1)
Table 1 evaluates the same WAR-CRAFT configuration on seven frozen upstream recommenders and 1,643 test mashups.
| Base | System | SetP | SetR | SetF1 | NDCG | MAP | Cov. | Eff. | CovF1 | Red. ↓ | Avg. | S | |
| WAR-R1 | Initial | 0.7282 | 0.6771 | 0.6800 | 0.7032 | 0.6643 | 0.4414 | 0.5874 | 0.4574 | 0.0657 | 1.70 | ||
| + WAR-CRAFT | 0.7739 | 0.6937 | 0.7038 | 0.7145 | 0.6737 | 0.4439 | 0.6138 | 0.4718 | 0.0444 | 1.50 | |||
| LLMAR | Initial | 0.7215 | 0.6734 | 0.6689 | 0.6909 | 0.6514 | 0.4363 | 0.5542 | 0.4370 | 0.0700 | 1.95 | ||
| + WAR-CRAFT | 0.7476 | 0.6927 | 0.6911 | 0.7076 | 0.6661 | 0.4509 | 0.5830 | 0.4617 | 0.0690 | 1.75 | |||
| GPT | Initial | 0.6544 | 0.5804 | 0.5912 | 0.5987 | 0.5641 | 0.4085 | 0.5532 | 0.4309 | 0.0417 | 1.35 | ||
| + WAR-CRAFT | 0.6727 | 0.6095 | 0.6139 | 0.6254 | 0.5892 | 0.4320 | 0.5708 | 0.4497 | 0.0483 | 1.45 | |||
| CVH-REC | Initial | 0.4537 | 0.5766 | 0.4698 | 0.5939 | 0.5479 | 0.4731 | 0.4994 | 0.4564 | 0.1231 | 2.00 | ||
| + WAR-CRAFT | 0.6863 | 0.6430 | 0.6271 | 0.6553 | 0.6082 | 0.4577 | 0.5985 | 0.4788 | 0.0644 | 1.60 | |||
| MARS | Initial | 0.5313 | 0.6724 | 0.5506 | 0.6970 | 0.6558 | 0.4991 | 0.5490 | 0.4812 | 0.1912 | 2.00 | ||
| + WAR-CRAFT | 0.6423 | 0.6790 | 0.6194 | 0.6977 | 0.6559 | 0.4992 | 0.5917 | 0.4971 | 0.1150 | 1.81 | |||
| WARBERT | Initial | 0.4169 | 0.5211 | 0.4274 | 0.5388 | 0.4965 | 0.4385 | 0.4754 | 0.4271 | 0.1492 | 2.00 | ||
| + WAR-CRAFT | 0.6621 | 0.6285 | 0.6096 | 0.6350 | 0.5888 | 0.4529 | 0.5930 | 0.4715 | 0.0704 | 1.61 | |||
| Tool-Embed-4B^FT | Initial | 0.4860 | 0.6518 | 0.5184 | 0.6804 | 0.6318 | 0.4481 | 0.5087 | 0.4452 | 0.2446 | 2.00 | ||
| + WAR-CRAFT | 0.7275 | 0.6309 | 0.6462 | 0.6557 | 0.6145 | 0.4323 | 0.6247 | 0.4700 | 0.0466 | 1.40 | |||
| Mean | Initial | 0.5703 | 0.6218 | 0.5580 | 0.6433 | 0.6017 | 0.4493 | 0.5325 | 0.4479 | 0.1265 | 1.86 | ||
| + WAR-CRAFT | 0.7018 | 0.6539 | 0.6444 | 0.6702 | 0.6281 | 0.4527 | 0.5965 | 0.4715 | 0.0654 | 1.59 |
Table 1. Results on 1,643 test mashups. + WAR-CRAFT denotes the final outputs. Metrics are macro-averaged over mashups; Mean reports the average across seven sources. Cov. and Eff. denote functional Coverage and Efficiency, CovF1 their per-mashup harmonic mean, and Red. functional Redundancy (↓). Bold marks the better quality score within each pair.
Recommendation accuracy. WAR-CRAFT improves SetF1 for all seven sources, raising the mean from 0.5580 to 0.6444 (+8.64 percentage points; 95% paired-bootstrap CI: [8.01, 9.28]). All source-specific intervals exclude zero (Appendix B). Gains are larger for the four ranking sources with top-2 improvements (+6.88–18.22 points) than for the three generative sources (+2.22–2.38 points).
Functional quality. Efficiency and CovF1 improve for all seven sources, with mean CovF1 rising from 0.4479 to 0.4715. Mean Redundancy falls from 0.1265 to 0.0654 and mean output size from 1.86 to 1.59 APIs, although GPT’s Redundancy increases. For GPT, adding APIs with shared functional labels can produce large ratio increases in small sets (Appendix B). Mean Coverage changes little (+0.34 percentage points), with decreases for CVH-REC and Tool-Embed-4B^FT despite their CovF1 gains.
Set size and membership. Since 53.8% of test mashups have one recorded API, we test whether prefix-based size adjustment explains the gains (Table 2). Controls truncate upstream top-10 lists for ranking sources and original generated lists otherwise to or (oracle size); shorter lists remain intact. Within each source, exact matching to uses only instances with at least that many candidates. On these instances (93.7% on average across sources), WAR-CRAFT exceeds equal-length prefixes by a mean 4.14 SetF1 percentage points (95% paired-bootstrap CI: [3.61, 4.67]). For multi-API mashups (), the corresponding eligible fraction and mean gain are 92.2% and 4.22 points (95% CI: [3.54, 4.89]). Across all multi-API mashups, WAR-CRAFT also improves mean SetF1 and Coverage over Initial while reducing mean output size from 2.09 to 1.93 APIs. The gains therefore extend to multi-API mashups and persist under exact size matching on eligible instances. WAR-CRAFT also exceeds truncation to the recorded set size in SetF1 (0.6444 versus 0.6240), although that oracle uses more APIs on multi-API mashups (2.71 versus 1.93).
| Method | All | Cov. | CovF1 | |||
| Initial | 0.5580 | 0.5960 | 0.5137 | 0.4699 | 0.4574 | 2.09 |
| Trunc. to | 0.6009 | 0.6767 | 0.5126 | 0.4423 | 0.4473 | 1.87 |
| WAR-CRAFT | 0.6444 | 0.7206 | 0.5557 | 0.4835 | 0.4790 | 1.93 |
| Trunc. to (oracle) | 0.6240 | 0.7105 | 0.5233 | 0.5031 | 0.4559 | 2.71 |
Table 2. Set size versus membership, averaged over seven sources. Results include instances without enough candidates for exact size matching. Cov., CovF1, and are on . The oracle uses recorded sizes and is excluded from bolding.
Refinement Baseline Comparison (RQ2)
Table 3 compares WAR-CRAFT with five independent baselines and four controlled variants across seven sources; per-source results are in Appendix B. WAR-CRAFT leads in mean SetF1 and CovF1, exceeding DPP-Cut, the best independent baseline by SetF1, by 1.92 percentage points (95% CI: [1.26, 2.60]).
| Method | SetF1 | CovF1 | Red. | Avg. | Calls |
| Initial | 0.5580 | 0.4479 | 0.1265 | 1.86 | 0 |
| Independent refinement | |||||
| Tuned- | 0.5982 | 0.4228 | 0.0227 | 1.23 | 0 |
| MMR-Cut | 0.6243 | 0.4349 | 0.0254 | 1.27 | 0 |
| DPP-Cut | 0.6252 | 0.4365 | 0.0238 | 1.26 | 0 |
| R&S | 0.5334 | 0.4631 | 0.0924 | 1.63 | 1.00 |
| PTR-Refine | 0.5625 | 0.4568 | 0.1118 | 1.76 | 2.71 |
| Controlled variants | |||||
| Direct-Edit | 0.6092 | 0.4677 | 0.1115 | 1.72 | 1.22 |
| + Self-Check | 0.6054 | 0.4621 | 0.1074 | 1.67 | 2.22 |
| Separated | 0.6250 | 0.4704 | 0.0874 | 1.66 | 3.20 |
| Ret-Abstain | 0.6174 | 0.4582 | 0.0583 | 1.48 | 2.15 |
| WAR-CRAFT | 0.6444 | 0.4715 | 0.0654 | 1.59 | 2.23 |
Table 3. Refinement results averaged over seven upstream sources. Calls reports mean GPT-5.6 Sol calls per mashup during refinement. Bold marks the best quality scores.
Independent refinement. On average, Tuned-, MMR-Cut, and DPP-Cut reduce set size and Redundancy but lower CovF1 relative to Initial. DPP-Cut outperforms WAR-CRAFT on single-API mashups (SetF1: 0.7355 versus 0.7206), but underperforms Initial on multi-API mashups (0.4967 versus 0.5137). LLM-Select also trails WAR-CRAFT on the four ranking sources (SetF1: 0.5759 versus 0.6256). R&S and PTR-Refine improve CovF1 over Initial, but R&S lowers SetF1 and PTR-Refine gains 0.45 points. PTR-Refine traces show that separate requirement matches can retain overlapping APIs, while approximate matches can suppress retrieval for unmet requirements. Multi-view voting without abstention or a separate documentation check can also admit unsupported additions.
Controlled variants. Self-Check does not improve Direct-Edit’s mean SetF1. Separated uses the most GPT calls (3.20 versus 2.23 per mashup for WAR-CRAFT). Direct-Edit, Self-Check, and Separated all have lower mean SetF1 and CovF1 and higher mean Redundancy than WAR-CRAFT. The traces illustrate differences in set auditing. On globebopor, Direct-Edit retains both Google Maps and Google Street View Image for street-level views, whereas the Role Writer keeps only Google Maps. Direct-Edit also requests fewer missing capabilities than the joint audit on six of seven sources, with traces showing omitted functions or composite queries. Separated can mistake application-side ranking for an unmet external-service requirement, whereas the joint audit identifies the missing external data. With the same inputs, supervision, and action mask, Ret-Abstain trails WAR-CRAFT on all seven sources, with a mean SetF1 gap of 2.70 points (95% CI: [2.26, 3.16]). The Verifier rejects 24.0% of its proposals, versus 4.6% for the Resolver. The Resolver’s accuracy advantage persists against this retriever with learned abstention.
Component Contributions (RQ3)
Table 4 reports cumulative results averaged over seven sources.
| Stage | SetF1 ↑ | CovF1 ↑ | Red. ↓ | Avg. |
| Initial | 0.5580 | 0.4479 | 0.1265 | 1.86 |
| + Role Writer | 0.5911 | 0.4396 | 0.0485 | 1.35 |
| + Audit | 0.6443 | 0.4715 | 0.0658 | 1.60 |
| + Verifier | 0.6444 | 0.4715 | 0.0654 | 1.59 |
Table 4. Cumulative refinement results averaged over seven upstream sources. Bold marks the best quality scores.
Set-aware filtering. The Role Writer raises mean SetF1 from 0.5580 to 0.5911 and reduces Redundancy from 0.1265 to 0.0485 (0.0520 excluding empty outputs), but lowers Coverage from 0.4493 to 0.4036 and CovF1 from 0.4479 to 0.4396. SetF1 gains are concentrated in the four ranking sources (+4.96 to +6.96 percentage points); for the generative sources, SetF1 changes by at most 0.26 points.
Coverage-driven addition. The Joint Requirement Auditor and Resolver then improve SetF1 for all seven sources, raising the mean to 0.6443. Mean Coverage returns to approximately its initial level (0.4532 versus 0.4493), while CovF1 rises to 0.4715. Redundancy increases to 0.0658 but remains below its initial level. Averaged over three runs, the Resolver returns <NO_ADD> for 69.7% of the queries it receives.
Commit verification. Averaged over three runs, the Commit Verifier rejects 4.6% of the 2,889 deduplicated proposals per run. A blinded human review of 100 rejected and 100 accepted proposals from one run upholds 85 rejections and 92 acceptances (Appendix C.2). Mean SetF1 changes only from 0.6443 to 0.6444, with CovF1 unchanged at the reported precision (0.4715). The Verifier primarily provides documentation-based screening and functional roles for accepted additions, rather than a substantial SetF1 gain.
Human audit of requirements and API support. To assess requirement level support directly, two authors annotated 140 random and 70 challenge source–mashup traces. Requirements were first drafted from requests alone, followed by API support scoring blind to upstream sources, system checklists and decisions, and recorded-set membership (Appendix C.2). On 139 assessable random traces, the share of reference requirements fully supported by at least one API rises from 55.2% in the initial sets to 62.5% in the final sets (+7.28 percentage points; 95% CI [4.24, 10.64]), while the share of unsupported API instances falls from 26.8% to 7.8%. Manual comparison with the system checklists identifies full matches for 82.8% of these 261 reference requirements, partial matches for 10.3%, and omissions for 6.9%.
Functional Role Quality (RQ4)
Table 5 reports final role quality under the source-blind judge (Appendix C.1). FRS ranges from 0.9296 to 0.9589 across sources. Grounding scores lowest for all seven sources, while clarity scores highest for six. Assigning zero scores to empty outputs lowers mean FRS from 0.9423 to 0.9115. We evaluate WAR-R1’s rationales [39] with the same judge and scoring rubric, replacing only “functional role” with “rationale” in the prompt. Its FRS is 0.9125 on recommended API instances with rationales, compared with 0.9521 for WAR-CRAFT’s refined outputs from the same source. However, 8.56% of WAR-R1’s recommended API instances lack rationales, while 7.53% of API instances with rationales are not recommended. Assigning zero scores to these unmatched instances following [39] lowers its FRS to 0.8471. By construction, every API in WAR-CRAFT’s output carries exactly one functional role.
| Source | Empty | Alignment | Grounding | Clarity | FRS |
| WAR-R1 | 83 | 0.9543 | 0.9349 | 0.9672 | 0.9521 |
| LLMAR | 39 | 0.9468 | 0.9210 | 0.9608 | 0.9429 |
| GPT | 115 | 0.9516 | 0.9223 | 0.9590 | 0.9443 |
| CVH-REC | 31 | 0.9320 | 0.9185 | 0.9539 | 0.9348 |
| MARS | 25 | 0.9334 | 0.9123 | 0.9549 | 0.9336 |
| WARBERT | 47 | 0.9279 | 0.9111 | 0.9498 | 0.9296 |
| Tool-Embed-4B^FT | 35 | 0.9696 | 0.9419 | 0.9650 | 0.9589 |
| Mean | – | 0.9451 | 0.9232 | 0.9587 | 0.9423 |
| Mean, zero-inclusive | – | 0.9142 | 0.8930 | 0.9274 | 0.9115 |
Table 5. Functional role quality after refinement. Scores are normalized to [0, 1] and averaged over APIs within each nonempty mashup, then over mashups. Mean rows average the seven sources. Empty counts mashups with no output APIs; the zero-inclusive row assigns them zero scores.
Human validation. Two authors independently rated 350 API–role instances (50 per source), blinded to the source, writing component, and judge scores, and resolved disagreements by discussion (Appendix § 11.§ 1). Across the three dimensions, the judge agrees exactly with the consensus on 89.71–91.14% of ratings, with mean absolute errors of 0.089–0.109 on the 0–2 scale. Quadratic-weighted Cohen’s ranges from 0.408 to 0.644, and Gwet’s AC2 from 0.963 to 0.974. Consensus ratings are concentrated at the top of the scale: 89.90% of dimension ratings receive the maximum score.
Sensitivity to Initial Set Size (RQ5)
We vary the initial cutoff for the four ranking sources while keeping the refinement configuration fixed. Figure 3 reports the final results, averaged over three runs. SetF1 exceeds the corresponding Initial in all 16 source–cutoff settings, by 1.25–30.56 percentage points. Although these gains increase with , final SetF1 peaks at or 2 and declines by 2.28–9.57 points from to . CovF1 exceeds its value at every larger cutoff, gaining 0.76–2.92 points at , while Redundancy and final set size also increase. Stage analysis shows that more initial APIs remain after filtering at larger , while accepted additions decrease by 46.2–75.2% from to across sources (Appendix § 10). The larger final sets therefore reflect more retained initial APIs despite fewer additions. These results reveal a trade-off between historical-set agreement, functional alignment, and output compactness.
Figure 3. Sensitivity to the initial cutoff under a fixed refinement configuration. (a) SetF1, (b) CovF1, (c) functional Redundancy, and (d) mean final set size.
Discussion and Limitations
WAR-CRAFT performs one refinement pass without re-auditing the final set. Composite queries may remain unresolved if the Auditor fails to decompose them, as the Resolver selects at most one API per query. The semantic stages can be adapted to open-weight LLMs for local deployment. Our experiments use the proprietary GPT-5.6 Sol to prioritize the quality of semantic judgments and functional roles. Historical API sets and coarse functional labels cannot fully assess valid alternatives, motivating broader expert annotation of acceptable APIs for each requirement. The LLM stages use a proprietary model that may have seen ProgrammableWeb during pretraining. In an exploratory check, it could not recover the functions of long-tail APIs from their names alone, but neither this check nor withholding API names rules out memorization.
Conclusion
We presented WAR-CRAFT for refining Web API recommendations without modifying upstream models. It filters unsupported or redundant APIs and jointly audits external-service requirements and their coverage. For each identified gap, the Resolver selects a catalog API or abstains; proposed additions are checked against API documentation. Each output API carries a request-specific functional role, with refinement decisions recorded in an audit trace. Experiments across seven heterogeneous recommenders show higher SetF1 and CovF1 and lower average functional redundancy. LLM-based evaluation and human validation support the mashup alignment, API grounding, and clarity of the generated roles.
References
Experimental Details
Models and Training
Upstream recommenders. Trained recommenders, including WAR-R1, use each method’s released code and default settings, with training on our training split and model selection on our validation split; MARS keeps its default Qwen2.5-14B-Instruct agents, and Tool-Embed-4BFT is Tool-Embed-4B further fine-tuned with LoRA (rank 32) for five epochs. Zero-shot GPT-5.6 Sol returns API names only. Names are normalized and matched to the catalog; for an unmatched name, GPT-5.6 Sol states the intended function, which is mapped to a catalog API or dropped if no API provides it. All upstream outputs are generated once and frozen. Empty initial sets (40 for LLMAR, 5 for WAR-R1, and 169 for GPT) skip the Role Writer.
Supervision construction. For the training states in § 5.1, GPT-5.6 Sol reviews each Auditor query against the mashup, current and recorded API sets, and catalog descriptions. Valid targets are recorded APIs outside the initial set that independently satisfy the query; multiple targets are allowed. A query receives <NO_ADD> if its capability is already covered, requires no separate API, or lacks full documented support from any eligible catalog API. Queries that only non-recorded APIs satisfy therefore receive neither label and are excluded (36 states), as are other unresolved ambiguous examples. Composite requirements are split when independently resolvable, and exact duplicate inputs are merged. The 436 catalog anchors (§ 5.1) cover APIs absent from the reviewed targets, including 143 appearing in recorded test sets. All anchors remain in training; the 10% validation subset contains only reviewed recommendation states.
Training details. We fine-tune the Resolver from Llama-3.2-3B-Instruct using LoRA with rank 32, , dropout 0.05, and a learning rate of . New API action-token embeddings are jointly updated. Training runs for four epochs on two RTX 4090 GPUs with seed 42, taking approximately 21 minutes. Token initialization and checkpoint selection follow § 5.1.
Computational cost. The Resolver averages 0.04 s of inference time per mashup. Mean input/output token counts per GPT call are 1,215/67 for the Role Writer, 1,059/210 for the Joint Requirement Auditor, and 924/76 for the Commit Verifier.
Refinement Baseline Details
LLM-based baselines use GPT-5.6 Sol with the decoding settings of § 5.1 and are run three times; retrieval and embeddings use Tool-Embed-4BFT. Hyperparameters are selected per source by validation macro SetF1; Table 6 lists the selected values.
| Source | Tuned- | MMR-Cut | DPP-Cut |
| LLMAR | 4 | (1, 0.75, 0.225) | (1, 0.90, 0.400) |
| CVH-REC | 1 | (1, 0.50, −0.075) | (1, 0.90, 0.475) |
| GPT | 4 | (1, 1, 0.550) | (0, 0.10, =10) |
| MARS | 1 | (0, 0.25, −0.425) | (0, 0.50, −0.125) |
| WARBERT | 1 | (1, 0.50, −0.075) | (1, 0.90, 0.475) |
| Tool-Embed-4BFT | 1 | (0, 0.50, −0.125) | (0, 0.50, −0.350) |
| WAR-R1 | 3 | (1, 0.75, 0.250) | (1, 0.90, 0.400) |
Table 6. Validation selected settings. Entries are , with for MMR and for DPP; denotes rank/semantic relevance. GPT’s DPP uses a cap of 10 instead of a threshold.
1) Tuned-k, MMR-Cut, and DPP-Cut. Tuned- retains up to APIs in upstream output order, . MMR-Cut and DPP-Cut use the upstream top-10 ranked list or original generated set, without additional retrieval. For each source and algorithm, validation selects either semantic relevance or discounted rank . API similarity is a shifted cosine Gram matrix regularized toward the identity with weight . MMR greedily trades relevance, weighted by , against maximum similarity to selected APIs; DPP greedily maximizes the log-determinant of a kernel combining relevance, weighted by , with API similarity. Both stop at a size cap in , bounded by the available candidates, or when marginal gain falls below (step 0.025).
2) LLM-Select. For each ranking recommender, the selector receives the request and the upstream’s top-10 APIs with their catalog descriptions in random order, and returns a supported, complementary subset, possibly empty, in the order it chooses.
3) Retrieve-and-Select. Retrieve-and-Select applies the LLM-Select procedure to , with distinct new APIs and initial-set membership hidden. Validation Recall@10 is 0.3713 for BM25, 0.5810 for released Tool-Embed-4B, and 0.7423 for Tool-Embed-4BFT, which is therefore used for all retrieval and embedding baselines.
4) PTR-Refine. PTR-Refine adapts PTR [7] to the same initial sets, omitting historical-bundle acquisition. One prompt extracts requirements from the request, and one per requirement identifies initial APIs that roughly fulfill it. Only APIs matched to at least one requirement are retained. Each unmatched requirement drives three catalog retrieval views: requirement-to-API similarity, APIs of similar training mashups, and API-to-API similarity from the top direct hit. The API receiving the most votes is added unless already present, with ties broken by direct similarity. Validation selects retrieval depth from .
5) Direct-Edit and Self-Check. Direct-Edit receives the request and initial APIs with their descriptions, without access to the rest of the catalog. One prompt jointly produces KEEP/DELETE decisions, roles for retained APIs, and queries for missing capabilities. These queries pass through the same Resolver and Commit Verifier as in WAR-CRAFT. Self-Check adds one prompt that reviews the input and Direct-Edit draft and returns a corrected draft before resolution.
6) Separated. Separated replaces the Joint Requirement Auditor with two prompts: requirement extraction from the request alone, followed by coverage assessment against the APIs retained by the Role Writer.
7) Ret-Abstain. Ret-Abstain replaces only the Resolver. A shared encoder maps the input in Eq. (6) and each API profile to normalized vectors and , respectively. With a fixed temperature , each available API scores , and a learned no-add candidate scores . It is trained for five epochs with cross-entropy on the Resolver’s supervision states, updating LoRA adapters on Tool-Embed-4B (rank 16, , learning rate ) and , (learning rate ); API embeddings are
refreshed every epoch, and the checkpoint with the best validation Hit@1 is kept. Ret-Abstain outputs <NO_ADD> when and otherwise proposes the top API, which is merged and verified as in WAR-CRAFT.
Metrics and Functional Annotations
Historical-set agreement. For mashup , let be the recorded API set and the predicted set. SetPrecision and SetRecall normalize by and , respectively, and SetF1 is their harmonic mean. All scores, including SetF1, are computed per mashup and macro-averaged. Empty predictions receive zero for the set and ranking metrics.
Ranking metrics. For the ordered output with , retained APIs keep their upstream order and accepted additions follow in audited requirement order. With , we compute NDCG, where and , and AP, where is the precision of the first outputs. Both use the complete output without a fixed cutoff and normalize by all recorded APIs. NDCG and MAP macro-average NDCG and AP across mashups.
Functional annotation. We use the fixed 53-label taxonomy from DOEM [28]. Of the 1,647 documented APIs, 1,151 inherit labels through exact normalized name matching, and 496 are reviewed from their catalog descriptions. A mashup inherits labels when its normalized name exactly matches a single DOEM mashup; unmatched mashups are annotated from their descriptions without access to historical API sets.
Functional metrics. Let be the mashup’s required labels and the labels of API , with and . Coverage is , Efficiency is , and CovF1 is their harmonic mean, set to 0 when both are 0. Redundancy is , the share of repeated label occurrences. We set for an empty provided-label set and for an empty recommendation set, and compute all four metrics per mashup before macro-averaging.
Additional Results
Overall effectiveness. Table 7 reports gains over Initial. For the primary comparisons, we first average per-mashup metric values across three runs for each GPT-based refinement method. We compute 95% paired bootstrap confidence intervals using 20,000 mashup-level resamples (seed 2026). Each resample uses the same mashup IDs across methods and upstream sources. For the Mean row, we average the seven source-specific gains within each resample. Mean Coverage shows no significant change. Coverage decreases for CVH-REC and Tool-Embed-4B, while Redundancy increases only for GPT. A paired analysis of GPT outputs finds increases in 55 of 1,643 mashups and decreases in 25. The increasing cases receive one (43 cases) or two (12 cases) additional APIs. Their mean Redundancy increase is 0.2502 (median 0.2500; interquartile range 0.1667–0.3333). Among these cases, 28 initially contain a single API and account for 57.2% of the summed positive changes, illustrating how a few additions sharing functional labels can sharply raise the ratio in GPT’s small initial sets.
| Source | SetF1 | CovF1 | Cov. | Red. |
| WAR-R1 | +2.38 [1.52, 3.24] | +1.45 [0.84, 2.06] | +0.25 [−0.50, 1.00] | −2.13 [−2.77, −1.51] |
| LLMAR | +2.22 [1.43, 3.02] | +2.47 [1.91, 3.03] | +1.46 [0.81, 2.12] | −0.10 [−0.54, 0.35] |
| GPT | +2.27 [1.58, 3.01] | +1.87 [1.40, 2.38] | +2.35 [1.83, 2.88] | +0.66 [0.40, 0.93] |
| CVH-REC | +15.74 [14.43, 17.01] | +2.24 [1.36, 3.13] | −1.54 [−2.54, −0.55] | −5.87 [−6.72, −5.01] |
| MARS | +6.88 [5.89, 7.84] | +1.59 [0.97, 2.20] | +0.01 [−0.70, 0.73] | −7.62 [−8.51, −6.76] |
| WARBERT | +18.22 [16.80, 19.65] | +4.44 [3.47, 5.44] | +1.44 [0.31, 2.60] | −7.88 [−8.80, −6.98] |
| Tool-Embed-4B{sup}FT` | +12.79 [11.49, 14.11] | +2.48 [1.60, 3.37] | −1.59 [−2.60, −0.59] | −19.80 [−20.72, −18.87] |
| Mean | +8.64 [8.01, 9.28] | +2.36 [1.97, 2.75] | +0.34 [−0.12, 0.80] | −6.11 [−6.49, −5.71] |
Table 7. Per-source gains of WAR-CRAFT over Initial in percentage points, with 95% paired-bootstrap confidence intervals. Negative Red. indicates reduced functional redundancy.
Refinement baselines. Table 8 reports SetF1 and CovF1 by upstream source. WAR-CRAFT achieves the highest SetF1 on all four ranking sources and LLMAR, while Separated leads on WAR-R1 and GPT. For CovF1, WAR-CRAFT leads on five sources; Retrieve-and-Select and Separated lead on GPT and CVH-REC, respectively.
| Method | SetF1 WAR-R1 | SetF1 LLMAR | SetF1 GPT | SetF1 CVH-REC | SetF1 MARS | SetF1 WARBERT | SetF1 Tool | CovF1 WAR-R1 | CovF1 LLMAR | CovF1 GPT | CovF1 CVH-REC | CovF1 MARS | CovF1 WARBERT | CovF1 Tool |
| Initial | 0.6800 | 0.6689 | 0.5912 | 0.4698 | 0.5506 | 0.4274 | 0.5184 | 0.4574 | 0.4370 | 0.4309 | 0.4564 | 0.4812 | 0.4271 | 0.4452 |
| Independent refinement | ||||||||||||||
| Tuned- | 0.6802 | 0.6689 | 0.5907 | 0.5225 | 0.6189 | 0.4737 | 0.6330 | 0.4570 | 0.4388 | 0.4314 | 0.4055 | 0.4277 | 0.3780 | 0.4214 |
| MMR-Cut | 0.6809 | 0.6702 | 0.5912 | 0.6077 | 0.6182 | 0.5618 | 0.6404 | 0.4507 | 0.4400 | 0.4302 | 0.4196 | 0.4492 | 0.4053 | 0.4493 |
| DPP-Cut | 0.6810 | 0.6659 | 0.5912 | 0.6109 | 0.6192 | 0.5646 | 0.6437 | 0.4580 | 0.4366 | 0.4309 | 0.4301 | 0.4438 | 0.4134 | 0.4429 |
| LLM-Select | – | – | – | 0.5875 | 0.5847 | 0.5565 | 0.5750 | – | – | – | 0.4400 | 0.4762 | 0.3987 | 0.4547 |
| R&S | 0.5647 | 0.5397 | 0.5242 | 0.5300 | 0.5279 | 0.5292 | 0.5183 | 0.4692 | 0.4616 | 0.4665 | 0.4615 | 0.4680 | 0.4609 | 0.4537 |
| PTR-Refine | 0.6502 | 0.6409 | 0.6058 | 0.5060 | 0.5441 | 0.4823 | 0.5084 | 0.4617 | 0.4481 | 0.4550 | 0.4678 | 0.4748 | 0.4442 | 0.4463 |
| Controlled variants | ||||||||||||||
| Direct-Edit | 0.6901 | 0.6794 | 0.6129 | 0.5905 | 0.5763 | 0.5644 | 0.5509 | 0.4716 | 0.4559 | 0.4531 | 0.4809 | 0.4913 | 0.4683 | 0.4529 |
| + Self-Check | 0.6906 | 0.6753 | 0.6004 | 0.5926 | 0.5714 | 0.5550 | 0.5525 | 0.4705 | 0.4559 | 0.4462 | 0.4721 | 0.4820 | 0.4542 | 0.4536 |
| Separated | 0.7065 | 0.6879 | 0.6242 | 0.5935 | 0.5880 | 0.5734 | 0.6017 | 0.4712 | 0.4613 | 0.4600 | 0.4814 | 0.4946 | 0.4626 | 0.4618 |
| Ret-Abstain | 0.6953 | 0.6801 | 0.5969 | 0.5775 | 0.6127 | 0.5715 | 0.5879 | 0.4675 | 0.4557 | 0.4431 | 0.4605 | 0.4916 | 0.4493 | 0.4397 |
| WAR-CRAFT | 0.7038 | 0.6911 | 0.6139 | 0.6271 | 0.6194 | 0.6096 | 0.6462 | 0.4718 | 0.4617 | 0.4497 | 0.4788 | 0.4971 | 0.4715 | 0.4700 |
Table 8. Per-source SetF1 and CovF1 of refinement methods. Tool denotes Tool-Embed-4B{sup}FT`, and R&S denotes Retrieve-and-Select. Bold marks the highest value in each column; – indicates not applicable.
Initial set size sensitivity. For , SetF1 gains over the corresponding initial sets, in percentage points, are (9.98, 15.74, 21.41, 26.90) for CVH-REC, (4.81, 6.88, 13.39, 20.99) for MARS, (14.80, 18.22, 22.42, 25.68) for WARBERT, and (1.25, 12.79, 21.08, 30.56) for Tool-Embed-4B. Across sources, mean retained size increases from 0.775–0.940 APIs at to 1.532–2.271 at . Over the same range, missing-query totals decrease by 18.6–47.5% and accepted API additions by 46.2–75.2%.
Additional controls. We tested two common retrieval replacements that take Tool-Embed-4B’s top API for each missing query: always adding it lowers SetF1 to 0.5807, below filtering alone (0.5911), and adding it only when its cosine similarity reaches a per-source validation threshold (0.35 for every source) yields 0.5937. Ret-Abstain (0.6174) is the strongest retrieval-based replacement we tested. We also tested reranking final sets containing at least two APIs with GPT-5.6 Sol. Mean NDCG and MAP changed only by -0.5 and -0.6 points, respectively. One possible explanation is that filtering removes poorly matched candidates while preserving useful upstream ordering, leaving limited benefit from further reranking.
Human Audit and Role Evaluation
Functional Role Evaluation
Scoring protocol. The source-blind Qwen3-235B-A22B-Instruct-2507 judge receives only the mashup description, API description, and role. Using the two descriptions as evidence, it evaluates the role without rewriting and independently scores mashup alignment, API grounding, and clarity in , adapting WAR-R1’s criteria [39]. Dimension scores are divided by two; FRS is the sum of the three raw scores divided by six. Scores are averaged over APIs within each mashup, then over mashups. A fixed prompt and temperature 0 are used across systems.
Human validation of the role judge. From one scored run’s final outputs, we randomly sample 50 API–role instances per source: 350 from 316 distinct mashups. Two authors independently score them using the same inputs and rubric, blind to the source, role-writing component, and judge scores. Pre-discussion exact agreement on alignment, grounding, and clarity is 91.43%, 92.00%, and 92.86%, respectively (quadratic-weighted 0.490, 0.733, and 0.579; AC2 0.972, 0.974, and 0.979). The 63 disagreements among 1,050 ratings, spanning 64/350 instances, are resolved by discussion; § 5.5 compares the consensus with the judge.
Negative control. For each of the 350 human-validated instances , we keep and uniformly sample a replacement role (seed 42) from the same source’s final outputs on other mashups. Donor API must satisfy , and must differ from ; every instance has an eligible donor. We reuse original scores and evaluate replacements with the same prompt and judge settings. Mean normalized FRS falls from 0.9486 to 0.2395, a drop of 0.7090 (95% CI [0.6890, 0.7281]; 10,000 paired bootstrap resamples, seed 42); FRS decreases for 343 of the 350 instances (98.0%) and is unchanged for the other seven. Alignment and grounding fall to 0.0271 and 0.0100, whereas clarity falls only to 0.6814, consistent with fluent but mismatched roles.
Requirements and Refinement Decisions
We sampled 20 random and 10 challenge traces per source (seed 2026), yielding 140 random and 70 challenge pairs covering 202 distinct mashups. Challenge selection used redundancy, filtering, and unmet-requirement signals, without recorded-set membership or performance scores. Two authors each annotated 130 traces, with 50 independently double-annotated. Requirements were drafted from requests alone before API scoring; later additions or scope revisions were logged for two random and three challenge entries. Shuffled, anonymized API descriptions concealed sources, ranks, system checklists and decisions, and recorded API sets.
API–requirement support was scored as absent (0), partial (1), or full (2). Full coverage requires one API scoring 2 for the entire reference entry, including compound entries; an unsupported API scores 0 against every entry. Excluding one unconfirmed empty reference leaves 139/70 assessable traces with 261/160 requirements (random/challenge). Final statistics include all stored APIs. An author assessed checklist alignment: full matches capture all actions, objects, and key constraints of a reference requirement; partial matches capture only part of this scope; omissions have no corresponding checklist content. Pre-discussion agreement on 383 comparable support scores was 353/383 (92.2%; quadratic-weighted ); disagreements were resolved through discussion. Proportions pool counts within each stratum. The coverage-gain CI uses 10,000 paired percentile-bootstrap resamples clustered by mashup (seed 42).
Among scope-comparable decisions, false COVERED judgments occurred in 8/90 random and 8/54 challenge cases; false MISSING judgments occurred in 13/74 and 7/48, respectively. Filtering lost full coverage of 3/144 random requirements (all recovered by later additions) and 1/84 challenge requirements (not recovered). Eligible alternatives were confirmed for 10/37 random and 12/30 challenge genuine gap abstentions. The remaining genuine-gap abstentions remain unclassified: review covered trajectory APIs and generally the retrieved top three, expanded to ten for 28 queries selected before scoring.
Review of verification decisions. We randomly sample and shuffle 100 rejected and 100 accepted proposals from one run. Two annotators independently apply the Verifier’s query-level criterion using only the request, API description, and originating queries, blind to API names, Verifier outputs, sources, and recorded-set membership. A proposal is supported if at least one query is supported. Pre-discussion agreement is 93.5% (Cohen’s ); the 13 disagreements are resolved by discussion, and reported counts use consensus labels. Of 33 sampled rejections of recorded APIs, 24 are upheld.
References
- [1]Hadeel Alhosaini, Sultan Alharbi, Xianzhi Wang, and Guangdong Xu. 2024. API recommendation for mashup creation: A comprehensive survey. Comput. J. 67, 5 (2024), 1920–1940.DOI
- [2]Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia) (SIGIR ’98). Association for Computing Machinery, New York, NY, USA, 335–336. doi:10.1145/290941.291025DOI
- [3]Laming Chen, Guoxin Zhang, and Eric Zhou. 2018. Fast greedy map inference for determinal point process to improve recommendation diversity. Advances in neural information processing systems 31 (2018).arxiv.org/abs/1709.05135
- [4]Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, MohammadHossein Bateni, Chen-Yu Lee, and Tomas Pfister. 2024. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2024. 4705–4726.arxiv.org/abs/2408.01875
- [5]Yu Du, Fangyun Wei, and Hongyang Zhang. 2024. Analytical Self-Reflective, Hierarchical Agents for Large-Scale API Calls. In Proceedings of the 41st International Conference on Machine Learning. 11812–11829.
- [6]Xiang Fei, Xiawu Zheng, and Hao Feng. 2025. Mcp-zero: Active tool discovery for autonomous llm agents. arXiv preprint arXiv:2506.01056 (2025).arxiv.org/abs/2506.01056
- [7]Hang Gao and Yongfeng Zhang. 2025. Task-Aligned Tool Recommendation for Large Language Models. In Proceedings of IJCNLP-AACL 2023–2045.
- [8]Jingtong Gao, Bo Chen, Xiangyu Zhao, Weiwen Liu, Xiangyang Li, Yichao Wang, Wanyu Wang, Huifeng Guo, and Ruiming Tang. 2025. Lmrank: Llm-based auto-reranking framework for recommendations. In Proceedings of the ACM on Web Conference 2025. 228–239.
- [9]Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16477–16508.arxiv.org/abs/2210.08726
- [10]Shanquan Gao, Yihui Wang, and Zhenwei Qu. 2026. CVH-REC: A novel method for web API recommendation based on cross-view HGNNs. IEEE Transactions on Software Engineering (2026).DOI
- [11]Wenwen Gong, Xuyuan Zhang, Yifei Chen, Qiang He, Amin Beheshti, Xiaolong Xu, Chao Yan, and Lianyong Qi. 2022. DAWAR: Diversity-aware Web APIs Recommendation for Mashup Creation based on Correlation Graph. In Proceedings of SIGIR. 395–404.
- [12]Qi Gu, Jian Cao, and Yancen Liu. 2021. CSBR: A compositional semantics-based service bundle recommendation approach for mashup development. IEEE Transactions on Services Computing 15, 6 (2021), 3170–3183.
- [13]Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems 36 (2023), 45870–45894.arxiv.org/abs/2305.11554
- [14]Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. In European conference on information retrieval. Springer, 364–381.
- [15]Xiangjun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. Knowledge-centric hallucination detection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 6953–6975.
- [16]Guosheng Kang, Jianjun Liu, Buqing Cao, and Manliang Cao. 2020. Nafm: neural and attentional factorization machine for web api recommendation. In 2020 IEEE international conference on web services (ICWS). IEEE, 330–337.DOI
- [17]Jiayu Li, Xinci Qiu, Guo-Heng Kang, Yan Li, and Jianxun Liu. 2025. TSSGCF: Textual similarity-supervised graph collaborative filtering for web api recommendation. In International Conference on Service-Oriented Computing. Springer, 102–117.DOI
- [18]Mingyi Liu, Zijie Yin, Congcong Tian, Shuang Yu, Taotao Cai, Zhihui Xu, and Zhongjie Wang. 2026. Mars: A multi-agent collaborative reasoning framework for service recommendation. IEEE Transactions on Services Computing (2026).
- [19]Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. 2026. Tools are under-documented: Simple document expansion boosts tool retrieval. In International Conference on Learning Representations, Vol. 2026. 141631–141653.arxiv.org/abs/2510.22670
- [20]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems 36 (2023), 46534–46594.
- [21]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FactScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of EMNLP. 12076–12100.
- [22]Shengye Pang, Yihang Zhang, Song Yang, Yixin Chen, Yanglan Gan, Shuigang Deng, and Guobing Zou. 2026. Leveraging Co-occurrence Bias in Web API Recommendation via Causality-Inspired Context-Adjusted Graph Learning. IEEE Transactions on Services Computing (2026).DOI
- [23]Lianyong Qi, Qiang He, Feifei Chen, Xuyun Zhang, Wanchun Dou, and Qiang Ni. 2020. Data-driven web APIs recommendation for building web applications. IEEE transactions on big data 8, 3 (2020), 685–698.DOI
- [24]Lianyong Qi, Houing Song, Xuyun Zhang, Gautam Srivastava, Xiaolong Xu, and Shui Yu. 2021. Compatibility-aware web API recommendation for mashup creation via textual description mining. ACM Transactions on Multimedia Computing Communications and Applications 17, 1s (2021), 1–19.
- [25]Shaowei Qin, Yiyi Zhao, Hao Wu, Lei Zhang, and Qiang He. 2025. Harnessing the power of large language model for effective web API recommendation. IEEE Transactions on Industrial Informatics 21, 7 (2025), 5360–5370.DOI
- [26]Yujia Qin, Shihiao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangyu Tang, Bill Qian, et al. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations. Vol. 2024. 9695–9717.arxiv.org/abs/2307.16789
- [27]Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaigiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. Towards completeness-oriented tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 1930–1940.DOI
- [28]Dunlei Rong, Mingyi Liu, Shuang Yu, Quan Z Sheng, Zhongjie Wang, and Xiaofei Xu. 2026. DOEM: A Novel Development-Oriented Evaluation Metric Framework for Service Recommendation. ACM Transactions on the Web 20, 4 (2026), 1–33.DOI
- [29]Dunlei Rong, Mingyi Liu, Yang Zhang, Quan Z Sheng, Lina Yao, Zhongjie Wang, and Xiaofei Xu. 2026. STAR: A Sequential Transformer-based Adaptive Reinforcement Learning Framework for Service Recommendation. IEEE Transactions on Services Computing (2026).DOI
- [30]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems 36 (2023), 8634–8652.arxiv.org/abs/2303.11366
- [31]Kaipu Sun, Wen Tang, Yecheng Jin, Meng Xi, Jiacheng Pan, Ying Li, and Jianwei Yin. 2026. LACL: Overcoming Semantic Sparsity in Mashup Development via LLM-Enhanced Service Bundle Recommendation. IEEE Transactions on Services Computing (2026).DOI
- [32]Weiwei Sun, Lingyang Yan, Xinyu Ma, Shuqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 conference on empirical methods in natural language processing. 14918–14937.
- [33]Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Sherry Wu. 2025. Checklists are better than reward models for aligning language models. Advances in Neural Information Processing Systems 38 (2025), 114728–114754.
- [34]Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. 2025. Toolgen: Unified tool retrieval and calling via generation. In International Conference on Learning Representations, Vol. 2025. 73473–73498.arxiv.org/abs/2410.03439
- [35]Shuyao Wang, Zhi Zheng, Guoxu Sui, and Hui Xiong. 2025. Unleashing the power of large language model for denoising recommendation. In Proceedings of the ACM on Web Conference 2025. 252–263.DOI
- [36]Hao Wu, Yunhao Duan, Kun Yue, and Lei Zhang. 2021. Mashup-oriented web API recommendation via multi-model fusion and multi-task learning. IEEE Transactions on Services Computing 15, 6 (2021), 3330–3343.DOI
- [37]Hui Wu and Lei Wang. 2025. SEFR: A Mashup Recommendation Approach for Crossover Service Convergence. IEEE Internet of Things Journal (2025).DOI
- [38]Zishuo Xu, Youhong Gu, and Dezhong Yao. 2026. WARBERT: A Hierarchical BERT-Based Model for Web API Recommendation. IEEE Transactions on Services Computing 19, 3 (2026), 2591–2604.arxiv.org/abs/2509.23175
- [39]Zishuo Xu, Dezhong Yao, and Yao Wan. 2026. From Ranking to Reasoning: Explainable Web API Recommendation via Semantic Reasoning. In Proc. of the IEEE 46th International Conference on Distributed Computing Systems (ICDCS). 1410–1420.arxiv.org/abs/2511.05820
- [40]Konstantin Yakovlev, Sergey Nikolenko, and Andrey Bout. 2024. Toolken+: Improving llm tool usage with reranking and a reject option. In Findings of the Association for Computational Linguistics: EMNLP 2024. 5967–5974.arxiv.org/abs/2410.12004
- [41]Chao Yan, Qinghe Yan, Jiahui Dong, Boyuan Yan, Lianyong Qi, and Weiyi Zhong. 2025. Optimal Subset Oracle-Based Web API Composition Recommendation for Mashup Creation. In International Conference on Service-Oriented Computing. Springer, 87–101.DOI
- [42]Mingdai Yang, Nurendra Choudhary, Jiangshu Du, Edward W Huang, Philip Yu, Karthik Subbian, and Danai Koutza. 2026. AgentDR: Dynamic Recommendation with Implicit Item-Item Relations via LLM-based Agents. In Proceedings of the ACM Web Conference 2026. 6230–6241.DOI
- [43]Song Yang, Guobing Zou, Shengxiang Hu, Shengye Pang, Yanglan Gan, Bofeng Zhang, and Yixin Chen. 2026. LMSR: LLM-enhanced Multi-perspective Service Feature Learning for Web API Recommendation. IEEE Transactions on Services Computing (2026).DOI
- [44]Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang. 2024. Toolrerank: Adaptive and hierarchy-aware reranking for tool retrieval. In Proceedings of the 2024 joint international conference on computational linguistics,arxiv.org/abs/2403.06551


