betaFeedback

WorksComputer Science

What Should an LLM Know? Knowledge-Guided Warm Starts for Software Optimization

Abstract

Large language models (LLMs) process gigabytes in minutes; human experts work far more slowly. Prior studies that explored combinations of human/AI optimizers ignore this gap. Those studies asked systems or humans to label too many examples (e.g., SynthCore needs over four dozen per trial). Here, we test human and AI knowledge under a budget where evaluation is practical (a few dozen labels or fewer). We compare five baselines (random search, RRP, EZR, few-shot prompting, and a budget-matched SynthCore variant) against three ways to give an LLM domain knowledge before it proposes its first configurations: (1) AMP lists the domain's rules, then checks each candidate against them; (2) HKMA adds scouted data or documentation to the prompt; and (3) H-DKP has human experts correct the LLM's rules. We find that domain knowledge improves the "start-up" phase of LLM optimization, but only when that knowledge is checked, by the LLM itself or by a human expert. Across 49 datasets, unchecked knowledge did no better than few-shot prompting or EZR. On the other hand, AMP's self-checking variant (AMP-4) reached the best statistical rank in 44 of 49 datasets (budget-matched SynthCore: 23 of 49), and H-DKP did so in 7 of 8 expert studies. These checks cost little: half a cent of LLM calls per trial, or 6 to 10 expert contacts. Hence we say LLMs may not need more data, but rather their knowledge needs checking (and checking is cheap).

Introduction

In this paper we assess the value of teaming up humans with LLMs to configure software. Modern software systems expose numerous tunable parameters that govern competing objectives: a database configuration trades query latency against memory footprint, while a machine learning pipeline balances predictive accuracy against training time [13, 50, 11]. Configuration is a complex task: configuration errors are widespread, and parameter spaces continue to grow in complexity [53, 52]. Accordingly we ask:

How to better control and configure complex software?\textit{How to better control and configure complex software?}

One approach is to configure systems using the knowledge contained inside large language models (LLMs). For example, in the “warm start” approach explored by Senthilkumar and Menzies [42], some oracle1oracle_1 (human or AI) passes a few labeled examples to a second oracle2oracle_2. In this paper, a label is the measured objective scores of one configuration (e.g., the runtime and memory use of one database setting); we also call this an evaluation. Labels are costly since each one needs the system to be run, or an expert to be asked.

Repeated studies show that if oracle1oracle_1 can generate good labels, oracle2oracle_2’s final performance can be dramatically improved; e.g., see the SynthCore study and prior work [42, 38].

This paper addresses two major drawbacks with the SynthCore study:

  • That prior analysis was fully automatic. Here, we conduct studies that combine AI and human decision making [45] (Ramírez et al. 2019).1

  • That prior analysis made unreasonable demands on the amount of available labeled data. SynthCore reduced conclusion variance by ensembling 15 prompting sessions, but it needs dozens of distinct labels per trial (on average). Here, we make conclusions using just 24 labels.

There are many reasons why it is prudent to study configuration using very limited budgets:

  • Experts are slow at labeling [28, 10, 19, 24, 48]. Knowledge engineers in the 1980s expected only 5–10 rules per expert per day [28]. And if rushed, the labels offered by experts become inaccurate [10]. Interactive optimizers face the same limit: users tire after only a few judgments [45] (see also § 2.5).

  • Automatic labeling methods are heuristic and error prone. Regular-expression heuristics are crude [16]; quality labeling tools like SZZ have a long list of known problems [14], and LLMs can assist but not replace human labelers [3].

  • Execution-based oracles can be slow: some labels need a compile and a benchmark run of slow and complex software, such as a full test suite [31, 6].

  • Evaluation is costly: exploring the 11 parameters of (e.g.) the x264 video encoder takes 1,536 hours [49], so budgets are often limited to dozens of evaluations [32, 7].

  • Configuration spaces are vast: MySQL’s 460 binary options define 24602^{460} configurations [52]. Even far larger budgets sample only a tiny part of such spaces.

  • Cheap labels are often wrong: historical logs can offer many misleading labels. For example, 90% of technical-debt “false positives” can be incorrect [54], with similar issues in security [51], static analysis [17], and defect data [43].

For these reasons, this paper focuses on very small budgets: 24 labels. At this scale, the human effort is realistic: it can fit within a standard work week without disrupting it2. This focus sets our work apart from much prior configuration and hyperparameter optimization research, which assumes (a) no human experimentation and (b) hundreds to thousands of evaluations [5, 4, 50, 33].

Within this budget, we compare three methods for giving an LLM domain knowledge before it proposes its first configurations (Figure 1). AMP has the LLM list the domain’s rules, then check each candidate against them [18, 25]. HKMA adds scouted data or documentation to the prompt [5, 20]. H-DKP has human experts correct the LLM’s rules [39]. Together, these methods draw on four sources of domain knowledge: the LLM itself, scouted data, external documents, and human experts.

Workflow showing inputs, domain knowledge, LLM warm start, sequential optimizer, and evaluation

Figure 1. Our trails starts with 4 labeled examples. Domain knowledge may be added before an LLM proposes 4 initial configurations. A sequential optimizer then selects the rest, for 24 labels in total.

We test these methods on 49 SE multi-objective optimization problems [29]. We ask:

RQ1: With only 24 labels, can domain knowledge help an LLM propose better initial configurations?

Only if that knowledge is checked. Unchecked knowledge (LLM-discovered rules, scouting, documents) did no better than direct few-shot prompting or the EZR active learner. But AMP-4, which checks each candidate against the rules, reached the best statistical rank on 44 of 49 datasets.

RQ2: Does increasing the number of configuration parameters change which kinds of knowledge help?

Yes. Checking matters most in high dimensions, where AMP-4 ranked best on 16 of 17 tasks, while documentation-only prompting fell from 12 of 15 (low) to 9 of 17 (high).

RQ3: Does human review of the LLM’s domain rules help?

Yes. H-DKP ranked best on 7 of 8 expert tasks, and beat direct prompting on 7 of 8. However, EZR also ranked best on 7 of 8, at zero cost.

RQ4: What does checked knowledge cost?

Very little. AMP-4 costs half a cent of LLM calls per trial (≤24\le24 labels) and H-DKP needs only 6 to 10 short expert emails. By contrast, restricting multi-session SynthCore to a comparable budget (SynthCore-28, 26.31 labels) drops its win rate to 23 of 49 tasks, trailing both AMP-4 (39 of 49) and zero-cost EZR (32 of 49).

In summary, this paper makes five contributions:

  • Evidence that domain knowledge improves LLM warm starts only when it is checked (§ 5.1 and § 5.2).

  • A human-in-the-loop study on eight tasks across nine expert tracks (§ 4.2 and § 5.3).

  • A cost analysis across configuration labels, LLM calls, and expert effort (§ 5.4).

  • A methodological lesson: LLM methods must be compared against simple baselines, since the model-free EZR active learner often does as well (§ 5.3 and § 7).

  • A replication package containing all our code, data, and prompts at https://github.com/srina1h/What-Should-an-LLM-Know.

Background and related work

Configuration search with few observations

Software configuration strongly affects performance across database management, software analytics, and compiler tuning. Van Aken et al. [50] use runtime measurements to recommend database settings, while studies in software analytics show that learner hyperparameters substantially alter prediction quality [11, 46]. Although simple heuristics sometimes suffice (Agrawal et al. 2019) and coarse performance models can still rank configurations effectively [31], performance models often transfer poorly across hardware platforms and workloads [49]. Consequently, optimizers must evaluate candidate configurations directly on the target system.

Because physical evaluations are costly, sequential optimizers select new configurations iteratively from a small set of prior observations. FLASH applies sequential model-based optimization to software configuration [32], and Bayesian optimization is widely used for compiler autotuning [7]. Tree-structured Parzen Estimation (TPE) finds promising and unpromising regions separately to guide candidate selection [5]. Other works study sampling-based baselines [6], landscape analysis from minimal data [24], and lightweight active learning via the EZR toolkit [27]. These studies define the data-light software engineering challenge [12]: find near-optimal configurations within about two dozen evaluations.

LLMs for optimization

Recent work explores using LLMs to guide black-box optimization. Several frameworks invoke the language model repeatedly throughout the search loop: LLAMBO uses LLMs for warm starts, surrogate modeling, and candidate sampling in Bayesian optimization [23], LMEA employs an LLM as a crossover and mutation operator in evolutionary search [22], and Huang et al. [15] evaluate iterative LLM optimization across continuous and discrete benchmarks. While iterative LLM-in-the-loop search can be effective, invoking a large model at every optimization step increases inference latency and token costs.

An alternative is to invoke the LLM only at initialization to warm-start a fast sequential optimizer. Senthilkumar and Menzies [41] show that prompting an LLM with task metadata and a few labeled examples can produce useful initial configurations; their direct few-shot approach serves as our BS-LLM baseline. However, single-pass few-shot generation is often unreliable when parameter interactions are complex. To improve robustness, SynthCore ensembles candidates over 15 independent few-shot generation sessions [42]. Although ensembling reduces variance, it needs more than twice our 24-label budget. Despite growing interest in LLM-assisted optimization [22, 15], it remains unclear which knowledge sources improve optimization quality when labels are scarce, whether passive context injection is sufficient or active constraint verification is required, and when the additional inference tokens or expert review effort are justified. over lightweight model-free optimizers. Rather than scaling up the labeling budget through multi-session ensembling, we investigate whether structuring and verifying domain knowledge allows an LLM to generate stronger warm starts within a strict 24-label budget.

Sources of domain knowledge

Knowledge engineering examines how domain information is elicited, structured, and verified for computational use [39]. In software configuration, relevant domain knowledge encompasses feature importance, feasible parameter bounds, and trade-offs among competing objectives. We study three mechanisms for supplying this knowledge to an LLM warm-start generator:

  1. Staged model reasoning and self-verification: Prompt decomposition breaks complex reasoning into sequential subtasks [18], while Self-Refine uses model-generated feedback to revise initial outputs [25]. AMP applies both ideas to warm starts (§ 3.2).

  1. External documentation and statistical scouting: Retrieval-augmented generation (RAG) grounds model outputs in external text corpora [20]. HKMA adds retrieved documentation, or a summary of TPE scouting, to the prompt (§ 3.3).

  1. Human expert review: Although LLMs can match human agreement on certain software annotation tasks [3], unverified model beliefs can still encode flawed domain assumptions. H-DKP asks human experts to check the LLM’s domain rules before optimization starts (§ 3.4).

Tabular benchmarks

We evaluate all methods using the MOOT repository [29], which curates tabular multi-objective optimization datasets from empirical software engineering studies, including measured system configurations such as those analyzed in PromiseTune [8] and software process simulations [35, 26]. Tabular benchmarks map pre-recorded configurations to measured objective values, enabling reproducible comparisons across many trials without re-executing costly system workloads. Because an LLM can propose arbitrary continuous or categorical feature combinations, each generated candidate is mapped to the nearest observed row in the benchmark table. Studies on surrogate and tabular benchmarks, such as YAHPO Gym [33] and NAS benchmarks [55], note that discrete table resolution can influence evaluation when proposals fall far from recorded rows. We account for this representation in our threats to validity and inspect proposal-to-row distances when constructing follow-up queries in our human expert study.

Humans in the optimization loop

Interactive search-based software engineering asks humans to guide or judge the search (Ramírez et al. 2019). A long-standing problem with this approach is user fatigue: people tire quickly, so an interactive optimizer can ask them for only a small number of judgments [45]. Knowledge acquisition has the same problem. Experts are slow to label examples [48, 24], and in the 1980s, knowledge engineers expected only 5–10 rules per expert per day [28]. Hence, human effort must be spent where it matters most. In H-DKP, experts do not label configurations; instead, they review a short list of rules proposed by the LLM across a few brief email exchanges (§ 4.2 and § 5.3).

Methods

Different optimization scenarios provide access to different sources of information. Hence our three knowledge-guided approaches elicit and enforce this knowledge through differing mechanisms: AMP relies on staged model reasoning and self-verification; HKMA relies on preliminary statistical scouting and external documentation; and H-DKP incorporates asynchronous human expert review.

Common warm-start generation and sequential search

We say that D={(xi,yi)}i=1N\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N} denotes a benchmark dataset where each configuration xix_i comprises pp input features and each outcome yiy_i comprises mm objective values with defined optimization directions (minimization or maximization). All single-session LLM warm-start methods (BS-LLM, AMP, HKMA, and H-DKP) share a common initialization and downstream optimization interface. Each trial begins by sampling Binitial=4B_{\mathrm{initial}} = 4 observed configurations from the benchmark dataset D\mathcal{D} and partitioning them by multi-objective quality into the best-performing observations (bestbest) and the remaining observations (restrest). Supplying both subsets gives the language model and the downstream sequential optimizer explicit examples of promising and unpromising parameter regions.

Given the task metadata MM (feature names, types, value ranges, and objective directions), the labeled best/rest examples SS, and any method-specific domain knowledge, the LLM is prompted to generate Bgenerated=4B_{\mathrm{generated}} = 4 feature-only candidate configurations: two expected to perform well and two expected to perform poorly. Requesting contrastive proposals encourages the model to span both high-performing and failure regions so the sequential surrogate can establish an informative decision boundary early in the search. Generated candidates are validated against the dataset schema for complete fields and valid data types, mapped to their nearest observed rows in D\mathcal{D} in normalized feature space, and evaluated by retrieving their objective values.

Following warm-start evaluation, sequential model-based optimization (SMO) iteratively selects and labels the remaining configurations up to the total budget Btotal=24B_{\mathrm{total}} = 24. At the start of SMO and after each subsequent evaluation, all observed configurations (the initial seeds plus the evaluated warm-start rows) are re-partitioned into bestbest and restrest subsets using their actual objective values rather than the LLM’s intended better/poorer polarity. Using the same lightweight active-learning core as EZR [27], SMO fits a two-class Naive Bayes surrogate with Gaussian likelihoods over the observed bestbest and restrest partitions and selects the next configuration via its exploit acquisition policy. For BS-LLM, AMP, HKMA-rag, and H-DKP,

SMO executes Bsearch=16B_{\mathrm{search}} = 16 sequential steps (4+4+16=244 + 4 + 16 = 24). For HKMA-scout and HKMA-combined, which allocate Bscout=10B_{\mathrm{scout}} = 10 evaluations to preliminary statistical scouting prior to LLM generation, SMO executes Bsearch=6B_{\mathrm{search}} = 6 sequential steps (4+10+4+6=244 + 10 + 4 + 6 = 24).

Adaptive Multi-Stage Prompting (AMP)

Direct few-shot prompting asks the LLM to infer parameter interactions and output numerical configurations in a single generation pass. In complex spaces, models frequently hallucinate parameter settings that violate basic domain constraints. Adaptive Multi-Stage Prompting (AMP) decomposes warm-start generation into sequential reasoning stages [18, 25] to test whether explicit constraint proposals and per-candidate verification improve configuration quality without requiring external data sources.

Table 1 and Algorithm 1 define the three stage ablations [44]. In the first stage, the model analyzes the task metadata MM and initial examples SS to summarize likely feature-objective trade-offs. AMP-2 passes this analysis directly to the candidate generation prompt (2 nominal LLM calls). AMP-3 introduces an intermediate stage that prompts the model to formulate explicit feasibility constraints and parameter rules CC from its analysis before generating the four candidates (3 nominal LLM calls). However, an LLM may articulate valid constraints in natural language yet still violate them when generating multi-dimensional vectors.

ConditionAnalysisConstraintsValidationNominal calls
AMP-2YesNoNo2
AMP-3YesYesNo3
AMP-4YesYesYes7

Table 1. AMP ablation conditions and nominal LLM calls required for a four-candidate warm start. Validation in AMP-4 is executed once per generated candidate.

AMP-4 addresses this gap by adding a dedicated validation stage: each of the four generated candidates is checked individually against the discovered constraints CC, and any candidate flagged as violating a constraint is revised before schema verification (7 nominal calls). All three variants operate within the same 24-label evaluation budget.

Algorithm 1 AMP generation and stage ablations

Require: Metadata MM, initial best/rest examples SS, mode m∈{2,3,4}m \in\{2,3,4\} Ensure: Four feature-only warm-start candidates EE

  1. A←LLM⁡(M,S,analyze features and trade-offs)A \leftarrow\operatorname{LLM}(M,S,\text{analyze features and trade-offs})

  2. C←∅C \leftarrow\varnothing

  3. if m≥3m \ge3 then

  4.     C←LLM⁡(A,discover constraints)C \leftarrow\operatorname{LLM}(A,\text{discover constraints})

  5. end if

  6. E←LLM⁡(M,S,A,C,generate two better and two poorer candidates)E \leftarrow\operatorname{LLM}(M,S,A,C,\text{generate two better and two poorer candidates})

  7. if m=4m=4 then

  8.     for each candidate e∈Ee \in E do

  9.         v←LLM⁡(C,e,validate and propose a revision if needed)v \leftarrow\operatorname{LLM}(C,e,\text{validate and propose a revision if needed})

  10.         if vv marks ee invalid and supplies revised features then

  11.             Replace ee with the proposed revision

  12.         end if

  13.     end for

  14. end if

  15. Apply common schema checks; reject malformed final sets

  16. return EE

Hybrid Knowledge-Model Approach (HKMA)

Rather than relying solely on internal model reasoning, the Hybrid Knowledge-Model Approach (HKMA) supplies external context derived from preliminary statistical scouting, technical documentation, or their combination (Algorithm 2) [44].

We evaluate three variants under the fixed 24-label budget:

  • HKMA-scout: A Tree-structured Parzen Estimator (TPE) [5] selects Bscout=10B_{\mathrm{scout}}=10 distinct observed configurations from D\mathcal{D}. The LLM is prompted to summarize empirical parameter patterns PP that distinguish the higher-performing and lower-performing scouted rows, and this summary is injected into the warm-start generation prompt alongside updated best/rest examples. Because the 10 scouting evaluations are part of the 24-label budget, HKMA-scout trades fewer downstream SMO iterations (Bsearch=6B_{\mathrm{search}}=6) for a richer empirical summary prior to generation.

  • HKMA-rag: Using task and feature descriptions as a query qq, a retrieval-augmented generation pipeline [20] extracts relevant passages RR from a curated index II of domain documentation and system manuals. The retrieved passages are supplied directly to the warm-start prompt without consuming any scouting labels, preserving Bsearch=16B_{\mathrm{search}}=16 SMO steps.

  • HKMA-combined: TPE scouting first acquires Bscout=10B_{\mathrm{scout}}=10 observations and generates the empirical summary PP. That summary is combined with the task metadata to form an informed retrieval query qq, and both PP and the retrieved passages RR are provided to the generation prompt, followed by Bsearch=6B_{\mathrm{search}}=6 SMO steps. Comparing HKMA-combined against HKMA-scout and HKMA-rag tests whether combining empirical scouting with documentation yields additive gains.

Algorithm 2 HKMA generation under alternative knowledge conditions

Require: Metadata MM, initial examples SS, document index II, condition mm Ensure: Four feature-only candidates EE and recorded scouting evaluations

1: P←∅P \leftarrow\varnothing; R←∅R \leftarrow\varnothing 2: if m∈{scout,combined}m \in\{\mathrm{scout}, \mathrm{combined}\} then 3: Sscout←\mathit{S}_{\mathrm{scout}} \leftarrow ten distinct observed rows selected by TPE scouting 4: Evaluate these rows and charge ten evaluations against the total labeling budget 5: P←LLM(Sscout,summarize observed best/rest differences)P \leftarrow\mathrm{LLM}(\mathit{S}_{\mathrm{scout}}, \text{summarize observed best/rest differences}) 6: S←S \leftarrow selected best/rest examples from S∪SscoutS \cup\mathit{S}_{\mathrm{scout}} 7: end if 8: if m∈{rag,combined}m \in\{\mathrm{rag}, \mathrm{combined}\} then 9: q←q \leftarrow feature/objective metadata, plus PP when available 10: R←Retrieve(I,q)R \leftarrow\mathrm{Retrieve}(I,q) 11: end if 12: E←LLM(M,S,P,R,generate two better and two poorer candidates)E \leftarrow\mathrm{LLM}(M,S,P,R,\text{generate two better and two poorer candidates}) 13: Apply common final-set schema checks 14: return EE and recorded scouting evaluations

Human-in-the-Loop Domain Knowledge Prompting (H-DKP)

Automated prompting and document retrieval can still introduce spurious correlations or misinterpret domain-specific parameters. Human-in-the-Loop Domain Knowledge Prompting (H-DKP) incorporates asynchronous expert review to filter, qualify, and refine LLM-generated domain claims before freezing the resulting knowledge for optimization (Algorithm 3) [44]. Participant demographics, ethics approval, and measured effort across the nine completed expert tracks are detailed in § 4.2.

As shown in Algorithm 3, H-DKP separates offline knowledge acquisition from online optimization. First, using only task metadata MM and source documentation CC, the LLM is prompted at temperature 0 to propose at most ten domain claims K0K_0, each stating a hypothesized feature-objective relationship, rationale, and confidence score. After automated screening to verify that all referenced feature names exist in MM, a human specialist classifies each claim as VALID, MODIFY, INVALID, or UNSURE; only claims marked VALID or supplied with explicit replacement conditions (MODIFY) enter the active rule set KK. Across subsequent asynchronous contacts (T≤10T \le10), the LLM generates candidate configurations from KK and presents one diagnostic question per exchange (either a failure-diagnosis query when an expected-better candidate underperforms the initial seeds, or a profile-plausibility query when the nearest benchmark row is too distant), updating KK only with explicit expert rules. Once frozen, KK conditions 20 independent warm-start trials under the standard 24-label budget.

Algorithm 3 H-DKP with asynchronous expert review

Require: Source context CC, metadata MM, expert HH, stopping rule T≤10T \le10

Ensure: Frozen knowledge KK and 20 independent four-candidate sets

  1. Propose initial claims K0←LLM⁡(C,M)K_0 \leftarrow\operatorname{LLM}(C, M) without observed rows

  2. Screen claim schema and send eligible claims to HH

  3. K←K \leftarrow explicitly accepted or revised claims from the reply

  4. for each subsequent expert exchange until the stopping rule is met do

  5. E←LLM⁡(M,K)E \leftarrow\operatorname{LLM}(M, K)

  6. Validate proposals and evaluate mapped observed rows

  7. if an interpretable failure candidate is available then

  8. Prepare a failure-diagnosis question using current knowledge KK

  9. else

  10. Prepare a profile-plausibility question for an unqueried candidate

  11. end if

  12. Send question to HH and await reply

  13. Update KK only with explicit accepted wording and conditions

  14. end for

  15. Freeze KK separately for this expert track

  16. for r=1r = 1 to 20 do

  17. Er←LLM⁡(M,K,trial examplesr)E_r \leftarrow\operatorname{LLM}(M, K, \text{trial examples}_r)

  18. Validate and score the four proposals, then run Bsearch=16B_{\mathrm{search}} = 16 SMO steps

  19. end for

  20. return KK, {Er}r=120\{E_r\}_{r=1}^{20}

Comparison baselines

We compare the knowledge-guided methods against three model-free baselines and two unguided LLM baselines:

  • Random search: Uniformly samples Bsearch=24B_{\mathrm{search}} = 24 distinct observed configurations from DD without replacement and returns the best configuration observed [4].

  • EZR: A lightweight active-learning optimizer [27] that initializes with Binitial=4B_{\mathrm{initial}} = 4 randomly selected configurations and iteratively selects Bsearch=20B_{\mathrm{search}} = 20 additional configurations using its exploit policy, yielding 24 distinct configuration labels per trial.

  • Recursive Random Projection (RRP): A sampling-based baseline that recursively partitions the configuration space using random projections, evaluating representative boundary instances at each split [6]. Its evaluation count depends on dataset size and recursion depth.

  • Direct few-shot prompting (BS-LLM): Following Senthilkumar and Menzies [41], BS-LLM prompts the language model with task metadata MM and Binitial=4B_{\mathrm{initial}} = 4 initial best/rest examples to propose Bgenerated=4B_{\mathrm{generated}} = 4 warm-start candidates in a single call, followed by Bsearch=16B_{\mathrm{search}} = 16 SMO evaluations (Btotal=24B_{\mathrm{total}} = 24).

  • Three-session budget-matched SynthCore (SynthCore-28): We run the Diversity ensemble routine from SynthCore [42]. It selects Binitial=16B_{\mathrm{initial}} = 16 initial representatives via recursive partitioning and executes 15 independent few-shot generation sessions, each proposing 4 candidates mapped to

observed rows in D\mathcal{D}, before selecting the top configuration across all sessions. This procedure issues 16+15×4=7616 + 15 \times4 = 76 nominal evaluation requests per trial. Because full SynthCore consumes more than double our 24-label budget, we evaluate a budget-matched restriction, SynthCore-28, constructed from the 16 initial representatives and three generation sessions of each trial (16+3×4=2816 + 3 \times4 = 28 nominal evaluation requests, averaging 26.31 distinct configuration labels). Comparing SynthCore-28 against BS-LLM and AMP-4 isolates whether multi-session ensembling remains competitive when restricted to roughly two dozen configuration labels.

Experimental setup

Our four research questions (§ 1) use different parts of this study. RQ1 and RQ2 evaluate the ten automated 24-label methods across 49 MOOT datasets. RQ3 evaluates H-DKP along with ten baselines across eight datasets with nine independent expert tracks. RQ4 examines labeling, computational, and human effort costs across all methods, along with a budget-matched three-session SynthCore baseline (SynthCore-28) to test whether multi-session ensembling remains competitive under small labeling budgets. Across all research questions, we evaluate the final configuration returned by each method using Chebyshev distance to the ideal point.

Datasets and dimensionality

To assess how methods scale with parameter space complexity (RQ2), we partition the 49 MOOT datasets by input feature count pp into three tiers: low dimensionality (p<6p < 6, 15 datasets), medium dimensionality (6≤p≤116 \le p \le11, 17 datasets), and high dimensionality (p>11p > 11, 17 datasets). The full 49-dataset corpus spans measured software configurations (such as database, compiler, and learner tuning), software process simulations, and general optimization tasks that broaden the range of feature interactions and objective trade-offs.

For more details on this data, see the MOOT documentation [29].

The human-in-the-loop study

Evaluating H-DKP (RQ3) requires human specialists to audit technical parameter relationships. Our protocol was approved by the North Carolina State University Institutional Review Board (protocol #28893). We recruited ten participants via email based on relevant software engineering, systems, or quantitative domain experience. Each candidate received an informed consent document by email and confirmed participation by email reply prior to the first round of questions. Participation was voluntary and uncompensated. Nine participants completed the protocol and one withdrew prior to completion (our records do not log the withdrawn participant’s assigned dataset), yielding nine completed expert tracks across eight datasets: two low-dimensional, four medium-dimensional, and two high-dimensional tasks (Table 2).

IDDatasetppmmTierRole / expertise basisYrsContactsPrep. (Gen. exch.)
E1Auto9343LowMechanical engineer4942 (7)
E2Project health53LowIndustrial repository maintainer2612 (2)
E3A2C CartPole74MedSoftware engineer4836 (6)
E4PostgreSQL91MedSoftware engineer4818 (3)
E5POM3a (Track 1)93MedPublished articles in area41012 (2)
E6POM3a (Track 2)93MedPublished articles in area301018 (3)
E7LLVM101MedResearcher in relevant area21024 (4)
E8Life expectancy202HighBiostatistician in public health41048 (8)
E9XOMO Flight234HighResearcher in relevant area2106 (1)
Total (8 datasets, 9 tracks)81216 (36)

Table 2. Participants and datasets in the H-DKP study. Each track represents one specialist’s asynchronous rule review followed by 20 optimization repeats using the same rules. Columns pp and mm give input features and objectives; Tier denotes dimensionality (Low p<6p < 6, Med 6≤p≤116 \le p \le11, High p>11p > 11); Contacts gives completed exchanges; Prep. gives benchmark accesses during offline follow-up question generation (Gen. exch.).

Table 2 summarizes the nine specialists (E1–E9), whose domain tenure ranged from 2 to 30 years. Within this cohort, the POM3 family models requirements prioritization under shifting stakeholder constraints [35], and XOMO Flight simulates autonomous software development risk and effort [26]. Two independent specialists (E5 and E6) evaluated POM3a starting from the same initial LLM claims to test whether different experts produce consistent optimization outcomes on the same task.

As formalized in § 3.4 (Algorithm 3), each track proceeded in two phases. In Phase A (offline knowledge acquisition), the expert reviewed up to ten LLM claims (VALID, MODIFY, INVALID, or UNSURE) and then answered one diagnostic question per follow-up email (either a failure diagnosis or a plausibility check) before freezing the knowledge base KK after 6–10 exchanges. Across the nine tracks, 36 follow-up exchanges involved generating and matching intermediate configurations against the benchmark table; under the nominal protocol of four seed rows and two candidate matches per generation exchange, offline preparation incurred an estimated 6–48 nominal table accesses per track (216 across all nine tracks; Table 2). Because seed rows and nearest-neighbor matches can repeat across exchanges, distinct offline labels are bounded above by these nominal counts. In Phase B (online optimization), the frozen rule set KK guided 20 independent trials, each strictly capped at the 24-label online budget (Binitial=4B_{\mathrm{initial}} = 4, Bgenerated=4B_{\mathrm{generated}} = 4, Bsearch=16B_{\mathrm{search}} = 16). Crucially, these offline accesses do not pass observed configurations or objective scores into Phase B: all intermediate configurations evaluated during Phase A are discarded after question generation, and only qualitative, natural-language directional rules explicitly confirmed by the specialist enter the frozen knowledge base KK.

Across all nine tracks, Phase A totaled 81 elicitation exchanges (6–10 per track, plus one recruitment and one thank-you email), completed over 5–17 calendar days per specialist. Because the first contact presented up to ten initial claims at once, initial calibration replies were longest (at most 19 lines), whereas subsequent single-question exchanges required only 1–3 lines per reply. Box 1 illustrates two claim-review exchanges.

Box 1: Sample expert claim-review exchanges from H-DKP (Phase A)

Example A – PostgreSQL (Rule R08, rejected):

  • Initial LLM claim: “If fsync is set to off, fullPageWrites should be kept on to mitigate increased risk of page corruption.”

  • Expert E4 verdict (INVALID): “Doesn’t make sense to keep FullPageWrites on if Fsync is already off if mitigation is the goal.”

  • Resulting rule treatment: R08 was removed from the frozen knowledge base (without inferring an unverified opposite rule).

Example B – POM3a (Rule R07, qualified vs. retained):

  • Initial LLM claim: “Higher Culture values (more requirements change) are likely to increase both simulated project cost (Cost−) and team idleness (Idle−).”

  • Expert E5 verdict (MODIFY): “Really depends on the change. If by change you mean tiny adoptable changes, the statement does not apply.”

  • Resulting frozen rule (Track 1): “If more requirements change, project cost and team idle time are likely to increase, subject to the scope of the change and existing foundations.” (By contrast, Expert E6 in Track 2 classified R07 as VALID and retained the broader rule.)

Budgets and model configuration

Figure 1 shows the study workflow. In the single-session LLM methods (BS-LLM, AMP, HKMA, and H-DKP), each online trial begins with Binitial=4B_{\mathrm{initial}} = 4 labeled examples partitioned into best and rest sets. The knowledge-construction procedure guides the LLM to propose Bgenerated=4B_{\mathrm{generated}} = 4 candidate configurations (two expected to perform well and two expected to perform poorly), which are mapped to their nearest observed rows in D\mathcal{D} and labeled. A sequential model-based optimizer (SMO) then selects and labels the remaining configurations up to a fixed total budget of Btotal=24B_{\mathrm{total}} = 24 evaluations (Bsearch=16B_{\mathrm{search}} = 16 for BS-LLM, AMP, HKMA-rag, and H-DKP; Bscout=10B_{\mathrm{scout}} = 10 and Bsearch=6B_{\mathrm{search}} = 6 for HKMA-scout and HKMA-combined). For the budget-matched ensemble sensitivity baseline, SynthCore-28 is configured as described in § 3.5. We run 20 independent trials per method and dataset across all experiments.

All LLM-guided methods use GPT-OSS-120B [1]3 with a decoding temperature of 0.7, top-p of 1.0, and a maximum of 4,000 output tokens per call during candidate generation. Initial domain claims in H-DKP are elicited at temperature 0 to ensure deterministic baseline beliefs for expert review (§ 4.2). For every completed trial, we log the number of LLM API calls, input and output tokens, monetary API charges, and elapsed wall-clock time.

To ensure a fair comparison, and following principles of experimental algorithmics (McGeoch 2012), we account for all online configuration evaluations consumed across every stage of an optimization run:

Btotal=Binitial+Bscout+Bgenerated+Bsearch.(1)B_{\mathrm{total}} = B_{\mathrm{initial}} + B_{\mathrm{scout}} + B_{\mathrm{generated}} + B_{\mathrm{search}}. \tag*{(1)}

Here, BinitialB_{\mathrm{initial}} denotes initial seed observations, BscoutB_{\mathrm{scout}} denotes preliminary TPE scouting evaluations, BgeneratedB_{\mathrm{generated}} denotes warm-start proposals mapped to benchmark rows, and BsearchB_{\mathrm{search}} denotes subsequent sequential search steps. Because mapping multiple generated candidates to their nearest neighbors in a finite table can select the same row more than once, we also track the number of distinct observed configurations evaluated per trial. Table 3 summarizes the online evaluation budgets across all methods; for H-DKP, one-time offline preparation accesses (6–48 nominal accesses per track; Table 2) are incurred once prior to freezing the knowledge base and are separate from the 24-label per-trial budget.

MethodBinitialB_{\mathrm{initial}}BscoutB_{\mathrm{scout}}Bgen.B_{\mathrm{gen.}}BsearchB_{\mathrm{search}}Nominal BtotalB_{\mathrm{total}}Distinct
Random000242424
EZR400202424
RRPRecursive partitioning by dataset sizeDataset-dep.Dataset-dep.
BS-LLM4041624≤24\le24
AMP-2 / 3 / 44041624≤24\le24
HKMA-rag4041624≤24\le24
HKMA-scout4104624≤24\le24
HKMA-combined4104624≤24\le24
H-DKP44041624≤24\le24
SynthCore-281601202826.31

Table 3. Configuration evaluation budgets across methods, decomposed by Equation (1). Nominal BtotalB_{\mathrm{total}} gives total evaluation requests per trial; Distinct gives the mean number of unique benchmark configurations labeled.

Final configuration quality

We evaluate each method by the multi-objective quality of the final configuration xx it returns. For each objective j∈{1,…,m}j \in\{1, \ldots, m\}, let aja_j and bjb_j denote the minimum and maximum observed values across the reference dataset D\mathcal{D}. Each nonconstant objective is min-max normalized to the unit interval:

uj(x)=yj(x)−ajbj−aj.(2)u_j(x) = \frac{y_j(x) - a_j}{b_j - a_j}. \tag*{(2)}

Let zjz_j denote the ideal normalized value for objective jj, where zj=0z_j = 0 for minimization objectives and zj=1z_j = 1 for maximization objectives. We measure the quality of configuration xx using its Chebyshev (L∞L_\infty) distance to the ideal point zz:

D(x)=max⁡1≤j≤m∣uj(x)−zj∣.(3)D(x) = \max_{1 \le j \le m} \lvert u_j(x) - z_j \rvert. \tag*{(3)}

Lower values of D(x)D(x) indicate better performance. Unlike averaging metrics that allow poor performance on one objective to be masked by gains on another, Chebyshev distance penalizes the worst normalized shortfall across all mm objectives [21]. Because the ideal point zz combines the best observed value of each individual objective in D\mathcal{D}, it typically lies outside the feasible Pareto front, meaning D(x)>0D(x) > 0 even for Pareto-optimal configurations.

During optimization, SMO, EZR, and SynthCore-28 rank candidate configurations internally using their native acquisition heuristic, Euclidean distance to heaven, m−1∑j=1m∣uj(x)−zj∣2\sqrt{m^{-1}\sum_{j=1}^{m}|u_j(x)-z_j|^2}. We preserve each optimizer’s internal L2L_2 ranking unmodified so that baselines operate as designed in prior work [27, 42], while scoring the final configuration returned by every method with Chebyshev distance D(x)D(x) in Equation (3) as an independent, external audit.

Statistical comparisons and grouping

Because optimization quality depends on how many configurations an algorithm is permitted to evaluate, comparing a 24-label method directly against a multi-session ensemble that consumes over four dozen labels confounds knowledge quality with sample size. We therefore structure our statistical comparisons around three scopes restricted to roughly two dozen evaluations:

  1. Primary benchmark comparison (RQ1 and RQ2): Compares the ten automated 24-label methods (Random, EZR, RRP, BS-LLM, AMP-2, AMP-3, AMP-4, HKMA-scout, HKMA-rag, and HKMA-combined) across all 49 datasets.

  2. Expert-study comparison (RQ3): Compares H-DKP against the same ten automated methods across the eight expert-study datasets, analyzing each of the two POM3a expert tracks independently.

  3. Budget-matched ensemble sensitivity comparison (RQ4): Evaluates the three-session SynthCore-28 baseline (28 nominal requests; 26.31 mean distinct labels) against the ten 24-label methods across all 49 datasets, alongside an analysis of API token costs, runtime, and human preparation effort.

Within each dataset and comparison scope, we rank methods using the Scott–Knott clustering procedure [40, 30]. Methods are sorted by their mean Chebyshev distance across the 20 trials and recursively partitioned into disjoint groups to maximize the between-group sum of squares. A candidate split is accepted only if the two resulting groups exhibit both a non-negligible effect size via Cliff’s delta (∣δ∣≥0.147|\delta| \ge0.147) [9, 36] and statistical significance (p<0.05p < 0.05) under a studentized bootstrap test [47] with 512 resamples under a common-mean null and an add-one pseudocount correction [34].^5

Methods assigned to Group 1 (the lowest-scoring cluster) on a given dataset are statistically indistinguishable winners for that task. Because Scott–Knott splits depend on the set of candidate distributions being partitioned, group memberships are computed separately for each comparison scope. Across datasets, we report both the number of tasks on which each method enters Group 1 (best-group membership) and the equal-weighted mean Chebyshev distance across the benchmark corpus.

Ablation and expert-rule analysis

To understand why specific knowledge strategies succeed or fail, our experimental design includes controlled ablations alongside the human expert audit. Comparing BS-LLM, AMP-2, AMP-3, and AMP-4 under an identical 24-label budget isolates the incremental effect of task analysis, constraint discovery, and per-candidate constraint validation. Similarly, comparing HKMA-scout, HKMA-rag, and HKMA-combined isolates the contribution of empirical scouting summaries versus retrieved documentation. For H-DKP, we record the expert classification (VALID, MODIFY, INVALID, or UNSURE) for every initial LLM claim, as well as the rules added or qualified during follow-up exchanges, allowing us to relate rule curation patterns to downstream optimization medians.

Results

Our experiments follow one design principle: human time is the scarce resource, so we use it last and only where it matters most. Hence, our experiments run in two stages:

  • Make elaborate use of cheap CPU time: We run 10 optimizers on 49 datasets and keep the best one. As seen below, the automatic analysis of RQ1 and RQ2 selected AMP-4.

  • Take more care with expensive human time: RQ3 compares our human-in-the-loop method (H-DKP) against the ten automated 24-label methods across eight datasets (nine expert tracks), and RQ4 evaluates the label, compute, and human effort costs across methods.

RQ1: Automated warm starts with 24 labels

RQ1: With only 24 labels, can domain knowledge help an LLM propose better initial configurations?

If RQ1 is true, then we would expect all our optimizers to perform better than some simpler baseline (i.e. BS-LLM’s single-shot prompting that uses no background domain knowledge).

As shown in Table 4, this expectation was not met:

MethodBest /49 (%)Low /15Medium /17High /17Mean DD (from Equation (3))
AMP-444 (89.8%)1315160.1862
BS-LLM36 (73.5%)1112130.1959
AMP-236 (73.5%)1112130.1965
AMP-336 (73.5%)1014120.1987
EZR34 (69.4%)1013110.1953
HKMA-scout33 (67.3%)1110120.1956
HKMA-combined33 (67.3%)1111110.1963
HKMA-rag33 (67.3%)121290.1971
Random23 (46.9%)8960.2226
RRP13 (26.5%)3460.2675

Table 4. Final Chebyshev distance (DD) and Scott–Knott Group 1 (best-group) membership counts across the 49 datasets for the ten automated 24-label methods. Lower DD values are better. Dimensionality tiers contain 15 low (p<6p < 6), 17 medium (6≤p≤116 \le p \le11), and 17 high (p>11p > 11) datasets.

  • BS-LLM was as good as, or better than, nearly everything else.

  • The sole exception was AMP-4. Among all compared approaches, AMP-4 achieves both the broadest statistical win rate, entering the best Scott–Knott group (Group 1) on 44 of 49 datasets (89.8%), and the lowest equal-weighted mean Chebyshev distance (0.1862).

To understand why AMP-4 does so well, it is insightful to compare it against its simpler variants:

  • Prompting the model to analyze feature trade-offs prior to generation (AMP-2) yields only 36 best-group memberships and a mean score of 0.1965, matching BS-LLM (36 memberships, mean 0.1959).

  • Adding explicit constraint discovery without validation (AMP-3) also does no better than 36 memberships and slightly increases the mean Chebyshev distance to 0.1987.

Hence we say that asking an LLM to formulate domain constraints in text provides no aggregate benefit if the model is still expected to satisfy those constraints in a single generation pass. Only when each generated candidate is explicitly checked and revised against the discovered constraints (AMP-4) does performance jump from 36 to 44 best-group memberships and drop in mean Chebyshev distance by nearly a full percentage point (0.1959 to 0.1862).

As to other methods: supplying external context without candidate verification failed to improve upon direct few-shot prompting.

  • Combining TPE statistical scouting with document retrieval (HKMA-combined, 33 memberships, mean 0.1963) provides no additive gain over either source alone.

  • For HKMA-scout and HKMA-combined, allocating 10 of the 24 evaluations to preliminary TPE scouting reduces the downstream sequential search budget from 16 steps to 6 steps, offsetting the benefit of the scouting summary.

Finally, the model-free baselines reveal the importance of benchmarking LLM warm starts against capable active learners rather than simple random sampling: while all LLM methods substantially outperform Random search (23/49, mean 0.2226) and RRP (13/49, mean 0.2675), the lightweight EZR active learner enters the best group on 34 datasets and achieves a lower mean Chebyshev score (0.1953) than BS-LLM, AMP-2, AMP-3, and all three HKMA variants. Among the compared methods, only AMP-4 outperforms both BS-LLM and EZR on best-group frequency as well as mean Chebyshev distance.

Hence, on the question of whether domain knowledge helps an LLM propose better initial configurations, we say: Only if that knowledge is checked. Unchecked knowledge (LLM-discovered rules, scouting, documents) did no better than direct few-shot prompting or the EZR active learner. But AMP-4, which checks each candidate against the rules, reached the best statistical rank on 44 of 49 datasets.

RQ2: Performance across dimensionality

RQ2: Does increasing the number of configuration parameters change which kinds of knowledge help?

If this were true, then any optimization oracle (LLM or symbolic, or human) might perform worse as the number of configuration interactions increased.

To check this, we look at the #configuration options columns of Table 4. Partitioning the 49 datasets across low (p<6p < 6), medium (6≤p≤116 \le p \le11), and high (p>11p > 11) parameter dimensionality reveals distinct scaling behaviors across knowledge strategies: some algorithms improve as dimensionality increases, while others degrade. Algorithms that improve as the number of configuration options increases are:

  • AMP-4 leads every tier and exhibits its largest advantage in high-dimensional spaces, entering the best group on 13 of 15 low-dimensional tasks (86.7%), 15 of 17 medium-dimensional tasks (88.2%), and 16 of 17 high-dimensional tasks (94.1%). By comparison, direct few-shot prompting (BS-LLM) and unvalidated analysis (AMP-2) remain between 70.6% and 76.5% across all three tiers.

  • HKMA-scout performs best in high-dimensional spaces relative to its other tiers, securing 12 of 17 best-group memberships (70.6%) compared with 9 of 17 for HKMA-rag, perhaps because TPE scouting supplies concrete empirical configurations observed directly in the high-dimensional space.

On the other hand, other algorithms are clearly challenged by an increasing number of options:

  • Retrieval-only prompting (HKMA-rag) performs well on low-dimensional tasks (12/15, 80.0%) and medium-dimensional tasks (12/17, 70.6%), but degrades sharply to 9 of 17 tasks (52.9%) in high-dimensional spaces.

  • Model-free methods also show sensitivity to dimensionality: Random search falls from 53.3% (8/15) in low dimensions to 35.3% (6/17) in high dimensions.

  • EZR declines from 76.5% (13/17) in medium dimensions to 64.7% (11/17) in high dimensions.

In any case, if we focus on the methods that did best in RQ1, on the question of whether more configuration parameters change which kinds of knowledge help, we say:

Yes. Checking matters most in high dimensions, where AMP-4 ranked best on 16 of 17 tasks, while documentation-only prompting fell from 12 of 15 (low) to 9 of 17 (high).

RQ3: Expert-reviewed knowledge

RQ3: Does human review of the LLM’s domain rules help?

MethodBest /8Share (%)
H-DKP7/887.5%
EZR7/887.5%
BS-LLM6/875.0%
AMP-45/862.5%
AMP-35/862.5%
HKMA-rag4/850.0%
HKMA-combined4/850.0%
HKMA-scout3/837.5%
AMP-23/837.5%
Random3/837.5%
RRP2/825.0%

Table 5. Scott–Knott Group 1 (best-group) membership counts across the eight expert-study datasets when comparing H-DKP against the ten automated 24-label methods. Separate analyses using POM3a Expert 1 and POM3a Expert 2 yield identical best-group counts.

Our pre-experimental intuition was that combining sources of intelligence (human and LLM) should improve our ability to configure software. If that were true, then our human-in-the-loop method (H-DKP) should outperform unguided prompting and match or exceed the automated baselines.

Table 5 compares H-DKP against the ten automated 24-label baselines across the eight expert-study datasets (nine independent expert tracks). In the eleven-method Scott–Knott analysis, H-DKP and model-free EZR tie for the highest best-group membership, each entering Group 1 on seven of the eight datasets (87.5%), ahead of BS-LLM (6/8) and AMP-4 and AMP-3 (5/8 each). These aggregate group counts are identical whether POM3a is evaluated using Expert Track 1 or Expert Track 2.

Because eight datasets with 20 trials each can place multiple methods into the same Scott–Knott cluster when within-task variance is high, we also inspect trial medians on the individual tracks to separate central tendency from outlier trials. Even where multiple methods share Group 1 on mean score, H-DKP achieves a lower 20-trial median Chebyshev distance than both BS-LLM and EZR on all seven non-tied datasets (and ties at the 0.2500 optimum on Auto93). Inspecting individual tasks also explains why AMP-4 enters Group 1 on five of the eight expert datasets compared with seven for H-DKP: on both Life Expectancy (p=20p = 20) and LLVM (p=10p = 10), AMP-4 matches or beats BS-LLM and EZR in median shortfall (0.0821 on Life Expectancy, tying H-DKP versus 0.1014 for BS-LLM and 0.0966 for EZR; 0.0151 on LLVM versus 0.0163 for BS-LLM and 0.0194 for EZR), but occasional high-shortfall outlier trials raise its mean score into Scott–Knott Group 2. Conditioning every run on a frozen set of expert-verified rules in H-DKP suppresses these outlier runs while preserving low median shortfall.

Examining the expert elicitation logs shows why this curation works. Across the nine tracks, specialists audited 61 initial zero-shot LLM claims, classifying 47 as VALID (77.0%), 8 as MODIFY (13.1%), 1 as INVALID (1.6%), and 5 as UNSURE (8.2%), which together with confirmed follow-up additions yielded 56 frozen rules (2 to 8 rules per track). Rather than rubber-stamping model beliefs, experts actively filtered flawed assumptions (Box 1), revealing two practical patterns:

  • Rule precision outweighs quantity: On XOMO Flight (p=23p = 23, the highest-dimensional task in the cohort), the specialist confirmed only two rules and marked two others uncertain, yet those two constraints guided H-DKP to a median score of 0.1138, beating BS-LLM (0.1239), EZR (0.1231), and AMP-4 (0.1172).

  • Equifinality across independent specialists: On POM3a (p=9p = 9), two experts reviewed the same initial LLM claims and produced different rule sets (Expert 1 validated six and modified two for eight rules; Expert 2 validated six and marked two uncertain for six rules). Despite generating different initial warm starts, both tracks converged to the exact same median benchmark configuration (D=0.1239D = 0.1239), improving substantially over BS-LLM (0.1648) and EZR (0.1547).

Hence, on the question of whether human review of the LLM’s domain rules helps, we say:

Yes. H-DKP ranked best on 7 of 8 expert tasks, and beat direct prompting on 7 of 8. However, EZR also ranked best on 7 of 8, at zero cost.

RQ4: Cost and budget-matched sensitivity

RQ4: What does checked knowledge cost?

A pressing concern with LLM-based methods is the evaluation and computational cost associated with running them. In particular, prior work such as SynthCore [42] pools 16 initial representatives with 15 few-shot generation sessions (76 nominal requests, averaging 55 distinct labels per trial), consuming more than twice the evaluation budget of our 24-label methods.

To test whether multi-session ensembling remains competitive without an expanded labeling budget, Table 6 compares the budget-matched three-session restriction, SynthCore-28 (28 nominal requests; 26.31 mean distinct labels, range 20–32), against the ten 24-label methods across all 49 datasets. When restricted to roughly the same evaluation budget as the 24-label methods, SynthCore-28 drops to ninth place, entering the best Scott–Knott group on only 23 of 49 datasets with a mean Chebyshev distance of 0.2063. Under equalized budgets, AMP-4 (39/49 in this eleven-method partitioning, mean 0.1862), BS-LLM (34/49, mean 0.1959), and model-free EZR (32/49, mean 0.1953) all substantially outperform SynthCore-28. Spending scarce labels on 16 initial representatives and unguided few-shot sessions without sequential search is therefore far less sample-efficient than pairing a single validated warm start with sequential optimization.

MethodDistinct labelsBest /49Mean DD
AMP-4≤24\le24390.1862
BS-LLM≤24\le24340.1959
EZR24320.1953
AMP-2≤24\le24320.1965
AMP-3≤24\le24320.1987
HKMA-rag≤24\le24290.1971
HKMA-scout≤24\le24280.1956
HKMA-combined≤24\le24260.1963
SynthCore-2826.31230.2063
Random24190.2226
RRPDataset-dep.120.2675

Table 6. Budget-matched sensitivity analysis comparing three-session SynthCore-28 (28 nominal requests, 26.31 mean distinct labels) against the ten 24-label methods across 49 datasets. Scott–Knott groups are recomputed across all eleven methods.

Beyond configuration evaluations, the methods differ in their computational and human preparation overheads (Table 7). Model-free baselines (Random, EZR, and RRP) incur zero LLM token costs and complete in a quarter of a second or less (≤0.23\le0.23 s). Among LLM-assisted methods, single-call approaches incur the lowest API charges per trial: HKMA-rag averages 1,219 tokens (0.02¢) and BS-LLM averages 1,688 tokens (0.03¢). Staged prompting increases token consumption alongside the number of reasoning steps: AMP-2 averages 4,648 tokens (0.07¢ across 2 calls), AMP-3 averages 10,279 tokens (0.15¢ across 3 calls), and AMP-4 averages 25,567 tokens (0.54¢ across 7 calls).

MethodLLM callsTokensAPI cost (¢)Time (s)
Random000<0.01
EZR0000.04
RRP0000.23
HKMA-rag11,2190.029.20
BS-LLM11,6880.032.40
HKMA-scout24,0330.0421.10
HKMA-combined24,1710.0520.20
AMP-224,6480.0739.80
AMP-3310,2790.1580.30
AMP-4725,5670.54209.80

Table 7. LLM calls and mean computational and inference cost per completed trial across the ten 24-label methods with GPT-OSS-120B, ordered by monetary API charge in US cents (¢). Tokens sum input and output tokens.

Although AMP-4 requires the largest token budget among the tested methods, its absolute inference cost remains roughly half of one US cent (0.54¢) and 3.5 minutes (209.80 s) of sequential API latency per trial. For H-DKP, upfront offline preparation required 6–10 short email exchanges (1–3 lines per follow-up reply over 5–17 calendar days) and an estimated 6–48 nominal benchmark accesses per track to formulate diagnostic questions (§ 4.2); once frozen, the rule set is reusable across optimization runs, requiring only a single online LLM call per trial comparable in token cost to BS-LLM.

Hence, on the question of what checked knowledge costs, we say:

Very little. AMP-4 costs half a cent of LLM calls per trial (≤24\le24 labels) and H-DKP needs only 6 to 10 short expert emails. By contrast, restricting multi-session SynthCore to a comparable budget (SynthCore-28, 26.31 labels) drops its win rate to 23 of 49 tasks, trailing both AMP-4 (39 of 49) and zero-cost EZR (32 of 49).

Threats to validity

Construct validity

We evaluate final optimization quality using Chebyshev (L∞L_{\infty}) distance to the ideal point, which penalizes the worst normalized shortfall across all objectives. Different multi-objective indicators, such as hypervolume or Euclidean distance to heaven, reflect alternative practitioner preferences over Pareto trade-offs [21]. Moreover, the downstream sequential optimizer (SMO), EZR, and SynthCore-28 rank candidates internally using Euclidean distance to heaven. Scoring the final configuration returned by every method with Chebyshev distance provides a uniform external criterion that requires balanced attainment across all objectives, although aligning internal search objectives with the external evaluation metric could further improve final scores.

Because MOOT is a collection of finite tabular benchmarks, objective normalization uses the minimum and maximum values recorded in each reference table, and LLM-generated configurations are evaluated by mapping them to the nearest observed row in normalized feature space. When a proposed configuration lies far from recorded table entries, nearest-neighbor lookup is only an approximation of true system behavior [33, 55]. To prevent tabular approximation artifacts from misleading human specialists during the H-DKP study, we inspected proposal-to-row distances prior to each contact and switched from failure-diagnosis queries to profile-plausibility queries whenever the nearest observed row was too distant to represent the generated candidate faithfully.

Internal validity

The compared methods differ in their prompt structures, knowledge sources, and labeling budgets, all of which contribute to final optimization performance. To prevent sample size from confounding the comparison of knowledge strategies, our primary benchmark evaluation equalizes the total budget at 24 configuration evaluations across Random search, EZR, BS-LLM, the three AMP variants, the three HKMA variants, and H-DKP (with RRP governed by its dataset-dependent recursive partitioning depth). For the budget-matched SynthCore-28 baseline, we explicitly track and report both total evaluation requests and mean distinct configurations labeled per trial. In H-DKP, the offline rule-elicitation phase used 36 generation-associated exchanges (an estimated 6–48 nominal table accesses per track, or 216 in total; Table 2) to construct diagnostic questions before freezing the rules; we report these one-time offline preparation costs separately from the 24-label online trial budget, and note that our effort logs record email counts, reply line lengths, and calendar turnaround (5–17 days) rather than active working minutes. Moreover, downstream H-DKP performance does not depend on heavy offline table querying: on XOMO Flight (p=23p = 23) and POM3a Track 1 (p=9p = 9), Phase A required only one and two generation-associated exchanges (6 and 12 nominal table accesses, respectively; Table 2), yet both tracks substantially outperformed BS-LLM and EZR across the 20 online trials.

Using pre-trained LLMs introduces potential threats from benchmark memorization, provider updates, and stochastic decoding variance [37]. Although GPT-OSS-120B may have been exposed to public documentation or descriptions of standard software systems during pre-training, all LLM baselines and knowledge-guided variants use the exact same underlying model and decoding parameters, ensuring that relative differences reflect the prompting and knowledge workflows rather than model selection. To account for stochastic generation and occasional malformed model outputs, we execute 20 independent trials per dataset and log all prompts, responses, and random seeds. When an LLM call returns unmappable feature values or fails schema parsing, the evaluation pipelines apply deterministic fallback rules (for example, SynthCore-28 falls back to a session’s initial representatives if no valid candidates are parsed in that session, and the AMP-4 evaluator retains prior valid settings when a revised field cannot be mapped). Consequently, the reported results reflect the end-to-end performance of each prompting and validation pipeline across completed trials.

Conclusion validity

Within each dataset, we group methods using Scott–Knott clustering gated by a non-parametric effect size threshold (Cliff’s ∣δ∣≥0.147|\delta| \ge0.147) and a studentized bootstrap test (p<0.05p < 0.05). All automated and expert-study comparisons are based on 20 trials per dataset. Because Scott–Knott partitions depend on the specific pool of methods included in a comparison, we recompute clusters separately for the ten-method primary scope, the eleven-method expert scope, and the SynthCore-28 sensitivity scope. Sharing Group 1 on a dataset indicates that differences among those methods are either statistically insignificant or small in effect size; we therefore report equal-weighted mean Chebyshev distances and trial medians alongside Group 1 membership counts to capture both statistical rank and shortfall magnitude.

External validity

Our 49-dataset benchmark corpus combines measured software system configurations (such as compilers, databases, and machine learning pipelines), software process simulations, and general optimization tasks from the MOOT repository. While this diverse corpus enables a broad evaluation across low, medium, and high parameter dimensionalities, dimensionality covaries with problem domain, table cardinality, and feature types across the benchmark. Furthermore, evaluating configurations via tabular lookup does not capture live execution variance, hardware interference, or workload drift on deployed production systems. Finally, recruitment for the H-DKP study depended on access to specialists with domain expertise in the target tasks. While our nine expert tracks across eight datasets (spanning 2 to 30 years of experience, including both system-specific authors/maintainers and general software engineering or quantitative researchers; Table 2) provide initial empirical evidence on human-in-the-loop rule curation, broader studies with larger pools of practitioners are needed to characterize inter-expert variability across industrial software systems.

Discussion

Our main result is simple: checking an LLM’s domain knowledge matters more than adding to it. Rules, documents, and scouted data did not help until each candidate was checked against them, either by the LLM itself (AMP-4) or by a human expert (H-DKP). The rest of this section explains why, what experts add, and what this means for practitioners.

Why constraint verification outperforms passive context injection

A central finding of our study is that domain knowledge improves LLM warm starts only when that knowledge is actively checked, either by the model itself (AMP-4) or by a human specialist (H-DKP). Under a fixed 24-label budget, prompting the model to analyze feature trade-offs (AMP-2, 36/49 best-group memberships, mean 0.1965) or to explicitly list parameter constraints before generation (AMP-3, 36/49, mean 0.1987) yields no aggregate improvement over direct few-shot prompting (BS-LLM, 36/49, mean 0.1959). However, when AMP-4 validates each generated candidate individually against those same model-discovered constraints and revises violations, best-group membership rises to 44 of 49 datasets (89.8%) and mean Chebyshev distance drops to 0.1862.

This contrast between AMP-3 and AMP-4 highlights an instruction-following gap between natural-language reasoning and numerical vector generation. An LLM can articulate sensible parameter dependencies in text, yet still violate its own stated constraints when asked to output four multi-dimensional configurations simultaneously in a single pass. Separating generation from per-candidate verification allows the model to audit one configuration vector at a time and repair infeasible or conflicting parameter assignments before physical evaluations are spent. This mechanism also explains why AMP-4 scales effectively with dimensionality, winning on 16 of 17 high-dimensional datasets (p>11p > 11, 94.1%) whereas unvalidated constraint discovery (AMP-3) drops from 14 of 17 in medium dimensions to 12 of 17 in high dimensions.

The HKMA results reinforce the limitation of passive context injection. All three HKMA variants enter the best group on 33 of 49 datasets, trailing BS-LLM (36/49) despite incorporating retrieved documentation, statistical scouting summaries, or both. Two opposing trade-offs explain this outcome:

  1. Scouting versus sequential search allocation: Under a strict 24-label cap, HKMA-scout and HKMA-combined spend 10 evaluations on TPE scouting prior to LLM invocation, leaving only 6 evaluations for downstream sequential optimization (compared with 16 SMO steps for BS-LLM and AMP). While scouting helps the LLM observe empirical patterns in high-dimensional spaces (where HKMA-scout achieves 12/17 best-group memberships versus 9/17 for HKMA-rag), starving the sequential optimizer of post-warm-start iterations offsets this advantage across the broader benchmark.

  1. Documentation specificity in high dimensions: Retrieval-only prompting (HKMA-rag) preserves all 16 SMO steps and performs well on low-dimensional tasks

(12/15, 80.0%), but falls to 9 of 17 (52.9%) on high-dimensional tasks. General documentation passages rarely specify joint settings across more than a dozen interacting parameters, and combining scouting summaries with retrieved text (HKMA-combined) adds prompt length without enforcing constraint satisfaction.

Human expertise, rule curation, and model-free baselines

The H-DKP study shows why active curation matters when incorporating human domain knowledge. Across the nine expert tracks, specialists did not merely rubber-stamp the LLM’s zero-temperature beliefs; they modified claims on five tracks, rejected an invalid claim on PostgreSQL, and flagged uncertain claims on three tracks. Incorporating these verified rules enabled H-DKP to achieve a lower median Chebyshev distance than direct prompting (BS-LLM) on seven of the eight tasks (tying on Auto93) and to enter the top Scott–Knott group on seven of eight tasks. Moreover, comparing H-DKP with AMP-3 and AMP-4 clarifies how human rule curation and automated candidate verification address the instruction-following gap in complementary ways. In AMP-3, the LLM formulates unconstrained rule lists and attempts to satisfy them in a single pass, often violating its own stated dependencies. Although H-DKP also uses single-pass generation during online trials (Phase B), it avoids AMP-3’s degradation for two reasons: (1) during offline elicitation (Phase A), candidate rules are iteratively tested against intermediate proposals and failure-diagnosis queries (Algorithm 3, lines 5–9), filtering out rules that the model misapplies before the knowledge base KK is frozen; and (2) specialists prune the knowledge base down to a compact set of two to eight relevant constraints per track, reducing instruction-following overload during generation and stabilizing trial-to-trial variance relative to self-discovered rules in AMP-4.

The expert tracks also demonstrate that the precision of domain constraints matters more than their quantity, and that distinct expert perspectives can converge to equivalent optimization outcomes:

  • High-leverage constraints in large spaces: On XOMO Flight (p=23p = 23), the specialist confirmed only two initial rules and marked two others uncertain. Despite leaving most of the 23 parameters unconstrained by explicit rules, those two verified constraints guided H-DKP to the lowest median Chebyshev score (0.1138) among all compared methods.

  • Equifinality across independent experts: On POM3a (p=9p = 9), two specialists independently reviewed the same initial LLM claims and produced distinct rule sets: Expert 1 confirmed eight rules (including two modified conditions), whereas Expert 2 confirmed six rules and marked two uncertain. Although these two knowledge bases produced different initial warm-start proposals and distinct trial distributions, subsequent sequential search guided both tracks to the exact same median benchmark configuration (D=0.1239D = 0.1239), outperforming both BS-LLM (0.1648) and EZR (0.1547). Different experts can therefore emphasize different valid domain relationships while steering sequential optimization into the same Pareto-optimal basin.

At the same time, evaluating LLM warm starts alongside the model-free EZR active learner provides an essential reality check. Many studies of LLMs for optimization compare primarily against Random search. Had we compared only against Random search (23/49 best-group memberships, mean 0.2226), every LLM prompting variant would appear to offer a substantial gain. In contrast, EZR uses the same 24-label budget at zero token cost and 0.04 seconds per trial, yet achieves a lower mean Chebyshev score (0.1953) than BS-LLM (0.1959), AMP-2 (0.1965), AMP-3 (0.1987), and all three HKMA variants, while matching H-DKP’s best-group count (7/8) on the expert cohort. Consequently, unverified LLM prompting barely improves upon a well-designed active-learning heuristic. Only active constraint validation (AMP-4 across the 49 datasets) and human rule curation (H-DKP on trial medians) consistently justify the added complexity of invoking a language model.

Trading inference compute against physical system evaluations

Comparing AMP-4 (≤24\le24 labels, 0.54\mathrm{\cent} per trial) against budget-matched SynthCore-28 (26.31 mean distinct labels) and model-free EZR (24 labels, 0\mathrm{\cent} per trial) clarifies the trade-off between LLM inference compute and physical configuration evaluations. While original SynthCore [42] relies on pooling 16 initial representatives with 15 generation sessions (averaging 55 distinct configuration labels per trial), our budget-matched sensitivity analysis (Table 6) shows that multi-session ensembling breaks down when restricted to roughly two dozen evaluations. Using 26.31 mean distinct labels, SynthCore-28 enters the best statistical group on only 23 of 49 datasets (mean DD = 0.2063), losing to 24-label AMP-4 (39/49, mean 0.1862), BS-LLM (34/49, mean 0.1959), and zero-cost EZR (32/49, mean 0.1953). Without sequential search feedback or constraint validation, allocating 16 labels to initial partitioning and 12 labels to unguided few-shot proposals uses a small evaluation budget inefficiently.

At the same time, the inference cost profile in Table 7 places AMP-4’s computational overhead relative to EZR. Whereas EZR requires zero API tokens and completes in 0.04 seconds, AMP-4 requires 7 LLM calls and 25,567 tokens per trial, corresponding to an average API cost of 0.54\mathrm{\cent} (roughly half of one US cent) and 3.5 minutes of sequential inference time. When LLM access is unavailable or sub-second latency is required, EZR provides a strong model-free default that matches or beats every unverified prompting strategy. However, in real-world software engineering tasks, such as compiling a large codebase like LLVM, executing database benchmarks in PostgreSQL, or training machine learning models, evaluating a single candidate configuration on the target system can require several minutes to several hours of compute time. Spending half a cent and 3.5 minutes of LLM inference on staged constraint validation to secure top-ranked configurations within 24 labels (39/49 in the sensitivity scope and 44/49 in the primary scope, versus 32/49 and 34/49 for EZR) represents a favorable trade-off whenever physical system evaluations dominate total runtime.

Conclusion

Configuring complex software is an important and difficult task: parameter spaces are large, configuration errors are common, and each evaluation can be slow and costly. Prior work on LLM warm starts, such as SynthCore [42], had two drawbacks. It was fully automatic, with no human in the loop, and it needed dozens of distinct labels per trial. Here, we studied what an LLM should know before it proposes its first configurations, using just 24 labels, and we tested human experts as one source of that knowledge.

Across 49 datasets and nine expert tracks, our lead result is that domain knowledge helps only when it is checked. AMP-4, where the LLM checks each candidate against its own rules, reached the best statistical rank on 44 of 49 datasets. H-DKP, where human experts correct the LLM’s rules, did so on 7 of 8 expert tasks.

Our empirical findings translate into concrete guidelines for selecting an optimization strategy based on evaluation cost, parameter dimensionality, and available domain assets. When system evaluations are expensive and labels are strictly capped (≤24\le24), use AMP-4. It requires no external documentation or human expert time, costs roughly half a cent per trial in LLM inference, and enters the top statistical group on 89.8% of tasks overall and 94.1% of high-dimensional tasks. When LLM access or inference latency is restricted, use model-free EZR instead. It executes in milliseconds, requires no API tokens, and matches or outperforms unvalidated LLM prompting (BS-LLM, AMP-2, AMP-3, and HKMA) in mean optimization quality. Finally, when domain specialists are available for recurring tuning tasks, use H-DKP. Although asynchronous elicitation requires an upfront investment of six to ten expert contacts, the resulting frozen rule set requires only a single LLM call per run, can be reused across repeated optimization campaigns, and consistently lowers median shortfall relative to unguided prompting.

This study suggests several directions for future work. First, our human-in-the-loop study covers eight datasets and nine expert tracks; larger expert cohorts would test whether the H-DKP results hold across more domains and more specialists per task. Second, all LLM-guided methods here use GPT-OSS-120B. Repeating the study with other models would show whether the gap between checked and unchecked knowledge is a property of the method or of one model. Third, AMP-4 checks only the four initial candidates, but the same per-candidate validation could also guide the sequential optimizer that selects the remaining labels. Lastly, MOOT tasks are tabular benchmarks, where each label is a table lookup. Future work should apply these methods to live systems, where each label is a real compile, benchmark run, or training job.

Statements and declarations

Data and code availability

The study uses tabular tasks from MOOT [29]. Available at: https://github.com/timm/moot

The replication package is available at: https://github.com/srina1h/What-Should-an-LLM-Know

References

  1. [1]Agarwal S, Ahmad L, Ai J, et al (2025) gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:250810925arxiv.org/abs/2508.10925
  2. [3]Ahmed T, Devanbu P, Treude C, et al (2025) Can llms replace manual annotation of software engineering artifacts? URL https://arxiv.org/abs/2408.05534, arXiv:2408.05534
  3. [4]Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. Journal of Machine Learning Research 13:281–305
  4. [5]Bergstra J, Bardenet R, Bengio Y, et al (2011) Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems, pp 2546–2554, URL https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization
  5. [6]Chen J, Nair V, Krishna R, et al (2018) “sampling” as a baseline optimizer for search-based software engineering. IEEE Transactions on Software Engineering 45(6):597–614arxiv.org/abs/1608.07617
  6. [7]Chen J, Xu N, Chen P, et al (2021) Efficient compiler autotuning via bayesian optimization. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE, pp 1198–1209DOI
  7. [8]Chen P, Chen T (2026) Promisetune: Unveiling causally promising and explainable configuration tuning. In: Proc. of the 48th IEEE/ACM Int. Conf. Softw. Eng.DOI
  8. [9]Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114(3):494–509. https://doi.org/10.1037/0033-2909.114.3.494
  9. [10]Easterby-Smith M (1980) The design, analysis and interpretation of repertory grids. International Journal of Man-Machine Studies 13(1):3–24. URL https://doi.org/10.1016/S0020-7373(80)80032-0
  10. [11]Fu W, Menzies T, Shen X (2016) Tuning for software analytics: Is it really necessary? Information and Software Technology 76:135–146. https://doi.org/10.1016/j.infsof.2016.04.017
  11. [12]Ganguly KK, Menzies T (2026) How low can you go? the data-light se challenge. In: FSE’26arxiv.org/abs/2512.13524
  12. [13]Harman M, Mansouri SA, Zhang Y (2012) Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys 45(1):11:1–11:61. https://doi.org/10.1145/2379776.2379787
  13. [14]Herbold S, Trautsch A, Trautsch F, et al (2022) Problems with szz and features: An empirical study of the state of practice of defect prediction data collection. Empir Softw Eng 27(2):42. URL https://doi.org/10.1007/s10664-021-10092-4
  14. [15]Huang B, Wu X, Zhou Y, et al (2024) Exploring the true potential: Evaluating the black-box optimization capability of large language models. arXiv preprint arXiv:240406290arxiv.org/abs/2404.06290
  15. [16]Kamei Y, Shihab E, Adams B, et al (2013) A large-scale empirical study of just-in-time quality assurance. IEEE Trans Softw Eng 39(6):757–773. URL https://doi.org/10.1109/TSE.2012.70
  16. [17]Kang HJ, Aw KL, Lo D (2022) Detecting false alarms from automatic static analysis tools: How far are we? In: 44th Int. Conf. Softw. Eng., URL https://doi.org/10.1145/3510003.3510214
  17. [18]Khot T, Trivedi H, Finlayson M, et al (2023) Decomposed prompting: A modular approach for solving complex tasks. In: Proc. ICLR, arXiv:2210.02406, originally 2022arxiv.org/abs/2210.02406
  18. [19]Kington A (2009) Defining Teachers’ Classroom Relationships. URL https://scholar.google.com/scholar?q=Kington+Defining+Teachers+Classroom+Relationships
  19. [20]Lewis P, Perez E, Piktus A, et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems, pp 9459–9474, URL https://arxiv.org/abs/2005.11401
  20. [21]Li M, Chen T, Yao X (2022) How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance. IEEE Trans Softw Eng 48(5):1771–1799. https://doi.org/10.1109/TSE.2020.3036108, URL https://doi.org/10.1109/TSE.2020.3036108
  21. [22]Liu S, Chen C, Qu X, et al (2024a) Large language models as evolutionary optimizers. In: 2024 IEEE Congress on Evolutionary Computation (CEC), IEEE, pp 1–8arxiv.org/abs/2310.19046
  22. [23]Liu T, Astorga N, Seedat N, et al (2024b) Large language models to enhance bayesian optimization. The Twelfth International Conference on Learning Representationsarxiv.org/abs/2402.03921
  23. [24]Lustosa A, Menzies T (2024) Learning from very little data: On the value of landscape analysis for predicting software project health. ACM Transactions on Software Engineering and Methodology 33(3):1–22arxiv.org/abs/2301.06577
  24. [25]Madaan A, Tandon N, Gupta P, et al (2023) Self-refine: Iterative refinement with self-feedback. In: Proc. NeurIPS McGeoch CC (2012) A guide to experimental algorithmics. Cambridge University Press
  25. [26]Menzies T, Richardson J (2005) Xomo: Understanding development options for autonomy. In: COCOMO forum
  26. [27]Menzies T, Srinivasan S (2026) Can ai be easy? lessons learned from the ezr. py toolkit. arXiv preprint arXiv:260603640DOI
  27. [28]Menzies T, Black J, Fleming J, et al (1992) An expert system for raising pigs. In: The First Conference on Practical Applications of Prolog, URL https://scholar.google.com/scholar?q=Menzies+expert+system+for+raising+pigs
  28. [29]Menzies T, Chen T, Ye Y, et al (2026) Moot: a repository of many multi-objective optimization tasksarxiv.org/abs/2511.16882
  29. [30]Mittas N, Angelis L (2012) Ranking and clustering software cost estimation models through a multiple comparisons algorithm. IEEE Transactions on software engineering 39(4):537–551
  30. [31]Nair V, Menzies T, Siegmund N, et al (2017) Using bad learners to find good configurations. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp 257–267, https://doi.org/10.1145/3106237.3106238
  31. [32]Nair V, Yu Z, Menzies T, et al (2020) Finding faster configurations using flash. IEEE Transactions on Software Engineering 46(7):794–811. https://doi.org/10.1109/TSE.2018.2870895
  32. [33]Pfisterer F, Schneider L, Moosbauer J, et al (2022) YAHPO Gym: An efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In: Proceedings of the First International Conference on Automated Machine Learning, pp 3/1–39, URL https://proceedings.mlr.press/v188/pfisterer22a.htmlarxiv.org/abs/2109.03670
  33. [34]Phipson B, Smyth GK (2016) Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. arXiv preprint arXiv:160305766DOI
  34. [35]Port D, Olkov A, Menzies T (2008) Using simulation to investigate requirements prioritization strategies. In: 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, IEEE, pp 268–277
  35. [36]Romano J, Kromrey JD, Coraggio J, et al (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’sd for evaluating group differences on the nsse and other surveys. In: annual meeting of the Florida Association of Institutional Research
  36. [37]Sallou J, Durieux T, Panichella A (2024) Breaking the silence: the threats of using llms in software engineering. In: Proceedings of the 2024 ACM/IEEE 44th International conference on software engineering: new ideas and emerging results, pp 102–106DOI
  37. [38]Sayyad AS, Ingram J, Menzies T, et al (2013) Scalable product line configuration: A straw to break the camel’s back. In: 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 465–474, https://doi.org/10.1109/ASE.2013.66963104
  38. [39]Schreiber G, Akkermans H, Anjewierden A, et al (2000) Knowledge Engineering and Management: The CommonKADS Methodology. MIT Press
  39. [40]Scott AJ, Knott M (1974) A cluster analysis method for grouping means in the analysis of variance. Biometrics 30(3):507–512. https://doi.org/10.2307/2529204
  40. [41]Senthilkumar L, Menzies T (2024) Can large language models improve SE active learning via warm-starts? arXiv:2501.00125, https://doi.org/10.48550/arXiv.2501.00125
  41. [42]Senthilkumar L, Menzies T (2026) From brittle to robust: Improving llm annotations for se optimization. Empirical Software Engineering 31(6):155
  42. [43]Shepperd M, Song Q, Sun Z, et al (2013) Data quality: Some comments on the nasa software defect datasets. IEEE Trans Softw Eng 39(9):1208–1215. URL https://doi.org/10.1109/TSE.2013.11
  43. [44]Srinivasan S, Menzies T (2026) Beyond the prompt: Assessing domain knowledge strategies for high-dimensional LLM optimization in software engineering. Registered report, arXiv:2602.02752v1, https://doi.org/10.48550/arXiv.2602.02752
  44. [45]Takagi H (2001) Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proceedings of the IEEE 89(9):1275–1296. URL https://doi.org/10.1109/5.949485
  45. [46]Tantithamthavorn C, McIntosh S, Hassan AE, et al (2016) Automated parameter optimization of classification techniques for defect prediction models. In: Proc. of the 38th Int. Conf. Softw. Eng., pp 321–332DOI
  46. [47]Tibshirani RJ, Efron B (1993) An introduction to the bootstrap. Monographs on statistics and applied probability 57(1):1–436
  47. [48]Valerdi R (2010) Heuristics for systems engineering cost estimation. IEEE Systems Journal 5(1):91–98. URL https://doi.org/10.1109/JSYST.2010.2065131
  48. [49]Valov P, Petkovich JC, Guo J, et al (2017) Transferring performance prediction models across different hardware platforms. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, pp 39–50DOI
  49. [50]Van Aken D, Pavlo A, Gordon GJ, et al (2017) Automatic database management system tuning through large-scale machine learning. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp 1009–1024, https://doi.org/10.1145/3035918.3064029
  50. [51]Wu X, Zheng W, Xia X, et al (2021) Data quality matters: A case study on data label correctness for security bug report prediction. IEEE Trans Softw Eng 48(7):2541–2556. URL https://doi.org/10.1109/TSE.2021.3063727
  51. [52]Xu T, Jin L, Fan X, et al (2015) Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2015, p 307–319, https://doi.org/10.1145/2786805.2786852, URL https://doi.org/10.1145/2786805.2786852
  52. [53]Yin Z, Ma X, Zheng J, et al (2011) An empirical study on configuration errors in commercial and open source systems. In: Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, pp 159–172DOI
  53. [54]Yu Z, Fahid FM, Tu H, et al (2021) Identifying self-admitted technical debts with jitterbug: A two-step approach. IEEE Trans Softw Eng URL https://doi.org/10.1109/TSE.2020.3031401
  54. [55]Zela A, Siems J, Zimmer L, et al (2022) Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. arXiv:2008.09777arxiv.org/abs/2008.09777

Paper details

vlnc:W_zf3b623c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents