Stacking-Based Weighting for Large Language Bayes in M-Open Settings

Introduction

Probabilistic inference is a fundamental tool for reasoning under uncertainty and making decisions from data. In practice, however, its performance depends critically on the chosen model, and specifying an appropriate probabilistic model is often the hardest part of the inference pipeline. This difficulty is especially pronounced in realistic settings, where the true data-generating process is unknown and any candidate model class is necessarily incomplete.

Recent work on Large Language Bayes (LLB) Domke [2] aims to reduce the burden of model specification by using a language model to generate candidate probabilistic programs from a textual description of the problem. Given a textual description of the task and observed data, LLB samples a collection of candidate models, performs inference within each model, and combines the resulting posterior distributions over target variables using Bayesian model averaging. This yields a flexible framework for probabilistic inference without requiring the user to manually specify a single exact model.

However, LLB is subject to model misspecification, where the set of candidate models may not include the true data-generating process, a setting referred to as $M$-open Yao et al. [13]. In this regime, Bayesian model averaging can behave more like model selection than model combination, concentrating its mass on a small subset of candidate models. In the context of LLB, this can lead to a collapse of the ensemble onto a single misspecified model, even when multiple models provide comparable predictive performance.

In this paper, we replace evidence-based weighting with stacking Yao et al. [13]. Rather than interpreting model weights as posterior probabilities, stacking selects weights to maximize out-of-sample predictive performance. This makes stacking well suited to the $M$-open regime induced by language-generated model classes, where the goal is not to recover a true model but to construct a predictive ensemble that remains robust under misspecification.

Because LLB operates with latent-variable models, the leave-one-out predictive quantities required by stacking are generally intractable. We therefore derive a tractable approximation based on a variational lower bound, allowing the stacking objective to be implemented using the same inference machinery already used within LLB. The resulting method preserves the model-generation and within-model inference pipeline of LLB while replacing its aggregation rule with a predictive weighting scheme that avoids degenerate concentration on a single model.

In summary, we propose a stacking-based aggregation method for LLB that replaces Bayesian model averaging with a leave-one-out predictive objective, together with a variational approximation that makes this objective tractable in latent-variable settings, enabling more reliable aggregation under model uncertainty in language-generated model spaces.

Background

This work builds on two main components: Large Language Bayes and stacking-based predictive model averaging. We introduce only the minimal setup required to describe our method.

Large Language Bayes

Large Language Bayes (LLB) (Domke, [2]) is an automated approach for performing probabilistic inference over latent variables given data and a textual description of the problem. Given a prompt $t$ describing the problem and observed data $\boldsymbol{x} = (x_1, \ldots, x_n)$, LLB samples a collection of candidate models

Each model $m^{(k)}$ defines a joint distribution over latent variables $z$ and observations $x$. Inference is performed independently within each model to obtain posterior distributions $p(z \mid x, m^{(k)}, t)$.

The overall objective is to approximate the posterior over latent variables by marginalizing over models:

In practice, sampling from $p(m \mid x, t)$ is intractable. Instead, LLB samples models from $p(m \mid t)$ and assigns importance weights based on the model evidence. This yields the approximation

where $w_k^{\mathrm{BMA}} \propto p(x \mid m^{(k)})$ are the standard Bayesian model averaging weights proportional to the marginal likelihood of each candidate model.

This construction corresponds to Bayesian model averaging over the sampled candidate models and therefore inherits the usual closed-world interpretation that one of the candidate models is expected to be adequate.

The $M$-open setting

In many practical scenarios, including LLB, the true data-generating process is not contained within the set of candidate models. This is known as the $M$-open setting, where the objective is not to recover a true model, but to obtain accurate predictions ([1] (Le & Clarke 2017)).

In this regime, Bayesian model averaging can concentrate its weight on a small subset of candidate models, effectively behaving more like model selection than model combination ([13] (Le & Clarke 2017)).

Since LLB generates models using a language model, there is no reason to expect that the true model lies within the candidate set. Therefore, LLB naturally operates in an $M$-open regime, where likelihood-based weighting can lead to undesirable weight concentration on a single misspecified model.

Stacking and predictive model averaging

Stacking provides an alternative framework for combining models that is explicitly designed for predictive performance. Rather than interpreting weights as posterior probabilities of models, stacking selects weights to maximize out-of-sample predictive accuracy under a leave-one-out scheme, typically measured using a proper scoring rule such as the log score ([13, 10]).

This leads to weights that directly target predictive accuracy rather than model identification. This predictive perspective makes stacking a natural replacement for the weighting scheme used in LLB, aligning the aggregation of models with the goal of accurate prediction under model misspecification.

Related Work

Our work sits at the intersection of language-conditioned probabilistic modeling and predictive aggregation under model misspecification.

Recent work has explored the use of natural language as a specification medium for probabilistic inference and prediction. (Wong et al., [12]) translates text into probabilistic programs and performs Bayesian inference in the resulting model, while (Requeima et al., [8]) uses language to condition numerical predictive distributions directly. (Saad et al., [9]) and (Li et al., 2024) further develop the idea of generating probabilistic programs or statistical models automatically from data and natural language. Large Language Bayes (Domke, [2]) builds on this line by treating a text description as inducing a distribution over formal probabilistic models and then performing inference within each sampled model. In this paper, we take the model-generation step of LLB as given and focus instead on how the resulting collection of candidate models should be aggregated.

A separate line of work studies how to combine predictive models when the true data-generating process is not assumed to be in the candidate class. Bayesian model averaging is the classical framework for accounting for model uncertainty [4, 11], but in the $M$-open setting it can concentrate too strongly on a small subset of candidate models and fail to exploit complementary predictive structure. Stacking addresses this issue by selecting weights that maximize out-of-sample predictive performance rather than posterior model probability [13, 10]. Le and Clarke Le & Clarke (2017) provide a Bayesian decision-theoretic interpretation of stacking in both $M$-complete and $M$-open settings, showing that stacking weights can be viewed as asymptotically minimizing posterior expected loss. Related extensions such as Bayesian hierarchical stacking further generalize stacking to input-dependent, partially pooled model weights [14]. In practice, the required leave-one-out predictive quantities can be efficiently approximated using Pareto-smoothed importance sampling [10].

Our setting combines these two perspectives. LLB provides a way to generate a finite set of candidate probabilistic programs from language, but the induced model class should not be assumed to contain the true data-generating process. We therefore replace evidence-based aggregation with a stacking objective defined over leave-one-out predictive approximations. This yields a predictive combination rule that is aligned with the $M$-open nature of language-generated model classes, while preserving the latent-variable structure and within-model inference machinery of LLB.

Method

We retain the LLB pipeline for model generation and inference [2], but replace the original evidence-based weighting rule with a regularized stacking-based objective that selects model weights according to predictive performance [13, 10].

Figure 1 illustrates the LLB pipeline together with the proposed stacking-based aggregation.

Stacking-Based Weighting Objective

Given a prompt $t$, LLB samples a finite set of candidate models

For each candidate model $m^{(k)}$, and for each observation $x_i$, we consider the leave-one-out predictive density

where $x_{-i}$ denotes the dataset with the $i$-th observation removed.

We define weights $w=(w_1,\ldots,w_K)$ constrained to the probability simplex,

The corresponding stacked predictive distribution for the held-out observation $x_i$ is

Following the stacking framework, we choose the weights by maximizing the aggregate leave-one-out log predictive score. To preserve the inductive bias encoded by the original LLB weighting rule, we add a KL regularizer that anchors the learned weights near the evidence-based BMA weights while still allowing the stacking objective to redistribute mass when this improves predictive performance:

This objective directly targets predictive performance under an $M$-open setting, where the true data-generating process is not assumed to lie in the candidate model class.

However, the quantities $p(x_i \mid x_{-i},m^{(k)},t)$ are generally intractable in the LLB setting because of latent variables. We therefore approximate these terms using a variational bound. The key result is summarized in the following proposition, with proof deferred to Appendix § 8.§ 1.

This is the objective we optimize in practice. It preserves the structure of stacking while replacing each intractable leave-one-out predictive term with a variational approximation that can be computed using the same inference machinery already used within LLB.

Full Algorithm

We now summarize the complete inference and weighting pipeline combining LLB-style model generation, variational leave-one-out estimation, stacking-based weight optimization, and posterior aggregation.

Algorithm 1 LLB with KL-Regularized Stacking-Based ELBO Weighting

Require: Text description $t$, dataset $x = (x_1,\ldots,x_n)$, number of models $K$

1. Sample models: $m^{(k)} \sim p(m \mid t)$, $k = 1,\ldots,K$

2. for each model $k = 1,\ldots,K$ do

3.     for each data point $i = 1,\ldots,n$ do

4.     Form leave-one-out dataset $x_{-i}$

5.     Construct variational posterior: $q_{i,k}(z) \approx p(z \mid x_i,x_{-i},m^{(k)},t)$

6.     Compute ELBO-based surrogate predictive score:

7.     end for

8. end for

9. Solve stacking optimization:

10. Compute final posterior mixture:

11. return $\widetilde{p}(z\mid x,t),\widehat{w}$

Experiments

Experimental Setup

We evaluate the proposed KL-regularized ELBO-stacking weighting scheme against the original LLB aggregation rule on a suite of probabilistic inference tasks. In line with the LLB framework, each task is specified by a textual problem description $t$ together with observed data $x$, while held-out observations are reserved for predictive evaluation.

We consider three county-level expected annual loss prediction tasks covering different natural hazard regimes: wildfire, inland flooding, and hurricane, derived from the FEMA National Risk Index dataset [3]. For each task, we construct a training set of $n = 20$ observed county-level loss values (in millions of USD), and evaluate predictive performance on 10 held-out counties. Across all tasks, losses are strictly positive and exhibit heavy-tailed variability.

For each task, candidate probabilistic programs are generated from the textual description $t$ following the LLB pipeline. Figure 1 illustrates the full pipeline from language prompt to posterior aggregation.

Method Details

For each task, we generate 250 candidate NumPyro programs using Qwen2.5 Coder 32B. Programs are generated from the textual description $t$ alone, deduplicated by normalized code hash, and filtered to retain only those that compile and pass shape validation.

Each valid candidate model is fit using the No-U-Turn Sampler (NUTS) [5], a gradient-based Markov Chain Monte Carlo method, producing posterior samples for downstream evaluation.

To approximate the leave-one-out predictive densities required for stacking, we construct $x_{-i}$ for each observation and compute a variational lower bound (ELBO) on $\log p(x_i\mid x_{-i},m^{(k)},t)$ using a Gaussian variational family. These ELBO values provide a tractable surrogate for the leave-one-out predictive objective and form the basis of weight optimization.

We compare the original LLB evidence-based aggregation rule with the KL-regularized stacking objective introduced in § 4. The KL term stabilizes optimization by encouraging proximity to the BMA weights while allowing deviations when supported by predictive performance. We select $\lambda$ from a small grid using validation predictive performance on a held-out validation split. We optimize this objective in log-space using SciPy’s BFGS with softmax parameterization ($w = \operatorname{softmax}(\theta)$) and analytical gradients.

Evaluation Metrics

We evaluate both the distribution of learned model weights and predictive performance. To assess weight concentration, we report the effective sample size (ESS), defined as $1/\sum_k w_k^2$, where values close to 1 indicate collapse to a single model and larger values indicate more distributed weighting. We also report the top model weights to directly visualize how mass is allocated across models.

Predictive performance is measured using expected log predictive density (ELPD) on held-out test data [10], where higher values indicate better generalization. This allows us to verify that distributing weight across models does not degrade predictive performance.

Results

Figure 2 and Table 1 show that BMA concentrates most of its mass on a single model, whereas stacking spreads weight across multiple candidate models.

Dataset	BMA top weights	Stacking top weights
Hurricane	$w_{139} = 1.000$	$w_1 = 0.759$<br>$w_{65} = 0.238$<br>$w_{129} = 0.003$
Inland Flood	$w_{24} = 1.000$	$w_{24} = 0.588$<br>$w_{13} = 0.412$
Wildfire	$w_{83} = 0.998$	$w_{138} = 0.666$<br>$w_{83} = 0.334$

This behavior is consistent across all three hazard regimes. In the BMA case, the posterior over models effectively collapses to a single candidate, indicating that evidence-based weighting behaves like model selection in practice. In contrast, stacking assigns non-trivial weight to multiple models, reflecting residual uncertainty over which misspecified model best approximates the data-generating process in the $M$-open setting.

Table 2 reports held-out predictive performance. Across all tasks, stacking matches Bayesian model averaging in terms of ELPD, indicating that redistributing weight across multiple models does not degrade predictive accuracy. This supports the interpretation that the primary difference between the two approaches is not predictive power, but rather the representation of model uncertainty.

Dataset	LLB (BMA)	Stacking
Hurricane	-5.983	-5.999
Inland Flood	-370.340	-370.340
Wildfire	-177.922	-177.897

Discussion and Limitations

Our results suggest that replacing evidence-based aggregation in LLB with a stacking-based objective yields weight distributions that are more consistent with the $M$-open setting. In particular, stacking produces less concentrated weights than Bayesian model averaging while maintaining comparable predictive performance. This supports our main claim: in language-generated model spaces, the goal is not to identify a single dominant model, but to maintain a meaningful mixture over plausible misspecified candidates. At the same time, our experiments do not establish that the learned weights recover the true epistemic uncertainty of the data-generating process. The resulting mixture still depends on the quality and diversity of candidate models generated by the language model. Future work includes evaluating the approach on more diverse problems and exploring alternative regularization strategies and input-dependent stacking schemes.

Conclusion

We proposed a stacking-based aggregation method for Large Language Bayes that replaces evidence-based weighting with a predictive objective while preserving the original language-model-driven model generation and within-model inference pipeline. Across our experiments, the method maintains comparable predictive performance to the LLB baseline while yielding a more distributed allocation of weights across candidate models. These results support the view that stacking is a natural fit for LLB-style inference in the $M$-open setting, where the goal is robust prediction and a more faithful representation of model uncertainty under misspecification.

Appendix

Proof of Proposition 4.1

Proof. Fix a candidate model $m^{(k)}$, an observation $x_i$, the remaining data $x_{-i}$, and the prompt $t$. Let $z$ denote the latent variables in the model, and let $q_{i,k}(z)$ be any variational distribution over $z$.

We begin with the leave-one-out predictive density:

We consider the Kullback–Leibler divergence from $q_{i,k}(z)$ to the leave-one-out posterior:

By Bayes’ rule,

Substituting (3) into (2) gives

Since $p(x_i \mid x_{-i},m^{(k)},t)$ does not depend on $z$, we may separate it from the integral:

Rearranging terms yields

We define the evidence lower bound:

Therefore,

Because the Kullback–Leibler divergence is non-negative,

it follows that

Applying the exponential function, which is monotone increasing, preserves the inequality:

Define the surrogate predictive density

Then (11) implies

Substituting this variational bound into the stacking objective gives the surrogate objective

This proves Proposition 4.1.

ELBO Convergence Diagnostics

To verify the quality of the variational approximations used in our stacking objective, we track the convergence of the ELBO values during optimization for each leave-one-out inference problem. Figure 3, Figure 4, and Figure 5 show ELBO trajectories for a sample of 6 data points from each of the three tasks.

We construct a variational approximation $q_{i,k}(z)$ and optimize the corresponding ELBO objective using gradient-based variational inference. The red solid lines show the ELBO values at each iteration, while the green dashed lines indicate the final converged values used in the stacking objective.

Across all three tasks, the ELBO values converge within 25 iterations, typically stabilizing after 10–15 iterations. The gray points in the background represent the raw ELBO evaluations during optimization. The convergence patterns suggest that the variational approximation provides stable and reliable surrogates for the intractable leave-one-out predictive densities required by the stacking objective.

Code

The link to the code is: Code