WorksComputer Science
Revealing the Biases of Large Language Models By Inferring Prior Distributions
Abstract
Large Language Models (LLMs) have been known to exhibit a wide range of social biases including name-nationality associations, occupation-gender associations and racial stereotypes. Existing approaches to evaluating these biases typically involve measuring surface-level behavior where a model is fed a dataset of prompts and its outputs are scored relative to ground-truth answers. Inspired by prior elicitation in cognitive science, we propose a different perspective that reconstructs latent bias as a prior underlying the model's behavior. We formulate the problem as a Bayesian inverse problem where, given prompt-response observations on a bias benchmark dataset, we infer a low-dimensional latent distribution that summarizes the model's stereotyping tendencies. This paper develops the mathematical foundation, a practical reconstruction algorithm, and experiments using the Bias Benchmark for QA (BBQ) benchmark to infer the prior distributions of LLMs over responses in ambiguous social contexts.
Introduction
LLMs often produce outputs that reflect potentially harmful stereotypes and biases present in training data [21, 12, 1, 20]. To evaluate social bias in NLP systems, prior work has developed association tests and benchmark datasets designed to elicit stereotyping [27, 26, 29]. While these benchmarks provide valuable measurements, they typically summarize behavior through scores computed from generated text or from the observable token probability assignments. Thus, they measure the extent to which a model exhibits certain biases, but fail to explain why the preferences arise.
In this paper, we propose a different perspective. Rather than treating bias benchmarks only as evaluation metrics, we take a Bayesian approach and use them to probe language models in a way that lets us reconstruct a latent space that describes the models’ prior beliefs. Concretely, we tackle the question: Can we reconstruct a latent prior distribution over possible biased responses that explains an LLM’s stereotypical preferences?
A Bayesian framing for this problem is natural because LLMs are inherently probabilistic systems that predict a probability distribution over tokens conditioned on previous ones [2, 30, 32]. The motivation for this approach comes from prior elicitation methods in cognitive science, where behavioral judgments have been used to infer the prior distributions that best explain human predictions [13, 28]. In this line of work, observed judgments can be used to diagnose which priors and prior parameters people use in certain task domains [16]. Related ideas have also been extended to LLMs where prompt-based probing has been used to elicit prior distributions from language models in specific domains (Zhu & Griffiths, 2024).
Building on these ideas, we introduce a Bayesian surrogate model in which an LLM behaves as if it first samples a discrete latent “bias state” and then generates outputs according to a state-conditional response model. The central unknown is the prior distribution that summarizes how the model allocates probability across bias hypotheses in social contexts. We interpret this distribution as summary of the systematic preference structure in an LLM. It is important to note that we are not assuming that LLMs are inherently Bayesian in that they are using Bayesian inference to construct their outputs. Instead, we use a Bayesian surrogate model to analyze and explain the latent prior beliefs that are induced by the model’s parameters.
Mathematically, the Bayesian inverse problem framework provides a principled theoretical background for latent prior reconstruction [8]. Given observations derived from token log-probabilities on candidate prompt completions, we define a likelihood function for the model and obtain a posterior distribution that approximates the desired latent prior. The main contributions of this paper are:
A mathematical formulation for latent prior reconstruction as a Bayesian inverse problem.
A practical reconstruction algorithm based on token log-probabilities using Bayesian inference.
An empirical demonstration using the Bias Benchmark for QA (BBQ) dataset [29].
Related Work
Quantifying Social Bias in Language Models. As language models become more powerful and increasingly widely used, AI ethics and safety have become central concerns [22, 11]. As a result, the literature on social biases in these models proposes a broad range of evaluation protocols analyzing model behavior on different levels from embeddings to probabilities and generated text [12]. Early work demonstrated that word embeddings encode implicit social associations analogous to those revealed by human implicit association tests and introduced methods for measuring such effects [4]. Subsequent work extended association testing to sentences, allowing LLM biases to be probed in a way closer to implicit association tests in psychology [7]. Another family of benchmarks measures bias directly from the model outputs, often through controlled choice tasks in which models select among a fixed set of options and are compared against an unbiased ground truth [29]. Other benchmark tests take this a step further and compare model probability assignments across minimally different texts or token options with different levels of stereotyping [27, 26].
Prior Elicitation in Cognitive Science and LLMs. In Bayesian models of cognition, priors are treated as structured assumptions that shape inductive judgments (Griffiths et al., 2008; Tenenbaum, 1998). Thus, reconstructing or eliciting a cognitive model’s prior can provide insight into the implicit beliefs of a system. The central idea in prior elicitation is that observed predictions and choices can be used to infer or at least validate the priors that best explain behavior in a certain task domain [17, 18]. Work on everyday judgments has shown that human predictions can often be well modeled as Bayesian updates under latent priors, helping to explain why predictions take the forms that they do [18]. More recently, these ideas have been extended to prior elicitation in LLMs through prompt-based iterated learning procedures that resemble Monte Carlo sampling schemes (Zhu & Griffiths, 2024). These experiments reflect those done on human judgment and provide evidence that a Bayesian framework can be useful for characterizing LLM predictions. We extend this line of work to analyze stereotype bias in LLMs. However, our approach differs in that we treat benchmark-derived log-probability observations as data from which we reconstruct a latent distribution via Bayesian inversion rather than through sampling and iterated interaction.
Bayesian Computation and Uncertainty Quantification. Once the problem of prior elicitation is posed as finding a posterior distribution from a set of observations, posterior computation and uncertainty quantification can be carried out using Bayesian inference through optimization-based approximations, sampling methods, or variational methods (Tipping, 2003; Goan & Fookes, 2020). The common optimization-based approach that will be used in this paper is to use a maximum a posteriori (MAP) estimate to provide a point summary [25] and then apply Laplace approximations by using local curvature (via the Hessian) around the MAP solution to approximate posterior uncertainty (Tierney & Kadane, 1986). On the other hand, sampling-based approaches like Markov Chain Monte Carlo (MCMC) and Hamiltonian Monte Carlo (HMC) can provide asymptotically exact posterior approximations under standard conditions [19, 10], but are often computationally expensive and reserved for settings requiring high precision. [31]. Variational inference offers a scalable optimization-based alternative by approximating the posterior within a tractable distribution family [3], and may be useful for future higher-dimensional prior elicitation. However, for our use case, Laplace approximation provides a simpler and more interpretable Hessian-based uncertainty estimate.
Bayesian Inverse Problem Framework
In this section we provide a high-level overview1 of the Bayesian inverse problem [8] (Stuart 2010). The goal of an inverse problem is to infer some unknown parameter from observations generated by a noisy forward model
where is a forward operator from the space of unknowns to the space of possible observations and represents observation noise.
In the Bayesian framework, is treated as a random variable so that we can characterize its posterior distribution given observations (Stuart, 2010). Specifically, a prior distribution is assigned to with density and Bayesian inference is used to combine this prior with a likelihood describing the probability of observing given . Assuming the noise has density , the likelihood is given by . Bayesian inference then gives a posterior distribution of the form
where the proportionality constant depends only on the observations . It is often convenient to express the likelihood in terms of a potential function . In this setting, this function is also often referred to as the negative log-likelihood. Rewriting (2) using this potential gives us
In this form, we can see that the negative log-likelihood encodes the agreement between the observations and those predicted by while the prior encodes our beliefs about plausible values of the unknown parameter. Together, this provides a complete characterization of the posterior distribution over , which serves as a solution to the inverse problem.
Our Formulation
We now interpret the abstract Bayesian inverse problem framework discussed above in the context of the problem of reconstructing an LLM’s latent bias prior from token-level probability observations on a controlled bias task. The purpose of this section is to identify, in the notation of Section 3, the unknown variable , the data space , the forward model , and the likelihood . In doing so, we obtain a concrete finite-dimensional Bayesian inverse problem whose solution is a posterior distribution over a latent probability vector summarizing the model’s systematic bias tendencies.
Unknown Variable and Observables
To begin, let us fix a bias domain in which we have mutually exclusive response categories. For instance, for a task related to gender biases, we may take , . We represent the latent bias prior as a probability vector where is the set of possible probability vectors in .
For computational convenience, we parameterize via log-ratios relative to a reference category , defining and , . For a given in our unknown parameter space , the corresponding probability vector can then be recovered via the inverse transformation
Thus, the unknown variable in our inverse problem is the vector which we interpret as an encoding of the latent bias prior over the candidate hypotheses in log-ratios. In the running example, if we transform via (4), then , , and represent the model’s latent prior mass on the hypotheses Male, Female, and Unknown, respectively. It is important to stress that this latent variable is not meant to represent an explicit internal object stored by the LLM but rather it is a low-dimensional surrogate parameter whose posterior distribution summarizes the systematic preference structure reflected in the model’s behavior.
To define the observational data and data space , let be prompts with the same shared answer set of candidate answers. For each , the model outputs log-probabilities across the possible answers. Passing to the same log-ratio representation, we define , . Thus, our full observable data take the form . This choice of representation in log-ratios removes the redundancy inherent in representing a probability by all coordinates simultaneously since the probabilities sum to one and thus only contain degrees of freedom. Furthermore, it allows us to work in the entirety of instead of the constrained simplex. In the next section, we will also see how this yields a particularly simple form for the forward operator.
Prompt-Dependent Evidence and Forward Operator
The key modeling issue is how the prompts should enter the inverse problem as they inherently affect the likelihood in the formulation. To tackle this, we model each prompt as contributing a prompt-specific evidence term while preserving a common latent prior across prompts.
Formally, let be a known map assigning each prompt to an evidence vector . We interpret as encoding the prompt-dependent evidence supplied by in favor of each candidate category. Thus represents a global latent bias prior shared across all prompts while captures the local way in which the prompt shifts or reweighs that prior. Passing again to log-ratios, we define the offset , , . This transformation ensures that only relative differences in the evidence affect the model and allows us to be consistent with the log-ratio representation.
We can now define the prompt-dependent forward operator directly in the log-ratio space. For each prompt , define by . Collecting these prompt-based maps together, we get the full forward operator as where .
We can now place the model into the exact form of (1) discussed in Section 3. We assume the observed log-ratios satisfy , where is some noise accounting for residual variability in the log-probability outputs of the LLM beyond what is explained by the prompt dependence and the forward operator . This is now precisely the finite-dimensional noisy inverse problem previously formalized in Section 3.
Prior, Likelihood, and Posterior
To obtain a tractable formulation, we specify the natural isotropic Gaussian prior measure on so that and assume independent isotropic Gaussian noise for all where is a precision parameter and is the variance. Let denote the corresponding prior density. Note that this prior induces a logistic-normal distribution on the implied probability vector . Equivalently, on the full data space , on . We then obtain from (2) that the posterior density satisfies
Substituting in for , we find that the negative log-likelihood satisfies the potential formulation from (3) where
In summary, we have now posed the bias prior elicitation problem as a Bayesian inverse problem. The unknown variable is the log-ratio vector and the observables are the log-ratios extracted from the token log-probabilities generated by the LLM. Furthermore, the forward operator is the map based on the evidence vectors and the likelihood is obtained from the noise model discussed in § 3.
Reconstruction Algorithm
We now face the computational problem of reconstructing the posterior distribution over from the observed data. Our approach consists of two stages: (1) computing a MAP estimate using gradient-based optimization, and (2) approximating the posterior distribution via a Laplace approximation around the MAP point.
Gradient-Based Optimization
Taking the negative log-likelihood from Equation (5) and combining with the Gaussian prior , our negative log-posterior, up to an additive constant, is
Thus, our MAP estimator is . The objective is smooth and convex, and can thus be optimized efficiently using first-order methods by computing the gradient [6]. In practice, we implement as a trainable parameter in a PyTorch model, and define a loss function corresponding to . The optimization proceeds over minibatches of data, where each batch contributes a stochastic estimate of the gradient. PyTorch’s automatic differentiation computes gradients via backpropagation, and the optimizer updates iteratively. The details of this procedure can be seen in Algorithm 1.
Algorithm 1: MAP Reconstruction via Gradient-Based Optimization
Input: Observed log-ratios , prompt offsets , noise level , prior precision
Output: MAP estimate
Initialize
for each epoch do
for each minibatch do
Compute predictions for
Compute loss
Update using Adam or SGD
end for
end for
return final value of as the MAP estimate
Laplace Approximation for Uncertainty Quantification
To capture uncertainty of the MAP estimate, we use a Laplace approximation centered at . Let denote the Hessian of the objective at the MAP solution. Then . We interpret this uncertainty on the simplex by transforming samples from this Gaussian approximation through the inverse log-ratio map in Equation (4).
In practice, after Algorithm 1 converges, we fit the local curvature of the loss using the training data. We implement this efficiently with laplace-torch [9], which computes an approximate Gaussian posterior over the unknown parameter . We then sample from this posterior to compute uncertainty estimates as summarized in Algorithm 2.
Algorithm 2: Laplace Approximation with laplace-torch
Input: Trained model with parameter , training data
Output: Approximate posterior over
Initialize Laplace approximation object
Fit curvature using training data
Obtain Gaussian approximation
Sample and compute
return posterior samples and uncertainty estimates
Experiments on Shift Ciphers
Before applying our method to large-scale bias benchmark datasets, we begin with a controlled experiment designed to validate whether our framework can recover a meaningful latent prior from LLM outputs. The goal of this experiment is to isolate a setting in which prompt-dependent effects are minimized so that we can test the basic structure of our algorithm.
Recent work has explored the presence of cognitive biases in language models across a variety of tasks including reasoning under uncertainty and structured problem-solving settings such as decoding shift ciphers [23 b]. Motivated by these observations, we construct a simple shift-cipher task in which the model is asked to infer the shift used to produce a given ciphertext. By carefully designing the inputs, we obtain a setting in which there is no meaningful evidence favoring a particular answer, allowing us to probe the model’s baseline preference over shifts. The code for our experiments is publicly available at https://github.com/duckpy/llm-prior-inference.git.
Problem Setup
We consider a simple shift cipher in which each plaintext letter is shifted by positions in the alphabet, wrapping from to . Rather than providing meaningful encrypted text, we generate prompts consisting of random lowercase strings. The candidate answers are the possible shifts , so this corresponds to the setting of § 4 with .
Since the strings are random, they do not encode any particular plaintext, so all shifts are equally plausible. In the notation of § 4, this corresponds to the special case in which the prompt-dependent evidence map is constant across all categories. This setting thus isolates the latent parameter as the sole systematic component of the model’s outputs making it an ideal test case for evaluating whether our method can recover a stable prior distribution.
For our experiments, we use the Google Gemma 3 (Team et al. (2025)) and Meta Llama 3.2 (Graffitiori et al. (2024)) models as our lightweight LLM models. A detailed review of the training procedure and hyperparameters as well as the training loss curves can be seen in Appendix § 13.
Results
We first validate our algorithm on a synthetic control dataset where we inject a known prior distribution into the ciphertext observation model while removing the LLM models. This experiment serves to verify that the algorithm behaves as intended under a controlled setting. Further details on this validation experiment can be found in Appendix § 12.
For the ciphertext experiments involving the lightweight LLMs, across all three models, the recovered prior distributions exhibit clear non-uniform structure despite the absence of prompt-dependent signal. Furthermore, the credible intervals are relatively tight, suggesting that the inferred priors are well identified. Notably, each model exhibits a distinct latent prior structure as shown in Figure 1.
Figure 1. Recovered prior distributions over shifts for Gemma 3 1B IT, Gemma 3 4B IT, and Llama 3.2 3B IT. Error bars denote 95% credible intervals obtained via Laplace approximation. Even in the absence of input signals, all models exhibit non-uniform prior distributions, with notable differences across architectures and scales.
In particular, the Gemma models display a sharper concentration on a small number of shifts, particularly at low rotations. This effect is especially pronounced for Gemma 3 4B IT, whose recovered prior has a dominant mode around a single shift. In this case, the bias might reflect the fact that the traditional Caesar cipher is performed with a shift of 3. In contrast, Llama 3.2 3B IT produces a more diffuse prior with greater probability mass on higher rotations than the Gemma models.
Interpretation
This experiment validates our framework in a setting where the prompt-dependent evidence is deliberately removed. The non-uniform priors suggest that LLMs can encode systematic preferences over answer categories independent of the input prompt. This aligns with prior observations of task-based cognitive biases in LLMs from McCoy et al. [23], but here we provide a quantitative framework for explicitly recovering and characterizing the latent prior beliefs.
While simplified, this experiment shows that we can successfully extract latent bias structure from model outputs. In more realistic settings where varies, this framework can help disentangle prompt-driven effects from latent bias, providing a principled tool for analyzing LLM behavior.
Experiments on BBQ
Having validated the reconstruction framework on a controlled task, we next apply it to a real bias benchmark setting. We use the Bias Benchmark for Question Answering (BBQ) dataset introduced by Parrish et al. [29]. BBQ is a multiple choice question answering benchmark for evaluating social bias in language models. BBQ is organized around nine social dimensions, including age, nationality, socioeconomic status, and sexual orientation. It also distinguishes between ambiguous contexts where the question lacks sufficient information to determine the answer, and disambiguated contexts where additional information resolves the ambiguity. We focus exclusively on the ambiguous context portion of BBQ. This subset plays the same conceptual role as the random string shift cipher experiment as it suppresses prompt-specific evidence which simplifies our model.
Problem Setup
The raw BBQ answer choices vary across examples, even within the same social dimension. To place all examples in a common latent prior framework, we map each item’s three answer choices into a reduced semantic answer space with three categories: , where S denotes the answer choice consistent with the intended Stereotyping, A denotes the answer choice that goes Against the intended stereotyping, and U denotes the answer choice corresponding to an Unknown answer.
Because social dimensions differ substantially in context, we define a separate latent prior for each BBQ dimension. Only six dimensions (age, disability status, gender identity, race / ethnicity, religion, sexual orientation) admit direct mappings to the reduced semantic answer space. We therefore restrict our analysis to these dimensions. For each dimension , we solve the inverse problem for the latent parameter from § 4 with . A detailed review of the training procedure and hyperparameters as well as the training loss curves can be seen in Appendix § 13.
Results
Figure 2 shows that the credible intervals for the recovered priors are relatively tight across all models and social dimensions, indicating that the inferred priors are well identified. This suggests that despite the prompt and answer choice variations that exist within each social dimension, the reduction to the answer space still produces stable and interpretable results.

Figure 2. Recovered latent priors over the reduced answer space for each BBQ social dimension, with 95% credible intervals derived from the Laplace approximation. Each model exhibited largely different latent priors across the different social dimensions.
We next compare the recovered latent priors with the standard BBQ bias score metric. In the BBQ framework, positive scores indicate preference for stereotype-consistent answers S, negative scores indicate preference for stereotype-inconsistent answers A, and values near zero indicate no clear systematic bias [29]. However, this scalar score combines two distinct behaviors: whether the model abstains under ambiguity, captured by probability mass on U, and whether the non-abstaining probability mass is directionally skewed towards S or A. Our reconstruction separates these effects by recovering the full latent distribution.
This distinction is visible in the Age social dimension. Figure 3, which visualizes the BBQ bias scores for each model, shows that both Gemma 3 4B IT and Llama 3.2 3B IT have large positive BBQ scores which indicate stereotype-consistent behavior. However, Figure 2 shows that the underlying priors differ substantially. While both models have low probability mass in U contributing to high bias scores, Gemma 3 4B IT assigns significantly more probability mass to S relative to A indicating a stronger directional bias not fully captured by the aggregate score metric.
Figure 3. BBQ bias scores for each model and social dimension. Positive values indicate bias towards the stereotype-consistent answer S, negative values indicate bias towards the stereotype-inconsistent answer A, and values near zero indicate no systematic bias in either direction.
A similar ambiguity appears when comparing the Disability Status and Religion dimensions for Gemma 3 4B IT. The BBQ scores for these dimensions are both positive and similar in magnitude but the recovered priors indicate a different mechanism of bias. In Disability Status, the model assigns low probability to U and substantial probability mass to both S and A which suggests overconfidence in the ambiguous setting. On the other hand, for the Religion dimension, the model assigns a majority of its probability mass to U, and the positive BBQ score arises primarily because the remaining probability mass is more skewed towards S than towards A.
The recovered priors also expose cases where near-balanced or moderate BBQ scores can hide low uncertainty. For example, we note that Gemma 3 1B IT often assigns comparable probability mass to S and A while failing to place a majority on U. Such behavior is not strongly directionally biased, yet is still undesirable because the model is making unsupported social inferences. These examples illustrate that similar BBQ benchmark scores can arise from fundamentally different underlying behaviors. Our framework resolves this ambiguity by recovering the full latent distribution over the answer choices .
Interpretation
The BBQ ambiguous context experiment gives us a concrete demonstration of how our method reveals the latent structure in model behavior that is not captured by the BBQ benchmark scores. Although benchmarks such as BBQ provide useful measures of accuracy and directional bias, they do not reveal the underlying mechanisms responsible for biased results. Our experiment shows that there are many components of bias corresponding to different behaviors in models such as directional bias and overconfidence that cannot be articulated by a single scalar score value.
More broadly, the experiment demonstrates that latent prior elicitation provides a more principled way to analyze biased model behavior. By constructing a reduced answer space to represent prompt variation, our approach was able to reconstruct a stable prior structure that represents the underlying preferences and tendencies in each model.
Limitations and Future Directions
Our approach, as it stands, still has several limitations. First, it relies on mapping raw answers to a reduced answer space, such as the representation used in our BBQ experiments. While this abstraction worked for the cases we analyzed, such a clean reduction may not be possible in general and may obscure nuances and finer-grained distinctions important for interpretation. Second, our experiments focused on settings where the prompt-dependent evidence term is constant, yielding a simplified forward model. This assumption is appropriate for some contexts, but does not capture the general setting in which prompts carry meaningful information.
These limitations suggest several directions for future work. One direction is to develop richer latent representations beyond the low-dimensional reduced answer spaces used here, allowing the framework to capture more nuanced model behavior. Another direction is to incorporate prompt-dependent evidence through , potentially learning from prompt embeddings or another representation.
More broadly, latent prior elicitation is not limited to bias analysis. The same framework could be applied to other domains where LLMs display systematic behavior under uncertainty including reasoning tasks or algorithmic tasks such as the shift cipher problem considered in this paper. Exploring these broader applications may reveal additional implicit structures encoded within LLMs.
Taken together, our results suggest that studying the latent distributions governing LLM behavior can provide valuable insight into the reasons why models behave the way they do. The Bayesian inverse problem formulation therefore offers a promising framework for uncovering these latent mechanisms underlying language model predictions.
Conclusion
In this work, we introduced a framework for eliciting latent priors from large language models by formulating the problem as a Bayesian inverse problem. Across both controlled synthetic experiments and real benchmark evaluations, we demonstrated that it is possible to recover stable and interpretable prior structures from model outputs. Furthermore, we demonstrated using the well-established BBQ bias benchmark that prior elicitation under this framework provides us with much more nuance in explaining model behavior that was not previously characterized through these benchmark datasets. While benchmarks like BBQ do provide useful measures of accuracy and directional bias, they do not capture the underlying reasons for the biased results. Our latent prior formulation provides an extra layer of explainability by providing the latent distribution which gives rise to these bias score results.
AI Use Statement
In this work, we used generative AI tools to edit the manuscript to improve grammar, readability, and conciseness. All AI-assisted edits were reviewed by the authors and the authors take full responsibility for the final content of this work.
Ethics Statement
We evaluate social bias in large language models using controlled experimental settings. In particular, model behavior in our experiments should not be interpreted as definitive statements about the broader societal impact of a model.
Reproducibility Statement
For reproducibility, the source code for the project is included in the supplementary materials. Furthermore, specific hyperparameters used for the algorithms outlined in Section 5 can be found in Section C.1.1. The specific experimental setups for each experiment can be found in Sections B.1, 6.1, and 7.1.
Acknowledgments
This work was supported by a gift from Google to Princeton as well as by funding from the Gemini Academic Program.
References
Measure-Theoretic Background
This section collects the measure-theoretic background needed for a rigorous statement of the inverse problem described in § 3. We will define some required measure-theoretic objects and present some results relevant to the Bayesian formulation of the inverse problem. The structure of this section parallels that of Stuart (2010); however, the derivations and proofs are our own.
Basic Preliminaries
We begin by reviewing the basic objects of study that we will be using throughout this paper.
Definition A.1. A measurable space is a pair , where is the sample space and is the -algebra of events on . A measure space is a triple , where is the measure. If , then is called a probability measure, and is called a probability space.
Definition A.2. A Banach space is a complete normed vector space . If is a topological space, then denotes the Borel -algebra generated by the open subsets of .
In our formulation of the Bayesian inverse problem, we will mainly be working under the assumption that in any measurable space is a separable Banach space and is the Borel -algebra . Now we introduce a basic measure-theoretic relation between measures.
Definition A.3. Let be a measurable space, and let be measures on it. We say that is absolutely continuous with respect to , and write , if
If both and , then and are called equivalent.
Additionally, it will be important to formally establish the idea of conditional distributions as our overarching goal is to reconstruct the latent bias priors conditioned on the dataset prompts and LLM output probabilities.
Definition A.4. Let and be measurable spaces, and let be a probability measure on . Denote the second marginal of by
A family of probability measures on is called a regular conditional distribution of given under if:
for every , the map is -measurable;
for every and ,
We note that when and are, as we assume, separable Banach spaces equipped with their Borel -algebras, regular conditional distributions exist and thus for the following subsections it is enough to assume existence when needed (Bovier [5]).
Finite-Dimensional Bayes’ Formula
With the basic definitions out of the way, the main goal is to rigorously derive Bayes’ formula in a way that is applicable to our prior reconstruction problem.
We begin by stating the following fundamental result from measure theory.
Theorem A.5 (Radon–Nikodym Theorem). Let and be measures on the same measurable space . If and is -finite then there exists a -measurable function such that
This function is unique -almost everywhere and is known as the Radon–Nikodym derivative of with respect to . The derivative is written as
This Radon–Nikodym derivative plays an important role in our derivation as it allows us to compare measures and express one in terms of another. It also helps us generalize the idea of a probability density which is essential for our formal derivation of the required tools to discuss the Bayesian inverse problem.
We continue with the derivation of the relevant finite-dimensional Bayes’ formula with the following theorem which utilizes the Radon–Nikodym derivative.
Theorem A.6 (Conditional Change of Measure). Let and be probability measures on , where and are measurable spaces. Let denote a generic point. Assume that , and that has a Radon–Nikodym derivative with respect to . Assume further that the conditional distribution of given under exists which we denote by . Then the conditional distribution of under , denoted , exists. Moreover,
and the Radon–Nikodym derivative is given by
where
Proof. Let and be the -marginals of and respectively. Since
and since decomposes as
we have, for measurable and ,
If we define
then, taking , we get
Thus, . We can then define for -a.e. ,
Notice, the set is -null since
Thus, may be defined arbitrarily on this set. Let us take for these values . Then is a probability measure for -a.e. and moreover
Therefore is the conditional distribution of given under . Finally, for -a.e. with ,
so and
For , since we took for this set, we obtain a Radon-Nikodym derivative of , which proves the claim.
As we will see, this conditional change of measure theorem is precisely what we need to formalize the finite-dimensional Bayes’ formula in rigorous measure-theoretic language.
Corollary A.7 (Finite-Dimensional Bayes’ Formula). Let and be random variables defined on a common probability space. Assume that the prior law of admits a Lebesgue density on , and that the conditional law of given admits a Lebesgue density on . Assume also that
Then the conditional law of given admits a density with respect to Lebesgue measure on , and
Proof. Let us define two measures on as follows. First, let
This is the product of the prior law of with the Lebesgue measure on . Second, let denote the joint law of . By definition of conditional density, has density
with respect to the Lebesgue measure , and thus
Therefore and the Radon-Nikodym derivative is clearly
Notice, under , the conditional law of given is simply the prior law of because is the product measure. Thus
Let us now apply Theorem A.6 to the pair of measures and on . Notice, while is not a probability measure, it is -finite, and the theorem extends to this setting. In particular, (7) then implies that the conditional law of given under satisfies
for . Substituting in , we obtain
Since has density with respect to the Lebesgue measure, it follows that has density
Thus, we get precisely
This gives a very powerful expression for the probability of a random variable given an observation of , which requires only the knowledge of the prior probability density and the conditional probability density . Both of these are readily available in most modeling scenarios as will be the case in our application.
Controlled Validation on Synthetic Data
Before applying this algorithm to LLMs, we provide a synthetic experiment with a known ground-truth prior to validate our method’s ability to reconstruct the latent parameter that it was designed to infer. We structure this experiment with the general context of ciphertext rotations which is the setting of the LLM-based experiment in § 6.
Experimental Setup
We begin with a known structured ground-truth prior and set varying prompt evidence for each synthetic data point. For our prior, we use a softmax sum of sines and cosines for a multimodal ground truth. Furthermore, each data point is set to have prompt evidence arbitrarily favoring shifts of 12 or 18. This could occur, for example, if the data in the probing dataset has a few particularly common default categories.
Since we want to control the output observations that we feed our model, in this experiment, observations are synthetic logits generated from the known prior, the varying prompt-evidence, and controlled Gaussian noise. We compare our method to the naive approach of taking the empirical mean of the outputs plus bootstrapped confidence intervals to see how our method converges to the true prior distribution more effectively than a basic average over the posterior distribution.
Reconstruction Accuracy
Figure 4 demonstrates that the distribution produced by our reconstruction algorithm closely matches the known prior distribution. In particular, the 95% credible intervals obtained from the Laplace-based uncertainty quantification contain the known prior.

Figure 4. The recovered prior using our reconstruction algorithm much more closely resembles the control prior distribution. Meanwhile, the naive empirical mean method is biased by the varying prompt evidence.
We also see clearly the advantage of our reconstruction algorithm over the naive empirical mean over the posterior. The empirical mean is influenced by varying prompt-specific evidence and incorrectly allocates probability mass away from the true known prior.
Overall, this control experiment provides evidence that our reconstruction algorithm is able to reproduce a distribution that closely matches a known set prior.
Experimental Details and Training Dynamics
In this section, we provide experimental details and training dynamics for the experiments conducted on shift ciphers (§ 6) and BBQ (§ 7). The code to replicate the experiments can be found in the supplementary materials. Furthermore, all experiments were run on MIG-partitioned NVIDIA A100 GPUs with 10 GB of GPU memory through an internal cluster. We obtained the underlying models from Hugging Face under model IDs google/gemma-3-1b-it, google/gemma-3-4b-it, and meta-llama/Llama-3.2-3B-Instruct.
Shift Ciphers
Training Procedure and Hyperparameters
Each prompt is constructed by sampling a string of random length with characters drawn uniformly from . These strings are then embedded into a fixed prompt template of the form:
"The following text was encrypted using a Caesar
cipher: <string>. What is the shift (0--25)? Answer
with a single integer."
For each prompt , we computed the model’s conditional log-probabilities over candidate answers corresponding to the integers .
We fit the MAP estimator described in § 5 using the PyTorch implementation outlined in Algorithm 1. The latent parameter is initialized by sampling from an isotropic Gaussian and optimized using the Adam optimizer.
In our experiments, we use the following hyperparameters: number of prompts , learning rate , number of epochs , prior precision . Since our prompt dependence is constant, we derive from the variation in the model’s output probabilities. After convergence, we compute the MAP estimate as well as the uncertainty as outlined in Algorithm 2 and transform the results into the probability vector using the inverse log-ratio map to obtain a distribution over shifts.
Training Dynamics
For the shift cipher experiments, we find that across all three language models, the training loss decreases smoothly over time as shown in Figure 5. This behavior confirms that the derived MAP objective is well conditioned in this setting.
Figure 5. Training loss history for the Adam-driven MAP optimization for Gemma 3 1B IT, Gemma 3 4B IT, and Llama 3.2 3B IT on the ciphertext problem. The smooth loss curves indicate a stable training objective.
BBQ Experiments
Training Procedure and Hyperparameters
For each BBQ item, we construct a prompt by concatenating the context, question, and answer options in a fixed multiple choice format. Concretely, if the three answer choices are , , and , we use a prompt template of the form:
"Context: <context>. Question: <question>. Options: A. <a1>, B. <a2>, C. <a3>. Answer with only A, B, or C."
For each model and social dimension , we evaluate the model’s conditional probabilities over the tokens corresponding to A, B, and C, and normalize these probabilities across the three choices before mapping to the reduced answer space . Then we estimate the latent parameter by minimizing the MAP objective from Equation (6).
In our experiments we optimize this objective for each social dimension using Adam with the same fixed hyperparameters from the ciphertext experiment across all models and dimensions. The number of prompts for these experiments, however, comes directly from the number of prompts in each social dimension of the BBQ dataset. Again, we derive from the variation in the model’s output probabilities and we run Algorithm 1 to find the MAP estimate before running Algorithm 2 for uncertainty quantification.
Training Dynamics
Examining the optimization dynamics of the MAP estimation procedure, Figure 6 shows the excess objective over training epochs for each model and social dimension. We plot the excess objective because the absolute scale of the objective differs across models and dimensions which is expected due to variations in the number of examples and variability of observations .
Figure 6. Training loss history for the Adam-driven MAP optimization for Gemma 3 1B IT, Gemma 3 4B IT, and Llama 3.2 3B IT. The training curves are shown separately for each social dimension. We note that each training run exhibits stable convergence.
Notably, the convergence behavior differs between models. The Gemma 3 1B IT model exhibits the smallest initial objective values and demonstrates the fastest convergence which is reasonable due to its small model size. On the other hand, the Gemma 3 4B IT model displays the largest initial objective values and takes the largest number of epochs to converge to a stable baseline.
Most importantly, however, we notice that across all models and all social dimensions, the optimization exhibits smooth decay towards zero, indicating stable convergence of our Adam-based optimization procedure.
References
- [1]Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122, 2025.DOI
- [2]Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3(Feb):1137–1155, 2003.
- [3]David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017.
- [4]Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D Lee, M Sugiyama, U Luxburg, I Guyon, and R Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.arxiv.org/abs/1607.06520
- [5]Anton Bovier. Stochastic processes, 2006.
- [6]Stephen P Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN 978-0-521-83378-3.
- [7]Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017.DOI
- [8]Masoumeh Dashti and Andrew M. Stuart. The Bayesian Approach to Inverse Problems, pp. 311–428. Springer International Publishing, Cham, 2017. ISBN 978-3-319-12385-1. doi: 10.1007/978-3-319-12385-1_7. URL https://doi.org/10.1007/978-3-319-12385-1_7.
- [9]Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux - effortless Bayesian deep learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20089–20103. Curran Associates, Inc., 2021.arxiv.org/abs/2106.14806
- [10]Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, 1987.DOI
- [11]Emilio Ferrara. Should ChatGPT be biased? challenges and risks of bias in large language models. First Monday, 28(11), Nov. 2023. doi: 10.5210/fm.v28i11.13346. URL https://firstmonday.org/ojs/index.php/fm/article/view/13346.DOI
- [12]Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179, 2024.arxiv.org/abs/2309.00770
- [13]Paul H Garthwaite, Joseph B Kadane, and Anthony O’Hagan. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470):680–701, 2005.DOI
- [16]Thomas L Griffiths and Joshua B Tenenbaum. Optimal predictions in everyday cognition. Psychological Science, 17(9):767–773, 2006.pubmed.ncbi.nlm.nih.gov/16984293
- [17]Thomas L Griffiths and Joshua B Tenenbaum. Predicting the future as Bayesian inference: People combine prior knowledge with observations when estimating duration and extent. Journal of Experimental Psychology: General, 140(4):725–743, 2011.
- [18]Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. Bayesian Models of Cognition, pp. 59–100. Cambridge Handbooks in Psychology. Cambridge University Press, 2008.
- [19]W K Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97–109, 1970.
- [20]Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander Van Der Linden, and Jon Roozenbeek. Generative language models exhibit social identity biases. Nature Computational Science, 5(1):65–75, 2025.
- [21]Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM Collective Intelligence Conference, pp. 12–24, 2023.DOI
- [22]David Leslie. Understanding artificial intelligence ethics and safety. arXiv preprint arXiv:1906.05684, 2019.DOI
- [23]R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121, October 2024a.DOI
- [25]Kevin P Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012. ISBN 978-0-262-01802-9.
- [26]Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356–5371, Online, August 2021. Association for Computational Linguistics.
- [27]Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953–1967, Online, November 2020. Association for Computational Linguistics.arxiv.org/abs/2010.00133
- [28]Anthony O’Hagan, Caitlin E Buck, Alireza Daneshkhah, J Richard Eiser, Paul H Garthwaite, David J Jenkinson, Jeremy E Oakley, and Tim Rakow. Uncertain Judgements: Eliciting Experts’ Probabilities. Wiley, 1 edition, July 2006. ISBN 978-0-470-02999-2 978-0-470-03331-9. doi: 10.1002/0470033312.DOI
- [29]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086–2105, Dublin, Ireland, May 2022. Association for Computational Linguistics.arxiv.org/abs/2110.08193
- [30]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 2019.
- [31]Tim Salimans, Diederik Kingma, and Max Welling. Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference on Machine Learning, pp. 1218–1226, 2015.arxiv.org/abs/1410.6460
- [32]Shoaib Ahmed Siddiqui, Yanzhi Chen, Juyeon Heo, Menglin Xia, and Adrian Weller. On evaluating LLMs’ capabilities as functional approximators: A Bayesian evaluation framework. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 5826–5835, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.arxiv.org/abs/2410.04541



