Embedding Semantic Entropy: NLI-Free Semantic Uncertainty Estimation with Geometry-Aware Clustering

Introduction

Large Language Models (LLMs) have demonstrated strong performance across a wide range of natural language processing tasks. Despite their fluency and coherence, these models frequently generate answers that are incorrect, fabricated, or unsupported by evidence while expressing high confidence. Such failures, commonly referred to as hallucinations or confabulations, pose serious risks in high-stakes domains where users may over-trust model outputs.

In this work, we focus on short-form question answering (QA), where model outputs are typically concise factual answers such as entities, dates, numerical values, or short statements.

Existing semantic uncertainty methods often rely on pairwise entailment models to group generated answers by meaning. Although effective, this approach introduces an external NLI component, requires many pairwise comparisons, and provides limited visibility into why responses are grouped together. This motivates an alternative formulation in which semantic groups are constructed directly from the geometry of sentence embeddings. Such a formulation can expose cluster compactness, representative medoids, and inter-cluster separation, making the uncertainty score easier to inspect.

Related Works

A foundational approach in this area is Semantic Entropy (SE) Farquhar et al. [5] which computes uncertainty over semantic meanings rather than surface-level token variations by clustering generated responses into semantically equivalent groups using bidirectional entailment. However, SE relies on hard clustering decisions derived from entailment models and does not capture fine-grained semantic relationships between responses. In particular, it does not explicitly model intra-cluster compactness or inter-cluster separation, which are critical for robust clustering Vipulanandan et al. [16].

To address the limitations of hard semantic grouping, kernel-based approaches have been proposed Nikitin et al. [11], which replaces discrete clustering with a kernel-based similarity matrix and computes uncertainty using von Neumann entropy Bengtsson and Życzkowski [2]. This formulation captures pairwise semantic similarities and provides more expressive uncertainty estimates. However, KLE operates on a kernel similarity matrix rather than explicit semantic clusters, it does not provide cluster-level structure such as representative responses, intra-group compactness, or separation between semantic groups. This limits its inspectability: although KLE can indicate that responses are semantically diverse, it does not directly show which semantic groups exist, which response represents each group, or how tightly grouped those responses are.

Quantum-inspired methods extend semantic entropy by modeling token-sequence probability uncertainty with quantum tensor networks and perturbation-based uncertainty quantification ?. While they improve robustness across generation lengths and quantization settings, they mainly calibrate sequence-probability uncertainty and do not explicitly address semantic cluster geometry.

Embedding-space methods such as Semantic Density Qiu and Miikkulainen [12] estimate uncertainty from the distribution of responses in embedding space, but do not directly incorporate cluster validation or geometric constraints such as intra-cluster compactness and inter-cluster separation.

Beyond single-dimensional approaches, multi-dimensional frameworks combine multiple uncertainty signals, such as semantic similarity between generated responses and knowledge consistency with respect to model outputs or external information Chen et al. [3]. These signals are computed independently and aggregated to produce a final uncertainty estimate, enabling improved detection of unreliable outputs in complex question answering settings. However, this approach introduces additional computational overhead and system complexity, as it requires multiple scoring components and aggregation mechanisms during inference.

Complementary work has explored hallucination detection using structured representations. Methods such as KEA Haskins and Adams [6] leverage knowledge graphs and graph kernel techniques to compare generated outputs with structured facts, enabling identification of inconsistencies at the level of explicit knowledge relationships. This provides interpretable signals by highlighting which parts of the generated response align or conflict with known facts. However, such approaches are less directly applicable to uncertainty estimation in short-form answer clustering.

Limitations of Existing Works. Despite these advances, existing methods exhibit several common limitations. First, clustering-based approaches such as SE rely on hard semantic equivalence and fail to capture fine-grained relationships between responses. Second, similarity-based methods such as KLE and Semantic Density do not explicitly enforce cluster geometry or validate clustering quality. These limitations motivate the need for a clustering framework that explicitly uses the geometry of semantic embedding space, including intra-cluster compactness and inter-cluster separation, to improve semantic grouping for uncertainty estimation.

Our Contribution. We propose a novel Embedding Semantic Entropy, a new embedding-based semantic uncertainty estimator that computes entropy over semantic groups constructed in embedding space rather than through entailment-based clustering.

• DeBERTa-free semantic clustering: We replace DeBERTa-based entailment clustering with embedding-based K-Medoids clustering. This shows that a strong external NLI classifier is not always necessary for semantic uncertainty estimation; established clustering techniques in embedding space can sustain, and in several settings improve, uncertainty estimation performance. By relying on standard distance-based clustering rather than pairwise entailment decisions, the proposed method also makes the semantic grouping process more transparent and easier to inspect. We also show that the external DeBERTa-based NLI component used for entailment clustering can be removed without substantial degradation in uncertainty estimation performance.

• Embedding-based white-box clustering: Prior NLI-based methods rely on DeBERTa ?, treating semantic grouping as a black-box decision. In contrast, the proposed method is a white-box approach that provides greater explainability by making the clustering process and cluster structure directly inspectable.

• Geometry-aware dynamic clustering: We adapt established cluster-validation criteria to the semantic-response clustering stage. Davies–Bouldin validation Davies and Bouldin [4] is used to evaluate the trade-off between intra-cluster dispersion and inter-cluster separation, a medoid-based compactness constraint prevents semantically broad clusters, and elbow-based selection ? avoids unnecessary over-splitting when additional clusters provide only marginal geometric improvement.

• Evaluation: We evaluate the method across QA datasets and open-weight LLMs using AUROC and AUARC.

Methodology

Our method improves the clustering stage of Semantic Entropy by replacing entailment-based grouping with embedding-based clustering. The resulting cluster distribution is then used to compute an entropy-based uncertainty score. The overall pipeline is summarized in Figure 1.

Input Preparation Given a question $q$, we sample generated answers $A = \{a_1,\ldots,a_n\}$. Each answer is combined with the question before embedding to preserve contextual meaning, i.e., $a_i' = \text{“Context: }q\text{ \textbackslash n Answer: }a_i\text{”}$. This context-aware formatting is important for short-form QA. A bare answer such as “London” may be ambiguous without the question, whereas the formatted input represents the answer in relation to the question being answered. We then apply basic preprocessing, including lowercasing, article removal, Unicode normalization, and noise cleaning, to reduce lexical variability.

Embedding Representation Each processed answer is encoded into a dense semantic vector $e_i = f(a_i')$, where $f(\cdot)$ denotes a sentence embedding encoder. In our implementation, we use supervised SimCSE with RoBERTa-large, princeton-nlp/sup-simcse-roberta-large ??. We select supervised SimCSE because the clustering stage requires an embedding space in which semantically similar answers are placed close together while semantically different answers are separated. Compared with generic sentence encoders, supervised SimCSE is explicitly trained using contrastive learning to improve sentence-level semantic similarity, making it suitable for grouping short generated answers. It also provides fixed-dimensional dense embeddings that can be efficiently clustered using distance-based methods such as K-Medoids. This is preferable for our setting because the clustering procedure depends directly on the geometry of the embedding space rather than on pairwise entailment labels. We then L2-normalize each embedding as $\hat{e}_i = e_i/\lVert e_i\rVert$.

Degeneracy Handling

Before the main clustering loop, we handle two degenerate cases. If $n = 1$, the only response is assigned to cluster 0. If all normalized embeddings are near-identical, all responses are also assigned to a single cluster. We detect this using

We use $\epsilon_{\mathrm{id}} = 10^{-8}$ only as a numerical tolerance for detecting duplicate or near-duplicate embeddings. Since embeddings are L2-normalized, Euclidean distance is directly related to cosine similarity:

Therefore, when the distance is below $10^{-8}$, the corresponding cosine similarity is essentially 1 up to floating-point precision. In this edge case, separating the responses into multiple clusters would be unstable and unnecessary. This step avoids unnecessary clustering when all generated answers have the same meaning. In such cases, all responses are assigned to one cluster, indicating zero semantic disagreement.

Clustering with K-Medoids

For non-degenerate cases, we cluster the normalized embeddings using K-Medoids. For each candidate number of clusters

Since typical experiments use 10 sampled responses per question, $K = 10$ is the theoretical maximum, corresponding to the case where each response forms its own cluster. However, cluster-validity metrics such as the Davies–Bouldin index are not meaningful when every point is assigned to a separate singleton cluster. Therefore, candidate clusterings are evaluated only when they produce at least two and fewer than $n$ non-empty clusters. This allows the method to explore meaningful semantic partitions while avoiding degenerate all-singleton solutions.

We use K-Medoids where the cluster representative is an actual generated answer. This makes each cluster more interpretable: the medoid can be inspected as the representative response for that semantic group. K-Means centroids, in contrast, are arithmetic averages that may not correspond to any real response and may fall inside the unit hypersphere after averaging normalized vectors.

In our implementation, K-Medoids uses PAM-style optimization Schubert and Rousseeuw [14], k-medoids++ initialization, Euclidean distance, and a fixed random seed. This choice is important because the clusters produced at this stage directly determine the probability masses used in the final entropy calculation. PAM-style optimization improves the quality of medoid selection by reducing within-cluster distance, which helps form compact semantic groups. The k-medoids++ initialization further improves stability by starting from well-separated candidate medoids, reducing sensitivity to poor initialization. Together, these choices make the semantic clusters more reliable and reproducible, which is essential because over-merging distinct meanings would underestimate uncertainty, while over-splitting paraphrases would overestimate uncertainty.

Dynamic Selection of Number of Clusters Selecting an appropriate number of clusters is critical for capturing the semantic structure of generated responses. Instead of fixing the number of clusters a priori, we evaluate multiple candidate values to identify the clustering configuration that best separates distinct answer meanings while keeping semantically equivalent responses within compact groups.

Rationale for Geometry-Aware Validation. The goal of the clustering stage is not to introduce a new clustering validity index, but to use established geometric criteria to construct reliable semantic groups for entropy estimation. Since the responses are represented as $L_2$-normalized embeddings, Euclidean distances reflect cosine-based semantic similarity. In our setting, each generated response is represented as an $L_2$-normalized embedding vector. Therefore, the structure of the embedding space can be analyzed through pairwise distances between response embeddings, distances from responses to their cluster medoids, and distances between cluster medoids. These distances are important because they determine whether responses assigned to the same cluster are semantically close and whether different clusters represent meaningfully different answer groups. If distinct meanings are merged into one loose cluster, the final entropy score may underestimate uncertainty. Conversely, if paraphrases are split into separate clusters, the entropy score may overestimate uncertainty. We therefore use distance-based validation to favor clusters that are internally compact and externally separated before computing the final entropy score.

Davies–Bouldin Index. For each candidate $K$, we compute the Davies–Bouldin (DB) index Davies and Bouldin [4]:

where $S_i$ denotes the intra-cluster dispersion of cluster $i$, and $M_{ij}$ represents the distance between clusters $i$ and $j$. The DB index jointly captures intra-cluster compactness and inter-cluster separation, with lower values indicating better clustering quality.

Davies–Bouldin validation is suitable because it jointly measures intra-cluster dispersion and inter-cluster separation without requiring ground-truth cluster labels Davies and Bouldin [4]. This is important in our setting because the true semantic clusters of sampled LLM responses are not known during inference. However, the Davies–Bouldin score alone can hide poor individual clusters because it summarizes clustering quality globally ?. We therefore add an explicit medoid-based compactness constraint to reject any candidate clustering that contains a loose semantic group.

Explicit Compactness Constraint. To address the limitation of relying only on a global DB score, we introduce an explicit compactness constraint that evaluates clustering quality at the level of individual clusters. For each cluster $C$, we compute the mean distance between its points and the corresponding medoid:

where $m_C$ is the medoid of cluster $C$, and $d(x,m_C)$ denotes the Euclidean distance between response embedding $x$ and medoid $m_C$ in the $L_2$-normalized embedding space. A clustering configuration is rejected if any cluster violates the compactness threshold, i.e., $\max_C D_C > \tau$. This prevents the merging of semantically dissimilar responses into overly broad clusters, which would otherwise reduce the final entropy score by assigning distinct meanings to the same semantic group.

Elbow-Based Selection ? Among the remaining valid candidates, we analyze the rate of improvement in DB scores to avoid selecting unnecessarily large values of $K$. For $K = 1$, there is only one cluster, so the DB score and elbow comparison are not meaningful because no inter-cluster separation exists. For $K = 2$, the DB score can be computed, but the elbow ratio cannot be formed because it requires three consecutive DB scores. For $K \ge3$, let:

We define the improvement ratio as:

where $\epsilon$ is a small constant for numerical stability. A sharp decrease in this ratio ($r$) indicates diminishing returns from increasing $K$. When $r < \gamma$, where $\gamma$ is the elbow sensitivity threshold, we identify an elbow point and select the previous number of clusters, i.e., $K^{*} = K-1$. The optimal number of clusters is therefore determined using a multi-stage procedure. Candidate values of $K$ that violate the compactness constraint are first discarded. The elbow criterion is then applied to the remaining candidates to detect diminishing improvements in DB scores. If no clear elbow point is observed, the value of $K$ that minimizes the DB index among the valid candidates is selected as a fallback. This fallback is justified because, after removing candidates with loose clusters, the remaining candidates already satisfy the compactness requirement; among them, the minimum DB score corresponds to the best available trade-off between intra-cluster compactness and inter-cluster separation. This procedure favors clusterings that are compact, separated, and not unnecessarily over-split, making the final semantic groups more reliable for entropy estimation.

Why Geometry-Aware Clustering is Needed. A direct replacement of NLI clustering with embedding clustering is not sufficient for semantic uncertainty estimation, because the number and quality of semantic groups directly determine the final entropy score. If embedding-based clustering merges distinct meanings into one broad cluster, the entropy score is underestimated; if it over-splits paraphrases into many small clusters, the entropy score is overestimated. Therefore, the clustering stage must not only group responses by embedding similarity, but also control the geometry of the resulting semantic groups. Emb-SE addresses this by combining medoid-based clustering, explicit compactness checking, Davies–Bouldin validation, and elbow-based model selection. These components serve complementary roles: K-Medoids provides representative generated answers as cluster centers, the compactness constraint prevents semantically loose clusters, the Davies–Bouldin index balances intra-cluster dispersion against inter-cluster separation, and the elbow criterion avoids unnecessary over-splitting. Together, this makes Emb-SE more than a straightforward embedding-based replacement of NLI clustering; it provides a geometry-aware mechanism for constructing semantic groups whose structure directly affects the uncertainty estimate.

Hyperparameter Configuration

We tune the clustering hyperparameters using a 5-fold cross-validation grid search for each model–dataset setting. The search is performed over the compactness threshold $\tau\in[0.10, 0.50]$ and the elbow sensitivity threshold $\gamma\in[0.01, 0.50]$. In the implementation, $\tau$ is searched from 0.10 to 0.50 with step size 0.05, while $\gamma$ is searched from 0.01 to 0.50 with step size 0.01. For each candidate configuration, the validation responses are clustered, the Embedding Semantic Entropy score is computed, and the configuration is evaluated using AUROC. The configuration with the highest mean cross-validation AUROC is selected for the corresponding model–dataset pair.

Appendix § 8, Table 1 summarizes the selected hyperparameter values. The compactness threshold $\tau$ is summarized by model, since it is mainly affected by model-specific response dispersion. The elbow sensitivity threshold $\gamma$ is summarized by dataset, since it reflects how clearly responses separate into semantic clusters for a given benchmark. Overall, Falcon-7B-Instruct tends to select stricter compactness values, while LLaMA-2-7B-Chat selects more relaxed compactness values. The selected $\gamma$ values vary more widely across datasets, indicating that the elbow behavior of the Davies–Bouldin curve depends on the structure of the answer space.

Parameter	Group	Avg. selected value	Selected range
$\gamma$	NQ	0.39	0.23–0.50
$\gamma$	SQuAD	0.34	0.23–0.39
$\gamma$	SVAMP	0.28	0.05–0.46
$\gamma$	TriviaQA	0.25	0.14–0.44
$\gamma$	BioASQ	0.21	0.01–0.42
$\tau$	Gemma-7B	0.16	0.15–0.20
$\tau$	LLaMA-2-7B-Chat	0.23	0.20–0.25
$\tau$	LLaMA-2-7B	0.13	0.10–0.20
$\tau$	Falcon-7B-Instruct	0.11	0.10–0.15

Uncertainty Estimation

After the final clustering step, we compute an entropy-based uncertainty score over the embedding-derived semantic clusters. The key idea is to first compute the probability of each generated sequence, then aggregate these probabilities within each semantic cluster, and finally compute entropy over the resulting cluster-level probability distribution.

Let $q$ denote the input question and let $A = \{a_1, \ldots, a_n\}$ be the sampled responses. We denote the token sequence corresponding to response $a_i$ as

where $i$ indexes the generated response, $N_i$ is the number of generated tokens in response $a_i$. The model assigns a conditional probability to each token given the question and the previously generated tokens:

where $r$ indexes the token position within that response. Then, we compute the sequence log-likelihood as

In practice, we use log-likelihoods for numerical stability, since multiplying many token probabilities can produce very small values.

The raw sequence probabilities are then normalized over the sampled responses:

Here, $P_{S_i}$ is the normalized probability mass assigned to sampled sequence $S_i$, and $\sum_{i=1}^{n} P_{S_i} = 1$.

Let $y_i \in\{1, \ldots, K^*\}$ denote the final cluster assignment of response $a_i$, where $K^*$ is the selected number of clusters. Each cluster is defined as a set of response indices:

The probability mass of cluster $C_k$ is obtained by summing the normalized sequence probabilities of all responses assigned to that cluster:

This aggregation prevents surface-level paraphrases from being treated as independent semantic outcomes.

The cluster probabilities are also normalized:

Therefore, the cluster probabilities form a valid probability distribution over the semantic clusters.

The Embedding Semantic Entropy score is then computed over the cluster probability distribution:

Low entropy indicates that most probability mass is concentrated in one embedding-derived semantic cluster, suggesting high confidence in a single semantic answer. High entropy indicates that probability mass is spread across multiple semantic clusters, reflecting greater semantic disagreement among the sampled responses.

Thus, Emb-SE estimates semantic uncertainty over groups rather than individual answer strings by using geometry-aware clustering in embedding space. The proposed method constructs these semantic groups using embedding-space distances, medoid-based compactness, and inter-cluster separation before computing the final entropy-based uncertainty score.

Experiments

We evaluate on short-form QA by generating multiple responses per question and estimating semantic uncertainty to identify unreliable outputs.

Datasets and Models. We evaluate on five short-form QA benchmarks: TriviaQA Joshi et al. [7], SQuAD Rajpurkar et al. [13], NQ-Open ?, SVAMP ?, and BioASQ ?. These datasets cover open-domain factual question answering, reading comprehension, arithmetic word problems, and biomedical question answering, allowing us to test whether the proposed clustering method is robust across different answer distributions.

Experiments are conducted using four open-weight LLMs: Falcon-7B-Instruct Almazrouei et al. [1], LLaMA-2-7B, LLaMA-2-7B-Chat Touvron et al. [15], and Gemma-7B ?. For each input question, $n = 10$ responses are sampled at temperature $T = 1.0$ using top-$K$ sampling ($K = 50$). An additional low-temperature sample ($T = 0.1$) is used to assess model accuracy against ground-truth answers. The exact answer-generation prompt format is provided in Appendix § 7.

Baselines. We compare against a range of uncertainty estimation methods spanning semantic, probabilistic, and embedding-based approaches. These include Semantic Entropy (SE), Discrete Semantic Entropy (DSE), and Embedding Regression (ER) Farquhar et al. [5], Kernel Language Entropy (KLE) Nikitin et al. [11], Predictive Entropy (PE) Malinin and Gales [9], and P(True) Kadavath et al. [8]. We additionally evaluate our proposed method, Embedding Semantic Entropy.

Evaluation Metrics. We evaluate performance using Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Accuracy-Rejection Curve (AUARC) Nadeem et al. [10]. These metrics are commonly used in state-of-the-art uncertainty estimation and hallucination detection evaluations [5, 11]. AUROC evaluates the ability to distinguish correct and incorrect predictions, whereas AUARC evaluates whether uncertainty estimates are useful for rejecting low-confidence predictions.

Results

Figure 2 and Figure 3 summarize the pairwise win-rate comparison across all 20 model–dataset settings, corresponding to four LLMs evaluated on five QA datasets. The complete numerical AUROC and AUARC results are reported in Appendix § 9, with model-wise results shown in Table 2–Table 5. Overall, Embedding Semantic Entropy performs competitively against strong semantic, probabilistic, and embedding-based uncertainty baselines, and is frequently the tied-best method.

Method	TriviaQA AUROC	TriviaQA AUARC	SQuAD AUROC	SQuAD AUARC	NQ AUROC	NQ AUARC	SVAMP AUROC	SVAMP AUARC	BioASQ AUROC	BioASQ AUARC
SE	0.791 ± 0.02	0.549 ± 0.03	0.665 ± 0.04	0.251 ± 0.03	0.716 ± 0.03	0.369 ± 0.03	0.698 ± 0.04	0.279 ± 0.04	0.831 ± 0.02	0.499 ± 0.03
DSE	0.779 ± 0.02	0.532 ± 0.03	0.655 ± 0.04	0.241 ± 0.03	0.714 ± 0.03	0.351 ± 0.03	0.711 ± 0.04	0.267 ± 0.03	0.828 ± 0.02	0.479 ± 0.03
PE	0.750 ± 0.03	0.522 ± 0.03	0.665 ± 0.04	0.257 ± 0.03	0.671 ± 0.03	0.348 ± 0.03	0.732 ± 0.04	0.290 ± 0.03	0.655 ± 0.03	0.406 ± 0.03
Pₜᵣᵤₑ	0.544 ± 0.03	0.390 ± 0.03	0.404 ± 0.04	0.133 ± 0.02	0.554 ± 0.03	0.280 ± 0.03	0.592 ± 0.04	0.226 ± 0.03	0.598 ± 0.03	0.358 ± 0.03
ER	0.719 ± 0.03	0.490 ± 0.03	0.628 ± 0.04	0.232 ± 0.03	0.658 ± 0.03	0.325 ± 0.03	0.683 ± 0.04	0.258 ± 0.03	0.829 ± 0.02	0.507 ± 0.03
KLE(K_HEAT)	0.771 ± 0.02	0.537 ± 0.03	0.602 ± 0.04	0.223 ± 0.03	0.712 ± 0.03	0.362 ± 0.03	0.706 ± 0.04	0.281 ± 0.03	0.785 ± 0.02	0.468 ± 0.03
Emb-SE	0.796 ± 0.02	0.549 ± 0.03	0.671 ± 0.04	0.255 ± 0.03	0.733 ± 0.03	0.378 ± 0.03	0.694 ± 0.04	0.277 ± 0.03	0.834 ± 0.02	0.504 ± 0.03

Method	TriviaQA AUROC	TriviaQA AUARC	SQuAD AUROC	SQuAD AUARC	NQ AUROC	NQ AUARC	SVAMP AUROC	SVAMP AUARC	BioASQ AUROC	BioASQ AUARC
SE	0.767 ± 0.02	0.705 ± 0.03	0.744 ± 0.03	0.364 ± 0.03	0.735 ± 0.03	0.528 ± 0.03	0.890 ± 0.02	0.763 ± 0.02	0.869 ± 0.02	0.702 ± 0.02
DSE	0.764 ± 0.02	0.695 ± 0.02	0.749 ± 0.03	0.350 ± 0.03	0.750 ± 0.03	0.522 ± 0.03	0.878 ± 0.02	0.749 ± 0.03	0.865 ± 0.02	0.683 ± 0.02
PE	0.744 ± 0.02	0.698 ± 0.02	0.737 ± 0.03	0.361 ± 0.03	0.725 ± 0.03	0.513 ± 0.03	0.858 ± 0.02	0.753 ± 0.03	0.825 ± 0.02	0.686 ± 0.02
Pₜᵣᵤₑ	0.739 ± 0.03	0.690 ± 0.03	0.644 ± 0.03	0.328 ± 0.03	0.684 ± 0.03	0.501 ± 0.03	0.710 ± 0.03	0.649 ± 0.03	0.808 ± 0.02	0.681 ± 0.02
ER	0.647 ± 0.02	0.625 ± 0.03	0.575 ± 0.03	0.298 ± 0.03	0.598 ± 0.03	0.416 ± 0.03	0.627 ± 0.02	0.604 ± 0.03	0.784 ± 0.02	0.547 ± 0.02
KLE(K_HEAT)	0.762 ± 0.02	0.701 ± 0.02	0.749 ± 0.03	0.340 ± 0.03	0.739 ± 0.03	0.526 ± 0.03	0.884 ± 0.02	0.760 ± 0.02	0.804 ± 0.02	0.670 ± 0.02
Emb-SE	0.781 ± 0.02	0.716 ± 0.02	0.767 ± 0.03	0.378 ± 0.03	0.745 ± 0.03	0.539 ± 0.03	0.891 ± 0.02	0.765 ± 0.02	0.875 ± 0.02	0.702 ± 0.02

Across the 20 model–dataset settings, Emb-SE achieves the best or tied-best AUROC in 15 settings and the best or tied-best AUARC in 13 settings. This indicates that the proposed embedding-based clustering method is not only competitive with entailment-based Semantic Entropy, but also consistently strong across diverse datasets and model families. The strongest performance is observed for Gemma-7B, where Emb-SE achieves the best or tied-best AUARC on all five datasets and the best AUROC on four of the five datasets. Emb-SE also performs strongly for Falcon-7B-Instruct and LLaMA-2-7B, especially on TriviaQA, NQ, and BioASQ.

Figure 2 shows the pairwise AUROC win-rate heatmap. Each cell reports the fraction of settings in which the row method outperforms the column method, with ties counted as 0.5. Emb-SE obtains the strongest overall pairwise AUROC profile. It outperforms or ties SE with a pairwise win rate of 0.80, DSE with 0.72, KLE($K_{\mathrm{HEAT}}$) with 0.90, PE with 0.90, and both P(True) and ER with 1.00.

The appendix tables provide a more detailed model-wise view. For Falcon-7B-Instruct, Emb-SE obtains the highest AUROC on TriviaQA, SQuAD, NQ, and BioASQ, while PE is strongest on SVAMP. For LLaMA-2-7B, Emb-SE is best on TriviaQA, NQ, and BioASQ, while DSE is strongest on SQuAD and SE is strongest on SVAMP. For LLaMA-2-7B-Chat, Emb-SE is best or tied-best on TriviaQA, SQuAD, NQ, and BioASQ, but weaker on SVAMP. For Gemma-7B, Emb-SE achieves the highest AUROC on TriviaQA, SQuAD, SVAMP, and BioASQ, while DSE is strongest on NQ.

These results suggest that embedding-space clustering provides discriminative semantic uncertainty scores comparable to, and often stronger than, entailment-based semantic grouping. However, performance still varies by dataset and model, showing that no single uncertainty estimator dominates every setting.

Figure 3 shows the pairwise AUARC win-rate heatmap. Emb-SE again has the strongest overall profile, with a pairwise AUARC win rate of 0.70 against SE, 0.95 against DSE, 0.90 against KLE($K_{\mathrm{HEAT}}$), 0.80 against PE, 1.00 against $P(\mathrm{True})$, and 0.95 against ER. This indicates that Emb-SE is especially effective in rejection-based evaluation, where the uncertainty score is used to rank unreliable responses for abstention.

The appendix tables show that Emb-SE achieves particularly strong AUARC performance for Gemma-7B, where it is best or tied-best on all five datasets. For LLaMA-2-7B, Emb-SE is best on TriviaQA, NQ, and BioASQ, while PE is strongest on SQuAD and SE is strongest on SVAMP. For LLaMA-2-7B-Chat, Emb-SE is best on TriviaQA, SQuAD, and BioASQ, while SE is slightly stronger on NQ and KLE($K_{\mathrm{HEAT}}$) is strongest on SVAMP. For Falcon-7B-Instruct, Emb-SE is best or tied-best on TriviaQA and NQ, while PE and ER remain stronger on some other datasets.

These results indicate that Emb-SE is useful for selective prediction: it can effectively identify outputs that should be rejected when the model is uncertain.

Comparison with Entailment-Based Approaches. Compared with SE and DSE, Emb-SE removes the external DeBERTa-based pairwise entailment clustering step while remaining competitive or stronger in many settings. The pairwise heatmaps show that Emb-SE has a higher AUROC win rate than both SE and DSE, and a higher AUARC win rate than both SE and DSE. This supports the central motivation of the proposed method: semantic uncertainty estimation can be performed using geometry-aware clustering in embedding space, without requiring pairwise NLI inference during uncertainty estimation.

A key advantage of Emb-SE is that the uncertainty score is computed over clusters formed in a continuous embedding space. Unlike entailment-based methods, where grouping depends on discrete NLI labels, the proposed method exposes the geometry behind each semantic group. Cluster medoids, response-to-medoid distances, inter-cluster distances, and compactness values can be inspected directly. This makes the resulting confidence or uncertainty score more interpretable: high uncertainty can be traced to probability mass being spread across well-separated semantic clusters, while low uncertainty corresponds to probability mass concentrated in a compact cluster. Thus, Emb-SE not only maintains competitive uncertainty estimation performance, but also provides a more explainable clustering structure than black-box entailment-based grouping.

At the same time, the appendix tables show that entailment-based or probabilistic baselines remain stronger in selected cases, especially on some SQuAD and SVAMP settings. Therefore, Emb-SE should not be interpreted as universally superior across all benchmarks. Rather, it provides a practical and more inspectable alternative to hard entailment-based clustering, while maintaining strong empirical performance.

Overall, the results show that Emb-SE is a competitive semantic uncertainty estimator across five QA datasets and four open-weight LLMs. The pairwise win-rate heatmaps show that Emb-SE has the strongest overall comparison profile for both AUROC and AUARC, while the appendix tables confirm that this performance is consistent across several models and datasets. These findings support the use of geometry-aware embedding clustering as a transparent alternative to DeBERTa-based semantic grouping.

Limitations

While the proposed method improves the semantic clustering stage used for entropy-based uncertainty estimation, several limitations remain. The method requires access to token or sequence probabilities to compute the probability-weighted entropy score. Therefore, it is less directly applicable to fully black-box LLMs that only return generated text without likelihood information. Emb-SE avoids pairwise NLI inference during clustering. For $n$ sampled responses, entailment-based clustering requires multiple pairwise comparisons, whereas Emb-SE embeds each response once and performs distance-based clustering in the embedding space. Although K-Medoids adds clustering overhead, the method avoids repeated DeBERTa inference and provides directly inspectable cluster geometry.

Conclusion

We proposed Embedding Semantic Entropy, a new embedding-based semantic uncertainty estimator that replaces entailment-based clustering with geometry-aware clustering in embedding space. The resulting cluster distribution is used to compute an entropy-based uncertainty score. By using K-Medoids clustering and dynamic cluster selection with compactness constraints and elbow-based analysis, the proposed method accounts for both intra-cluster coherence and inter-cluster separation.

Experimental results across multiple LLMs and QA datasets show that the proposed approach achieves competitive performance across AUROC and AUARC. These findings suggest that incorporating embedding-space geometry can improve semantic grouping for uncertainty estimation. Moreover, the results indicate that DeBERTa-based semantic clustering is not strictly necessary for competitive semantic uncertainty estimation, as embedding-based clustering achieves comparable performance while offering a more transparent clustering process.

Answer Generation Prompt

This appendix describes the prompt format used to generate model answers for the uncertainty estimation experiments. For each dataset example, the prompt consists of a brief instruction, optional few-shot demonstrations, and the target question. The brief instruction used in the default setting is:

Answer the following question as briefly as possible.

For few-shot prompting, examples are sampled from the training set. Each few-shot example is formatted as a question–answer pair, optionally including context when the dataset setting requires it. The general few-shot format is:

Question: <question>
Answer: <answer>

When context is enabled, the example format becomes:

Context:  <context>
Question: <question>
Answer:  <answer>

For the target validation example, the same format is used, but the answer field is left open for the model to complete. Without context, the final query prompt is:

Question: <question>
Answer:

With context, the final query prompt is:

Context:  <context>
Question: <question>
Answer:

Thus, the full prompt used for generation is:

In the experiments, five few-shot examples are used by default. For each target question, one low-temperature response is generated with temperature $T = 0.1$ and used for accuracy evaluation. In addition, ten high-temperature responses are generated with temperature $T = 1.0$ and used for semantic uncertainty estimation. The maximum generation length is set to 50 new tokens. This setup follows the uncertainty estimation protocol in which the low-temperature response represents the model’s most likely answer, while the high-temperature responses provide a sample of possible semantic alternatives.

Hyperparameter Selection Details

Table 1 summarizes the hyperparameter values selected by 5-fold cross-validation. The search space is fixed for all model–dataset settings, with $\tau\in[0.10, 0.50]$ and $\gamma\in[0.01, 0.50]$. For each model–dataset setting, cross-validation selects the values that maximize validation AUROC.

The table is a compact summary of these model–dataset-specific selections rather than a full listing of every selected pair. The compactness threshold $\tau$ is grouped by model because it mainly reflects model-specific variation in response dispersion within the embedding space. The elbow sensitivity threshold $\gamma$ is grouped by dataset because it reflects how clearly generated answers separate into semantic clusters for each benchmark. Although both parameters may be influenced by both model and dataset, this grouping highlights the dominant variation observed in the selected values.

In Table 1, the range reports the minimum and maximum values chosen by cross-validation within each group, not the full grid-search range. The avg selected value is computed over all cross-validation-selected hyperparameter values within each group and is therefore not necessarily equal to the midpoint of the selected range. These selected values are used for the final Embedding Semantic Entropy evaluation reported in the main results.

Full Experimental Results

This appendix provides the complete numerical results corresponding to the AUROC and AUARC bar plots presented in the main text. Table 2–Table 5 report the mean performance and standard deviation for each uncertainty estimation method across all model–dataset settings. These results include the proposed Embedding Semantic Entropy method together with the baseline methods SE, DSE, KLE, PE, $P(\mathrm{True})$, and ER. The best-performing method for each model–dataset setting is highlighted in bold.

Method	TriviaQA AUROC	TriviaQA AUARC	SQuAD AUROC	SQuAD AUARC	NQ AUROC	NQ AUARC	SVAMP AUROC	SVAMP AUARC	BioASQ AUROC	BioASQ AUARC
SE	0.784 ± 0.03	0.745 ± 0.02	0.785 ± 0.03	0.405 ± 0.03	0.775 ± 0.02	0.541 ± 0.03	0.878 ± 0.02	0.649 ± 0.03	0.852 ± 0.02	0.642 ± 0.03
DSE	0.778 ± 0.02	0.742 ± 0.02	0.798 ± 0.03	0.398 ± 0.03	0.776 ± 0.02	0.527 ± 0.03	0.870 ± 0.02	0.636 ± 0.03	0.848 ± 0.02	0.618 ± 0.03
PE	0.751 ± 0.03	0.730 ± 0.02	0.757 ± 0.03	0.408 ± 0.03	0.750 ± 0.03	0.533 ± 0.03	0.841 ± 0.02	0.632 ± 0.03	0.819 ± 0.02	0.620 ± 0.03
Pₜᵣᵤₑ	0.607 ± 0.03	0.668 ± 0.03	0.628 ± 0.03	0.331 ± 0.03	0.678 ± 0.03	0.480 ± 0.03	0.751 ± 0.03	0.555 ± 0.04	0.672 ± 0.03	0.521 ± 0.03
ER	0.619 ± 0.03	0.664 ± 0.03	0.573 ± 0.03	0.288 ± 0.03	0.569 ± 0.03	0.403 ± 0.03	0.758 ± 0.03	0.574 ± 0.03	0.748 ± 0.02	0.583 ± 0.03
KLE(K_HEAT)	0.775 ± 0.03	0.743 ± 0.02	0.710 ± 0.03	0.381 ± 0.03	0.780 ± 0.02	0.542 ± 0.03	0.867 ± 0.02	0.642 ± 0.03	0.818 ± 0.02	0.623 ± 0.03
Emb-SE	0.801 ± 0.02	0.755 ± 0.02	0.778 ± 0.03	0.403 ± 0.03	0.785 ± 0.02	0.544 ± 0.03	0.868 ± 0.02	0.644 ± 0.03	0.858 ± 0.02	0.644 ± 0.03

Method	TriviaQA AUROC	TriviaQA AUARC	SQuAD AUROC	SQuAD AUARC	NQ AUROC	NQ AUARC	SVAMP AUROC	SVAMP AUARC	BioASQ AUROC	BioASQ AUARC
SE	0.787 ± 0.02	0.731 ± 0.02	0.750 ± 0.03	0.360 ± 0.03	0.724 ± 0.03	0.514 ± 0.03	0.841 ± 0.02	0.590 ± 0.03	0.842 ± 0.02	0.640 ± 0.03
DSE	0.788 ± 0.02	0.730 ± 0.02	0.755 ± 0.03	0.356 ± 0.03	0.726 ± 0.03	0.510 ± 0.03	0.845 ± 0.02	0.588 ± 0.03	0.840 ± 0.02	0.632 ± 0.03
PE	0.758 ± 0.02	0.722 ± 0.02	0.719 ± 0.03	0.359 ± 0.03	0.715 ± 0.03	0.510 ± 0.03	0.835 ± 0.02	0.589 ± 0.03	0.678 ± 0.03	0.554 ± 0.03
Pₜᵣᵤₑ	0.670 ± 0.03	0.662 ± 0.03	0.598 ± 0.04	0.297 ± 0.03	0.647 ± 0.03	0.465 ± 0.03	0.573 ± 0.03	0.422 ± 0.04	0.790 ± 0.02	0.633 ± 0.03
ER	0.599 ± 0.02	0.636 ± 0.03	0.562 ± 0.03	0.266 ± 0.03	0.570 ± 0.03	0.412 ± 0.03	0.803 ± 0.03	0.575 ± 0.03	0.729 ± 0.03	0.569 ± 0.03
KLE(K_HEAT)	0.783 ± 0.02	0.732 ± 0.02	0.707 ± 0.03	0.349 ± 0.03	0.715 ± 0.03	0.508 ± 0.03	0.845 ± 0.02	0.591 ± 0.03	0.805 ± 0.02	0.623 ± 0.03
Emb-SE	0.788 ± 0.02	0.733 ± 0.02	0.762 ± 0.03	0.368 ± 0.03	0.728 ± 0.03	0.513 ± 0.03	0.827 ± 0.02	0.579 ± 0.03	0.845 ± 0.02	0.646 ± 0.03