betaFeedback

WorksComputer Science

Sentence-Level Topic Modeling with Lexical and Contextual Evidence

Abstract

Topic modeling faces a fundamental granularity dilemma: probabilistic topic models assign topics at the word level, whereas transformer-based methods operate at the document level. The latter overlooks that real-world documents typically cover multiple topics, whereas the former relies on individual words, which rarely provide sufficient context to identify them. We introduce SentenceETM, a neural topic model that assigns a topic to each sentence by jointly using lexical and contextual evidence. Each topic is described both by a distribution over the vocabulary, defined through word embeddings, and by a Gaussian over sentence embeddings concentrated on a topic-specific low-dimensional subspace. The topic of a sentence generates both its words and its embedding and is marginalized exactly, so that its posterior combines the two sources of evidence. Document proportions are inferred with amortized variational inference, while the Gaussian parameters are re-estimated in closed form. Experiments show that SentenceETM consistently improves sentence-level topic assignment over state-of-the-art approaches, and a case study of a presidential debate shows how sentence-level topics reveal how candidates handle the same subjects.

Introduction

Topic models are well-established methods for extracting the main themes from a collection of documents. Regardless of their underlying formulation, these methods produce a set of topics that characterize the corpus, where a topic is broadly understood as a group of semantically coherent words. A document is characterized by a mixture of these topics. However,

such a mixture tells us which topics the documents contain but not where they are localized. Indeed, real documents rarely contain a single theme, and locating them requires deciding which textual unit a topic is attached to and from which evidence it is inferred.

We refer to lexical evidence as the information carried by words occurring in a text segment1, represented as counts over a fixed vocabulary, and to contextual evidence as the dense representation of that text segment produced by a pretrained neural encoder. Lexical evidence is directly interpretable as a distribution over words, but it is blind to word order and to the sense a word takes in a given occurrence. Contextual evidence can capture the overall semantics of a document more faithfully, but at the cost of interpretability, since the words that characterize a topic are no longer part of its representation and must be recovered separately.

The two main families of topic models each build primarily on one of these types of evidence, because the textual unit they adopt admits primarily one of them. Bag-of-words approaches, e.g. [4], treat the word as the fundamental unit: each word receives a latent topic assignment, and a document becomes a mixture over topics. A word is represented independently of its surrounding context, so contextual evidence from the document is not available to the model at this level. Embedding-based approaches, e.g. [12], take the opposite stance, treating the document as the fundamental unit and clustering the encoder’s representations. Contextual evidence is exploited directly, but lexical evidence re-enters only post hoc, through a descriptor-extraction step playing no role in the topic assignment itself. The result is that one family is too fine-grained to use context, the other too coarse-grained to localize the topics.

Sentence-level topic models lie in between these two extremes. A sentence is semantically self-contained, short enough that its words and its embedding are concerned with a common theme, and numerous enough within a document that assigning a topic to each sentence reveals how topics are distributed throughout the text. Most importantly, at the sentence level, lexical and contextual evidence became available at the same textual unit. While sentence-level topic models have been proposed in the past [1, 7], each model still relies on a single type of evidence for topic assignment. To fill this gap, we make the following contributions.

(i) We introduce SentenceETM, a neural sentence-level topic model that jointly uses lexical and contextual evidence to assign a topic to each sentence. Each topic is characterized by both a distribution over the vocabulary, defined via word embeddings, and a region of the sentence-embedding space, modeled as a Gaussian concentrated on a topic-specific low-dimensional subspace, thereby keeping the model tractable in the high-dimensional space of sentence encoders. Each sentence draws a topic from the document’s topic proportions, and that topic generates both the sentence’s words and its contextual embedding. The sentence topic is marginalized exactly, so that its posterior combines the two sources of evidence, and the parameters of the contextual side are estimated in closed form.

(ii) We present a novel evaluation pipeline for sentence-level topic models. We build synthetic corpora from existing topic-labeled document collections, in which every sentence has a known topic and document-level topic mixtures are controlled by construction.

(iii) We show that SentenceETM can be adapted to different domains through an interchangeable sentence encoder, without modifying the generative model. We illustrate this on the transcript of a televised presidential debate. The sentence-level representation allows us to examine how candidates distribute topics within the same debate segment and how frequently new topics are introduced during speaking turns.

Related works

The use of sentences as a textual unit in topic modeling has been previously investigated mainly as an extension of LDA [4]. Early approaches incorporated sentence- or segment-level structure without making the segment the unit of topic assignment [6, 22, 15]. Differently, senLDA [1] moves the topic assignment from words to sentences: one topic is drawn for each sentence, and the words in that sentence are drawn from the selected topic. The topic–word association is preserved, with each topic represented as a distribution over the vocabulary. More recently, Cha and Lee (2024) introduced SentenceLDA, which replaces the word with the sentence as the basic unit of Gaussian LDA [8], while Schneider (2023) uses sentence embeddings for topic assignment through hard expectation maximization. In these methods, contextual information is used directly for topic assignment, whereas lexical information is used only to describe the resulting topics. Our contextual component is related to model-based clustering in high dimensions [5]: instead of the full-covariance Gaussians of Gaussian LDA, each topic is modeled by a Gaussian concentrated on its own low-dimensional subspace, which remains tractable for sentence embeddings with hundreds of dimensions. A parallel line of work combines lexical and contextual evidence without changing the textual unit. Contextualized Topic Models [2 b] combine bag-of-words representations with pretrained contextual document embeddings, while TNTM [19] models topics as distributions over transformer representations. Both combine the two forms of evidence at the document level and return a topic per document.

The proposed approach

SentenceETM (Figure 1) is built on a single question:

Overview of SentenceETM

Figure 1. Overview of SentenceETM. Document-level topic proportions (green) are combined with the contextual and lexical evidence of each sentence (red) into the posterior γmk\gamma_{mk}, which re-estimates each topic’s subspace Gaussian in closed form (dashed arrow).

“How likely is it to observe a given sentence embedding together with its words, given that the sentence belongs to a document with topic proportions θ(d)\boldsymbol{\theta}^{(d)}?”

Let wm(d)=(wm1(d),…,wmNm(d)(d))\mathbf{w}^{(d)}_m = (w^{(d)}_{m1}, \ldots, w^{(d)}_{mN^{(d)}_m}) denote the words of the mm-th sentence of document dd and e~m(d)\tilde{\mathbf{e}}^{(d)}_m its normalized sentence embedding. Given θ(d)\boldsymbol{\theta}^{(d)}, the model assigns to each sentence the likelihood

p(wm(d),e~m(d)∣θ(d))=∑k=1Kθk(d)N(e~m(d);μk,Σk)∏n=1Nm(d)βk,wmn(d).(1)p(\mathbf{w}^{(d)}_m, \tilde{\mathbf{e}}^{(d)}_m \mid\boldsymbol{\theta}^{(d)}) = \sum_{k=1}^{K} \theta^{(d)}_k \mathcal{N}(\tilde{\mathbf{e}}^{(d)}_m; \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k) \prod_{n=1}^{N^{(d)}_m} \beta_{k,w^{(d)}_{mn}} . \tag*{(1)}

Each term of the sum is the product of the prevalence of topic kk in the document, the likelihood of the sentence embedding under the topic, and the likelihood of the sentence words under the topic. Since both views (i.e., evidence) are generated by the same topic, a topic contributes to Equation (1) only when it explains the embedding and the words simultaneously. Normalizing the terms of the sum yields the posterior over the topic of the sentence, γmk(d)=p(zm(d)=k∣wm(d),e~m(d),θ(d))∝θk(d)N(e~m(d);μk,Σk)∏n=1Nm(d)βk,wmn(d)\gamma^{(d)}_{mk} = p(z^{(d)}_m = k \mid\mathbf{w}^{(d)}_m, \tilde{\mathbf{e}}^{(d)}_m, \boldsymbol{\theta}^{(d)}) \propto\theta^{(d)}_k \mathcal{N}(\tilde{\mathbf{e}}^{(d)}_m; \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k) \prod_{n=1}^{N^{(d)}_m} \beta_{k,w^{(d)}_{mn}}, which provides the sentence-level topic assignment. In practice, the lexical and contextual terms can have different numerical scales; during training and inference, we weight the contextual log-likelihood by c>0c > 0, with c=1c = 1 recovering Equation (1).

Generative model

A corpus consists of DD documents. Document dd is a collection of M(d)M^{(d)} sentences, and sentence mm has the words wm(d)\mathbf{w}^{(d)}_m, with wmn(d)∈{1,…,V}w^{(d)}_{mn} \in\{1,\ldots,V\}. Each sentence is observed through two views. The lexical view is its bag of words, while the contextual view is the embedding em(d)∈RE\mathbf{e}^{(d)}_m \in\mathbb{R}^{E} of the raw sentence, produced by a frozen sentence encoder. Since sentence encoders are trained to compare sentences by cosine similarity, we ℓ2\ell_2-normalize these embeddings, e~m(d)=em(d)/∥em(d)∥\tilde{\mathbf{e}}^{(d)}_m = \mathbf{e}^{(d)}_m/\lVert\mathbf{e}^{(d)}_m\rVert, so that Euclidean distances between them are a monotone function of their cosine similarity. Finally, ρ∈Rp×V\boldsymbol{\rho} \in\mathbb{R}^{p \times V} are pre-trained word embeddings.

Lexical view. Each topic k∈{1,…,K}k \in\{1,\ldots,K\} has a topic embedding αk∈Rp\boldsymbol{\alpha}_k \in\mathbb{R}^{p} in the word embedding space, which defines a distribution over the vocabulary, βk=softmax⁡(ρ⊤αk)\boldsymbol{\beta}_k = \operatorname{softmax}(\boldsymbol{\rho}^{\top}\boldsymbol{\alpha}_k), as in ETM [9].

Contextual view. Each topic is also associated with a region of the sentence embedding space, described by a Gaussian distribution with mean μk∈RE\boldsymbol{\mu}_k \in\mathbb{R}^{E} and covariance Σk\boldsymbol{\Sigma}_k. Sentence embeddings are high-dimensional, and a full covariance matrix would require O(E2)O(E^2) parameters per topic. Following [5], we instead adopt a flexible and parsimonious modeling which assumes that the sentences of each topic vary mainly within a topic-specific subspace of dimension hk≪Eh_k \ll E, spanned by the orthonormal columns of Vk∈RE×hk\mathbf{V}_k \in\mathbb{R}^{E \times h_k}, and that the remaining variation is isotropic noise:

Σk=VkΛkVk⊤+σk2IE,(2)\boldsymbol{\Sigma}_k = \mathbf{V}_k\boldsymbol{\Lambda}_k\mathbf{V}_k^{\top} + \sigma_k^2\mathbf{I}_E, \tag*{(2)}

where Λk=diag⁡(λk1−σk2,…,λkhk−σk2)\boldsymbol{\Lambda}_k = \operatorname{diag}(\lambda_{k1}-\sigma_k^2,\ldots,\lambda_{kh_k}-\sigma_k^2). Along the hkh_k directions of its subspace, topic kk has variances λk=(λk1,…,λkhk)\boldsymbol{\lambda}_k = (\lambda_{k1},\ldots,\lambda_{kh_k}), with λk1≥⋯≥λkhk\lambda_{k1} \ge\cdots\ge\lambda_{kh_k}; in every other direction, it has variance σk2<λkhk\sigma_k^2 < \lambda_{kh_k}. A sentence is thus likely under a topic when it lies close to the topic subspace, even if it is far from its mean along the directions in which the topic varies. Since each topic has its own subspace, different topics can vary along different directions of the embedding space. The covariance in Equation (2) requires O(Ehk)O(Eh_k) parameters per topic, and both its inverse and its determinant have closed forms that never involve E×EE \times E matrices (Section A.3). We collect the model parameters in Θ={Θlex,Θctx}\boldsymbol{\Theta} = \{\boldsymbol{\Theta}_{\mathrm{lex}}, \boldsymbol{\Theta}_{\mathrm{ctx}}\}, with Θlex={αk}k=1K\boldsymbol{\Theta}_{\mathrm{lex}} = \{\boldsymbol{\alpha}_k\}_{k=1}^{K} and Θctx={μk,Vk,λk,σk2}k=1K\boldsymbol{\Theta}_{\mathrm{ctx}} = \{\boldsymbol{\mu}_k,\mathbf{V}_k,\boldsymbol{\lambda}_k,\sigma_k^2\}_{k=1}^{K}.

Generative process. For each document, topic proportions θ(d)=softmax⁡(δ(d))\boldsymbol{\theta}^{(d)} = \operatorname{softmax}(\boldsymbol{\delta}^{(d)}) are drawn from a logistic-normal prior, with δ(d)∼N(0,IK)\boldsymbol{\delta}^{(d)} \sim\mathcal{N}(\mathbf{0},\mathbf{I}_K). Each sentence then receives a single topic zm(d)∼Mult⁡(1;θ(d))z^{(d)}_m \sim\operatorname{Mult}(1;\boldsymbol{\theta}^{(d)}), which generates both of its views: the normalized embedding, e~m(d)∣zm(d)=k∼N(μk,Σk)\tilde{\mathbf{e}}^{(d)}_m \mid z^{(d)}_m = k \sim\mathcal{N}(\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k), and every word of the sentence, wmn(d)∣zm(d)=k∼Mult⁡(1;βk)w^{(d)}_{mn} \mid z^{(d)}_m = k \sim\operatorname{Mult}(1;\boldsymbol{\beta}_k). The full process is summarized in Algorithm 1 (Section A.4). Given the topic of a sentence, the two views are assumed to be conditionally independent: p(wm(d),e~m(d)∣zm(d)=k)=N(e~m(d);μk,Σk)∏n=1Nm(d)βk,wmn(d)p(\mathbf{w}^{(d)}_m,\tilde{\mathbf{e}}^{(d)}_m \mid z^{(d)}_m = k) = \mathcal{N}(\tilde{\mathbf{e}}^{(d)}_m;\boldsymbol{\mu}_k,\boldsymbol{\Sigma}_k)\prod_{n=1}^{N^{(d)}_m}\beta_{k,w^{(d)}_{mn}}. Marginalizing the topic of the sentence with respect to the document proportions yields the sentence likelihood of Equation (1). Since sentences are conditionally independent given θ(d)\boldsymbol{\theta}^{(d)}, the joint distribution of a document and its latent proportions is p(W(d),E~(d),δ(d)∣Θ)=N(δ(d);0,IK)∏m=1M(d)p(wm(d),e~m(d)∣θ(d))p(\mathbf{W}^{(d)},\tilde{\mathbf{E}}^{(d)},\boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta}) = \mathcal{N}(\boldsymbol{\delta}^{(d)};\mathbf{0},\mathbf{I}_K)\prod_{m=1}^{M^{(d)}}p(\mathbf{w}^{(d)}_m,\tilde{\mathbf{e}}^{(d)}_m \mid\boldsymbol{\theta}^{(d)}), where θ(d)=softmax⁡(δ(d))\boldsymbol{\theta}^{(d)} = \operatorname{softmax}(\boldsymbol{\delta}^{(d)}). The sentence topics zm(d)z^{(d)}_m are summed out exactly within each sentence likelihood; the only latent variable is the document-level δ(d)\boldsymbol{\delta}^{(d)}, whose posterior we approximate in Section 2.2.

Inference model

The marginal likelihood p(W(d),E~(d)∣Θ)p(\mathbf{W}^{(d)},\tilde{\mathbf{E}}^{(d)} \mid\boldsymbol{\Theta}) of document dd is

∫N(δ(d);0,IK)∏m=1M(d)p(wm(d),e~m(d)∣θ(d)) dδ(d),(3)\int\mathcal{N}(\boldsymbol{\delta}^{(d)};\mathbf{0},\mathbf{I}_K)\prod_{m=1}^{M^{(d)}}p(\mathbf{w}^{(d)}_m,\tilde{\mathbf{e}}^{(d)}_m \mid\boldsymbol{\theta}^{(d)})\,\mathrm{d}\boldsymbol{\delta}^{(d)}, \tag*{(3)}

which is intractable because of the logistic-normal prior. We therefore approximate the posterior of δ(d)\boldsymbol{\delta}^{(d)} with variational inference, and fit the model parameters Θ\boldsymbol{\Theta} by maximizing a lower bound on Equation (3). For any distribution q(δ(d))q(\boldsymbol{\delta}^{(d)}), the log marginal likelihood decomposes as log⁡p(W(d),E~(d)∣Θ)=L(q;Θ)+DKL(q(δ(d))∥p(δ(d)∣W(d),E~(d),Θ))\log p(\mathbf{W}^{(d)},\widetilde{\mathbf{E}}^{(d)} \mid\Theta)=\mathcal{L}(q;\Theta)+D_{\mathrm{KL}}\left(q(\delta^{(d)})\mathbin{\|}p(\delta^{(d)}\mid\mathbf{W}^{(d)},\widetilde{\mathbf{E}}^{(d)},\Theta)\right), where L(q;Θ)=Eq[log⁡p(W(d),E~(d),δ(d)∣Θ)−log⁡q(δ(d))]\mathcal{L}(q;\Theta)=\mathbb{E}_{q}\left[\log p(\mathbf{W}^{(d)},\widetilde{\mathbf{E}}^{(d)},\delta^{(d)}\mid\Theta)-\log q(\delta^{(d)})\right] (proof in Appendix A.1). Since the KL divergence is non-negative, L\mathcal{L} is a lower bound (ELBO) of the log marginal likelihood, and maximizing it with respect to qq brings qq closer to the true posterior.

Variational assumptions. We restrict qq to Gaussians with diagonal covariance, amortized by an inference network, to approximate the intractable posterior p(δ(d)∣W(d),E~(d),Θ)p(\delta^{(d)}\mid\mathbf{W}^{(d)},\widetilde{\mathbf{E}}^{(d)},\Theta): qϕ(δ(d)∣W(d))=N(δ(d);μϕ(u(d)),diag⁡(σϕ2(u(d))))q_{\phi}(\delta^{(d)}\mid\mathbf{W}^{(d)})=\mathcal{N}(\delta^{(d)};\mu_{\phi}(\mathbf{u}^{(d)}),\operatorname{diag}(\sigma_{\phi}^{2}(\mathbf{u}^{(d)}))), where u(d)∈RV\mathbf{u}^{(d)}\in\mathbb{R}^{V} is the ℓ1\ell_{1}-normalized bag of words of the document and μϕ,σϕ\mu_{\phi},\sigma_{\phi} are the outputs of a multilayer perceptron with parameters ϕ\phi, as in ETM (Dieng et al. [9]). The inference network conditions only on the words of the document; the contextual view enters the inference through the likelihood. The sentence topics require no variational approximation, since they are summed out exactly in Equation (1) and, once θ(d)\theta^{(d)} is known, their posterior γmk(d)\gamma_{mk}^{(d)} follows by normalizing its terms.

Objective and optimization. For a document dd, our objective is (derivation in Appendix A.2):

Ld(Θ,ϕ)=Eqϕ[∑m=1M(d)log⁡p(wm(d),e~m(d)∣θ(d))]−DKL(qϕ(δ(d)∣W(d))∥N(0,IK))(4)\mathcal{L}_{d}(\Theta,\phi)=\mathbb{E}_{q_{\phi}}\left[\sum_{m=1}^{M^{(d)}}\log p\left(w_{m}^{(d)},\widetilde{e}_{m}^{(d)}\mid\theta^{(d)}\right)\right]-D_{\mathrm{KL}}\left(q_{\phi}(\delta^{(d)}\mid\mathbf{W}^{(d)})\mathbin{\|}\mathcal{N}(0,I_{K})\right) \tag*{(4)}

where Eqϕ\mathbb{E}_{q_{\phi}} is the expectation under qϕ(δ(d)∣W(d))q_{\phi}(\delta^{(d)}\mid\mathbf{W}^{(d)}). The first term rewards topics that explain both views of every sentence; each summand is the logarithm of the sentence likelihood of Equation (1), computed stably with the log-sum-exp operator. The second term keeps the variational distribution close to the prior and has the closed form 12∑k=1K(σϕ,k2+μϕ,k2−1−log⁡σϕ,k2)\frac{1}{2}\sum_{k=1}^{K}(\sigma_{\phi,k}^{2}+\mu_{\phi,k}^{2}-1-\log\sigma_{\phi,k}^{2}).

We maximize ∑d=1DLd\sum_{d=1}^{D}\mathcal{L}_{d} by alternating two steps (Algorithm 2). (i) Inference network and lexical parameters. We update ϕ\phi and Θlex\Theta_{\mathrm{lex}} by stochastic gradient ascent on mini-batches of documents, estimating the expectation in Equation (4) with a single sample and the reparameterization δ(d)=μϕ+σϕ⊙ϵ\delta^{(d)}=\mu_{\phi}+\sigma_{\phi}\odot\epsilon, with ϵ∼N(0,IK)\epsilon\sim\mathcal{N}(0,I_{K}). (ii) Contextual parameters. At the end of each epoch, we compute the posteriors γmk(d)\gamma_{mk}^{(d)} of all sentences by normalizing the terms of Equation (1), using the mean of qϕq_{\phi}, θ(d)=softmax⁡(μϕ(u(d)))\theta^{(d)}=\operatorname{softmax}(\mu_{\phi}(\mathbf{u}^{(d)})), and update Θctx\Theta_{\mathrm{ctx}} in closed form. For fixed posteriors, the maximum-likelihood estimates of the parameters of Equation (2) are: the estimate of Vk\mathbf{V}_{k} collects the hkh_{k} eigenvectors of Sk=1nk∑d,mγmk(d)(e~m(d)−μ^k)(e~m(d)−μ^k)TS_{k}=\frac{1}{n_{k}}\sum_{d,m}\gamma_{mk}^{(d)}(\widetilde{\mathbf{e}}_{m}^{(d)}-\hat{\mu}_{k})(\widetilde{\mathbf{e}}_{m}^{(d)}-\hat{\mu}_{k})^{\mathsf{T}} and

σ^k2=1E−hk∑j=hk+1Eλ^kj=tr⁡(Sk)−∑j=1hkλ^kjE−hk,(5)\hat{\sigma}_{k}^{2}=\frac{1}{E-h_{k}}\sum_{j=h_{k}+1}^{E}\hat{\lambda}_{kj}=\frac{\operatorname{tr}(S_{k})-\sum_{j=1}^{h_{k}}\hat{\lambda}_{kj}}{E-h_{k}}, \tag*{(5)}

where λ^k1≥⋯≥λ^kE\hat{\lambda}_{k1}\geq\cdots\geq\hat{\lambda}_{kE} are the ordered eigenvalues of SkS_{k}, μ^k=1nk∑d,mγmk(d)e~m(d)\hat{\mu}_{k}=\frac{1}{n_{k}}\sum_{d,m}\gamma_{mk}^{(d)}\widetilde{\mathbf{e}}_{m}^{(d)} and nk=∑d,mγmk(d)n_{k}=\sum_{d,m}\gamma_{mk}^{(d)}. We update the contextual parameters in closed form, conditional on the current responsibilities γmk(d)\gamma_{mk}^{(d)}. This update requires no learning rate and keeps V^k\widehat{\mathbf{V}}_{k} orthonormal by construction. The contextual parameters are initialized by running k-means on all normalized sentence embeddings and applying the same updates with hard assignments, γmk(d)∈{0,1}\gamma_{mk}^{(d)}\in\{0,1\}. After training, we compute θ(d)\theta^{(d)} from the mean of qϕq_{\phi} and assign each sentence to arg⁡max⁡kγmk(d)\arg\max_{k}\gamma_{mk}^{(d)}. In practice, we use in all experiments the same subspace dimension for every topic, i.e., hk=hh_{k}=h for all kk.

Experimental Setting

Prior work evaluates sentence-level topic models based on their induced document-level topic representations and therefore does not directly assess how topics are distributed across sentences. For example, senLDA (Balikas et al., [1]) is evaluated using held-out perplexity and the F1F_{1} score of a classifier trained on document topic proportions, while SentenceLDA (Cha and Lee, [7]) is evaluated based on the quality of the document representations it induces. We argue that evaluating sentence-level topic models requires, first and foremost, datasets with sentence-level ground-truth topic labels. In the absence of suitable existing datasets, we propose a synthetic corpora generation strategy that provides such labels while allowing us to control the document-level topic mixtures. We remind readers that the labels are used only afterward for model evaluation and are not used during training (all methods are unsupervised). Details on the used datasets, model training, and metrics in Appendix B.

Synthetic corpora. Let LL be the number of sentences of the desired document to create, KK the number of topics it mixes, and pp the percentage of the document that belongs to its dominant topic. Each synthetic document is assembled by drawing m1=p⋅Lm_{1}=p\cdot L sentences from a document of the dominant topic and m2=L(1−p)/(K−1)m_{2}=L(1-p)/(K-1) sentences from (random) documents of each of the remaining K−1K-1 topics, all drawn from a corpus whose documents carry a single class label. The new document mixture is therefore fixed by construction. Varying KK at fixed LL and pp therefore varies how the same L⋅(1−p)L\cdot(1-p) non-dominant-topic sentences are fragmented: at K=2K=2 they form a single block of L⋅(1−p)L\cdot(1-p) sentences, at the largest admissible KK they are L⋅(1−p)L\cdot(1-p) isolated sentences, one per topic. Three constraints delimit the admissible KK. Every topic must contribute at least one sentence, m2≥1m_{2}\geq1, hence K≤1+L(1−p)K\leq1+L(1-p); the dominant topic must dominate, m1>m2m_{1}>m_{2}, i.e., p>1/Kp>1/K; and K≤Kmax⁡K\leq K_{\max}, the number of classes in the source corpus.

Figure 2. SentenceETM sentence-level topic assignments. BBC News, K=5K = 5. Background color marks the predicted topic (business / entertainment / politics / sport / tech); misassignment in red.

We build the synthetic corpora from: BBC News [11], 20 Newsgroups2, and Wikipedia [13, 17]. We set L=20L = 20 and p=0.6p = 0.6 for all three corpora. The dominant topic then contributes m1=p⋅L=12m_1 = p \cdot L = 12 sentences, which must come from a single article, so LL is bounded by the length of the source articles: m1=12m_1 = 12 is not binding for Wikipedia (median ∼110\sim110 sentences per article), is available in about half of the BBC articles (median ∼13\sim13), and in about a quarter of the 20 Newsgroups posts, whose lengths are heavily skewed (median ∼6\sim6, mean 11.1). A larger LL would leave too few eligible articles in BBC and 20 Newsgroups, while a smaller one would make sentence topics harder to recover, even at K=2K = 2. With p=0.6p = 0.6, the dominant topic covers a clear majority of every document, while the remaining L⋅(1−p)=8L \cdot(1-p) = 8 sentences allow the non-dominant mass to be split across up to eight topics. We use K∈{2,3,5,9}K \in\{2, 3, 5, 9\} for 20 Newsgroups and Wikipedia, and K∈{2,3,5}K \in\{2, 3, 5\} for BBC News (as it has a maximum of 5 classes). The synthetic corpus has 700 documents, each of 20 sentences.

Political debates dataset. ElecDeb60to20 (Gofredo et al., 2023) comprises 44 U.S. televised debates (1960–2020), collected from the Commission on Presidential Debates3, which provides transcripts of presidential and vice-presidential debates. For our experiments, we consider the debate between Biden and Trump on the 29th of October 2020, as it is the longest. Specifically, the debates are organized into 6 segments (see Appendix § 7), each presenting a main topic, along with the journalists’ and the candidates’ answers.

Baselines. We test SentenceETM against the main candidates of each of the three families of topic models. At the word-level, we consider LDA [4] and ETM [9]; at the document-level, BERTopic [12], which clusters document embeddings, and finally at the sentence-level, senLDA [1], an LDA in which all words of a sentence share one topic, and SentenceLDA [7], a Gaussian LDA over sentence embeddings (cf. § 1.1). All models use the same preprocessing and vocabulary, and all embedding-based models (BERTopic, SentenceLDA, SentenceETM) use the same sentence encoder, either all-MiniLM-L6-v2 or all-mpnet-base-v2. Our evaluation requires one topic per sentence, which word- and document-level models do not output directly. Therefore, we derive sentence-level assignments as follows: for LDA and ETM, every word ww of document dd takes its most probable topic given the fitted model, arg⁡max⁡kθdkϕkw\arg\max_k \theta_{dk}\phi_{kw}, and a sentence takes the topic receiving the majority of its word assignments. For BERTopic, each document is represented by the mean embedding of its sentences, using the same encoder as the sentence-level models to avoid truncation and give each sentence equal weight. Documents are clustered with UMAP and HDBSCAN; the resulting clusters are reduced to KK topics, outliers are reassigned to the nearest topic, and every sentence inherits the topic of its document.

Experimental Results

Figure 2 shows a randomly selected document for K=5K = 5 on BBC News synthetic (cf. § 3). SentenceETM switches topics from one sentence to the next according to the actual content. The red sentence “It’s what I want to do for the rest of my life” highlights the model mistake. Notably, such a sentence carries no topic-specific information and is therefore not representative of any of the topics. Moreover, in Figure 4 we show the structure of the latent space for a randomly chosen run on Wikipedia (full details in Appendix C.2). Finally, we refer the reader to Appendix C.5 for results on the original datasets.

Figure 4. Sentence-level topic assignments (Wikipedia, all-mpnet-base-v2, K=9K = 9). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel.

Results on synthetic corpora

Additional analysis of the model, including sensitivity analyses for the sentence encoder and document length, is available in Appendices C.3 to C.4. Computational cost in Appendix B.5. SenClu [20] is in Section C.1 due to space constraints.

Ablation study. To assess the contribution of the lexical and contextual evidence, we vary the weight cc of the contextual likelihood and a common subspace dimension hh, setting hk=hh_k = h for all topics. We consider two reference models: one that uses lexical evidence only (c=0c = 0, where the subspace dimension plays no role) and one that uses contextual evidence only. Since SentenceETM initializes its contextual component from a kk-means partition of the normalized sentence embeddings, we additionally report this partition as a strong contextual baseline. We use BBC News with K=5K = 5 and all-mpnet-base-v2, and repeat each configuration on 3 synthetic corpora sampled with different seeds. Figure 3 reports ARI and AMI as a function of cc for each hkh_k. For c≤0.003c \le0.003, the contextual evidence has almost no influence: the model behaves like the lexical-only reference (ARI ≈0.30\approx0.30) and varies strongly across corpora. Between c=0.003c = 0.003 and c=0.03c = 0.03, the contextual evidence takes over, earlier for larger hkh_k, whose richer subspaces make it more informative. For c≥0.03c \ge0.03, performance is largely stable (ARI approximately 0.62–0.66), with the best results around c=0.1c = 0.1 for every tested subspace dimension. The gain over lexical evidence alone is large (about +0.33 ARI), whereas the gain over contextual evidence alone depends on hkh_k: it is about +0.08 for hk=4h_k = 4 and progressively decreases as the subspace becomes more expressive, down to about +0.01 for hk=64h_k = 64. Thus, lexical and contextual evidence are complementary: lexical information contributes most when the contextual model is constrained, while for larger subspaces the contextual representation already explains much of the sentence structure. The lexical component additionally yields the topic–word distributions used to interpret the topics. The setting used in the main experiments (hk=32h_k = 32 for all kk, c=0.5c = 0.5) lies within the stable region. The AMI results show the same behavior.

Figure 3. Ablation on BBC News (K=5K = 5, all-mpnet-base-v2). ARI and AMI as a function of cc, for the common subspace dimensions hk=h∈{4,16,32,64}h_k = h \in\{4, 16, 32, 64\}. Lexical evidence only in gray; diamonds denote contextual-only models.

Analysis in terms of ARI and AMI

Figure 6 reports clustering quality analysis as a function of the number of topics KK (all-mpnet-base-v2). The complete set of numerical results is available in Table 5 to 7. Two groups of methods clearly emerge. SentenceETM, senLDA, and SentenceLDA, the other baselines lag behind, as their ARI and AMI are close to zero across all three datasets and increase only slightly as KK grows. This follows from the construction of the corpora: at K=2K = 2, every document contains both topics in proportions 0.6/0.4, so document-level information provides little information about individual sentences; as KK grows, the dominant topic stands out more clearly against each of the others, making document-level assignments marginally informative. Overall, the proposed model achieves the best ARI and AMI across almost all settings, regardless of the encoder or dataset. A substantial part of the variability in the results comes from the choice of class subset, indeed some combinations of classes are intrinsically easier to separate than others (see Table 1 to 3). To isolate the contribution of the model, Figure 5 compares methods on the same subsets. For each subset of KK classes, we average scores over seeds and compute Δ=sSentenceETM−max⁡bsb\Delta= s_{\mathrm{SentenceETM}} - \max_b s_b, where the maximum is taken over all baselines on that subset. A positive Δ\Delta means that SentenceETM outperforms every baseline on that subset. In both ARI and AMI, SentenceETM outperforms all baselines across all 31 subsets, with median gains ranging from about 0.03 to 0.2, depending on the dataset and KK. The gain is largest at small KK and decreases as KK grows, as the strongest baseline, typically SentenceLDA, becomes more competitive. The most challenging dataset turned out to be 20 Newsgroups, possibly due to its less-curated, more conversational nature. Unlike BBC News and Wikipedia, whose documents are generally centered on a well-defined subject, 20 Newsgroups consists of user-written discussion posts that may include digressions, quoted text, or generic conversational content. As a result, individual sentences can be only weakly indicative of the document-level topic.

Figure 5. SentenceETM per-subset gain, all-mpnet-base-v2. Each point corresponds to a subset of (KK) ground-truth classes, with (Δ\Delta) measured relative to the best-performing baseline for that subset (SentenceLDA) and averaged over seeds.

Figure 6. ARI and AMI as function of the number of topics KK, all-mpnet-base-v2. Markers indicate the median across runs, while bars show the interquartile range. Higher is better.

KKRunSeedTopics
206142entertainment, politics
216142business, entertainment
226142politics, tech
306142entertainment, politics, tech
316142business, entertainment, tech
326142entertainment, sport, tech
506142business, entertainment, politics, sport, tech
516143business, entertainment, politics, sport, tech
526144business, entertainment, politics, sport, tech

Table 1. BBC News. For each number of topics KK and each repetition, the classes were sampled from BBC News to compose the synthetic documents. Every document draws its dominant topic uniformly among the KK listed. When K=Kmax⁡K = K_{\max}, a single subset exists, so repetitions vary the corpus seed instead of the classes.

ModelARI K=2K = 2ARI K=3K = 3ARI K=5K = 5AMI K=2K = 2AMI K=3K = 3AMI K=5K = 5
mpnet
SentenceETM0.76 ±0.040.69 ±0.030.63 ±0.010.67 ±0.040.62 ±0.030.61 ±0.01
SentenceLDA0.52 ±0.120.58 ±0.030.54 ±0.070.50 ±0.080.54 ±0.020.52 ±0.05
senLDA0.52 ±0.080.43 ±0.110.46 ±0.070.45 ±0.060.39 ±0.100.45 ±0.05
SenClu0.26 ±0.220.25 ±0.120.23 ±0.040.25 ±0.200.26 ±0.110.26 ±0.04
BERTopic0.00 ±0.000.01 ±0.020.08 ±0.010.00 ±0.000.01 ±0.010.11 ±0.01
ETM0.01 ±0.010.03 ±0.020.15 ±0.040.01 ±0.010.03 ±0.010.15 ±0.04
LDA0.00 ±0.000.00 ±0.000.02 ±0.000.00 ±0.000.00 ±0.000.02 ±0.00
MiniLM
SentenceETM0.72 ±0.040.64 ±0.030.44 ±0.010.63 ±0.040.58 ±0.040.46 ±0.01
SentenceLDA0.02 ±0.010.04 ±0.010.12 ±0.030.11 ±0.010.19 ±0.030.34 ±0.03
senLDA0.52 ±0.080.43 ±0.110.46 ±0.070.45 ±0.060.39 ±0.100.45 ±0.05
SenClu0.39 ±0.160.26 ±0.130.25 ±0.060.37 ±0.140.26 ±0.130.27 ±0.04
BERTopic0.00 ±0.000.03 ±0.020.10 ±0.040.00 ±0.000.03 ±0.020.12 ±0.03
ETM0.01 ±0.010.03 ±0.020.15 ±0.040.01 ±0.010.03 ±0.010.15 ±0.04
LDA0.00 ±0.000.00 ±0.000.02 ±0.000.00 ±0.000.00 ±0.000.02 ±0.00

Table 5. BBC News: ARI and AMI. Results are reported as mean ± standard deviation over 3×33 \times3 runs. Bold the best results within each sentence encoder; underlined the overall best results.

Results on ElecDeb60to20

We illustrate SentenceETM ’s flexibility by applying it to a specific use case. We consider the first 2020 U.S. presidential debate between Donald Trump and Joe Biden, in which the moderator divided the discussion into six segments, and treat each speaking turn as a document and its sentences as the units to which topics are assigned. Because SentenceETM treats the sentence encoder as a separate, interchangeable component, the encoder can be chosen to suit the application. Here, we want topics to capture what is discussed rather than how or by whom: a sentence attacking an opponent’s record on the economy and one defending one’s own should fall under the same topic.

topic. We therefore use the instruction-following encoder Qwen3-Embedding-0.6B and prepend to every sentence the instruction “Given a sentence spoken in a U.S. presidential debate, identify the policy issue it discusses, regardless of the speaker’s stance or tone”. We consider K=20K = 20 topics and an extended stopword list that includes words specific to spoken debate.

Topics across debate segments. Figure 7 compares how the two candidates distribute their sentences over topics within each segment. For each segment and topic, we compute the share of each candidate’s sentences assigned to that topic and report the difference ΔP\Delta P (Trump − Biden); red cells indicate topics used more by Trump, blue cells indicate topics used more by Biden, and differences of at least 0.10 are annotated. The two topics below the dashed line capture generic conversational sentences. The largest differences reveal different emphases within the same segment. On race, Biden speaks more about racism towards African Americans (−0.12). On the economy, Biden focuses more on jobs and businesses (−0.19), while Trump discusses lockdowns (shut, country, close, +0.14). On election integrity, Biden is on voting and ballot counting (−0.15). In the COVID-19 segment, Biden devotes sentences to masks and vaccines (−0.13), while Trump emphasizes the timing of the response and comparisons with other countries (top-words in Appendix C.7).

Figure 7. Difference in probabilities of topic use between Trump and Biden across debate segments. ΔP=P(T∣S,Trump)−P(T∣S,Biden)\Delta P = P(T \mid S, \mathrm{Trump}) - P(T \mid S, \mathrm{Biden}).

Topic diversity within speaking turns. Figure 8 measures how often a candidate introduces new topics within a speaking turn. Within a turn, we say that a sentence introduces a new topic if its topic has not appeared in any previous sentence of that turn. The first sentence of a turn is excluded, since it cannot introduce a new topic. The index is the fraction of the remaining sentences that introduce a new topic, computed over all the turns of a candidate in a segment: ∑d(T(d)−1)∑d(M(d)−1)\frac{\sum_d \left(T^{(d)}-1\right)}{\sum_d \left(M^{(d)}-1\right)}, where M(d)M^{(d)} is the number of sentences of turn dd and T(d)T^{(d)} the number of distinct topics it contains. The index is 0 when every turn stays on a single topic and 1 when every sentence introduces a new topic. Computed over all the turns of the debate, the index is almost identical for the two candidates (0.45 for Trump, 0.43 for Biden): around two sentences out of five introduce a new topic. We use this value as each candidate’s average, and the figure reports, for each segment, the difference between the candidate’s index in that segment and their average. A positive value means that, in that segment, the candidate introduces new topics more often than usual; a negative value indicates lower within-turn topic diversity. Differences are larger across segments than between speakers. COVID-19 is the least cohesive segment for both candidates (+0.15 for Trump, +0.26 for Biden). Election integrity is the most cohesive segment for Biden (−0.20), whose turns stay focused on voting, while Trump’s most cohesive segment is the candidates’ records (−0.10).

Figure 8. New topics per additional sentence within speaking turns, by segment and candidate, as a difference from each candidate’s average. Positive values: more topics than usual; negative values: fewer.

Conclusion

We introduced SentenceETM, a sentence-level topic model that combines lexical evidence from topic–word distributions with contextual evidence from topic-specific low-dimensional subspaces of the sentence-embedding space. Document-level topic proportions act as a prior, and Gaussian parameters are updated in closed form. We also proposed a synthetic corpus protocol in which each sentence has a known topic. SentenceETM achieves the best ARI and AMI in almost all settings and remains robust to a smaller sentence encoder, unlike SentenceLDA. In a U.S. presidential debate, it reveals differences in topic emphasis between candidates and within-turn topic diversity. Future work includes modeling sentence order and topic transitions, and complementing synthetic labels with human annotations.

AI use statement

In this work, we used generative AI tools for tasks requiring disclosure, including: method implementation, by drafting portions of the code, assisting with notebooks, and helping integrate existing state-of-the-art implementations; interpretation of results, by discussing experimental outcomes and possible explanations reported in the paper; qualitative analysis, by proposing short labels for the debate topics in Table 9. The model itself, including the use of high-dimensional Gaussian models for contextual evidence, was conceived by the authors. Generative AI was not used to generate synthetic data, formulate hypotheses or novel mathematical claims, or clean the datasets. All AI-assisted material was reviewed by the authors. All code was inspected and executed by the authors; all numerical results reported in the paper were obtained from experimental outputs rather than generated by AI. We take full responsibility for the final content of this work, including any text, claims, code, or other artifacts produced with the assistance of generative AI.

TopicLabelTop words
0Lockdownsshut, close, country, open, allow, state, democrats, destroy, wrong, throw
1Accusationsman, plan, lie, fraud, office, name, liar, run, wait, acknowledge
2Mail-in votingballot, vote, count, send, poll, mail, solicit, state, open, director
3Electric carscar, cheap, expensive, electric, safe, cost, standard, automobile, tiny, least
4Conversationalguess, wrong, hit, honest, suppose, hey, excuse, surprised, feel, plenty
5Achievementsjob, win, top, thank, proud, testify, set, every, happy, administration
6Foreign countrieschina, head, die, world, russia, india, eat, billion, ukraine, lunch
7Racismafrican, racist, americans, wing, police, community, close, group, racism, whistle
8Health caredrug, care, obamacare, healthcare, affordable, pre, existing, condition, act, million
9Mixeddeal, record, back, god, send, wait, extent, november, soon, period
10Economyjob, economy, create, business, economic, company, history, leave, million, recovery
11Climate and energybuild, plant, deal, paris, accord, energy, emit, hard, environment, problem
12Law enforcementlaw, enforcement, police, violence, appropriate, order, crime, support, percent, officer
13Election outcomeelection, vote, win, elect, outcome, run, determine, shut, left, party
14Forest firesforest, tree, management, city, tear, billion, stop, burn, major, else
15Mixedmilitary, school, bring, support, back, country, church, include, name, class
16Taxestax, dollar, billion, eliminate, pay, taxis, million, money, fortune, cut
17Deaths and suburbsdie, suburb, life, place, million, town, home, trouble, far, agree
18Supreme Courtcourt, justice, hold, leave, supreme, united, states, vote, term, elect
19COVID-19 measuresmask, vaccine, wear, social, speak, test, distancing, end, america, johnson

Table 9. Topics learned on the presidential debate. For each topic, the ten words with the highest probability under the learned topic–word distribution, and a short label assigned by inspecting them.

Proofs

This section provides the proofs supporting Section 2. We also provide algorithms summarizing both the generative process and the inference procedure.

ELBO decomposition

By the product rule,

p(W(d),E~(d),δ(d)∣Θ)=p(W(d),E~(d)∣Θ)p(δ(d)∣W(d),E~(d),Θ).p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta}) = p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)} \mid\boldsymbol{\Theta})p(\boldsymbol{\delta}^{(d)} \mid\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\Theta}).

Therefore, for every value of δ(d)\boldsymbol{\delta}^{(d)} at which the posterior is positive,

log⁡p(W(d),E~(d)∣Θ)=log⁡p(W(d),E~(d),δ(d)∣Θ)−log⁡p(δ(d)∣W(d),E~(d),Θ).\log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)} \mid\boldsymbol{\Theta}) = \log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta}) - \log p(\boldsymbol{\delta}^{(d)} \mid\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\Theta}).

Taking the expectation of both sides under qq leaves the left-hand side unchanged, as it does not depend on δ(d)\boldsymbol{\delta}^{(d)}:

log⁡p(W(d),E~(d)∣Θ)=Eq[log⁡p(W(d),E~(d),δ(d)∣Θ)]\log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)} \mid\boldsymbol{\Theta}) = \mathbb{E}_{q}\left[\log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta})\right]
−Eq[log⁡p(δ(d)∣W(d),E~(d),Θ)].- \mathbb{E}_{q}\left[\log p(\boldsymbol{\delta}^{(d)} \mid\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\Theta})\right].

Finally, adding and subtracting Eq[log⁡q(δ(d))]\mathbb{E}_{q}[\log q(\boldsymbol{\delta}^{(d)})] and regrouping,

log⁡p(W(d),E~(d)∣Θ)=Eq[log⁡p(W(d),E~(d),δ(d)∣Θ)q(δ(d))]+Eq[log⁡q(δ(d))p(δ(d)∣W(d),E~(d),Θ)]=L(q;Θ)+DKL(q(δ(d))∥p(δ(d)∣W(d),E~(d),Θ)).\begin{aligned} \log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)} \mid\boldsymbol{\Theta}) = \mathbb{E}_{q}\left[\log\frac{p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta})}{q(\boldsymbol{\delta}^{(d)})}\right] + \mathbb{E}_{q}\left[\log\frac{q(\boldsymbol{\delta}^{(d)})}{p(\boldsymbol{\delta}^{(d)} \mid\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\Theta})}\right] \\ &= \mathcal{L}(q;\boldsymbol{\Theta}) + D_{\mathrm{KL}}\left(q(\boldsymbol{\delta}^{(d)}) \mathbin{\|} p(\boldsymbol{\delta}^{(d)} \mid\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\Theta})\right). \end{aligned}

Derivation of eq:elbo

Recall from the ELBO decomposition that

L(q;Θ)=Eq[log⁡p(W(d),E~(d),δ(d)∣Θ)−log⁡q(δ(d))].\mathcal{L}(q;\boldsymbol{\Theta}) = \mathbb{E}_{q}\left[\log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta}) - \log q(\boldsymbol{\delta}^{(d)})\right].

Moreover,

log⁡p(W(d),E~(d),δ(d)∣Θ)=log⁡N(δ(d);0,IK)+∑m=1M(d)log⁡p(wm(d),e~m(d)∣θ(d)).\log p(\mathbf{W}^{(d)}, \widetilde{\mathbf{E}}^{(d)}, \boldsymbol{\delta}^{(d)} \mid\boldsymbol{\Theta}) = \log\mathcal{N}(\boldsymbol{\delta}^{(d)};0,I_K) + \sum_{m=1}^{M^{(d)}} \log p(\mathbf{w}_{m}^{(d)}, \widetilde{\mathbf{e}}_{m}^{(d)} \mid\boldsymbol{\theta}^{(d)}).

Substituting into L(q;Θ)\mathcal{L}(q;\boldsymbol{\Theta}), using the linearity of the expectation and the definition of the Kullback–Leibler divergence, DKL(q∥p)=Eq[log⁡q−log⁡p]D_{\mathrm{KL}}(q \mathbin{\|} p) = \mathbb{E}_{q}[\log q - \log p],

L(q;Θ)=Eq[log⁡N(δ(d);0,IK)+∑m=1M(d)log⁡p(wm(d),e~m(d)∣θ(d))−log⁡q(δ(d))]=Eq[∑m=1M(d)log⁡p(wm(d),e~m(d)∣θ(d))]+Eq[log⁡N(δ(d);0,IK)−log⁡q(δ(d))]=Eq[∑m=1M(d)log⁡p(wm(d),e~m(d)∣θ(d))]−DKL(q(δ(d))∥N(0,IK)),\begin{aligned} \mathcal{L}(q;\boldsymbol{\Theta}) = \mathbb{E}_{q}\left[\log\mathcal{N}(\boldsymbol{\delta}^{(d)};0,I_K) + \sum_{m=1}^{M^{(d)}} \log p(\mathbf{w}_{m}^{(d)}, \widetilde{\mathbf{e}}_{m}^{(d)} \mid\boldsymbol{\theta}^{(d)}) - \log q(\boldsymbol{\delta}^{(d)})\right] \\ &= \mathbb{E}_{q}\left[\sum_{m=1}^{M^{(d)}} \log p(\mathbf{w}_{m}^{(d)}, \widetilde{\mathbf{e}}_{m}^{(d)} \mid\boldsymbol{\theta}^{(d)})\right] + \mathbb{E}_{q}\left[\log\mathcal{N}(\boldsymbol{\delta}^{(d)};0,I_K) - \log q(\boldsymbol{\delta}^{(d)})\right] \\ &= \mathbb{E}_{q}\left[\sum_{m=1}^{M^{(d)}} \log p(\mathbf{w}_{m}^{(d)}, \widetilde{\mathbf{e}}_{m}^{(d)} \mid\boldsymbol{\theta}^{(d)})\right] - D_{\mathrm{KL}}\left(q(\boldsymbol{\delta}^{(d)}) \mathbin{\|} \mathcal{N}(0,I_K)\right), \end{aligned}

which, with q=qϕq = q_{\phi}, is (4).

Computing the contextual likelihood

The covariance of Equation (2) can be equivalently written as

Σk=Vkdiag⁡(λk1,…,λkhk)VkT+σk2(IE−VkVkT),\Sigma_k = \mathbf{V}_k \operatorname{diag}(\lambda_{k1},\ldots,\lambda_{kh_k})\mathbf{V}_k^{\mathsf{T}}+\sigma_k^2(\mathbf{I}_E-\mathbf{V}_k\mathbf{V}_k^{\mathsf{T}}),

since Vkdiag⁡(σk2,…,σk2)VkT=σk2VkVkT\mathbf{V}_k\operatorname{diag}(\sigma_k^2,\ldots,\sigma_k^2)\mathbf{V}_k^{\mathsf{T}}=\sigma_k^2\mathbf{V}_k\mathbf{V}_k^{\mathsf{T}}. As Vk\mathbf{V}_k has orthonormal columns, Σk\Sigma_k has eigenvalues λk1,…,λkhk\lambda_{k1},\ldots,\lambda_{kh_k} along the columns of Vk\mathbf{V}_k and σk2\sigma_k^2 along the E−hkE-h_k orthogonal directions. Hence

Σk−1=Vkdiag⁡(λk1−1,…,λkhk−1)VkT+σk−2(IE−VkVkT),\Sigma_k^{-1}=\mathbf{V}_k\operatorname{diag}(\lambda_{k1}^{-1},\ldots,\lambda_{kh_k}^{-1})\mathbf{V}_k^{\mathsf{T}}+\sigma_k^{-2}(\mathbf{I}_E-\mathbf{V}_k\mathbf{V}_k^{\mathsf{T}}),
log⁡∣Σk∣=∑j=1hklog⁡λkj+(E−hk)log⁡σk2.\log|\Sigma_k|=\sum_{j=1}^{h_k}\log\lambda_{kj}+(E-h_k)\log\sigma_k^2.

Let r=e~m(d)−μk\mathbf{r}=\tilde{\mathbf{e}}_m^{(d)}-\boldsymbol{\mu}_k and a=VkTr∈Rhk\mathbf{a}=\mathbf{V}_k^{\mathsf{T}}\mathbf{r}\in\mathbb{R}^{h_k} be the coordinates of r\mathbf{r} in the topic subspace. Since ∥r∥2=∥a∥2+∥(IE−VkVkT)r∥2\|\mathbf{r}\|^2=\|\mathbf{a}\|^2+\|(\mathbf{I}_E-\mathbf{V}_k\mathbf{V}_k^{\mathsf{T}})\mathbf{r}\|^2,

log⁡N(e~m(d);μk,Σk)=−12[Elog⁡2π+log⁡∣Σk∣+∑j=1hkaj2λkj+∥r∥2−∥a∥2σk2].\log\mathcal{N}(\tilde{\mathbf{e}}_m^{(d)};\boldsymbol{\mu}_k,\Sigma_k)=-\frac{1}{2}\left[E\log2\pi+\log|\Sigma_k|+\sum_{j=1}^{h_k}\frac{a_j^2}{\lambda_{kj}}+\frac{\|\mathbf{r}\|^2-\|\mathbf{a}\|^2}{\sigma_k^2}\right].

The third term measures the distance of the sentence from the topic mean within the topic subspace, weighted by the topic’s variance along its directions; the fourth term measures its distance from the subspace, weighted by the noise variance σk2\sigma_k^2. The cost is O(Ehk)O(Eh_k) per sentence and topic, and no E×EE\times E matrix is ever formed. In practice, we lower-bound λkj\lambda_{kj} and σk2\sigma_k^2 by a small constant to prevent degenerate topics.

Generative process

Algorithm 1  Generative process of SentenceETM
1:  for each topic k = 1, ..., K do
2:      β_k ← softmax(ρ^⊤α_k)
3:      Σ_k ← V_k diag(λ_{k1} − σ_k², ..., λ_{kh_k} − σ_k²)V_k^⊤ + σ_k²I_E
4:  end for
5:  for each document d = 1, ..., D do
6:      Draw δ^(d) ∼ N(0, I_K) and set θ^(d) ← softmax(δ^(d))
7:      for each sentence m = 1, ..., M^(d) do
8:          Draw the topic z_m^(d) ∼ Mult(1; θ^(d))
9:          Draw the embedding ẽ_m^(d) ∼ N(μ_{z_m^(d)}, Σ_{z_m^(d)})
10:         for each word n = 1, ..., N_m^(d) do
11:             Draw the word w_mn^(d) ∼ Mult(1; β_{z_m^(d)})
12:         end for
13:     end for
14: end for

Inference process

Algorithm 2 Inference in SentenceETM

Require: documents {W(d)}d=1D\{\mathbf{W}^{(d)}\}_{d=1}^{D} with sentence embeddings {em(d)}\{\mathbf{e}^{(d)}_m\}; number of topics KK; topic-specific subspace dimensions {hk}k=1K\{h_k\}_{k=1}^{K}; number of epochs TT

  1. e~m(d)←em(d)/∥em(d)∥\tilde{\mathbf{e}}^{(d)}_m \leftarrow\mathbf{e}^{(d)}_m / \lVert\mathbf{e}^{(d)}_m \rVert for every sentence

  2. Run KK-means with KK clusters on {e~m(d)}\{\tilde{\mathbf{e}}^{(d)}_m\}; set γmk(d)=1\gamma^{(d)}_{mk}=1 if sentence mm of dd is in cluster kk, 00 otherwise

  3. Initialize Θctx\Theta_{\mathrm{ctx}} from the hard assignments using the closed-form estimates in § 2.2; initialize ϕ\phi and Θlex\Theta_{\mathrm{lex}} at random

  4. for t=1,…,Tt=1,\ldots,T do

  5. &nbsp;&nbsp;&nbsp;&nbsp;for each mini-batch B\mathcal{B} of documents do

  6. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for d∈Bd \in\mathcal{B} do

  7. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;δ(d)←μϕ(u(d))+σϕ(u(d))⊙ϵ\boldsymbol{\delta}^{(d)} \leftarrow\boldsymbol{\mu}_{\phi}(\mathbf{u}^{(d)})+\boldsymbol{\sigma}_{\phi}(\mathbf{u}^{(d)})\odot\boldsymbol{\epsilon}, ϵ∼N(0,IK)\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},I_K)

  8. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;θ(d)←softmax⁡(δ(d))\boldsymbol{\theta}^{(d)}\leftarrow\operatorname{softmax}(\boldsymbol{\delta}^{(d)})

  9. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute Ld\mathcal{L}_d with Equation (4), summing out each zm(d)z^{(d)}_m in Equation (1)

  10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end for

  11. &nbsp;&nbsp;&nbsp;&nbsp;Update ϕ\phi and Θlex\Theta_{\mathrm{lex}} by a gradient step on ∑d∈BLd\sum_{d\in\mathcal{B}}\mathcal{L}_d ▷ Θctx\Theta_{\mathrm{ctx}} fixed

  12. &nbsp;&nbsp;&nbsp;&nbsp;end for

  13. &nbsp;&nbsp;&nbsp;&nbsp;θ(d)←softmax⁡(μϕ(u(d)))\boldsymbol{\theta}^{(d)}\leftarrow\operatorname{softmax}(\boldsymbol{\mu}_{\phi}(\mathbf{u}^{(d)})) for every document

  14. &nbsp;&nbsp;&nbsp;&nbsp;Compute γmk(d)\gamma^{(d)}_{mk} for every sentence

  15. &nbsp;&nbsp;&nbsp;&nbsp;Update Θctx\Theta_{\mathrm{ctx}} using the closed-form estimates in § 2.2 ▷ closed form

  16. end for

  17. return ϕ\phi, Θ\Theta, and the sentence topics arg max⁡kγmk(d)\operatorname*{arg\,max}_k\gamma^{(d)}_{mk}

Additional Materials to Section 3

Synthetic dataset construction pipeline

Synthetic corpora generation

Figure 9. Synthetic corpora generation. Example with K=3K=3, L=10L=10 and p=0.6p=0.6.

The construction of a synthetic corpus proceeds in four steps, repeated independently for every run.

  1. As detailed in § 3, we start with existing topic modeling datasets that provide a single topic label per document: BBC News (Greene and Cunningham, [11]), 20 Newsgroups4, and Wikipedia (Merity et al., [17]). Specifically,

(a) BBC News dataset is a corpus consisting of 2.225 articles published on the BBC News website corresponding to 2004-2005 (in English). Each article is labeled under one of 5 categories: business, entertainment, politics, sport or tech.

(b) 20 Newsgroups dataset comprises around 18.000 newsgroups posts on 20 topics (in English): rec.sport.hockey, comp.sys.ibm.pc.hardware, talk.politics.mideast, comp.sys.mac.hardware, sci.electronics, talk.religion.misc, sci.crypt, sci.med, alt.atheism, rec.motorcycles, rec.autos, comp.windows.x, comp.graphics, sci.space, talk.politics.guns, misc.forsale, rec.sport.baseball, talk.politics.misc, comp.os.ms-windows.misc, soc.religion.christian.

(c) We use the training split of the Wikipedia corpus from Hoyle et al. [13], derived from WikiText-103 ([17]). This dataset is a general-audience corpus consisting of 14,000 English Wikipedia articles. We use the articles’ supercategory labels (15): Language and literature, Natural sciences, Art and architecture, Music, Engineering and technology, Media and drama, History, Warfare, Sports and recreation, Geography and places, Social sciences and society, Philosophy and religion, Video games, Agriculture, food, and drink, Mathematics.

  1. For a given KK, we draw KK distinct topic labels from those available; each of the three runs uses a different subset. How KK is chosen is discussed in § 3.

  1. For each of the nn documents, we draw the dominant topic uniformly among the KK selected labels. We then sample one source article of that topic among those with at least m1m_1 sentences, and take m1m_1 of its sentences. The same is done for each of the K−1K - 1 non-dominant topics, taking m2m_2 sentences from an article with at least m2m_2 of them. Sentences are drawn without replacement within an article and keep their original order; source articles are reused across documents only when the pool is smaller than nn. The drawn sentences are finally concatenated by topic, so a document consists of KK contiguous blocks.

  1. Data preprocessing is applied on the resulting corpus as described in Section B.1.1.

In Table 1 to Table 3 we provide the topic compositions of the synthetic corpora for each of the runs.

KKRunSeedTopics
206142Mathematics; Music
216142Language and literature; Music
226142Agriculture, food, and drink; Language and literature
306142Language and literature; Music; Warfare
316142Agriculture, food, and drink; Engineering and technology; Language and literature
326142Language and literature; Social sciences and society; Video games
506142History; Language and literature; Mathematics; Music; Sports and recreation
516142Language and literature; Media and drama; Philosophy and religion; Social sciences and society; Sports and recreation
526142Art and architecture; History; Philosophy and religion; Social sciences and society; Warfare
906142Agriculture, food, and drink; Engineering and technology; Geography and places; History; Language and literature; Mathematics; Music; Natural sciences; Social sciences and society
916142Art and architecture; Geography and places; Language and literature; Mathematics; Media and drama; Music; Social sciences and society; Video games; Warfare
926142Agriculture, food, and drink; Art and architecture; Geography and places; History; Language and literature; Mathematics; Music; Sports and recreation; Warfare

Table 3. Wikipedia. For each number of topics KK and each repetition, the classes were sampled from Wikipedia to compose the synthetic documents. Every document draws its dominant topic uniformly among the KK listed. When K=Kmax⁡K = K_{\max}, a single subset exists, so repetitions vary the corpus seed instead of the classes.

Preprocessing

Each document is represented as (i) a bag-of-words over a fixed vocabulary and (ii) a sequence of sentences.

Lexical view. We follow standard ETM preprocessing ([9]). Documents are lemmatized with en_core_web_lg; NLTK stop words, generic verbs and modals, punctuation, non-alphabetic tokens, and tokens shorter than three characters are removed. We also discard terms occurring in more than τmax⁡=0.2\tau_{\max} = 0.2 of the documents or in only one document. The resulting lemmas are intersected with the lowercase alphabetic GloVe vocabulary, retaining the V=15000V = 15000 most frequent words. Thus, all columns of ρ∈Rp×V\boldsymbol{\rho} \in\mathbb{R}^{p \times V} (p=300p = 300) are initialized

KKRunSeedTopics
206142rec.motorcycles, sci.crypt
216142misc.forsale, sci.crypt
226142alt.atheism, rec.autos
306142rec.autos, sci.crypt, talk.religion.misc
316142alt.atheism, comp.sys.mac.hardware, rec.autos
326142rec.motorcycles, soc.religion.christian, talk.politics.mideast
506142misc.forsale, rec.autos, rec.sport.baseball, sci.crypt, talk.politics.mideast
516142rec.autos, rec.motorcycles, rec.sport.baseball, soc.religion.christian, talk.politics.guns
526142comp.os.ms-windows.misc, misc.forsale, sci.space, talk.politics.guns, talk.politics.misc
906142alt.atheism, comp.sys.ibm.pc.hardware, comp.windows.x, rec.autos, rec.sport.baseball, sci.med, sci.space, soc.religion.christian, talk.politics.mideast
916142comp.graphics, comp.sys.mac.hardware, comp.windows.x, rec.motorcycles, rec.sport.hockey, sci.crypt, sci.med, soc.religion.christian, talk.politics.misc
926142alt.atheism, comp.os.ms-windows.misc, comp.sys.mac.hardware, misc.forsale, rec.motorcycles, sci.med, talk.politics.guns, talk.politics.mideast, talk.politics.misc

Table 2. 20 Newsgroups. For each number of topics KK and each repetition, the classes were sampled from 20 Newsgroups to compose the synthetic documents. Every document draws its dominant topic uniformly among the KK listed. When K=Kmax⁡K = K_{\max}, a single subset exists, so repetitions vary the corpus seed instead of the classes.

with pre-trained GloVe vectors and fine-tuned during training. Documents with fewer than five in-vocabulary tokens are discarded, and counts are normalized before being fed to the inference network.

Contextual view. Raw documents are segmented with the Punkt tokenizer as w(d)=(w1(d),…,wM(d)(d))\mathbf{w}^{(d)} = (\mathbf{w}^{(d)}_1, \ldots, \mathbf{w}^{(d)}_{M^{(d)}}); if segmentation returns no sentence, the whole document is used as one sentence. Each sentence has a discrete bag-of-words representation over the vocabulary above, used by the lexical evidence, and a continuous representation obtained by encoding the raw sentence with a pre-trained encoder (all-mpnet-base-v2 and all-MiniLM-L6-v2 for the synthetic and real corpora, and Qwen3-Embedding-0.6B for the debate). The resulting embedding em(d)\mathbf{e}^{(d)}_m is ℓ2\ell_2-normalized, e~m(d)=em(d)/∥em(d)∥\tilde{\mathbf{e}}^{(d)}_m = \mathbf{e}^{(d)}_m / \lVert\mathbf{e}^{(d)}_m \rVert, and used as contextual evidence (Section 2). Embeddings are computed once and cached; the encoders are not fine-tuned.

ElecDeb60to20

The organization of the debate is indicated by the journalist at the beginning of the debate:

“Good evening from the Health Education Campus of Case Western Reserve University and the Cleveland Clinic. I’m Chris Wallace of Fox News and I welcome you to the first of the 2020 presidential debates between President Donald J. Trump and former Vice President Joe Biden. This debate is sponsored by the Commission on Presidential Debates. The Commission has designed the format, six roughly 15-minute segments with two-minute answers from each candidate to the first question, then open discussion for the rest of each segment. Both campaigns have agreed to these rules. For the record, I decided the topics and the questions in each topic. I can assure you none of the questions has been shared with the Commission or the two candidates. This debate is being conducted under health and safety protocols designed by the Cleveland Clinic, which is serving as the health security advisor to the Commission for all four debates. As a precaution, both campaigns have agreed the candidates will not shake hands at the beginning of tonight’s debate. The audience here in the hall has promised to remain silent. No cheers, no boos, or other interruptions so we, and more importantly you, can focus on what the candidates have to say. No noise except right now, as we welcome the Republican nominee, President Trump, and the Democratic nominee Vice President Biden.”

Overall, the segments present the following topics:

  1. Index 4 – 146: Supreme Court / Healthcare / Obamacare / Filibuster / Court packing.

  1. Index 147 – 272: COVID19 / Vaccine / Scientists / Reopening schools and economy / Masks / Disease spread at events.

  1. Index 273 – 475: Economy / Unemployment rate / Recovery / Taxes / Economic plan.

  1. Index 476 – 626: Race issues / Racial sensitivity training / Law and Order / Black Lives Matter / Protests.

  1. Index 627 – 779: Trump and Biden records / Climate change / Global warming / Green New Deal.

  1. Index 780 – 853: Election integrity / Mail-in voting.

Model training

Sentence embeddings are computed once with a frozen encoder: all-mpnet-base-v2 in the main experiments and all-MiniLM-L6-v2 in the encoder comparison. For models using word embeddings, ρ\boldsymbol{\rho} is initialized with GloVe and fine-tuned. SentenceETM is trained for 300 epochs with Adam (learning rate 2⋅10−32 \cdot10^{-3}, weight decay 1.2⋅10−61.2 \cdot10^{-6}, gradient clipping 2.0) and batches of 64 documents. The inference network has hidden size 256, topic embeddings have dimension 300, the topic subspace dimension is (for all the topics) hk=32h_k = 32, and contextual evidence is weighted by c=0.5c = 0.5. Topic Gaussians are initialized by k-means (10 restarts) on normalized sentence embeddings and re-estimated in closed form after each epoch, rather than optimized by Adam. Eigenvalues and noise variances are lower-bounded by 10−610^{-6}. For the baselines, we use the hyperparameters from the original implementations, sharing the batch size and learning rate across neural models. ETM [9] is trained for 150 epochs with hidden size 800, no encoder dropout, and weight decay 1.2⋅10−61.2 \cdot10^{-6}. LDA [4] uses 201 batch iterations; senLDA [1], 201 Gibbs sweeps with α=β=1/K\alpha= \beta= 1/K; and SentenceLDA ([7]), 20 sweeps with α=κ=0.1\alpha= \kappa= 0.1. For the Gibbs samplers, we use the final state without averaging, as in the original implementations. BERTopic ([12]) uses UMAP with 15 neighbors, 5 components, and cosine distance, is constrained to KK topics, and reassigns outliers by embedding similarity. Each configuration is repeated 3 times, varying source classes or, when only one class subset is available, the sampled documents and sentences. Each model is trained with 3 seeds per repetition.

Metrics

We measure assignment quality against the ground-truth sentence topics with the Adjusted Mutual Information ([21]) and the Adjusted Rand Index ([14]), both corrected for chance. Since every sentence has a known topic, these metrics directly evaluate the quantity that the models infer, without resorting to proxies for topic quality.

Computational cost

The cost of SentenceETM has three parts. Let S=∑dM(d)S = \sum_d M^{(d)} be the number of sentences in the corpus.

(i) Encoding. Each sentence is encoded once by the frozen encoder, and the embeddings are cached. This cost is shared with every method that relies on sentence embeddings; with all-mpnet-base-v2, encoding the 14,000 sentences of a synthetic corpus takes about 30 seconds (≈2\approx2 ms per sentence).

(ii) Gradient steps. At each epoch, the inference network costs O(DVH)O(DVH), with HH its hidden size; the lexical evidence costs O(KVp)O(KVp) to compute β\boldsymbol{\beta} plus one term per word token and topic; the contextual evidence costs O(SE∑k=1Khk)O(SE\sum_{k=1}^{K} h_k), which reduces to O(SKEh)O(SKEh) when hk=hh_k = h for all kk.

(iii) Closed-form update. Once per epoch, the topic Gaussians are re-estimated from the responsibilities, which requires O(SKE2)O(SKE^2) operations to accumulate the covariances and O(KE3)O(KE^3) for their eigendecompositions. This step does not depend on the vocabulary and grows linearly with SS and KK.

Table 4 reports the training time of all models on all the datasets. On average over KK, SentenceETM trains in 340 to 391 seconds per run, against 1717 to 3118 seconds for SentenceLDA, the other sentence-level model that uses sentence embeddings whose Gibbs sampler must update every sentence assignment sequentially. It is slower than the document-level models and senLDA, which use neither embeddings nor a per-sentence posterior over KK Gaussians. Its cost grows linearly with KK. Most of this time is spent in the closed-form update, which our implementation runs on the CPU; it could be moved to the GPU or performed every few epochs. Once the model is trained, assigning a topic to a new sentence requires one forward pass of the inference network and the evaluation of KK Gaussian densities.

Additional Materials to Section 4

Numerical results

Tables 5 to 7 report its ARI and AMI on the synthetic corpora. The tables also present the results for SenClu ([20]), which assigns sentences to topics by hard expectation-maximization over sentence embeddings, weighted by document-level topic proportions. We use the provided official implementation with its default hyperparameters (40 epochs, α=1/K\alpha= 1/\sqrt{K}) and feed it the same sentences and sentence embeddings as the other models. SentenceETM outperforms it in every setting. SenClu varies strongly across runs (standard deviation up to 0.32 ARI on Wikipedia), which we attribute to its random initialization of the topic centroids.

Structure of the latent space

Figures 4 and 10 show, for one run per dataset, the sentences projected onto the first two principal components of their embeddings and colored by the topic assigned by each model. The three most probable words of each topic are listed below each panel. With SentenceETM, sentences assigned to the same topic largely form distinct regions of the embedding space, and the top words identify the corresponding classes (e.g., bank, economy, rate for Business in BBC News, church, jesus, bible for Christianity in 20 Newsgroups, music, chart, video for Music in Wikipedia). In contrast, BERTopic and ETM produce substantially more mixed assignments. BERTopic assigns one topic per document, so every sentence inherits its document’s topic regardless of its content; ETM assigns sentence topics from the words of document-level topics. Their top words reveal further limitations: BERTopic produces topics that mix or duplicate classes (e.g., game, season, team and game, league, play on Wikipedia), while ETM topics are often dominated by rare, specific words (Ise, boerse, gerrad; tercel, diesel, mustang; chili, doughnut, steak). Finally, we notice that SentenceETM on 20 Newsgroups, the topic matched to Atheism is dominated by conversational words (list, answer, sure), while the one matched to X window has uninformative top words and few sentences, consistent with the lower scores on this dataset. In Appendix § 8.6 we provide some examples of the misassigned sentences.

DatasetKKSentenceETMSentenceLDAsenLDASenCluBERTopicETMLDA
BBC News22811946861.61.64.326
BBC News33281391871.91.64.423
BBC News54101815882.31.74.624
BBC NewsAvg.3401717871.91.64.425
20 Newsgroups22101913811.21.54.022
20 Newsgroups32552116811.91.54.121
20 Newsgroups53351817822.01.54.120
20 Newsgroups94975309822.71.54.219
20 NewsgroupsAvg.3242789822.01.54.121
Wikipedia22682308851.81.53.726
Wikipedia33282304841.81.53.824
Wikipedia54132684842.21.53.923
Wikipedia95545176882.21.44.123
WikipediaAvg.3913118852.01.53.924

Table 4. Training time in seconds (median over 9 runs; 700 documents, 14,000 sentences), excluding corpus construction and sentence encoding. All experiments were run on NVIDIA H100 GPUs. Neural models (SentenceETM, ETM, BERTopic, SenClu) run on a GPU; LDA, senLDA, and SentenceLDA, whose released implementations are CPU-only, run on CPU.

Sensitivity to the sentence encoder

Replacing all-mpnet-base-v2 with the smaller all-MiniLM-L6-v2 affects the embedding-based sentence models very differently. For SentenceETM, the drop is at most 0.05 in both ARI and AMI, except on BBC News at K=5K = 5 (0.19 ARI and 0.15 AMI), where its ARI falls slightly below that of senLDA. SentenceLDA, in contrast, degrades sharply: its ARI falls to at most 0.16 in every setting (e.g., from 0.52 to 0.02 on BBC News at K=2K = 2), and its AMI decreases by up to 0.39. With all-MiniLM-L6-v2, SentenceLDA obtains lower ARI than senLDA, which does not use embeddings, in every setting. This degradation cannot be attributed solely to the weaker embeddings: SenClu, using the same all-MiniLM-L6-v2 embeddings, loses at most 0.06 ARI relative to all-mpnet-base-v2 and even improves in most settings. A possible explanation lies in how the models represent topics in the embedding space. SentenceLDA uses full-covariance Gaussians, making its clustering depend on the geometry of the entire space. In SentenceETM, each covariance is instead constrained to a few principal directions plus isotropic noise and re-estimated from soft responsibilities; this parsimonious representation may be less sensitive to directions carrying little topical information. Note that senLDA, ETM, and LDA do not use the sentence encoder, so their results are identical for both encoders (cf. Table 5 to Table 7).

ModelARI K=2K=2ARI K=3K=3ARI K=5K=5ARI K=9K=9AMI K=2K=2AMI K=3K=3AMI K=5K=5AMI K=9K=9
mpnet — SentenceETM0.69 ±0.130.57 ±0.080.45 ±0.150.47 ±0.040.61 ±0.150.55 ±0.090.47 ±0.150.55 ±0.02
mpnet — SentenceLDA0.44 ±0.110.41 ±0.070.38 ±0.150.45 ±0.050.45 ±0.090.48 ±0.090.42 ±0.150.50 ±0.04
mpnet — senLDA0.53 ±0.190.47 ±0.080.31 ±0.090.29 ±0.040.46 ±0.180.43 ±0.080.34 ±0.090.38 ±0.03
mpnet — SenClu0.37 ±0.320.40 ±0.160.22 ±0.150.27 ±0.040.34 ±0.280.38 ±0.150.23 ±0.150.34 ±0.03
mpnet — BERTopic0.00 ±0.000.00 ±0.000.04 ±0.020.11 ±0.050.00 ±0.000.00 ±0.000.08 ±0.010.18 ±0.03
mpnet — ETM0.02 ±0.020.17 ±0.060.14 ±0.070.20 ±0.020.02 ±0.020.16 ±0.050.15 ±0.060.24 ±0.01
mpnet — LDA0.00 ±0.000.01 ±0.010.02 ±0.010.03 ±0.010.00 ±0.000.01 ±0.010.03 ±0.020.05 ±0.01
MiniLM — SentenceETM0.67 ±0.140.57 ±0.100.43 ±0.150.44 ±0.050.60 ±0.150.53 ±0.110.45 ±0.140.51 ±0.02
MiniLM — SentenceLDA0.03 ±0.020.10 ±0.050.11 ±0.010.16 ±0.040.11 ±0.030.27 ±0.060.29 ±0.050.38 ±0.03
MiniLM — senLDA0.53 ±0.190.47 ±0.080.31 ±0.090.29 ±0.040.46 ±0.180.43 ±0.080.34 ±0.090.38 ±0.03
MiniLM — SenClu0.52 ±0.300.46 ±0.180.27 ±0.150.31 ±0.030.46 ±0.280.42 ±0.160.28 ±0.150.38 ±0.02
MiniLM — BERTopic0.00 ±0.000.00 ±0.000.03 ±0.010.13 ±0.040.00 ±0.000.00 ±0.000.08 ±0.010.19 ±0.03
MiniLM — ETM0.02 ±0.020.17 ±0.060.14 ±0.070.20 ±0.020.02 ±0.020.16 ±0.050.15 ±0.060.24 ±0.01
MiniLM — LDA0.00 ±0.000.01 ±0.010.02 ±0.010.03 ±0.010.00 ±0.000.01 ±0.010.03 ±0.020.05 ±0.01

Table 7. Wikipedia: ARI and AMI. Results are reported as mean ± standard deviation over 3×33 \times3 runs. Bold the best results within each sentence encoder; _underlined_ the overall best results.

Sensitivity to document length

We vary the number of sentences per document M(d)∈{20,40,60,80}M^{(d)} \in\{20, 40, 60, 80\} on Wikipedia with K=9K = 9 and

ModelARI K=2K=2ARI K=3K=3ARI K=5K=5ARI K=9K=9AMI K=2K=2AMI K=3K=3AMI K=5K=5AMI K=9K=9
mpnet — SentenceETM0.56 ±0.050.46 ±0.030.35 ±0.040.27 ±0.020.48 ±0.030.44 ±0.020.41 ±0.050.37 ±0.04
mpnet — SentenceLDA0.16 ±0.040.23 ±0.020.30 ±0.070.25 ±0.030.26 ±0.030.37 ±0.010.39 ±0.050.33 ±0.03
mpnet — senLDA0.22 ±0.130.30 ±0.020.21 ±0.030.17 ±0.050.18 ±0.080.26 ±0.010.22 ±0.030.21 ±0.05
mpnet — SenClu0.36 ±0.190.35 ±0.070.28 ±0.030.23 ±0.040.32 ±0.160.33 ±0.060.33 ±0.030.31 ±0.05
mpnet — BERTopic0.00 ±0.000.02 ±0.030.09 ±0.020.06 ±0.080.00 ±0.000.03 ±0.030.12 ±0.020.10 ±0.10
mpnet — ETM0.02 ±0.020.12 ±0.020.11 ±0.010.11 ±0.030.02 ±0.020.11 ±0.020.12 ±0.020.15 ±0.04
mpnet — LDA0.00 ±0.000.01 ±0.010.01 ±0.000.02 ±0.010.00 ±0.000.01 ±0.010.02 ±0.010.03 ±0.01
MiniLM — SentenceETM0.54 ±0.040.43 ±0.020.32 ±0.010.25 ±0.030.46 ±0.020.41 ±0.020.36 ±0.030.34 ±0.04
MiniLM — SentenceLDA0.01 ±0.000.02 ±0.010.03 ±0.010.03 ±0.010.07 ±0.000.13 ±0.020.22 ±0.030.25 ±0.04
MiniLM — senLDA0.22 ±0.130.30 ±0.020.21 ±0.030.17 ±0.050.18 ±0.080.26 ±0.010.22 ±0.030.21 ±0.05
MiniLM — SenClu0.40 ±0.080.29 ±0.040.25 ±0.050.20 ±0.040.33 ±0.070.28 ±0.020.28 ±0.040.28 ±0.04
MiniLM — BERTopic0.00 ±0.000.02 ±0.030.07 ±0.060.06 ±0.080.00 ±0.000.03 ±0.030.09 ±0.070.10 ±0.09
MiniLM — ETM0.02 ±0.020.12 ±0.020.11 ±0.010.11 ±0.030.02 ±0.020.11 ±0.020.12 ±0.020.15 ±0.04
MiniLM — LDA0.00 ±0.000.01 ±0.010.01 ±0.000.02 ±0.010.00 ±0.000.01 ±0.010.02 ±0.010.03 ±0.01

Table 6. 20 Newsgroup: ARI and AMI. Results are reported as mean ± standard deviation over 3×33 \times3 runs. Bold the best results within each sentence encoder; _underlined_ the overall best results.

all-mpnet-base-v2, while decreasing the number of documents accordingly (700, 350, 233, and 175), so that each corpus contains approximately 14,000 sentences. All other settings are the same as in the main experiments.

As shown in Figure 11, SentenceETM obtains the highest ARI and AMI at every document length, while remaining stable across the four settings: ARI ranges from 0.46 to 0.49 and AMI from 0.54 to 0.56, with differences within one standard deviation. The other sentence-level models are similarly stable but consistently lower, with SentenceLDA ranging from 0.42 to 0.45 ARI and senLDA remaining around 0.30. Among the document-level models, ETM gradually degrades as document length increases (ARI from 0.18 to 0.14), BERTopic fluctuates between 0.08 and 0.12, and LDA remains close to zero. These results suggest that sentence-level assignments are largely insensitive to document length in this setting, even when the same amount of sentence-level information is distributed across fewer, longer documents.

Figure 11. Sensitivity to document length (Wikipedia, K=9K = 9, all-mpnet-base-v2). ARI (left) and AMI (right) as a function of the number of sentences per document M(d)M^{(d)} (3×33 \times3 runs).

Results on original corpora

We evaluate the lexical quality of the learned topics (10 top-words) through Topic Coherence (TC) with NPMI [18, 16] and Topic Diversity (TD) [9]. We use the topics learned in the same runs as the synthetic-corpus experiments; no model is retrained on the original datasets. For each run, we extract the top words of the learned topics and evaluate them against the corresponding full original corpus. This choice avoids measuring coherence on the synthetic documents used for training, whose co-occurrence statistics are partly induced by our construction procedure: each synthetic document deliberately combines sentences drawn from different source classes, and words from otherwise unrelated topics may therefore co-occur simply because their sentences were placed in the same document. Computing TC on these documents could consequently reward topic-word sets that reflect the synthetic mixing procedure rather than genuine associations in the source corpus. Evaluating the same learned top words on the original corpus instead measures whether they correspond to word associations that occur naturally in the underlying data. TD does not require a reference corpus, since it depends only on the overlap among the top words of the learned topics. We therefore compute it directly from the topics obtained in each run. Figure 12 shows that the sentence-level models produce consistently coherent and diverse topics across datasets. SentenceETM maintains high TC on BBC News, 20 Newsgroups, and Wikipedia, with comparatively limited variation across datasets. SentenceLDA and senLDA show a similarly stable behavior, whereas the document-level baselines are substantially more dataset-dependent: in particular, ETM ranges from low or negative coherence on BBC News and 20 Newsgroups to among the highest values on Wikipedia. BERTopic and LDA exhibit similar, although less pronounced, variations. TD is close to one for most methods and configurations, especially for the sentence-level models, indicating little overlap among their top words. It therefore provides less discrimination than TC in this setting; importantly, SentenceETM combines this high diversity with consistently high coherence.

Figure 12. Topic coherence (TC) and topic diversity (TD) on the original corpora. Topics are those learned in the synthetic-corpus experiments, while TC is evaluated using word co-occurrences from the corresponding full original dataset. TD is computed directly from the learned top words. Higher is better.

Examples of misassigned sentences

In Table 8, we randomly select eight sentences assigned to an incorrect topic by SentenceETM for each dataset, using the same runs as in Figure 4 and Figure 10. Most errors fall into three groups. First, many misassigned sentences contain little topical information: generic statements, reported speech, or discourse-like utterances whose label is inherited from the source document but cannot be reliably inferred from the sentence alone (e.g., “So to win something like this really feels great”, “I won’t argue too much about anything”). In such cases, the predicted topic often reflects cues in the wording itself, such as win suggesting sport or an argumentative tone suggesting a religion-related forum. Second, several sentences are semantically closer to the predicted topic than to the label of their source document. For example, a business article on Turkish firms excluded for national-security reasons discusses politics; an alt.atheism post mentioning Soviet and French tanks discusses the Middle East conflict; and Wikipedia sentences mentioning boxing or cricket naturally evoke sport. These cases highlight the noise introduced by propagating document-level labels to individual sentences. Third, some disagreements involve inherently overlapping categories, such as Warfare and History, or Geography and places and Art and architecture for a sentence describing a historical house. Only a few cases lack a clear semantic explanation, such as a sentence on the history of Calakmul assigned to Language and literature or a bibliographic reference assigned to Geography and places.

Figure 10. Sentence-level topic assignments (all-mpnet-base-v2). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel. Wikipedia is in Figure 4.

True → PredictedSentence
BBC News
Entertainment →\to SportThere were no specific details on the injuries.
Business →\to TechIt will do so at a presentation in New York on 26 January.
Business →\to Politics“I will give up my role as soon as the board of directors asks me to do so,” he said.
Business →\to PoliticsIn both cases, the justification has been national security, amid allegations that the Turkish firms are too close to Israel.
Business →\to Politics“First and foremost this is a human tragedy.”
Tech →\to BusinessTaipei’s TFC 101 Tower is more than 50m taller than the Petronas Towers in Kuala Lumpur, Malaysia—formerly the world’s tallest skyscraper.
Entertainment →\to Sport“So to win something like this really feels great.”
Business →\to TechIt is the second time Vodafone has come under the Commission’s scrutiny.
20 Newsgroups
alt.atheism →\to soc.religion.christianWhen did I acquire lack of belief in the Easter Bunny?
alt.atheism →\to talk.politics.mideastWe liberated it from Spain.
rec.sport.baseball →\to sci.med(Not to mention the possibility that no organ is particularly crucial in some third species.)
comp.windows.x →\to soc.religion.christianCan anyone shoot obvious holes in this theory?
alt.atheism →\to soc.religion.christianI, for one, don’t regret it.
alt.atheism →\to talk.politics.mideast(He ignores the challenge that extreemists for peace tend to be quite insistent that everyone accept their ideals for the world, and have even turned quite violent.)
rec.autos →\to soc.religion.christianI won’t argue too much about anything.
alt.atheism →\to talk.politics.mideastTheir tanks are almost all soviet, with about 100 French tanks (older ones).
Wikipedia
Language and lit. →\to MusicTokyopop licensed the series in North America, and released the first two volumes in 2007.
Warfare →\to HistoryDevereux became adjutant to Illinois National Guard Maj. Simon Bolivar Buckner, later a Confederate general during the Civil War and eventually a governor of Kentucky.
History →\to Sports and recr.A friend of English captain Douglas Jardine from their Oxford days, Kent Hughes publicly defended Jardine’s tactics of sustained short pitched bowling against the Australian batsmen, arguing that Australia used similar tactics against England during the 1921 Ashes tour.
History →\to Language and lit.The early history of Calakmul is obscure, although a dynastic list has been pieced together that extends back into an ancestral past.
Geography →\to Art and arch.The original house was constructed about 1696 by William Morgan, around the time of his marriage to Margaret Avery.
Mathematics →\to GeographyStevens, Peter S. Patterns in Nature.
Language and lit. →\to Sports and recr.She later became a Distinguished Alumna of the University of North Carolina and the University of Iowa.
Sports and recr. →\to HistoryThe delegation was also accompanied by the head boxing coach Lolo Heimuli.

Table 8. Examples of topic disagreements with the source labels. We report the topic inherited by the sentence from its source document and the predicted topic (SentenceETM). The misassignments are on generic sentences, cases where the predicted topic better matches the sentence content, and overlaps between categories.

Top-words in ElecDeb60to20

Table 9 lists the twenty topics learned on the ElecDeb60to20 debate between Biden and Trump on 29/10/2020. Most of them correspond to identifiable themes of the discussion, often more specific than the moderator’s segments: within the COVID-19 segment, the model separates preventive measures (topic 19) from lockdowns (topic 0); within the candidates’ records, it isolates climate and energy (topic 11), forest fires (topic 14) and electric cars (topic 3). A second group of topics captures the register rather than the subject of the debate: conversational interjections (topic 4), personal accusations (topic 1), and claims of achievement (topic 5). These topics are a consequence of modeling individual sentences: in a debate, many sentences carry no policy content, and the model groups them into topics of their own rather than spreading them across the thematic ones. Finally, two topics (9 and 15) contain unrelated words and lack a clear interpretation.

References

  1. [1]Balikas, G., Amini, M.-R., and Clausel, M. (2016). On a topic model for sentences. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 921–924.arxiv.org/abs/1606.00253
  2. [2]Bianchi, F., Terragni, S., and Hovy, D. (2021a). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 759–766.
  3. [4]Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022.
  4. [5]Bouveyron, C., Girard, S., and Schmid, C. (2007). High-dimensional data clustering. Computational statistics & data analysis, 52(1):502–519.DOI
  5. [6]Boyd-Graber, J. and Blei, D. (2008). Syntactic topic models. Advances in neural information processing systems, 21.
  6. [7]Cha, T. and Lee, D. (2024). SentenceLDA: Discriminative and robust document representation with sentence level topic model. In Graham, Y. and Purver, M., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta. Association for Computational Linguistics.
  7. [8]Das, R., Zaheer, M., and Dyer, C. (2015). Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 795–804.DOI
  8. [9]Dieng, A. B., Ruiz, F. J., and Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453.
  9. [11]Greene, D. and Cunningham, P. (2006). Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd international conference on Machine learning, pages 377–384.DOI
  10. [12]Grootendorst, M. (2022). Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.arxiv.org/abs/2203.05794
  11. [13]Hoyle, A. M., Calvo-Bartolomé, L., Boyd-Graber, J. L., and Resnik, P. (2025). Proxann: Use-oriented evaluations of topic models and document clustering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15872–15897.
  12. [14]Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of classification, 2(1):193–218.DOI
  13. [15]Jiang, H., Zhou, R., Zhang, L., Wang, H., and Zhang, Y. (2019). Sentence level topic models for associated topics extraction. World Wide Web, 22(6):2545–2560.DOI
  14. [16]Lau, J. H., Newman, D., and Baldwin, T. (2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530–539.DOI
  15. [17]Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.arxiv.org/abs/1609.07843
  16. [18]Mimno, D., Wallach, H., Talley, E., Leenders, M., and McCallum, A. (2011). Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 262–272.
  17. [19]Reuter, A., Thielmann, A., Weisser, C., Säfken, B., and Kneib, T. (2025). Probabilistic topic modeling with transformer representations. IEEE Transactions on Neural Networks and Learning Systems, 36(8):14551–14565.
  18. [20]Schneider, J. (2023). Efficient and flexible topic modeling using pretrained embeddings and bag of sentences. arXiv preprint arXiv:2302.03106.arxiv.org/abs/2302.03106
  19. [21]Vinh, N. X., Epps, J., and Bailey, J. (2009). Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pages 1073–1080.
  20. [22]Wang, D., Zhu, S., Li, T., and Gong, Y. (2009). Multi-document summarization using sentence-based topic models. In Proceedings of the ACL-IJCNLP 2009 conference short papers, pages 297–300.DOI

Paper details

vlnc:W_4c3daee7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents