WorksComputer Science
Sentence-Level Topic Modeling with Lexical and Contextual Evidence
Abstract
Topic modeling faces a fundamental granularity dilemma: probabilistic topic models assign topics at the word level, whereas transformer-based methods operate at the document level. The latter overlooks that real-world documents typically cover multiple topics, whereas the former relies on individual words, which rarely provide sufficient context to identify them. We introduce SentenceETM, a neural topic model that assigns a topic to each sentence by jointly using lexical and contextual evidence. Each topic is described both by a distribution over the vocabulary, defined through word embeddings, and by a Gaussian over sentence embeddings concentrated on a topic-specific low-dimensional subspace. The topic of a sentence generates both its words and its embedding and is marginalized exactly, so that its posterior combines the two sources of evidence. Document proportions are inferred with amortized variational inference, while the Gaussian parameters are re-estimated in closed form. Experiments show that SentenceETM consistently improves sentence-level topic assignment over state-of-the-art approaches, and a case study of a presidential debate shows how sentence-level topics reveal how candidates handle the same subjects.
Introduction
Topic models are well-established methods for extracting the main themes from a collection of documents. Regardless of their underlying formulation, these methods produce a set of topics that characterize the corpus, where a topic is broadly understood as a group of semantically coherent words. A document is characterized by a mixture of these topics. However,
such a mixture tells us which topics the documents contain but not where they are localized. Indeed, real documents rarely contain a single theme, and locating them requires deciding which textual unit a topic is attached to and from which evidence it is inferred.
We refer to lexical evidence as the information carried by words occurring in a text segment1, represented as counts over a fixed vocabulary, and to contextual evidence as the dense representation of that text segment produced by a pretrained neural encoder. Lexical evidence is directly interpretable as a distribution over words, but it is blind to word order and to the sense a word takes in a given occurrence. Contextual evidence can capture the overall semantics of a document more faithfully, but at the cost of interpretability, since the words that characterize a topic are no longer part of its representation and must be recovered separately.
The two main families of topic models each build primarily on one of these types of evidence, because the textual unit they adopt admits primarily one of them. Bag-of-words approaches, e.g. [4], treat the word as the fundamental unit: each word receives a latent topic assignment, and a document becomes a mixture over topics. A word is represented independently of its surrounding context, so contextual evidence from the document is not available to the model at this level. Embedding-based approaches, e.g. [12], take the opposite stance, treating the document as the fundamental unit and clustering the encoder’s representations. Contextual evidence is exploited directly, but lexical evidence re-enters only post hoc, through a descriptor-extraction step playing no role in the topic assignment itself. The result is that one family is too fine-grained to use context, the other too coarse-grained to localize the topics.
Sentence-level topic models lie in between these two extremes. A sentence is semantically self-contained, short enough that its words and its embedding are concerned with a common theme, and numerous enough within a document that assigning a topic to each sentence reveals how topics are distributed throughout the text. Most importantly, at the sentence level, lexical and contextual evidence became available at the same textual unit. While sentence-level topic models have been proposed in the past [1, 7], each model still relies on a single type of evidence for topic assignment. To fill this gap, we make the following contributions.
(i) We introduce SentenceETM, a neural sentence-level topic model that jointly uses lexical and contextual evidence to assign a topic to each sentence. Each topic is characterized by both a distribution over the vocabulary, defined via word embeddings, and a region of the sentence-embedding space, modeled as a Gaussian concentrated on a topic-specific low-dimensional subspace, thereby keeping the model tractable in the high-dimensional space of sentence encoders. Each sentence draws a topic from the document’s topic proportions, and that topic generates both the sentence’s words and its contextual embedding. The sentence topic is marginalized exactly, so that its posterior combines the two sources of evidence, and the parameters of the contextual side are estimated in closed form.
(ii) We present a novel evaluation pipeline for sentence-level topic models. We build synthetic corpora from existing topic-labeled document collections, in which every sentence has a known topic and document-level topic mixtures are controlled by construction.
(iii) We show that SentenceETM can be adapted to different domains through an interchangeable sentence encoder, without modifying the generative model. We illustrate this on the transcript of a televised presidential debate. The sentence-level representation allows us to examine how candidates distribute topics within the same debate segment and how frequently new topics are introduced during speaking turns.
Related works
The use of sentences as a textual unit in topic modeling has been previously investigated mainly as an extension of LDA [4]. Early approaches incorporated sentence- or segment-level structure without making the segment the unit of topic assignment [6, 22, 15]. Differently, senLDA [1] moves the topic assignment from words to sentences: one topic is drawn for each sentence, and the words in that sentence are drawn from the selected topic. The topic–word association is preserved, with each topic represented as a distribution over the vocabulary. More recently, Cha and Lee (2024) introduced SentenceLDA, which replaces the word with the sentence as the basic unit of Gaussian LDA [8], while Schneider (2023) uses sentence embeddings for topic assignment through hard expectation maximization. In these methods, contextual information is used directly for topic assignment, whereas lexical information is used only to describe the resulting topics. Our contextual component is related to model-based clustering in high dimensions [5]: instead of the full-covariance Gaussians of Gaussian LDA, each topic is modeled by a Gaussian concentrated on its own low-dimensional subspace, which remains tractable for sentence embeddings with hundreds of dimensions. A parallel line of work combines lexical and contextual evidence without changing the textual unit. Contextualized Topic Models [2 b] combine bag-of-words representations with pretrained contextual document embeddings, while TNTM [19] models topics as distributions over transformer representations. Both combine the two forms of evidence at the document level and return a topic per document.
The proposed approach
SentenceETM (Figure 1) is built on a single question:

Figure 1. Overview of SentenceETM. Document-level topic proportions (green) are combined with the contextual and lexical evidence of each sentence (red) into the posterior , which re-estimates each topic’s subspace Gaussian in closed form (dashed arrow).
“How likely is it to observe a given sentence embedding together with its words, given that the sentence belongs to a document with topic proportions ?”
Let denote the words of the -th sentence of document and its normalized sentence embedding. Given , the model assigns to each sentence the likelihood
Each term of the sum is the product of the prevalence of topic in the document, the likelihood of the sentence embedding under the topic, and the likelihood of the sentence words under the topic. Since both views (i.e., evidence) are generated by the same topic, a topic contributes to Equation (1) only when it explains the embedding and the words simultaneously. Normalizing the terms of the sum yields the posterior over the topic of the sentence, , which provides the sentence-level topic assignment. In practice, the lexical and contextual terms can have different numerical scales; during training and inference, we weight the contextual log-likelihood by , with recovering Equation (1).
Generative model
A corpus consists of documents. Document is a collection of sentences, and sentence has the words , with . Each sentence is observed through two views. The lexical view is its bag of words, while the contextual view is the embedding of the raw sentence, produced by a frozen sentence encoder. Since sentence encoders are trained to compare sentences by cosine similarity, we -normalize these embeddings, , so that Euclidean distances between them are a monotone function of their cosine similarity. Finally, are pre-trained word embeddings.
Lexical view. Each topic has a topic embedding in the word embedding space, which defines a distribution over the vocabulary, , as in ETM [9].
Contextual view. Each topic is also associated with a region of the sentence embedding space, described by a Gaussian distribution with mean and covariance . Sentence embeddings are high-dimensional, and a full covariance matrix would require parameters per topic. Following [5], we instead adopt a flexible and parsimonious modeling which assumes that the sentences of each topic vary mainly within a topic-specific subspace of dimension , spanned by the orthonormal columns of , and that the remaining variation is isotropic noise:
where . Along the directions of its subspace, topic has variances , with ; in every other direction, it has variance . A sentence is thus likely under a topic when it lies close to the topic subspace, even if it is far from its mean along the directions in which the topic varies. Since each topic has its own subspace, different topics can vary along different directions of the embedding space. The covariance in Equation (2) requires parameters per topic, and both its inverse and its determinant have closed forms that never involve matrices (Section A.3). We collect the model parameters in , with and .
Generative process. For each document, topic proportions are drawn from a logistic-normal prior, with . Each sentence then receives a single topic , which generates both of its views: the normalized embedding, , and every word of the sentence, . The full process is summarized in Algorithm 1 (Section A.4). Given the topic of a sentence, the two views are assumed to be conditionally independent: . Marginalizing the topic of the sentence with respect to the document proportions yields the sentence likelihood of Equation (1). Since sentences are conditionally independent given , the joint distribution of a document and its latent proportions is , where . The sentence topics are summed out exactly within each sentence likelihood; the only latent variable is the document-level , whose posterior we approximate in Section 2.2.
Inference model
The marginal likelihood of document is
which is intractable because of the logistic-normal prior. We therefore approximate the posterior of with variational inference, and fit the model parameters by maximizing a lower bound on Equation (3). For any distribution , the log marginal likelihood decomposes as , where (proof in Appendix A.1). Since the KL divergence is non-negative, is a lower bound (ELBO) of the log marginal likelihood, and maximizing it with respect to brings closer to the true posterior.
Variational assumptions. We restrict to Gaussians with diagonal covariance, amortized by an inference network, to approximate the intractable posterior : , where is the -normalized bag of words of the document and are the outputs of a multilayer perceptron with parameters , as in ETM (Dieng et al. [9]). The inference network conditions only on the words of the document; the contextual view enters the inference through the likelihood. The sentence topics require no variational approximation, since they are summed out exactly in Equation (1) and, once is known, their posterior follows by normalizing its terms.
Objective and optimization. For a document , our objective is (derivation in Appendix A.2):
where is the expectation under . The first term rewards topics that explain both views of every sentence; each summand is the logarithm of the sentence likelihood of Equation (1), computed stably with the log-sum-exp operator. The second term keeps the variational distribution close to the prior and has the closed form .
We maximize by alternating two steps (Algorithm 2). (i) Inference network and lexical parameters. We update and by stochastic gradient ascent on mini-batches of documents, estimating the expectation in Equation (4) with a single sample and the reparameterization , with . (ii) Contextual parameters. At the end of each epoch, we compute the posteriors of all sentences by normalizing the terms of Equation (1), using the mean of , , and update in closed form. For fixed posteriors, the maximum-likelihood estimates of the parameters of Equation (2) are: the estimate of collects the eigenvectors of and
where are the ordered eigenvalues of , and . We update the contextual parameters in closed form, conditional on the current responsibilities . This update requires no learning rate and keeps orthonormal by construction. The contextual parameters are initialized by running k-means on all normalized sentence embeddings and applying the same updates with hard assignments, . After training, we compute from the mean of and assign each sentence to . In practice, we use in all experiments the same subspace dimension for every topic, i.e., for all .
Experimental Setting
Prior work evaluates sentence-level topic models based on their induced document-level topic representations and therefore does not directly assess how topics are distributed across sentences. For example, senLDA (Balikas et al., [1]) is evaluated using held-out perplexity and the score of a classifier trained on document topic proportions, while SentenceLDA (Cha and Lee, [7]) is evaluated based on the quality of the document representations it induces. We argue that evaluating sentence-level topic models requires, first and foremost, datasets with sentence-level ground-truth topic labels. In the absence of suitable existing datasets, we propose a synthetic corpora generation strategy that provides such labels while allowing us to control the document-level topic mixtures. We remind readers that the labels are used only afterward for model evaluation and are not used during training (all methods are unsupervised). Details on the used datasets, model training, and metrics in Appendix B.
Synthetic corpora. Let be the number of sentences of the desired document to create, the number of topics it mixes, and the percentage of the document that belongs to its dominant topic. Each synthetic document is assembled by drawing sentences from a document of the dominant topic and sentences from (random) documents of each of the remaining topics, all drawn from a corpus whose documents carry a single class label. The new document mixture is therefore fixed by construction. Varying at fixed and therefore varies how the same non-dominant-topic sentences are fragmented: at they form a single block of sentences, at the largest admissible they are isolated sentences, one per topic. Three constraints delimit the admissible . Every topic must contribute at least one sentence, , hence ; the dominant topic must dominate, , i.e., ; and , the number of classes in the source corpus.
Figure 2. SentenceETM sentence-level topic assignments. BBC News, . Background color marks the predicted topic (business / entertainment / politics / sport / tech); misassignment in red.
We build the synthetic corpora from: BBC News [11], 20 Newsgroups2, and Wikipedia [13, 17]. We set and for all three corpora. The dominant topic then contributes sentences, which must come from a single article, so is bounded by the length of the source articles: is not binding for Wikipedia (median sentences per article), is available in about half of the BBC articles (median ), and in about a quarter of the 20 Newsgroups posts, whose lengths are heavily skewed (median , mean 11.1). A larger would leave too few eligible articles in BBC and 20 Newsgroups, while a smaller one would make sentence topics harder to recover, even at . With , the dominant topic covers a clear majority of every document, while the remaining sentences allow the non-dominant mass to be split across up to eight topics. We use for 20 Newsgroups and Wikipedia, and for BBC News (as it has a maximum of 5 classes). The synthetic corpus has 700 documents, each of 20 sentences.
Political debates dataset. ElecDeb60to20 (Gofredo et al., 2023) comprises 44 U.S. televised debates (1960–2020), collected from the Commission on Presidential Debates3, which provides transcripts of presidential and vice-presidential debates. For our experiments, we consider the debate between Biden and Trump on the 29th of October 2020, as it is the longest. Specifically, the debates are organized into 6 segments (see Appendix § 7), each presenting a main topic, along with the journalists’ and the candidates’ answers.
Baselines. We test SentenceETM against the main candidates of each of the three families of topic models. At the word-level, we consider LDA [4] and ETM [9]; at the document-level, BERTopic [12], which clusters document embeddings, and finally at the sentence-level, senLDA [1], an LDA in which all words of a sentence share one topic, and SentenceLDA [7], a Gaussian LDA over sentence embeddings (cf. § 1.1). All models use the same preprocessing and vocabulary, and all embedding-based models (BERTopic, SentenceLDA, SentenceETM) use the same sentence encoder, either all-MiniLM-L6-v2 or all-mpnet-base-v2. Our evaluation requires one topic per sentence, which word- and document-level models do not output directly. Therefore, we derive sentence-level assignments as follows: for LDA and ETM, every word of document takes its most probable topic given the fitted model, , and a sentence takes the topic receiving the majority of its word assignments. For BERTopic, each document is represented by the mean embedding of its sentences, using the same encoder as the sentence-level models to avoid truncation and give each sentence equal weight. Documents are clustered with UMAP and HDBSCAN; the resulting clusters are reduced to topics, outliers are reassigned to the nearest topic, and every sentence inherits the topic of its document.
Experimental Results
Figure 2 shows a randomly selected document for on BBC News synthetic (cf. § 3). SentenceETM switches topics from one sentence to the next according to the actual content. The red sentence “It’s what I want to do for the rest of my life” highlights the model mistake. Notably, such a sentence carries no topic-specific information and is therefore not representative of any of the topics. Moreover, in Figure 4 we show the structure of the latent space for a randomly chosen run on Wikipedia (full details in Appendix C.2). Finally, we refer the reader to Appendix C.5 for results on the original datasets.
Figure 4. Sentence-level topic assignments (Wikipedia, all-mpnet-base-v2, ). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel.
Results on synthetic corpora
Additional analysis of the model, including sensitivity analyses for the sentence encoder and document length, is available in Appendices C.3 to C.4. Computational cost in Appendix B.5. SenClu [20] is in Section C.1 due to space constraints.
Ablation study. To assess the contribution of the lexical and contextual evidence, we vary the weight of the contextual likelihood and a common subspace dimension , setting for all topics. We consider two reference models: one that uses lexical evidence only (, where the subspace dimension plays no role) and one that uses contextual evidence only. Since SentenceETM initializes its contextual component from a -means partition of the normalized sentence embeddings, we additionally report this partition as a strong contextual baseline. We use BBC News with and all-mpnet-base-v2, and repeat each configuration on 3 synthetic corpora sampled with different seeds. Figure 3 reports ARI and AMI as a function of for each . For , the contextual evidence has almost no influence: the model behaves like the lexical-only reference (ARI ) and varies strongly across corpora. Between and , the contextual evidence takes over, earlier for larger , whose richer subspaces make it more informative. For , performance is largely stable (ARI approximately 0.62–0.66), with the best results around for every tested subspace dimension. The gain over lexical evidence alone is large (about +0.33 ARI), whereas the gain over contextual evidence alone depends on : it is about +0.08 for and progressively decreases as the subspace becomes more expressive, down to about +0.01 for . Thus, lexical and contextual evidence are complementary: lexical information contributes most when the contextual model is constrained, while for larger subspaces the contextual representation already explains much of the sentence structure. The lexical component additionally yields the topic–word distributions used to interpret the topics. The setting used in the main experiments ( for all , ) lies within the stable region. The AMI results show the same behavior.
Figure 3. Ablation on BBC News (, all-mpnet-base-v2). ARI and AMI as a function of , for the common subspace dimensions . Lexical evidence only in gray; diamonds denote contextual-only models.
Analysis in terms of ARI and AMI
Figure 6 reports clustering quality analysis as a function of the number of topics (all-mpnet-base-v2). The complete set of numerical results is available in Table 5 to 7. Two groups of methods clearly emerge. SentenceETM, senLDA, and SentenceLDA, the other baselines lag behind, as their ARI and AMI are close to zero across all three datasets and increase only slightly as grows. This follows from the construction of the corpora: at , every document contains both topics in proportions 0.6/0.4, so document-level information provides little information about individual sentences; as grows, the dominant topic stands out more clearly against each of the others, making document-level assignments marginally informative. Overall, the proposed model achieves the best ARI and AMI across almost all settings, regardless of the encoder or dataset. A substantial part of the variability in the results comes from the choice of class subset, indeed some combinations of classes are intrinsically easier to separate than others (see Table 1 to 3). To isolate the contribution of the model, Figure 5 compares methods on the same subsets. For each subset of classes, we average scores over seeds and compute , where the maximum is taken over all baselines on that subset. A positive means that SentenceETM outperforms every baseline on that subset. In both ARI and AMI, SentenceETM outperforms all baselines across all 31 subsets, with median gains ranging from about 0.03 to 0.2, depending on the dataset and . The gain is largest at small and decreases as grows, as the strongest baseline, typically SentenceLDA, becomes more competitive. The most challenging dataset turned out to be 20 Newsgroups, possibly due to its less-curated, more conversational nature. Unlike BBC News and Wikipedia, whose documents are generally centered on a well-defined subject, 20 Newsgroups consists of user-written discussion posts that may include digressions, quoted text, or generic conversational content. As a result, individual sentences can be only weakly indicative of the document-level topic.
Figure 5. SentenceETM per-subset gain, all-mpnet-base-v2. Each point corresponds to a subset of () ground-truth classes, with () measured relative to the best-performing baseline for that subset (SentenceLDA) and averaged over seeds.
Figure 6. ARI and AMI as function of the number of topics , all-mpnet-base-v2. Markers indicate the median across runs, while bars show the interquartile range. Higher is better.
| Run | Seed | Topics | |
| 2 | 0 | 6142 | entertainment, politics |
| 2 | 1 | 6142 | business, entertainment |
| 2 | 2 | 6142 | politics, tech |
| 3 | 0 | 6142 | entertainment, politics, tech |
| 3 | 1 | 6142 | business, entertainment, tech |
| 3 | 2 | 6142 | entertainment, sport, tech |
| 5 | 0 | 6142 | business, entertainment, politics, sport, tech |
| 5 | 1 | 6143 | business, entertainment, politics, sport, tech |
| 5 | 2 | 6144 | business, entertainment, politics, sport, tech |
Table 1. BBC News. For each number of topics and each repetition, the classes were sampled from BBC News to compose the synthetic documents. Every document draws its dominant topic uniformly among the listed. When , a single subset exists, so repetitions vary the corpus seed instead of the classes.
| Model | ARI | ARI | ARI | AMI | AMI | AMI |
| mpnet | ||||||
| SentenceETM | 0.76 ±0.04 | 0.69 ±0.03 | 0.63 ±0.01 | 0.67 ±0.04 | 0.62 ±0.03 | 0.61 ±0.01 |
| SentenceLDA | 0.52 ±0.12 | 0.58 ±0.03 | 0.54 ±0.07 | 0.50 ±0.08 | 0.54 ±0.02 | 0.52 ±0.05 |
| senLDA | 0.52 ±0.08 | 0.43 ±0.11 | 0.46 ±0.07 | 0.45 ±0.06 | 0.39 ±0.10 | 0.45 ±0.05 |
| SenClu | 0.26 ±0.22 | 0.25 ±0.12 | 0.23 ±0.04 | 0.25 ±0.20 | 0.26 ±0.11 | 0.26 ±0.04 |
| BERTopic | 0.00 ±0.00 | 0.01 ±0.02 | 0.08 ±0.01 | 0.00 ±0.00 | 0.01 ±0.01 | 0.11 ±0.01 |
| ETM | 0.01 ±0.01 | 0.03 ±0.02 | 0.15 ±0.04 | 0.01 ±0.01 | 0.03 ±0.01 | 0.15 ±0.04 |
| LDA | 0.00 ±0.00 | 0.00 ±0.00 | 0.02 ±0.00 | 0.00 ±0.00 | 0.00 ±0.00 | 0.02 ±0.00 |
| MiniLM | ||||||
| SentenceETM | 0.72 ±0.04 | 0.64 ±0.03 | 0.44 ±0.01 | 0.63 ±0.04 | 0.58 ±0.04 | 0.46 ±0.01 |
| SentenceLDA | 0.02 ±0.01 | 0.04 ±0.01 | 0.12 ±0.03 | 0.11 ±0.01 | 0.19 ±0.03 | 0.34 ±0.03 |
| senLDA | 0.52 ±0.08 | 0.43 ±0.11 | 0.46 ±0.07 | 0.45 ±0.06 | 0.39 ±0.10 | 0.45 ±0.05 |
| SenClu | 0.39 ±0.16 | 0.26 ±0.13 | 0.25 ±0.06 | 0.37 ±0.14 | 0.26 ±0.13 | 0.27 ±0.04 |
| BERTopic | 0.00 ±0.00 | 0.03 ±0.02 | 0.10 ±0.04 | 0.00 ±0.00 | 0.03 ±0.02 | 0.12 ±0.03 |
| ETM | 0.01 ±0.01 | 0.03 ±0.02 | 0.15 ±0.04 | 0.01 ±0.01 | 0.03 ±0.01 | 0.15 ±0.04 |
| LDA | 0.00 ±0.00 | 0.00 ±0.00 | 0.02 ±0.00 | 0.00 ±0.00 | 0.00 ±0.00 | 0.02 ±0.00 |
Table 5. BBC News: ARI and AMI. Results are reported as mean ± standard deviation over runs. Bold the best results within each sentence encoder; underlined the overall best results.
Results on ElecDeb60to20
We illustrate SentenceETM ’s flexibility by applying it to a specific use case. We consider the first 2020 U.S. presidential debate between Donald Trump and Joe Biden, in which the moderator divided the discussion into six segments, and treat each speaking turn as a document and its sentences as the units to which topics are assigned. Because SentenceETM treats the sentence encoder as a separate, interchangeable component, the encoder can be chosen to suit the application. Here, we want topics to capture what is discussed rather than how or by whom: a sentence attacking an opponent’s record on the economy and one defending one’s own should fall under the same topic.
topic. We therefore use the instruction-following encoder Qwen3-Embedding-0.6B and prepend to every sentence the instruction “Given a sentence spoken in a U.S. presidential debate, identify the policy issue it discusses, regardless of the speaker’s stance or tone”. We consider topics and an extended stopword list that includes words specific to spoken debate.
Topics across debate segments. Figure 7 compares how the two candidates distribute their sentences over topics within each segment. For each segment and topic, we compute the share of each candidate’s sentences assigned to that topic and report the difference (Trump − Biden); red cells indicate topics used more by Trump, blue cells indicate topics used more by Biden, and differences of at least 0.10 are annotated. The two topics below the dashed line capture generic conversational sentences. The largest differences reveal different emphases within the same segment. On race, Biden speaks more about racism towards African Americans (−0.12). On the economy, Biden focuses more on jobs and businesses (−0.19), while Trump discusses lockdowns (shut, country, close, +0.14). On election integrity, Biden is on voting and ballot counting (−0.15). In the COVID-19 segment, Biden devotes sentences to masks and vaccines (−0.13), while Trump emphasizes the timing of the response and comparisons with other countries (top-words in Appendix C.7).
Figure 7. Difference in probabilities of topic use between Trump and Biden across debate segments. .
Topic diversity within speaking turns. Figure 8 measures how often a candidate introduces new topics within a speaking turn. Within a turn, we say that a sentence introduces a new topic if its topic has not appeared in any previous sentence of that turn. The first sentence of a turn is excluded, since it cannot introduce a new topic. The index is the fraction of the remaining sentences that introduce a new topic, computed over all the turns of a candidate in a segment: , where is the number of sentences of turn and the number of distinct topics it contains. The index is 0 when every turn stays on a single topic and 1 when every sentence introduces a new topic. Computed over all the turns of the debate, the index is almost identical for the two candidates (0.45 for Trump, 0.43 for Biden): around two sentences out of five introduce a new topic. We use this value as each candidate’s average, and the figure reports, for each segment, the difference between the candidate’s index in that segment and their average. A positive value means that, in that segment, the candidate introduces new topics more often than usual; a negative value indicates lower within-turn topic diversity. Differences are larger across segments than between speakers. COVID-19 is the least cohesive segment for both candidates (+0.15 for Trump, +0.26 for Biden). Election integrity is the most cohesive segment for Biden (−0.20), whose turns stay focused on voting, while Trump’s most cohesive segment is the candidates’ records (−0.10).
Figure 8. New topics per additional sentence within speaking turns, by segment and candidate, as a difference from each candidate’s average. Positive values: more topics than usual; negative values: fewer.
Conclusion
We introduced SentenceETM, a sentence-level topic model that combines lexical evidence from topic–word distributions with contextual evidence from topic-specific low-dimensional subspaces of the sentence-embedding space. Document-level topic proportions act as a prior, and Gaussian parameters are updated in closed form. We also proposed a synthetic corpus protocol in which each sentence has a known topic. SentenceETM achieves the best ARI and AMI in almost all settings and remains robust to a smaller sentence encoder, unlike SentenceLDA. In a U.S. presidential debate, it reveals differences in topic emphasis between candidates and within-turn topic diversity. Future work includes modeling sentence order and topic transitions, and complementing synthetic labels with human annotations.
AI use statement
In this work, we used generative AI tools for tasks requiring disclosure, including: method implementation, by drafting portions of the code, assisting with notebooks, and helping integrate existing state-of-the-art implementations; interpretation of results, by discussing experimental outcomes and possible explanations reported in the paper; qualitative analysis, by proposing short labels for the debate topics in Table 9. The model itself, including the use of high-dimensional Gaussian models for contextual evidence, was conceived by the authors. Generative AI was not used to generate synthetic data, formulate hypotheses or novel mathematical claims, or clean the datasets. All AI-assisted material was reviewed by the authors. All code was inspected and executed by the authors; all numerical results reported in the paper were obtained from experimental outputs rather than generated by AI. We take full responsibility for the final content of this work, including any text, claims, code, or other artifacts produced with the assistance of generative AI.
| Topic | Label | Top words |
| 0 | Lockdowns | shut, close, country, open, allow, state, democrats, destroy, wrong, throw |
| 1 | Accusations | man, plan, lie, fraud, office, name, liar, run, wait, acknowledge |
| 2 | Mail-in voting | ballot, vote, count, send, poll, mail, solicit, state, open, director |
| 3 | Electric cars | car, cheap, expensive, electric, safe, cost, standard, automobile, tiny, least |
| 4 | Conversational | guess, wrong, hit, honest, suppose, hey, excuse, surprised, feel, plenty |
| 5 | Achievements | job, win, top, thank, proud, testify, set, every, happy, administration |
| 6 | Foreign countries | china, head, die, world, russia, india, eat, billion, ukraine, lunch |
| 7 | Racism | african, racist, americans, wing, police, community, close, group, racism, whistle |
| 8 | Health care | drug, care, obamacare, healthcare, affordable, pre, existing, condition, act, million |
| 9 | Mixed | deal, record, back, god, send, wait, extent, november, soon, period |
| 10 | Economy | job, economy, create, business, economic, company, history, leave, million, recovery |
| 11 | Climate and energy | build, plant, deal, paris, accord, energy, emit, hard, environment, problem |
| 12 | Law enforcement | law, enforcement, police, violence, appropriate, order, crime, support, percent, officer |
| 13 | Election outcome | election, vote, win, elect, outcome, run, determine, shut, left, party |
| 14 | Forest fires | forest, tree, management, city, tear, billion, stop, burn, major, else |
| 15 | Mixed | military, school, bring, support, back, country, church, include, name, class |
| 16 | Taxes | tax, dollar, billion, eliminate, pay, taxis, million, money, fortune, cut |
| 17 | Deaths and suburbs | die, suburb, life, place, million, town, home, trouble, far, agree |
| 18 | Supreme Court | court, justice, hold, leave, supreme, united, states, vote, term, elect |
| 19 | COVID-19 measures | mask, vaccine, wear, social, speak, test, distancing, end, america, johnson |
Table 9. Topics learned on the presidential debate. For each topic, the ten words with the highest probability under the learned topic–word distribution, and a short label assigned by inspecting them.
Proofs
This section provides the proofs supporting Section 2. We also provide algorithms summarizing both the generative process and the inference procedure.
ELBO decomposition
By the product rule,
Therefore, for every value of at which the posterior is positive,
Taking the expectation of both sides under leaves the left-hand side unchanged, as it does not depend on :
Finally, adding and subtracting and regrouping,
Derivation of eq:elbo
Recall from the ELBO decomposition that
Moreover,
Substituting into , using the linearity of the expectation and the definition of the Kullback–Leibler divergence, ,
which, with , is (4).
Computing the contextual likelihood
The covariance of Equation (2) can be equivalently written as
since . As has orthonormal columns, has eigenvalues along the columns of and along the orthogonal directions. Hence
Let and be the coordinates of in the topic subspace. Since ,
The third term measures the distance of the sentence from the topic mean within the topic subspace, weighted by the topic’s variance along its directions; the fourth term measures its distance from the subspace, weighted by the noise variance . The cost is per sentence and topic, and no matrix is ever formed. In practice, we lower-bound and by a small constant to prevent degenerate topics.
Generative process
Algorithm 1 Generative process of SentenceETM
1: for each topic k = 1, ..., K do
2: β_k ← softmax(ρ^⊤α_k)
3: Σ_k ← V_k diag(λ_{k1} − σ_k², ..., λ_{kh_k} − σ_k²)V_k^⊤ + σ_k²I_E
4: end for
5: for each document d = 1, ..., D do
6: Draw δ^(d) ∼ N(0, I_K) and set θ^(d) ← softmax(δ^(d))
7: for each sentence m = 1, ..., M^(d) do
8: Draw the topic z_m^(d) ∼ Mult(1; θ^(d))
9: Draw the embedding ẽ_m^(d) ∼ N(μ_{z_m^(d)}, Σ_{z_m^(d)})
10: for each word n = 1, ..., N_m^(d) do
11: Draw the word w_mn^(d) ∼ Mult(1; β_{z_m^(d)})
12: end for
13: end for
14: end for
Inference process
Algorithm 2 Inference in SentenceETM
Require: documents with sentence embeddings ; number of topics ; topic-specific subspace dimensions ; number of epochs
for every sentence
Run -means with clusters on ; set if sentence of is in cluster , otherwise
Initialize from the hard assignments using the closed-form estimates in § 2.2; initialize and at random
for do
for each mini-batch of documents do
for do
,
Compute with Equation (4), summing out each in Equation (1)
end for
Update and by a gradient step on ▷ fixed
end for
for every document
Compute for every sentence
Update using the closed-form estimates in § 2.2 ▷ closed form
end for
return , , and the sentence topics
Additional Materials to Section 3
Synthetic dataset construction pipeline

Figure 9. Synthetic corpora generation. Example with , and .
The construction of a synthetic corpus proceeds in four steps, repeated independently for every run.
As detailed in § 3, we start with existing topic modeling datasets that provide a single topic label per document: BBC News (Greene and Cunningham, [11]), 20 Newsgroups4, and Wikipedia (Merity et al., [17]). Specifically,
(a) BBC News dataset is a corpus consisting of 2.225 articles published on the BBC News website corresponding to 2004-2005 (in English). Each article is labeled under one of 5 categories: business, entertainment, politics, sport or tech.
(b) 20 Newsgroups dataset comprises around 18.000 newsgroups posts on 20 topics (in English): rec.sport.hockey, comp.sys.ibm.pc.hardware, talk.politics.mideast, comp.sys.mac.hardware, sci.electronics, talk.religion.misc, sci.crypt, sci.med, alt.atheism, rec.motorcycles, rec.autos, comp.windows.x, comp.graphics, sci.space, talk.politics.guns, misc.forsale, rec.sport.baseball, talk.politics.misc, comp.os.ms-windows.misc, soc.religion.christian.
(c) We use the training split of the Wikipedia corpus from Hoyle et al. [13], derived from WikiText-103 ([17]). This dataset is a general-audience corpus consisting of 14,000 English Wikipedia articles. We use the articles’ supercategory labels (15): Language and literature, Natural sciences, Art and architecture, Music, Engineering and technology, Media and drama, History, Warfare, Sports and recreation, Geography and places, Social sciences and society, Philosophy and religion, Video games, Agriculture, food, and drink, Mathematics.
For a given , we draw distinct topic labels from those available; each of the three runs uses a different subset. How is chosen is discussed in § 3.
For each of the documents, we draw the dominant topic uniformly among the selected labels. We then sample one source article of that topic among those with at least sentences, and take of its sentences. The same is done for each of the non-dominant topics, taking sentences from an article with at least of them. Sentences are drawn without replacement within an article and keep their original order; source articles are reused across documents only when the pool is smaller than . The drawn sentences are finally concatenated by topic, so a document consists of contiguous blocks.
Data preprocessing is applied on the resulting corpus as described in Section B.1.1.
In Table 1 to Table 3 we provide the topic compositions of the synthetic corpora for each of the runs.
| Run | Seed | Topics | |
| 2 | 0 | 6142 | Mathematics; Music |
| 2 | 1 | 6142 | Language and literature; Music |
| 2 | 2 | 6142 | Agriculture, food, and drink; Language and literature |
| 3 | 0 | 6142 | Language and literature; Music; Warfare |
| 3 | 1 | 6142 | Agriculture, food, and drink; Engineering and technology; Language and literature |
| 3 | 2 | 6142 | Language and literature; Social sciences and society; Video games |
| 5 | 0 | 6142 | History; Language and literature; Mathematics; Music; Sports and recreation |
| 5 | 1 | 6142 | Language and literature; Media and drama; Philosophy and religion; Social sciences and society; Sports and recreation |
| 5 | 2 | 6142 | Art and architecture; History; Philosophy and religion; Social sciences and society; Warfare |
| 9 | 0 | 6142 | Agriculture, food, and drink; Engineering and technology; Geography and places; History; Language and literature; Mathematics; Music; Natural sciences; Social sciences and society |
| 9 | 1 | 6142 | Art and architecture; Geography and places; Language and literature; Mathematics; Media and drama; Music; Social sciences and society; Video games; Warfare |
| 9 | 2 | 6142 | Agriculture, food, and drink; Art and architecture; Geography and places; History; Language and literature; Mathematics; Music; Sports and recreation; Warfare |
Table 3. Wikipedia. For each number of topics and each repetition, the classes were sampled from Wikipedia to compose the synthetic documents. Every document draws its dominant topic uniformly among the listed. When , a single subset exists, so repetitions vary the corpus seed instead of the classes.
Preprocessing
Each document is represented as (i) a bag-of-words over a fixed vocabulary and (ii) a sequence of sentences.
Lexical view. We follow standard ETM preprocessing ([9]). Documents are lemmatized with en_core_web_lg; NLTK stop words, generic verbs and modals, punctuation, non-alphabetic tokens, and tokens shorter than three characters are removed. We also discard terms occurring in more than of the documents or in only one document. The resulting lemmas are intersected with the lowercase alphabetic GloVe vocabulary, retaining the most frequent words. Thus, all columns of () are initialized
| Run | Seed | Topics | |
| 2 | 0 | 6142 | rec.motorcycles, sci.crypt |
| 2 | 1 | 6142 | misc.forsale, sci.crypt |
| 2 | 2 | 6142 | alt.atheism, rec.autos |
| 3 | 0 | 6142 | rec.autos, sci.crypt, talk.religion.misc |
| 3 | 1 | 6142 | alt.atheism, comp.sys.mac.hardware, rec.autos |
| 3 | 2 | 6142 | rec.motorcycles, soc.religion.christian, talk.politics.mideast |
| 5 | 0 | 6142 | misc.forsale, rec.autos, rec.sport.baseball, sci.crypt, talk.politics.mideast |
| 5 | 1 | 6142 | rec.autos, rec.motorcycles, rec.sport.baseball, soc.religion.christian, talk.politics.guns |
| 5 | 2 | 6142 | comp.os.ms-windows.misc, misc.forsale, sci.space, talk.politics.guns, talk.politics.misc |
| 9 | 0 | 6142 | alt.atheism, comp.sys.ibm.pc.hardware, comp.windows.x, rec.autos, rec.sport.baseball, sci.med, sci.space, soc.religion.christian, talk.politics.mideast |
| 9 | 1 | 6142 | comp.graphics, comp.sys.mac.hardware, comp.windows.x, rec.motorcycles, rec.sport.hockey, sci.crypt, sci.med, soc.religion.christian, talk.politics.misc |
| 9 | 2 | 6142 | alt.atheism, comp.os.ms-windows.misc, comp.sys.mac.hardware, misc.forsale, rec.motorcycles, sci.med, talk.politics.guns, talk.politics.mideast, talk.politics.misc |
Table 2. 20 Newsgroups. For each number of topics and each repetition, the classes were sampled from 20 Newsgroups to compose the synthetic documents. Every document draws its dominant topic uniformly among the listed. When , a single subset exists, so repetitions vary the corpus seed instead of the classes.
with pre-trained GloVe vectors and fine-tuned during training. Documents with fewer than five in-vocabulary tokens are discarded, and counts are normalized before being fed to the inference network.
Contextual view. Raw documents are segmented with the Punkt tokenizer as ; if segmentation returns no sentence, the whole document is used as one sentence. Each sentence has a discrete bag-of-words representation over the vocabulary above, used by the lexical evidence, and a continuous representation obtained by encoding the raw sentence with a pre-trained encoder (all-mpnet-base-v2 and all-MiniLM-L6-v2 for the synthetic and real corpora, and Qwen3-Embedding-0.6B for the debate). The resulting embedding is -normalized, , and used as contextual evidence (Section 2). Embeddings are computed once and cached; the encoders are not fine-tuned.
ElecDeb60to20
The organization of the debate is indicated by the journalist at the beginning of the debate:
“Good evening from the Health Education Campus of Case Western Reserve University and the Cleveland Clinic. I’m Chris Wallace of Fox News and I welcome you to the first of the 2020 presidential debates between President Donald J. Trump and former Vice President Joe Biden. This debate is sponsored by the Commission on Presidential Debates. The Commission has designed the format, six roughly 15-minute segments with two-minute answers from each candidate to the first question, then open discussion for the rest of each segment. Both campaigns have agreed to these rules. For the record, I decided the topics and the questions in each topic. I can assure you none of the questions has been shared with the Commission or the two candidates. This debate is being conducted under health and safety protocols designed by the Cleveland Clinic, which is serving as the health security advisor to the Commission for all four debates. As a precaution, both campaigns have agreed the candidates will not shake hands at the beginning of tonight’s debate. The audience here in the hall has promised to remain silent. No cheers, no boos, or other interruptions so we, and more importantly you, can focus on what the candidates have to say. No noise except right now, as we welcome the Republican nominee, President Trump, and the Democratic nominee Vice President Biden.”
Overall, the segments present the following topics:
Index 4 – 146: Supreme Court / Healthcare / Obamacare / Filibuster / Court packing.
Index 147 – 272: COVID19 / Vaccine / Scientists / Reopening schools and economy / Masks / Disease spread at events.
Index 273 – 475: Economy / Unemployment rate / Recovery / Taxes / Economic plan.
Index 476 – 626: Race issues / Racial sensitivity training / Law and Order / Black Lives Matter / Protests.
Index 627 – 779: Trump and Biden records / Climate change / Global warming / Green New Deal.
Index 780 – 853: Election integrity / Mail-in voting.
Model training
Sentence embeddings are computed once with a frozen encoder: all-mpnet-base-v2 in the main experiments and all-MiniLM-L6-v2 in the encoder comparison. For models using word embeddings, is initialized with GloVe and fine-tuned. SentenceETM is trained for 300 epochs with Adam (learning rate , weight decay , gradient clipping 2.0) and batches of 64 documents. The inference network has hidden size 256, topic embeddings have dimension 300, the topic subspace dimension is (for all the topics) , and contextual evidence is weighted by . Topic Gaussians are initialized by k-means (10 restarts) on normalized sentence embeddings and re-estimated in closed form after each epoch, rather than optimized by Adam. Eigenvalues and noise variances are lower-bounded by . For the baselines, we use the hyperparameters from the original implementations, sharing the batch size and learning rate across neural models. ETM [9] is trained for 150 epochs with hidden size 800, no encoder dropout, and weight decay . LDA [4] uses 201 batch iterations; senLDA [1], 201 Gibbs sweeps with ; and SentenceLDA ([7]), 20 sweeps with . For the Gibbs samplers, we use the final state without averaging, as in the original implementations. BERTopic ([12]) uses UMAP with 15 neighbors, 5 components, and cosine distance, is constrained to topics, and reassigns outliers by embedding similarity. Each configuration is repeated 3 times, varying source classes or, when only one class subset is available, the sampled documents and sentences. Each model is trained with 3 seeds per repetition.
Metrics
We measure assignment quality against the ground-truth sentence topics with the Adjusted Mutual Information ([21]) and the Adjusted Rand Index ([14]), both corrected for chance. Since every sentence has a known topic, these metrics directly evaluate the quantity that the models infer, without resorting to proxies for topic quality.
Computational cost
The cost of SentenceETM has three parts. Let be the number of sentences in the corpus.
(i) Encoding. Each sentence is encoded once by the frozen encoder, and the embeddings are cached. This cost is shared with every method that relies on sentence embeddings; with all-mpnet-base-v2, encoding the 14,000 sentences of a synthetic corpus takes about 30 seconds ( ms per sentence).
(ii) Gradient steps. At each epoch, the inference network costs , with its hidden size; the lexical evidence costs to compute plus one term per word token and topic; the contextual evidence costs , which reduces to when for all .
(iii) Closed-form update. Once per epoch, the topic Gaussians are re-estimated from the responsibilities, which requires operations to accumulate the covariances and for their eigendecompositions. This step does not depend on the vocabulary and grows linearly with and .
Table 4 reports the training time of all models on all the datasets. On average over , SentenceETM trains in 340 to 391 seconds per run, against 1717 to 3118 seconds for SentenceLDA, the other sentence-level model that uses sentence embeddings whose Gibbs sampler must update every sentence assignment sequentially. It is slower than the document-level models and senLDA, which use neither embeddings nor a per-sentence posterior over Gaussians. Its cost grows linearly with . Most of this time is spent in the closed-form update, which our implementation runs on the CPU; it could be moved to the GPU or performed every few epochs. Once the model is trained, assigning a topic to a new sentence requires one forward pass of the inference network and the evaluation of Gaussian densities.
Additional Materials to Section 4
Numerical results
Tables 5 to 7 report its ARI and AMI on the synthetic corpora. The tables also present the results for SenClu ([20]), which assigns sentences to topics by hard expectation-maximization over sentence embeddings, weighted by document-level topic proportions. We use the provided official implementation with its default hyperparameters (40 epochs, ) and feed it the same sentences and sentence embeddings as the other models. SentenceETM outperforms it in every setting. SenClu varies strongly across runs (standard deviation up to 0.32 ARI on Wikipedia), which we attribute to its random initialization of the topic centroids.
Structure of the latent space
Figures 4 and 10 show, for one run per dataset, the sentences projected onto the first two principal components of their embeddings and colored by the topic assigned by each model. The three most probable words of each topic are listed below each panel. With SentenceETM, sentences assigned to the same topic largely form distinct regions of the embedding space, and the top words identify the corresponding classes (e.g., bank, economy, rate for Business in BBC News, church, jesus, bible for Christianity in 20 Newsgroups, music, chart, video for Music in Wikipedia). In contrast, BERTopic and ETM produce substantially more mixed assignments. BERTopic assigns one topic per document, so every sentence inherits its document’s topic regardless of its content; ETM assigns sentence topics from the words of document-level topics. Their top words reveal further limitations: BERTopic produces topics that mix or duplicate classes (e.g., game, season, team and game, league, play on Wikipedia), while ETM topics are often dominated by rare, specific words (Ise, boerse, gerrad; tercel, diesel, mustang; chili, doughnut, steak). Finally, we notice that SentenceETM on 20 Newsgroups, the topic matched to Atheism is dominated by conversational words (list, answer, sure), while the one matched to X window has uninformative top words and few sentences, consistent with the lower scores on this dataset. In Appendix § 8.6 we provide some examples of the misassigned sentences.
| Dataset | SentenceETM | SentenceLDA | senLDA | SenClu | BERTopic | ETM | LDA | |
| BBC News | 2 | 281 | 1946 | 86 | 1.6 | 1.6 | 4.3 | 26 |
| BBC News | 3 | 328 | 1391 | 87 | 1.9 | 1.6 | 4.4 | 23 |
| BBC News | 5 | 410 | 1815 | 88 | 2.3 | 1.7 | 4.6 | 24 |
| BBC News | Avg. | 340 | 1717 | 87 | 1.9 | 1.6 | 4.4 | 25 |
| 20 Newsgroups | 2 | 210 | 1913 | 81 | 1.2 | 1.5 | 4.0 | 22 |
| 20 Newsgroups | 3 | 255 | 2116 | 81 | 1.9 | 1.5 | 4.1 | 21 |
| 20 Newsgroups | 5 | 335 | 1817 | 82 | 2.0 | 1.5 | 4.1 | 20 |
| 20 Newsgroups | 9 | 497 | 5309 | 82 | 2.7 | 1.5 | 4.2 | 19 |
| 20 Newsgroups | Avg. | 324 | 2789 | 82 | 2.0 | 1.5 | 4.1 | 21 |
| Wikipedia | 2 | 268 | 2308 | 85 | 1.8 | 1.5 | 3.7 | 26 |
| Wikipedia | 3 | 328 | 2304 | 84 | 1.8 | 1.5 | 3.8 | 24 |
| Wikipedia | 5 | 413 | 2684 | 84 | 2.2 | 1.5 | 3.9 | 23 |
| Wikipedia | 9 | 554 | 5176 | 88 | 2.2 | 1.4 | 4.1 | 23 |
| Wikipedia | Avg. | 391 | 3118 | 85 | 2.0 | 1.5 | 3.9 | 24 |
Table 4. Training time in seconds (median over 9 runs; 700 documents, 14,000 sentences), excluding corpus construction and sentence encoding. All experiments were run on NVIDIA H100 GPUs. Neural models (SentenceETM, ETM, BERTopic, SenClu) run on a GPU; LDA, senLDA, and SentenceLDA, whose released implementations are CPU-only, run on CPU.
Sensitivity to the sentence encoder
Replacing all-mpnet-base-v2 with the smaller all-MiniLM-L6-v2 affects the embedding-based sentence models very differently. For SentenceETM, the drop is at most 0.05 in both ARI and AMI, except on BBC News at (0.19 ARI and 0.15 AMI), where its ARI falls slightly below that of senLDA. SentenceLDA, in contrast, degrades sharply: its ARI falls to at most 0.16 in every setting (e.g., from 0.52 to 0.02 on BBC News at ), and its AMI decreases by up to 0.39. With all-MiniLM-L6-v2, SentenceLDA obtains lower ARI than senLDA, which does not use embeddings, in every setting. This degradation cannot be attributed solely to the weaker embeddings: SenClu, using the same all-MiniLM-L6-v2 embeddings, loses at most 0.06 ARI relative to all-mpnet-base-v2 and even improves in most settings. A possible explanation lies in how the models represent topics in the embedding space. SentenceLDA uses full-covariance Gaussians, making its clustering depend on the geometry of the entire space. In SentenceETM, each covariance is instead constrained to a few principal directions plus isotropic noise and re-estimated from soft responsibilities; this parsimonious representation may be less sensitive to directions carrying little topical information. Note that senLDA, ETM, and LDA do not use the sentence encoder, so their results are identical for both encoders (cf. Table 5 to Table 7).
| Model | ARI | ARI | ARI | ARI | AMI | AMI | AMI | AMI |
| mpnet — SentenceETM | 0.69 ±0.13 | 0.57 ±0.08 | 0.45 ±0.15 | 0.47 ±0.04 | 0.61 ±0.15 | 0.55 ±0.09 | 0.47 ±0.15 | 0.55 ±0.02 |
| mpnet — SentenceLDA | 0.44 ±0.11 | 0.41 ±0.07 | 0.38 ±0.15 | 0.45 ±0.05 | 0.45 ±0.09 | 0.48 ±0.09 | 0.42 ±0.15 | 0.50 ±0.04 |
| mpnet — senLDA | 0.53 ±0.19 | 0.47 ±0.08 | 0.31 ±0.09 | 0.29 ±0.04 | 0.46 ±0.18 | 0.43 ±0.08 | 0.34 ±0.09 | 0.38 ±0.03 |
| mpnet — SenClu | 0.37 ±0.32 | 0.40 ±0.16 | 0.22 ±0.15 | 0.27 ±0.04 | 0.34 ±0.28 | 0.38 ±0.15 | 0.23 ±0.15 | 0.34 ±0.03 |
| mpnet — BERTopic | 0.00 ±0.00 | 0.00 ±0.00 | 0.04 ±0.02 | 0.11 ±0.05 | 0.00 ±0.00 | 0.00 ±0.00 | 0.08 ±0.01 | 0.18 ±0.03 |
| mpnet — ETM | 0.02 ±0.02 | 0.17 ±0.06 | 0.14 ±0.07 | 0.20 ±0.02 | 0.02 ±0.02 | 0.16 ±0.05 | 0.15 ±0.06 | 0.24 ±0.01 |
| mpnet — LDA | 0.00 ±0.00 | 0.01 ±0.01 | 0.02 ±0.01 | 0.03 ±0.01 | 0.00 ±0.00 | 0.01 ±0.01 | 0.03 ±0.02 | 0.05 ±0.01 |
| MiniLM — SentenceETM | 0.67 ±0.14 | 0.57 ±0.10 | 0.43 ±0.15 | 0.44 ±0.05 | 0.60 ±0.15 | 0.53 ±0.11 | 0.45 ±0.14 | 0.51 ±0.02 |
| MiniLM — SentenceLDA | 0.03 ±0.02 | 0.10 ±0.05 | 0.11 ±0.01 | 0.16 ±0.04 | 0.11 ±0.03 | 0.27 ±0.06 | 0.29 ±0.05 | 0.38 ±0.03 |
| MiniLM — senLDA | 0.53 ±0.19 | 0.47 ±0.08 | 0.31 ±0.09 | 0.29 ±0.04 | 0.46 ±0.18 | 0.43 ±0.08 | 0.34 ±0.09 | 0.38 ±0.03 |
| MiniLM — SenClu | 0.52 ±0.30 | 0.46 ±0.18 | 0.27 ±0.15 | 0.31 ±0.03 | 0.46 ±0.28 | 0.42 ±0.16 | 0.28 ±0.15 | 0.38 ±0.02 |
| MiniLM — BERTopic | 0.00 ±0.00 | 0.00 ±0.00 | 0.03 ±0.01 | 0.13 ±0.04 | 0.00 ±0.00 | 0.00 ±0.00 | 0.08 ±0.01 | 0.19 ±0.03 |
| MiniLM — ETM | 0.02 ±0.02 | 0.17 ±0.06 | 0.14 ±0.07 | 0.20 ±0.02 | 0.02 ±0.02 | 0.16 ±0.05 | 0.15 ±0.06 | 0.24 ±0.01 |
| MiniLM — LDA | 0.00 ±0.00 | 0.01 ±0.01 | 0.02 ±0.01 | 0.03 ±0.01 | 0.00 ±0.00 | 0.01 ±0.01 | 0.03 ±0.02 | 0.05 ±0.01 |
Table 7. Wikipedia: ARI and AMI. Results are reported as mean ± standard deviation over runs. Bold the best results within each sentence encoder; _underlined_ the overall best results.
Sensitivity to document length
We vary the number of sentences per document on Wikipedia with and
| Model | ARI | ARI | ARI | ARI | AMI | AMI | AMI | AMI |
| mpnet — SentenceETM | 0.56 ±0.05 | 0.46 ±0.03 | 0.35 ±0.04 | 0.27 ±0.02 | 0.48 ±0.03 | 0.44 ±0.02 | 0.41 ±0.05 | 0.37 ±0.04 |
| mpnet — SentenceLDA | 0.16 ±0.04 | 0.23 ±0.02 | 0.30 ±0.07 | 0.25 ±0.03 | 0.26 ±0.03 | 0.37 ±0.01 | 0.39 ±0.05 | 0.33 ±0.03 |
| mpnet — senLDA | 0.22 ±0.13 | 0.30 ±0.02 | 0.21 ±0.03 | 0.17 ±0.05 | 0.18 ±0.08 | 0.26 ±0.01 | 0.22 ±0.03 | 0.21 ±0.05 |
| mpnet — SenClu | 0.36 ±0.19 | 0.35 ±0.07 | 0.28 ±0.03 | 0.23 ±0.04 | 0.32 ±0.16 | 0.33 ±0.06 | 0.33 ±0.03 | 0.31 ±0.05 |
| mpnet — BERTopic | 0.00 ±0.00 | 0.02 ±0.03 | 0.09 ±0.02 | 0.06 ±0.08 | 0.00 ±0.00 | 0.03 ±0.03 | 0.12 ±0.02 | 0.10 ±0.10 |
| mpnet — ETM | 0.02 ±0.02 | 0.12 ±0.02 | 0.11 ±0.01 | 0.11 ±0.03 | 0.02 ±0.02 | 0.11 ±0.02 | 0.12 ±0.02 | 0.15 ±0.04 |
| mpnet — LDA | 0.00 ±0.00 | 0.01 ±0.01 | 0.01 ±0.00 | 0.02 ±0.01 | 0.00 ±0.00 | 0.01 ±0.01 | 0.02 ±0.01 | 0.03 ±0.01 |
| MiniLM — SentenceETM | 0.54 ±0.04 | 0.43 ±0.02 | 0.32 ±0.01 | 0.25 ±0.03 | 0.46 ±0.02 | 0.41 ±0.02 | 0.36 ±0.03 | 0.34 ±0.04 |
| MiniLM — SentenceLDA | 0.01 ±0.00 | 0.02 ±0.01 | 0.03 ±0.01 | 0.03 ±0.01 | 0.07 ±0.00 | 0.13 ±0.02 | 0.22 ±0.03 | 0.25 ±0.04 |
| MiniLM — senLDA | 0.22 ±0.13 | 0.30 ±0.02 | 0.21 ±0.03 | 0.17 ±0.05 | 0.18 ±0.08 | 0.26 ±0.01 | 0.22 ±0.03 | 0.21 ±0.05 |
| MiniLM — SenClu | 0.40 ±0.08 | 0.29 ±0.04 | 0.25 ±0.05 | 0.20 ±0.04 | 0.33 ±0.07 | 0.28 ±0.02 | 0.28 ±0.04 | 0.28 ±0.04 |
| MiniLM — BERTopic | 0.00 ±0.00 | 0.02 ±0.03 | 0.07 ±0.06 | 0.06 ±0.08 | 0.00 ±0.00 | 0.03 ±0.03 | 0.09 ±0.07 | 0.10 ±0.09 |
| MiniLM — ETM | 0.02 ±0.02 | 0.12 ±0.02 | 0.11 ±0.01 | 0.11 ±0.03 | 0.02 ±0.02 | 0.11 ±0.02 | 0.12 ±0.02 | 0.15 ±0.04 |
| MiniLM — LDA | 0.00 ±0.00 | 0.01 ±0.01 | 0.01 ±0.00 | 0.02 ±0.01 | 0.00 ±0.00 | 0.01 ±0.01 | 0.02 ±0.01 | 0.03 ±0.01 |
Table 6. 20 Newsgroup: ARI and AMI. Results are reported as mean ± standard deviation over runs. Bold the best results within each sentence encoder; _underlined_ the overall best results.
all-mpnet-base-v2, while decreasing the number of documents accordingly (700, 350, 233, and 175), so that each corpus contains approximately 14,000 sentences. All other settings are the same as in the main experiments.
As shown in Figure 11, SentenceETM obtains the highest ARI and AMI at every document length, while remaining stable across the four settings: ARI ranges from 0.46 to 0.49 and AMI from 0.54 to 0.56, with differences within one standard deviation. The other sentence-level models are similarly stable but consistently lower, with SentenceLDA ranging from 0.42 to 0.45 ARI and senLDA remaining around 0.30. Among the document-level models, ETM gradually degrades as document length increases (ARI from 0.18 to 0.14), BERTopic fluctuates between 0.08 and 0.12, and LDA remains close to zero. These results suggest that sentence-level assignments are largely insensitive to document length in this setting, even when the same amount of sentence-level information is distributed across fewer, longer documents.
Figure 11. Sensitivity to document length (Wikipedia, , all-mpnet-base-v2). ARI (left) and AMI (right) as a function of the number of sentences per document ( runs).
Results on original corpora
We evaluate the lexical quality of the learned topics (10 top-words) through Topic Coherence (TC) with NPMI [18, 16] and Topic Diversity (TD) [9]. We use the topics learned in the same runs as the synthetic-corpus experiments; no model is retrained on the original datasets. For each run, we extract the top words of the learned topics and evaluate them against the corresponding full original corpus. This choice avoids measuring coherence on the synthetic documents used for training, whose co-occurrence statistics are partly induced by our construction procedure: each synthetic document deliberately combines sentences drawn from different source classes, and words from otherwise unrelated topics may therefore co-occur simply because their sentences were placed in the same document. Computing TC on these documents could consequently reward topic-word sets that reflect the synthetic mixing procedure rather than genuine associations in the source corpus. Evaluating the same learned top words on the original corpus instead measures whether they correspond to word associations that occur naturally in the underlying data. TD does not require a reference corpus, since it depends only on the overlap among the top words of the learned topics. We therefore compute it directly from the topics obtained in each run. Figure 12 shows that the sentence-level models produce consistently coherent and diverse topics across datasets. SentenceETM maintains high TC on BBC News, 20 Newsgroups, and Wikipedia, with comparatively limited variation across datasets. SentenceLDA and senLDA show a similarly stable behavior, whereas the document-level baselines are substantially more dataset-dependent: in particular, ETM ranges from low or negative coherence on BBC News and 20 Newsgroups to among the highest values on Wikipedia. BERTopic and LDA exhibit similar, although less pronounced, variations. TD is close to one for most methods and configurations, especially for the sentence-level models, indicating little overlap among their top words. It therefore provides less discrimination than TC in this setting; importantly, SentenceETM combines this high diversity with consistently high coherence.
Figure 12. Topic coherence (TC) and topic diversity (TD) on the original corpora. Topics are those learned in the synthetic-corpus experiments, while TC is evaluated using word co-occurrences from the corresponding full original dataset. TD is computed directly from the learned top words. Higher is better.
Examples of misassigned sentences
In Table 8, we randomly select eight sentences assigned to an incorrect topic by SentenceETM for each dataset, using the same runs as in Figure 4 and Figure 10. Most errors fall into three groups. First, many misassigned sentences contain little topical information: generic statements, reported speech, or discourse-like utterances whose label is inherited from the source document but cannot be reliably inferred from the sentence alone (e.g., “So to win something like this really feels great”, “I won’t argue too much about anything”). In such cases, the predicted topic often reflects cues in the wording itself, such as win suggesting sport or an argumentative tone suggesting a religion-related forum. Second, several sentences are semantically closer to the predicted topic than to the label of their source document. For example, a business article on Turkish firms excluded for national-security reasons discusses politics; an alt.atheism post mentioning Soviet and French tanks discusses the Middle East conflict; and Wikipedia sentences mentioning boxing or cricket naturally evoke sport. These cases highlight the noise introduced by propagating document-level labels to individual sentences. Third, some disagreements involve inherently overlapping categories, such as Warfare and History, or Geography and places and Art and architecture for a sentence describing a historical house. Only a few cases lack a clear semantic explanation, such as a sentence on the history of Calakmul assigned to Language and literature or a bibliographic reference assigned to Geography and places.
Figure 10. Sentence-level topic assignments (all-mpnet-base-v2). Sentences projected onto the first two embedding PCs and colored by assigned topic; top-3 words below each panel. Wikipedia is in Figure 4.
| True → Predicted | Sentence |
| BBC News | |
Entertainment Sport | There were no specific details on the injuries. |
Business Tech | It will do so at a presentation in New York on 26 January. |
Business Politics | “I will give up my role as soon as the board of directors asks me to do so,” he said. |
Business Politics | In both cases, the justification has been national security, amid allegations that the Turkish firms are too close to Israel. |
Business Politics | “First and foremost this is a human tragedy.” |
Tech Business | Taipei’s TFC 101 Tower is more than 50m taller than the Petronas Towers in Kuala Lumpur, Malaysia—formerly the world’s tallest skyscraper. |
Entertainment Sport | “So to win something like this really feels great.” |
Business Tech | It is the second time Vodafone has come under the Commission’s scrutiny. |
| 20 Newsgroups | |
alt.atheism soc.religion.christian | When did I acquire lack of belief in the Easter Bunny? |
alt.atheism talk.politics.mideast | We liberated it from Spain. |
rec.sport.baseball sci.med | (Not to mention the possibility that no organ is particularly crucial in some third species.) |
comp.windows.x soc.religion.christian | Can anyone shoot obvious holes in this theory? |
alt.atheism soc.religion.christian | I, for one, don’t regret it. |
alt.atheism talk.politics.mideast | (He ignores the challenge that extreemists for peace tend to be quite insistent that everyone accept their ideals for the world, and have even turned quite violent.) |
rec.autos soc.religion.christian | I won’t argue too much about anything. |
alt.atheism talk.politics.mideast | Their tanks are almost all soviet, with about 100 French tanks (older ones). |
| Wikipedia | |
Language and lit. Music | Tokyopop licensed the series in North America, and released the first two volumes in 2007. |
Warfare History | Devereux became adjutant to Illinois National Guard Maj. Simon Bolivar Buckner, later a Confederate general during the Civil War and eventually a governor of Kentucky. |
History Sports and recr. | A friend of English captain Douglas Jardine from their Oxford days, Kent Hughes publicly defended Jardine’s tactics of sustained short pitched bowling against the Australian batsmen, arguing that Australia used similar tactics against England during the 1921 Ashes tour. |
History Language and lit. | The early history of Calakmul is obscure, although a dynastic list has been pieced together that extends back into an ancestral past. |
Geography Art and arch. | The original house was constructed about 1696 by William Morgan, around the time of his marriage to Margaret Avery. |
Mathematics Geography | Stevens, Peter S. Patterns in Nature. |
Language and lit. Sports and recr. | She later became a Distinguished Alumna of the University of North Carolina and the University of Iowa. |
Sports and recr. History | The delegation was also accompanied by the head boxing coach Lolo Heimuli. |
Table 8. Examples of topic disagreements with the source labels. We report the topic inherited by the sentence from its source document and the predicted topic (SentenceETM). The misassignments are on generic sentences, cases where the predicted topic better matches the sentence content, and overlaps between categories.
Top-words in ElecDeb60to20
Table 9 lists the twenty topics learned on the ElecDeb60to20 debate between Biden and Trump on 29/10/2020. Most of them correspond to identifiable themes of the discussion, often more specific than the moderator’s segments: within the COVID-19 segment, the model separates preventive measures (topic 19) from lockdowns (topic 0); within the candidates’ records, it isolates climate and energy (topic 11), forest fires (topic 14) and electric cars (topic 3). A second group of topics captures the register rather than the subject of the debate: conversational interjections (topic 4), personal accusations (topic 1), and claims of achievement (topic 5). These topics are a consequence of modeling individual sentences: in a debate, many sentences carry no policy content, and the model groups them into topics of their own rather than spreading them across the thematic ones. Finally, two topics (9 and 15) contain unrelated words and lack a clear interpretation.
References
- [1]Balikas, G., Amini, M.-R., and Clausel, M. (2016). On a topic model for sentences. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 921–924.arxiv.org/abs/1606.00253
- [2]Bianchi, F., Terragni, S., and Hovy, D. (2021a). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 759–766.
- [4]Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022.
- [5]Bouveyron, C., Girard, S., and Schmid, C. (2007). High-dimensional data clustering. Computational statistics & data analysis, 52(1):502–519.DOI
- [6]Boyd-Graber, J. and Blei, D. (2008). Syntactic topic models. Advances in neural information processing systems, 21.
- [7]Cha, T. and Lee, D. (2024). SentenceLDA: Discriminative and robust document representation with sentence level topic model. In Graham, Y. and Purver, M., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta. Association for Computational Linguistics.
- [8]Das, R., Zaheer, M., and Dyer, C. (2015). Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 795–804.DOI
- [9]Dieng, A. B., Ruiz, F. J., and Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453.
- [11]Greene, D. and Cunningham, P. (2006). Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd international conference on Machine learning, pages 377–384.DOI
- [12]Grootendorst, M. (2022). Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.arxiv.org/abs/2203.05794
- [13]Hoyle, A. M., Calvo-Bartolomé, L., Boyd-Graber, J. L., and Resnik, P. (2025). Proxann: Use-oriented evaluations of topic models and document clustering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15872–15897.
- [14]Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of classification, 2(1):193–218.DOI
- [15]Jiang, H., Zhou, R., Zhang, L., Wang, H., and Zhang, Y. (2019). Sentence level topic models for associated topics extraction. World Wide Web, 22(6):2545–2560.DOI
- [16]Lau, J. H., Newman, D., and Baldwin, T. (2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530–539.DOI
- [17]Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.arxiv.org/abs/1609.07843
- [18]Mimno, D., Wallach, H., Talley, E., Leenders, M., and McCallum, A. (2011). Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 262–272.
- [19]Reuter, A., Thielmann, A., Weisser, C., Säfken, B., and Kneib, T. (2025). Probabilistic topic modeling with transformer representations. IEEE Transactions on Neural Networks and Learning Systems, 36(8):14551–14565.
- [20]Schneider, J. (2023). Efficient and flexible topic modeling using pretrained embeddings and bag of sentences. arXiv preprint arXiv:2302.03106.arxiv.org/abs/2302.03106
- [21]Vinh, N. X., Epps, J., and Bailey, J. (2009). Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pages 1073–1080.
- [22]Wang, D., Zhu, S., Li, T., and Gong, Y. (2009). Multi-document summarization using sentence-based topic models. In Proceedings of the ACL-IJCNLP 2009 conference short papers, pages 297–300.DOI









