betaFeedback

WorksComputer Science

Interpretable Text Embeddings Based on Hierarchical Questions

Abstract

QA-based text embeddings represent texts as vectors consisting of 1/0 answers to yes/no questions. Although these embeddings provide detailed information for interpretation and achieve high task performance, they often require humans to examine a large number of questions. Furthermore, the relationship between the number of questions and the appropriate level of question granularity has not been sufficiently studied. To address these challenges, we propose a method for constructing hierarchical questions for text embeddings. As the hierarchy deepens, questions become more specific and their number increases. During interpretation, we reduce the number of questions that humans need to examine by replacing questions at deeper levels with aligned questions at shallower levels. The question hierarchy is constructed through recursive clustering, contrastive prompting, and question alignment. Experimental results on general-purpose tasks show that our method improves interpretability while maintaining task performance. It is also found that, when only a limited number of questions are available, questions at shallower levels tend to achieve higher task performance than those at deeper levels, even with the same number of questions.

Introduction

Text embeddings, which represent sentences, paragraphs, and documents as vectors, are fundamental to a wide range of natural language processing (NLP) applications, such as information retrieval [40] and enhancing the generative capabilities of large language models (LLMs) [24]. For this reason, methods for obtaining high-quality text embeddings have been extensively studied [35, 17, 30]. However, these embeddings remain difficult to interpret, making it challenging for humans to

understand which semantic aspects contribute to the similarity or dissimilarity between two texts. This lack of interpretability is particularly critical in high-stakes domains, such as legal and medical applications, where transparent NLP systems are highly desirable [29].

To develop interpretable text embeddings, various methods have been proposed [45, 7, 36]. In particular, QA-based text embeddings [10, 37, 38] shown in Figure 1 (a) are highly interpretable because each dimension of an embedding corresponds to a yes/no question and represents its binary (1/0) answer. While conventional QA-based text embeddings represent texts using answers to relatively detailed questions, they tend to require a large number of questions for human interpretation. Furthermore, it remains unclear how many questions are needed and how specific they should be for general-purpose tasks.

Overview of QA-based text embeddings and interpreting each embedding

Figure 1. Overview of QA-based text embeddings and interpreting each embedding.

This study organizes questions used for text embeddings into a hierarchical structure, where more specific questions are allocated at the lower levels. This approach has two advantages. First, it can flexibly change the number of dimensions of embeddings, i.e., the number of questions. It is preferable, since the optimal number of dimensions may differ across downstream tasks for which the text embeddings are applied. Second, it can alleviate the burden on humans to interpret text embeddings. Instead of reading all questions, a human can refer to a small number of general questions, as illustrated in Figure 1 (b), where the specific questions q1q_1, q2q_2, and q4q_4 are summarized by the general question q9q_9.

To generate questions of varying granularity, as shown in Figure 2, we recursively apply the question-generation framework proposed by Sun et al. [37], which combines clustering of large-scale text corpora with contrastive prompting. Contrastive prompting generates highly discriminative questions, and recursive clustering is expected to result in questions at shallower levels summarizing questions at deeper levels. The alignment between a question at a given level and a question at the preceding level is determined based on their entailment relationship.

Overview of question generation

Figure 2. Overview of question generation.

Experiments on three tasks, semantic textual similarity (STS) [5], text clustering [1], and information retrieval [40] show that our method substantially improves the Pareto front between embedding performance and cognitive load. Our analysis also shows that questions become more specific at deeper levels of the hierarchy, and that aligned questions have an entailment relation with high probability. We further find that, when only a limited number of questions are available, questions at shallower levels produce better text embeddings than questions randomly selected at deeper levels.

QA-based Text Embeddings

QA-based text embeddings [10, 37, 38] represent a text through its responses to a set of natural-language questions. Each dimension of the embedding corresponds to a yes/no question, with a value of 1 or 0 indicating a “yes” or “no” response, respectively. For instance, the text “A study shows that exercise is good for health.” may be represented as a vector of [1,0,1][1, 0, 1] for the questions “Is it about health?”, “Is it about politics?”, and “Does it describe re-

search?”. Thus, QA-based embeddings allow texts to be compared in semantic space while preserving the interpretability of individual dimensions.

Let Q={q1,q2,…,qm}Q = \{q_1, q_2, \ldots, q_m\} be a set of yes/no questions. The QA-based embedding e(x)e(x) of an input text xx is expressed as follows:

e(x)=[a1(x),a2(x),…,am(x)],(1)\mathbf{e}(x) = [a_1(x), a_2(x), \ldots, a_m(x)], \tag*{(1)}

where ai(x)a_i(x) denotes the binary answer to question qiq_i for xx. The model fi(x)f_i(x) predicts the probability that the answer to question qiq_i for sentence xx is “yes”, and ai(x)a_i(x) is determined as follows,

ai(x)={1 (yes)if fi(x)>τ0 (no)otherwise,(2)a_i(x) = \begin{cases} 1\ (\text{yes}) & \text{if } f_i(x) > \tau\\ 0\ (\text{no}) & \text{otherwise} \end{cases}, \tag*{(2)}

where τ\tau is a threshold. The model fi(x)f_i(x) is trained using a collection of sample texts and LLM-generated answers to those samples.

Since questions directly affect the quality of text embeddings, the method to create questions is a critical element in QA-based text embedding. Benara et al. [10] proposed QA-Emb, a method uses LLMs to generate yes/no questions with task-specific prompts containing examples. Sun et al. [37] further proposed CQG-MBQA, a framework for generating more discriminative questions for QA-based text embeddings. It first groups texts into clusters based on the semantic similarity and then uses LLMs to generate yes/no questions that distinguish between the clusters. Since this framework can generate interpretable text embeddings comparable to those produced by black-box text embedding methods, we build our question generation method based on this approach.

Algorithm 1 Question Generation
1: # Step1. Recursive Clustering
2: $\mathcal{S} \leftarrow \{s_1, s_2, \ldots\}$
3: $\mathcal{C}^{1} \leftarrow \operatorname{kmeans}(\mathcal{S}, k_{\mathrm{first}})$
4: for $l \in \{2, 3, \ldots, l_{\max}\}$ do
5: $\mathcal{C}^{l} \leftarrow \bigcup_{\mathcal{C}_{i}^{l-1}\in\mathcal{C}^{l-1}} \operatorname{kmeans}(\mathcal{C}_{i}^{l-1}, k)$
6: end for
7: # Step2. Contrastive Question Generation
8: for $l \in \{1, 2, \ldots, l_{\max}\}$ do
9: $\widetilde{\mathcal{R}}^{l} \leftarrow \bigcup_{\mathcal{C}_{i}^{l}\in\mathcal{C}^{l}} \{\widetilde{\mathcal{R}}_{i}^{l} \mid \widetilde{\mathcal{R}}_{i}^{l} \mathrel{\mathop{\mathrm{def}}=} \operatorname{cqg}(\mathcal{C}_{i}^{l}, \mathcal{C}^{l})\}$
10: $\widetilde{\mathcal{Q}}^{l} \leftarrow \bigcup_{\widetilde{\mathcal{R}}_{i}^{l}\in\widetilde{\mathcal{R}}^{l}} \widetilde{\mathcal{R}}_{i}^{l}$
11: end for
12: # Step3. Question Filtering
13: for $l \in \{1, 2, \ldots, l_{\max}\}$ do
14: $\mathcal{R}^{l} \leftarrow \operatorname{filtering}(\widetilde{\mathcal{R}}^{l})$
15: $\mathcal{Q}^{l} \leftarrow \bigcup_{\mathcal{R}_{i}^{l}\in\mathcal{R}^{l}} \mathcal{R}_{i}^{l}$
16: end for

Proposed Method

Our proposed method consists of three processes: question generation, question alignment, and model training. In the question generation process, we combine recursive clustering with the question-generation method proposed by Sun et al. [37] to generate highly discriminative questions at various levels of granularity (§ 3.1). In question alignment, each question at a given hierarchical level is matched with a question at the superior level with which it has an entailment relationship (§ 3.2). In the model training process, for each generated question, we train a model that predicts the probability to answer “yes” for a given sentence (§ 3.3).

Question Generation

Question generation process consists of three steps: recursive clustering, contrastive question generation, and question filtering. A simplified overview is shown in Algorithm 1, and each step is explained in detail below.

Step1. Recursive Clustering We first use the MEDI2 dataset1 (Muennighoff et al. (2025)), which contains approximately 6.8 million texts, as a large-scale corpus S\mathcal{S}. The recursive kk-means clustering is applied to S\mathcal{S}, where each text is encoded using an existing black-box text embedding model. At the first level, S\mathcal{S} is grouped into kfirstk_{\mathrm{first}} clusters C1\mathcal{C}^{1}. At subsequent levels, texts in each cluster from the previous level Cil−1\mathcal{C}_{i}^{l-1} are grouped into kk clusters, and Cl\mathcal{C}^{l} is obtained by merging them. Thus, the number of clusters increases by a factor of kk at each level, i.e., ∣Cl∣=k∣Cl−1∣\lvert\mathcal{C}^{l}\rvert= k\lvert\mathcal{C}^{l-1}\rvert. The parameter kfirstk_{\mathrm{first}} is set larger than kk to make a relatively large number of clusters at the top level.

Step2. Contrastive Question Generation For each cluster Cil\mathcal{C}_{i}^{l} at the level ll, questions are generated by the contrastive question generation (CQG) framework proposed by Sun et al. [37]. First, positive and negative samples are collected. The positive samples are npn_p texts randomly sampled from Cil\mathcal{C}_{i}^{l}. The negative samples include hard negatives and easy negatives. For the hard negatives, nhn_h texts are randomly sampled from the ncn_c clusters in Cl\mathcal{C}^{l} that are closest to Cil\mathcal{C}_{i}^{l}. For the easy negatives, nen_e texts are randomly sampled from the remaining clusters. The sampled positive and negative texts are then included in a prompt that instructs the LLM to generate 10 questions that yield “yes” for the positive samples and “no” for the negative samples. The prompt is provided in Appendix § 8. As a result of this step, 10 times as many questions are generated as there are clusters. In Algorithm 1, R~l\widetilde{\mathcal{R}}^{l} is a collection of question sets, where each question set is generated from a single cluster, while Q~l\widetilde{\mathcal{Q}}^{l} is a set of all questions at level ll.

Step3. Filtering Questions Many of the generated questions are semantically similar, resulting in redundancy that increases the computational cost of constructing QA-based text embeddings. Therefore, we apply a filtering process to the generated questions following a probing mechanism proposed by Sun et al. [37]. First, we calculate the quality for each question q∈Q~lq \in\widetilde{\mathcal{Q}}^{l} at the level ll. As in CQG, we first randomly sample ppp_p positive probes (samples) from the cluster Cil\mathcal{C}_{i}^{l} where qq is generated, php_h hard negative probes from ncn_c clusters close to Cil\mathcal{C}_{i}^{l}, and pep_e easy negative probes from other clusters. After that, the answers to qq are generated by a LLM for each probe, and the quality of qq is calculated as follows:

quality⁡(q)=# “yes” for positive probespp−# “yes” for negative probesph+pe.(3)\operatorname{quality}(q) = \frac{\#\text{ “yes” for positive probes}}{p_p} -\frac{\#\text{ “yes” for negative probes}}{p_h+p_e}. \tag*{(3)}

Afterward, all questions are encoded using a black-box text embedding method, and questions whose cosine similarity exceeds θ\theta are removed. In this filtering process, it is ensured that higher-quality Algorithm 2 Question Alignment

  1. TquestionT_{\mathrm{question}}: lmax⁡×∣Qlmax⁡∣l_{\max} \times\lvert Q^{l_{\max}}\rvert matrix

  2. for l∈{2,3,…,lmax⁡}l \in\{2, 3, \ldots, l_{\max}\} do

  3.     for Ril∈RlR_i^l \in R^l do

  4.         j←Tcluster[l][i]j \leftarrow T_{\mathrm{cluster}}[l][i]

  5.         for ql∈Rilq^l \in R_i^l do

  6.             q~l−1←\tilde{q}^{l-1} \leftarrow

  7.             argmax⁡ql−1∈Rjl−1entail⁡(ql,ql−1)\underset{q^{l-1}\in R_j^{l-1}}{\operatorname{argmax}}\operatorname{entail}(q^l,q^{l-1})

  8.             a←a \leftarrow index of qlq^l in QlQ^l

  9.             b←b \leftarrow index of q~l−1\tilde{q}^{l-1} in Ql−1Q^{l-1}

  10.             Tquestion[l][a]←bT_{\mathrm{question}}[l][a] \leftarrow b

  11.         end for

  12.     end for

  13. end for

questions are prioritized and retained. The detailed algorithm for this filtering process is described in Appendix § 9.

Question Alignment

We align questions at level ll to the most related question at the superior level, called the “parent question” hereafter. The alignment table Tquestion[l][a]T_{\mathrm{question}}[l][a] is defined as an index of the parent question of the aa-th question in QlQ^l, and computed by Algorithm 2. Let TclusterT_{\mathrm{cluster}} denote the precomputed alignment table that records an index of the parent cluster. For each question cluster RilR_i^l at level ll, an index of the parent cluster (jj) is obtained using TclusterT_{\mathrm{cluster}}. Next, the parent question q~l−1\tilde{q}^{l-1} is chosen from the questions ql−1q^{l-1} in Rjl−1R_j^{l-1}. Specifically, qlq^l and ql−1q^{l-1} are converted into declarative sentences dld^l and dl−1d^{l-1}, respectively, and a natural language inference (NLI) model predicts the probability that dld^l entails dl−1d^{l-1}. This score represents the validity of replacing qlq^l with ql−1q^{l-1} when a human interprets text embeddings. Then, the question with the highest entailment probability is selected, and its index in Ql−1Q^{l-1} is recorded in TquestionT_{\mathrm{question}}.

Model Training

The QA model fl(x)f_l(x) described in § 2 is trained based on the multi-task binary question answering (MBQA) framework proposed by Sun et al. [37]. For each question qq at the level ll, the training data for qq is created as the following procedure. First, 50 texts from the cluster CilC_i^l where question qq is generated, 30 texts from the ncn_c clusters closest to CilC_i^l within ClC^l, and 20 texts from all other clusters are randomly sampled. Next, the answers to a question qq for these texts are predicted by LLMs. After that, the pairs of the black-box embeddings from the sampled texts and their corresponding 1/0 answers are used as the training data to train a regression model. Training details are provided in Appendix § 10.

The QA-based text embedding for a given input text is obtained by feeding a black-box embedding into the regression models for all questions.

Level1234Baseline
# Clusters402001,0004,9965,000
# Questions2138613,46213,91416,419

caption: Number of clusters and questions.

:::

Experiments

This section demonstrates the effectiveness of our method by comparing it with the baseline. We first describe the experimental settings, such as the baseline and hyperparameters (§ 4.1). After that, we explain the metrics for measuring the quality and interpretability of QA-based text embeddings (§ 4.2). Finally, we evaluate the quality and interpretability of our method and the baseline on general-purpose tasks (§ 4.3).

Experimental Setup

We use CQG-MBQA (Sun et al., [37]) as the baseline. CQG-MBQA uses a single clustering step instead of the recursive clustering used in our method. For the recursive clustering described in § 3.1, we set the parameters kfirstk_{\mathrm{first}} and kk to 40 and 5, respectively. We use UAE-large-v1{2} for text embedding, gpt-4o-mini{3} for question generation, and mDeBERTa-base-v3{4} as the NLI model for question alignment. Table 1 shows the number of clusters and questions generated by our method and the baseline. The number of clusters at the fourth level is slightly smaller than 5,000 because some clusters at the third level contain fewer than five samples and therefore cannot be further divided into five clusters. The number of generated questions increases in proportion to the number of clusters. The distribution of the number of texts in each cluster is provided in Appendix § 11.

Table 2 shows the hyperparameters for question generation in our method and the baseline. For direct comparison, we use the same hyperparameter settings for the fourth level as for the baseline. As the hierarchical level becomes shallower, we increase npn_p by 2 and nhn_h and nen_e by 6 each. This is because clusters at shallow levels contain more samples and therefore cover broader regions of the semantic space, but using too few contrastive samples for these clusters may cause the generated questions to focus on only a limited region of the semantic space.

Levelnpn_pnhn_hnen_encn_cppp_pphp_hpep_eθ\theta
112363635320.8
210303035320.8
38242435320.8
46181835320.8
Baseline6181835320.8

Table 2. Hyperparameters for question generation.

Our method and the baseline are evaluated on the three tasks, STS ([5]), text clustering ([1]), and information retrieval ([40]), using the datasets included in the massive text embedding benchmark (MTEB) ([32]). Details of the datasets are provided in Appendix § 12.

Metrics

Embedding Quality Based on the basic settings in MTEB ([32]), the STS task uses Spearman’s rank correlation coefficient, the text clustering task uses the V-Measure, and the information retrieval task uses nDCG@10 as evaluation metrics for the quality of text embeddings.

Interpretability As a metric for measuring interpretability, we define cognitive load as the number of questions a human must ask during interpretation. Specifically, it is defined for each of the three tasks as follows.

STS: Let Ists\mathcal{I}_{\mathrm{sts}} be the indices of questions answered “yes” in both sentence vectors v1\mathbf{v}_1 and v2\mathbf{v}_2. The cognitive load is defined as the size of Ists\mathcal{I}_{\mathrm{sts}}.

Ists={i∣v1i=1∧v2i=1}(4)\mathcal{I}_{\mathrm{sts}} = \{i \mid v_{1i}=1 \land v_{2i}=1\} \tag*{(4)}
clsts(v1,v2)=∣Ists∣(5)\mathrm{cl}_{\mathrm{sts}}(\mathbf{v}_1,\mathbf{v}_2) = |\mathcal{I}_{\mathrm{sts}}| \tag*{(5)}

Clustering: Let Iclustering\mathcal{I}_{\mathrm{clustering}} be the indices of questions answered “yes” in all text embeddings in a cluster V\mathcal{V}, then clclustering(V)=∣Iclustering∣\mathrm{cl}_{\mathrm{clustering}}(\mathcal{V}) = |\mathcal{I}_{\mathrm{clustering}}|.

Iclustering={i∣∀vj∈V:vji=1}(6)\mathcal{I}_{\mathrm{clustering}} = \{i \mid\forall v_j \in\mathcal{V}: v_{ji}=1\} \tag*{(6)}

Retrieval: Let Iretrieval\mathcal{I}_{\mathrm{retrieval}} be the indices of questions answered “yes” in both the query vector u\mathbf{u} and the top-1 passage v\mathbf{v}, then clretrieval(u,v)=∣Iretrieval∣\mathrm{cl}_{\mathrm{retrieval}}(\mathbf{u},\mathbf{v}) = |\mathcal{I}_{\mathrm{retrieval}}|.

Iretrieval={i∣ui=1∧vi=1}(7)\mathcal{I}_{\mathrm{retrieval}} = \{i \mid u_i=1 \land v_i=1\} \tag*{(7)}

Here, the cognitive load for STS is the same as that defined by Sun et al. [37], while the cognitive loads for clustering and retrieval are newly added in this study.

Furthermore, we assume that question alignment reduces cognitive load by replacing the question with its aligned abstract question at the higher level, thereby reducing the number of questions a human must check. We define the cognitive load of embeddings with question alignment as

clxlqa(⋅)=∣{Tquestion[l][i]∣i∈Ix}∣,(8)\mathrm{cl}_{x}^{\mathrm{lqa}}(\cdot) = |\{T_{\mathrm{question}}[l][i] \mid i \in\mathcal{I}_x\}|, \tag*{(8)}

where x∈{sts,clustering,retrieval}x \in\{\mathrm{sts}, \mathrm{clustering}, \mathrm{retrieval}\} and ll is the question level.

Result and Discussion

Performance vs. Cognitive Load Figure 3 shows the performance and cognitive load on the STS, text clustering, and information retrieval tasks when the threshold τ\tau is 0.3. Several common trends can be observed across the three tasks. First, the performance and cognitive load of the fourth level are nearly identical to those of the baseline. This suggests that questions generated through recursive clustering provide similar embedding quality and cognitive load to those generated using a single clustering step. Second, there is a trade-off between task performance and cognitive load. In other words, obtaining higher-quality text embeddings requires more questions, increasing cognitive load. A user can choose the level of embeddings, considering a balance between the performance of the downstream task and the cognitive load of interpreting the model’s prediction. In this experiment, since cognitive load increases significantly from the third level to the fourth level while performance improves only slightly, the third level appears to offer the best balance between task performance and cognitive load. Finally, question alignment can reduce cognitive load while maintaining task performance. As a result, the Pareto front of task performance and cognitive load is significantly improved, outperforming the baseline. Results for other values of the threshold τ\tau are given in Appendix § 14, indicating a similar trade-off between performance and cognitive load across all values.5

Figure 3. Task performance vs. cognitive load (τ\tau is 0.3). Q-Align indicates the results after question alignment, shown as white-filled shapes.

Impact of Thresholds on Performance and Cognitive Load

Figure 4 and Figure 5 show the task performance and cognitive load for the threshold τ\tau, respectively. As shown in Figure 4, performance peaks around a threshold of 0.3 for all three tasks and gradually decreases thereafter. Furthermore, the difference in performance between the third and fourth level is not particularly significant, indicating that high-quality text embeddings can be generated for general-purpose tasks even with the number and granularity of questions at the third level. As shown in Figure 5, cognitive load decreases significantly as the threshold increases. Furthermore, the effect of reduction of cognitive load by question alignment is greater when the level is deep or the threshold is low. However, if the threshold is too low, the discrepancy between the model’s and human’s standards, such as a human interpreting a question the model answered “yes” to as “no”, may become greater, potentially reducing the reliability of the generated text embeddings.

Figure 4. Task performance for the different threshold τ\tau.

Figure 5. Cognitive load for the different threshold τ\tau. Q-Align indicates the results after question alignment, shown as white-filled shapes.

Analysis

This section analyzes our method from various perspectives: entailment relationships between aligned questions (§ 5.1), the appropriate question granularity (§ 5.2), and a case study (§ 5.3).

Entailment Relationships between Aligned Questions

To reduce cognitive load through question alignment, the target question needs to semantically summarize the source questions. To verify this, the entailment relationship between the aligned questions is estimated. Specifically, for the ii-th source question qilsq_i^{l_s} at the level lsl_s and the jj-th target question qjltq_j^{l_t} at the level ltl_t (where ls>ltl_s > l_t), we convert them into declarative sentences dilsd_i^{l_s} and djltd_j^{l_t}. After that, the probability p(djlt∣dils)p(d_j^{l_t}|d_i^{l_s}) that dilsd_i^{l_s} entails djltd_j^{l_t}, and the probability p(dils∣djlt)p(d_i^{l_s}|d_j^{l_t}) that djltd_j^{l_t} entails dilsd_i^{l_s} is estimated by an NLI model4. If p(djlt∣dils)p(d_j^{l_t}|d_i^{l_s}) is high (i.e., qilsq_i^{l_s} entails qjltq_j^{l_t}) and p(dils∣djlt)p(d_i^{l_s}|d_j^{l_t}) is largely lower than p(djlt∣dils)p(d_j^{l_t}|d_i^{l_s}) (i.e., qjltq_j^{l_t} is more abstract than qilsq_i^{l_s}), qjltq_j^{l_t} can be considered suitable for replacing qilsq_i^{l_s}. When the difference between lsl_s and ltl_t is 2 or greater, jj is determined by recursively consulting the alignment table TquestionT_{\mathrm{question}}. For example, if lsl_s is 3 and ltl_t is 1, jj is Tquestion[2][Tquestion[3][i]]T_{\mathrm{question}}[2][T_{\mathrm{question}}[3][i]].

Figure 6 shows the proportion of combinations of questions across different levels for which the entailment probability exceeded 50%. For many level pairs, more than 67% of lower-level questions entail upper-level questions. On the other hand, the proportion of upper-level questions that entail lower-level questions is small. These results indicate that the granularity of the questions becomes finer as the level deepens, while aligned

Heatmap showing entailment probabilities across premise and hypothesis levels

Figure 6. Proportion of cases where the probability of “entailment” exceeded 50%.

questions preserve semantic coherence. Therefore, our method generates hierarchical questions in which upper-level questions summarize lower-level questions. The distribution of the probability of entailment is provided in Appendix § 13.

Impact of Granularity of Questions

We investigate how question granularity affects task performance and cognitive load. Figure 3 shows that embeddings encoded by the questions at the fourth level achieve the highest performance, but this may stem from using the largest number of questions. To evaluate the appropriateness of the questions themselves, questions from deeper levels are randomly chosen so that the number of sampled questions matches that of the upper levels, and the resulting text embeddings are evaluated using the same metrics as in § 4. Figure 7 shows the mean Spearman’s rank correlation coefficient and cognitive load for the STS task across five different seeds when the threshold τ\tau is 0.3 and 0.7. When τ\tau is 0.3, there is little difference in performance across question granularities. In contrast, when τ\tau is 0.7, task performance at upper levels tends to be higher than that at lower levels with the same number of questions. This trend is particularly pronounced when the number of questions is small. When the threshold is high and “yes” answers are less likely, fine-grained questions are answered “no” in most cases, making the resulting text embeddings less informative, although cognitive cost is reduced. In such cases, abstract questions are more appropriate for QA-based text embeddings. Results for other values of τ\tau and other tasks are provided in Appendix § 16.

Task performance versus cognitive load for two values of $\tau$

Figure 7. Task performance vs. cognitive load when the number of questions is matched across levels by random sampling.

Case Study

As a case study, we present an example of interpreting text embeddings using our method. The two input texts are sampled from the MS MARCO dataset [8] for the retrieval task, and the threshold τ\tau is set to 0.5. Questions at the fourth level for which the model answers “yes” for both texts are replaced with their aligned questions at the third level. The results shown in Table 3 illustrate the interpretability of our method. For questions related to concepts shared by two texts (e.g., education and medical care), the models answer “yes” for both. In contrast, for questions related to concepts that differ between the two texts (e.g., specific numerical thresholds and unsafe conditions), the models answer “yes” only for the corresponding text. Furthermore, the aligned questions summarize the questions for which the model answers “yes” for both texts. This makes it easier for humans to interpret the text embeddings.

Text 1:what level is blood pressure too low
Text 2:As long as you are not feeling dizzy or passing out, and have a decent quality of life, it is totally okay to have that level of blood pressure. There are a lot of normal people walking around with blood pressures of 90/60 on no medication and it is just normal for them. The only time doctors treat low blood pressure is when there are any symptoms, such as dizziness, light-headedness, or the experience of passing out. Any of these could result in a potentially disastrous fall, so if you experience one of these symptoms then you do want to tell your doctor right away.
Source QuestionA1A2Target Question
Is the article likely to be used in academic settings or coursework?YesYesIs the article concerned with enhancing understanding or skills?
Is there a clear intent to explain a concept or process?YesYes
Is the information presented in a clinical or educational context?YesYesDoes the article discuss physical or anatomical concepts?
Is the information provided related to health or medical conditions?YesYes
Is there mention of commonly accepted blood pressure thresholds?NoYes
Is there an absence of alarming or unsafe conditions in the description?YesNo

Table 3. Case study of interpreting text embeddings using question alignment. A1 and A2 denote the answers of the model for Text 1 and Text 2, respectively.

Related Work

General Text Embeddings Text embeddings have been actively studied in the field of NLP [15, 35, 20]. Many recent text embedding models use Transformer encoders [16, 26, 41] trained using a contrastive learning framework [13, 17, 44]. More recently, methods that use the hidden states of LLMs as text embeddings have also been proposed. These include methods that make the attention in LLMs bidirectional and apply contrastive learning [9, 21] (Muennighoff et al. 2025), as well as methods that insert input text to specific prompt templates without updating the parameters of the LLMs [19, 23, 25].

Interpretable Text Embeddings Various methods have been proposed for interpretable text embeddings [18, 39, 7]. Many of these methods assign specific roles to each dimension of the text embeddings [29, 34, 36]. Opitz and Frank [33] mapped each dimension of the text embeddings to an abstract meaning representation metrics. Wang et al. [45] mapped each dimension of the text embeddings to an anchor text and defined its value as the similarity between the corresponding anchor text and the input sentence. However, because these methods produce dense vectors, their interpretations may vary across individuals. In contrast, QA-based text embeddings [10, 37, 38] are sparse, making such ambiguity less likely.

Conclusion

In this paper, we proposed an interpretable text embedding framework with hierarchical questions to improve the interpretability of QA-based text embeddings. Experiments across multiple tasks showed that our method improved the Pareto front derived from the trade-off between task performance and cognitive load. Our analysis further showed that the aligned questions frequently exhibited entailment relationships, and the questions in shallower levels can be more effective when only a limited number of questions are available.

Limitations

Although our method reduces cognitive load during interpretation, manually inspecting all generated questions remains difficult due to their large number. Therefore, our qualitative analysis is limited to sampled questions, and a more comprehensive human evaluation is needed to assess the quality and interpretability of the generated questions.

Another limitation is that our method focuses on improving the Pareto front derived from the trade-off between task performance and cognitive load rather than directly improving task performance. Future work could study how the question hierarchy can be used to improve embedding quality while keeping the embeddings interpretable.

In addition, we do not evaluate our method on multiple languages. The quality and granularity of generated questions may differ across languages, so it is still unclear whether our method works well in other languages. Evaluating our method in multilingual settings is therefore an important direction for future work.

Prompts

The prompt templates used in our experiments are shown in Figure 8–Figure 10. We use gpt-4o-mini3 for all LLM-based processing described in this section.

Prompt template for QA-based text embedding
Evaluate the following text chunk based on
the yes/no questions provided.

**Text Chunk:**
{text chunk}

**Questions:**
{questions}

**Instruction for the model:** Please read
the provided text chunk and answer each of
the questions with either “yes” or “no”.
Format the responses as follows:
1. yes/no
2. yes/no

Figure 8. Prompt template for QA-based text embedding.

Prompt template for contrastive question generation
Generate 10 simple yet insightful yes/no
questions that determine properties of an
article, where for all questions, the answer
will be “yes” for ALL the positive articles
and “no” for ALL the negative articles. Keep
questions concise and avoid using complex
sentence structures with “and” or “or” unless
necessary.

**Positive Articles:**
{positive articles}

**Negative Articles:**
{negative articles}

**Instruction:** Based on the excerpts
provided, generate 10 simple yet insightful
yes/no questions that can accurately
differentiate the positive articles from the
negative articles. Each question should be
concise and framed in such a way that it will
elicit a “yes” response for ALL positive
articles and a “no” response for ALL negative
articles. Avoid using complex sentence
structures with “and” or “or” unless
absolutely necessary. Format the questions
in a numbered list as shown below:
1. [First simple yes/no question]
2. [Second simple yes/no question]

Figure 10. Prompt template for contrastive question generation.

Prompt: QA-based Text Embedding

Figure 8 shows the prompt template used to obtain QA-based text embeddings. The prompt takes two inputs: an input text and a set of yes/no questions. The LLM is instructed to answer each question with either “yes” or “no” in the same order as the questions. The resulting answers are used as labels for the corresponding text–question pairs.

Prompt: Question-to-Declarative Conversion

Figure 9 shows the prompt template used for question-to-declarative conversion. The prompt takes a yes/no question as input. The LLM is instructed to convert the question into a corresponding declarative sentence and to output only the converted sentence.

Prompt template for question-to-declarative conversion
Convert the following question into a
declarative sentence.

**Question:**
{question}

**Instruction:** Please output only the
declarative sentence derived from the given
interrogative sentence.

Figure 9. Prompt template for question-to-declarative conversion.

Prompt: Contrastive Question Generation

Figure 10 shows the prompt template used for contrastive question generation. The prompt takes two sets of articles as input: positive articles and negative articles. The LLM is instructed to generate 10 concise yes/no questions that yield “yes” for all positive articles and “no” for all negative articles. To encourage simple question structures, we additionally instruct the LLM to avoid using “and” or “or” unless necessary.

Details of Filtering

Algorithm 3 shows the detailed procedure for question filtering. It consists of two steps, and Rstep1lR_{\mathrm{step1}}^l and Rstep2lR_{\mathrm{step2}}^l are collections of sets of questions after filtering in Step 1 and Step 2, respectively. It is basically the same as the method proposed by Sun et al. [37]. However, in this study, the flag topttop_t at line 14 is set to false to avoid selecting the top tt questions, because it may eliminate promising candidates.

Details of Training QA Models

The structure of the QA model is the same as that of Sun et al. [37]. The regression model is a multilayer perceptron that passes a 1024-dimensional vector through a (1024, 8) linear layer, a ReLU activation function, and an (8, 1) linear layer, in that order. The output logit is passed through a sigmoid function to obtain a probability of a “yes” answer. For training hyperparameters, we set the batch size to 20, the number of epochs to 10, and the learning rate to 1e-3. The model is trained using NVIDIA DGX Spark (128GB CPU).

Analysis of Clusters

The distribution of the number of texts in each cluster is shown in Figure 11.

Distribution of the number of texts in each cluster

Figure 11. Distribution of the number of texts in each cluster.

Evaluation Data

The evaluation data for each task is as follows:

STS Seven datasets: SemEval STS tasks 2012-2016 ([5, 6, 3, 2, 4]), STS Benchmark ([12]), and SICK-Relatedness ([28]).

Algorithm 3 Question Filtering

  1. Step1. Filtering for each cluster

  2. Rstep1l←∅R^{l}_{\mathrm{step1}} \leftarrow\emptyset

  3. for R∈Rl\mathcal{R} \in R^{l} do

  4. Rexclude←∅\mathcal{R}_{\mathrm{exclude}} \leftarrow\emptyset

  5. A←{(qi,qj)∈R×R∣qi≠qj}\mathcal{A} \leftarrow\{(q_i,q_j) \in\mathcal{R} \times\mathcal{R} \mid q_i \ne q_j\}

  6. for qi,qj∈Aq_i, q_j \in\mathcal{A} do

  7. if sim⁡(qi,qj)>θ\operatorname{sim}(q_i,q_j) > \theta then

  8. if quality⁡(qi)<quality⁡(qj)\operatorname{quality}(q_i) < \operatorname{quality}(q_j) then

  9. Rexclude←Rexclude∪{qi}\mathcal{R}_{\mathrm{exclude}} \leftarrow\mathcal{R}_{\mathrm{exclude}} \cup\{q_i\}

  10. end if

  11. end if

  12. end for

  13. R←R∖Rexclude\mathcal{R} \leftarrow\mathcal{R} \setminus\mathcal{R}_{\mathrm{exclude}}

  14. if top_t\mathit{top\_t} then

  15. R←top_t(R,t)\mathcal{R} \leftarrow\mathit{top\_t}(\mathcal{R},t)

  16. end if

  17. Rstep1l←Rstep1l∪{R}R^{l}_{\mathrm{step1}} \leftarrow R^{l}_{\mathrm{step1}} \cup\{\mathcal{R}\}

  18. end for 19.

  19. Step2. Filtering across clusters

  20. Rstep2l←∅R^{l}_{\mathrm{step2}} \leftarrow\emptyset

  21. for Rtar∈Rstep1l\mathcal{R}_{\mathrm{tar}} \in R^{l}_{\mathrm{step1}} do

  22. Rexclude←∅\mathcal{R}_{\mathrm{exclude}} \leftarrow\emptyset

  23. for Rref∈Rstep1l\mathcal{R}_{\mathrm{ref}} \in R^{l}_{\mathrm{step1}} do

  24. A←{(qi,qj)∈Rtar×Rref∣qi≠qj}\mathcal{A} \leftarrow\{(q_i,q_j) \in\mathcal{R}_{\mathrm{tar}} \times\mathcal{R}_{\mathrm{ref}} \mid q_i \ne q_j\}

  25. for qi,qj∈Aq_i, q_j \in\mathcal{A} do

  26. if sim⁡(qi,qj)>θ\operatorname{sim}(q_i,q_j) > \theta then

  27. if quality⁡(qi)<quality⁡(qj)\operatorname{quality}(q_i) < \operatorname{quality}(q_j) then

  28. Rexclude←Rexclude∪{qi}\mathcal{R}_{\mathrm{exclude}} \leftarrow\mathcal{R}_{\mathrm{exclude}} \cup\{q_i\}

  29. end if

  30. end if

  31. end for

  32. end for

  33. Rtar←Rtar∖Rexclude\mathcal{R}_{\mathrm{tar}} \leftarrow\mathcal{R}_{\mathrm{tar}} \setminus\mathcal{R}_{\mathrm{exclude}}

  34. Rstep2l←Rstep2l∪{Rtar}R^{l}_{\mathrm{step2}} \leftarrow R^{l}_{\mathrm{step2}} \cup\{\mathcal{R}_{\mathrm{tar}}\}

  35. end for

  36. return Rstep2lR^{l}_{\mathrm{step2}}

Clustering Seven datasets: TwentyNewsgroupsClustering6, StackExchangeClusteringP2P7, BiorxivClusteringP2P8, BiorxivClusteringS2S9, MedrxivClusteringP2P10, Medrxiv-ClusteringS2S11, and RedditClusteringP2P12

Retrieval Six datasets: MS MARCO ([8]), ArguAna ([42]), FiQA-2018 ([27]), NFCorpus ([11]), SCIDOCS ([14]), and SciFact ([43]).

Analysis of Questions

Length of Questions

The distribution of the number of characters in questions is shown in Figure 12. Even across different levels, the length of questions does not vary much.

Distribution of the number of characters in questions

Figure 12. Distribution of the number of characters in questions.

Concreteness of Questions

We calculate the concreteness of questions using wordtangible13. In wordtangible, a predefined dictionary, where the keys are words and the values are their concreteness scores, is used to compute the average concreteness of words as the concreteness of the sentence. In the original wordtangible setting, words not included in the dictionary are ignored. However, we believe that words not in the dictionary, such as proper nouns, should be considered concrete, so we assign them the maximum concreteness score (5 points). The distribution of the concreteness of questions is shown in Figure 13. As the level becomes deeper, the average concreteness tends to increase.

Distribution of the concreteness of questions

Figure 13. Distribution of the concreteness of questions.

Probability of Entailment

Figure 14 shows the distribution of entailment probabilities when the deeper-level sentences are the premise and the shallower-level sentences are the hypothesis, while Figure 15 shows the distribution of entailment probabilities when the premise and hypothesis are swapped.

Distribution of probabilities of entailment

Figure 14. Distribution of probabilities of entailment (deeper-level →\to shallower-level).

Distribution of probabilities of entailment

Figure 15. Distribution of probabilities of entailment (shallower-level →\to deeper-level).

Full Results of Performance vs. Cognitive Load

Figure 16–Figure 18 show the full results of the performance and cognitive load on the STS, text clustering, and information retrieval tasks, respectively.

Task performance versus cognitive load for $\tau$ is 0.1

(a) (a) τ\tau is 0.1

Task performance versus cognitive load for $\tau$ is 0.2

(b) (b) τ\tau is 0.2

Task performance versus cognitive load for $\tau$ is 0.3

(c) (c) τ\tau is 0.3

Task performance versus cognitive load for $\tau$ is 0.4

(d) (d) τ\tau is 0.4

Task performance versus cognitive load for $\tau$ is 0.5

(e) (e) τ\tau is 0.5

Task performance versus cognitive load for $\tau$ is 0.6

(f) (f) τ\tau is 0.6

Task performance versus cognitive load for $\tau$ is 0.7

(g) (g) τ\tau is 0.7

Task performance versus cognitive load for $\tau$ is 0.8

(h) (h) τ\tau is 0.8

Task performance versus cognitive load for $\tau$ is 0.9

(i) (i) τ\tau is 0.9

Figure 16. Task performance vs. cognitive load on the STS task.

Task performance vs. cognitive load for $\tau$ is 0.1

(a) τ\tau is 0.1

Task performance vs. cognitive load for $\tau$ is 0.2

(b) τ\tau is 0.2

Task performance vs. cognitive load for $\tau$ is 0.3

(c) τ\tau is 0.3

Task performance vs. cognitive load for $\tau$ is 0.4

(d) τ\tau is 0.4

Task performance vs. cognitive load for $\tau$ is 0.5

(e) τ\tau is 0.5

Task performance vs. cognitive load for $\tau$ is 0.6

(f) τ\tau is 0.6

Task performance vs. cognitive load for $\tau$ is 0.7

(g) τ\tau is 0.7

Task performance vs. cognitive load for $\tau$ is 0.8

(h) τ\tau is 0.8

Task performance vs. cognitive load for $\tau$ is 0.9

(i) τ\tau is 0.9

Figure 18. Task performance vs. cognitive load on the information retrieval task.

Task Performance by Dataset

Table 4–Table 6 present the performance on individual datasets for the STS, text clustering, and information retrieval tasks. CQG-MBQA (Repro.) denotes

TypeModelSTS12STS13STS14STS15STS16STS-BSICK-RAvg.
Black-boxBERT38.7857.9857.9863.1561.0646.3558.4054.81
Black-boxGloVe54.6469.1660.8172.3165.3461.5455.4362.74
Black-boxUSE64.4967.8064.6176.8373.1874.9276.6971.22
Black-boxSimCSE (Unsup.)66.0581.4973.6179.7278.1276.5272.2475.39
Black-boxSBERT (Ori.)74.5377.0073.1881.8576.8279.1074.2976.68
Black-boxSimCSE (Sup.)75.3084.6780.1985.4080.8284.2568.3879.86
Black-boxWhitenedCSE74.6585.7977.4984.7180.3381.4875.3479.97
Black-boxSBERT (New)73.0882.1376.7385.5880.2383.0979.3280.02
Black-boxOpenAI72.8486.181.1588.4985.0883.5679.0082.31
Black-boxAnglE79.0989.6285.0289.5186.6189.0682.6285.93
InterpretableBag-of-Tokens44.7552.0654.7868.6560.5954.8557.8756.22
InterpretableQAEmb-MBQA59.4063.1957.6869.2963.1871.3372.3365.20
InterpretableCQG-MBQA (Ori.)69.2180.1973.9180.6678.3082.6978.2177.60
InterpretableCQG-MBQA (Repro.)74.9579.9675.6682.4780.4784.2677.1679.28
InterpretableOurs (Level 4)75.5880.2375.6382.7580.2783.7376.7479.27
InterpretableOurs (Level 3)74.1381.3275.4682.0279.2683.1176.0778.77
InterpretableOurs (Level 2)67.2875.5270.8377.5074.7779.6074.4774.28
InterpretableOurs (Level 1)58.2869.8259.6872.4367.3672.0568.9666.94

Table 4. Task performance by dataset on the STS task.

TypeModelMS MARCOArguAnaFQANFCSCIDOCSSciFactAvg.
Black-boxBERT16.8628.292.194.302.8213.3411.30
Black-boxSimCSE (Unsup.)44.6338.349.849.885.5025.7222.32
Black-boxGloVe44.2736.3010.0913.878.0429.5823.69
Black-boxSimCSE (Sup.)47.8638.3310.4112.427.5329.5924.36
Black-boxSBERT (New)88.7447.1337.2732.2521.8262.6448.31
Black-boxAnglE90.4366.1544.8438.6522.9874.0756.19
Black-boxOpenAI92.1858.0555.0042.0723.1177.7758.03
InterpretableBag-of-Tokens29.7934.253.9921.516.7947.3623.95
InterpretableBM2568.4249.2825.1432.0815.7868.7043.23
InterpretableQAEmb-MBQA40.5134.758.233.873.7412.0117.18
InterpretableCQG-MBQA (Ori.)62.2147.7518.639.748.6732.8029.97
InterpretableCQG-MBQA (Repro.)67.7955.5716.647.5412.8725.1630.93
InterpretableOurs (Level 4)69.6655.0418.638.3612.6525.4031.62
InterpretableOurs (Level 3)61.8049.0015.127.0710.1820.2127.29
InterpretableOurs (Level 2)38.2935.645.625.475.699.5416.71
InterpretableOurs (Level 1)12.1614.542.293.112.124.116.39

Table 6. Task performance by dataset on the information retrieval task.

our reimplementation of CQG-MBQA, which corresponds to the baseline described in the main text. The results for all other methods, except CQG-MBQA (Repro.) and the proposed method, are taken from Sun et al. [37]. For CQG-MBQA (Ori.), CQG-MBQA (Repro.), and the proposed method, the threshold τ\tau is set to 0.5.

Full Results of Analysis of Appropriate Granularity of Questions

Figure 19–Figure 21 show the full results of the performance and cognitive load on the STS, text clustering, and information retrieval tasks when the number of questions is matched across levels by random sampling, respectively.

Figure 19. Task performance vs. cognitive load on the STS task when the number of questions is matched across levels by random sampling.

Task performance versus cognitive load for $\tau$ is 0.1

(a) τ\tau is 0.1

Task performance versus cognitive load for $\tau$ is 0.2

(b) τ\tau is 0.2

Task performance versus cognitive load for $\tau$ is 0.3

(c) τ\tau is 0.3

Task performance versus cognitive load for $\tau$ is 0.4

(d) τ\tau is 0.4

Task performance versus cognitive load for $\tau$ is 0.5

(e) τ\tau is 0.5

Task performance versus cognitive load for $\tau$ is 0.6

(f) τ\tau is 0.6

Image not extracted

(g) τ\tau is 0.7

Image not extracted

(h) τ\tau is 0.8

Image not extracted

(i) τ\tau is 0.9

Figure 21. Task performance vs. cognitive load on the information retrieval task when the number of questions is matched across levels by random sampling.

Task performance vs. cognitive load for $\tau$ is 0.1

(a) (a) τ\tau is 0.1

Task performance vs. cognitive load for $\tau$ is 0.2

(b) (b) τ\tau is 0.2

Task performance vs. cognitive load for $\tau$ is 0.3

(c) (c) τ\tau is 0.3

Task performance vs. cognitive load for $\tau$ is 0.4

(d) (d) τ\tau is 0.4

Task performance vs. cognitive load for $\tau$ is 0.5

(e) (e) τ\tau is 0.5

Task performance vs. cognitive load for $\tau$ is 0.6

(f) (f) τ\tau is 0.6

Task performance vs. cognitive load for $\tau$ is 0.7

(g) (g) τ\tau is 0.7

Task performance vs. cognitive load for $\tau$ is 0.8

(h) (h) τ\tau is 0.8

Task performance vs. cognitive load for $\tau$ is 0.9

(i) (i) τ\tau is 0.9

Figure 17. Task performance vs. cognitive load on the text clustering task.

TypeModelTNGSE-P2PBR-P2BR-S2SMR-P2PMR-S2SRD-P2PAvg.
Black-boxSimCSE (Unsup.)23.2128.5024.9019.5523.6021.9745.1426.70
Black-boxGloVe25.8328.5129.2719.1826.1220.3835.8226.44
Black-boxBERT23.3526.5530.1224.7726.0923.6043.3228.26
Black-boxSimCSE (Sup.)34.8629.4530.1524.6726.2524.1247.7431.03
Black-boxSBERT (New)47.4733.1336.9933.2134.2532.2454.8038.87
Black-boxAnglE51.7236.7239.3837.2333.2231.1865.3542.11
Black-boxOpenAI58.1436.8838.0336.5332.7031.2767.9643.07
InterpretableBag-of-Tokens8.5217.644.703.3211.3913.0515.6710.61
InterpretableQAEmb-MBQA36.7225.6824.6621.1625.5322.8546.5729.02
InterpretableCQG-MBQA (Ori.)40.0028.2234.8831.1331.0228.7154.4035.48
InterpretableCQG-MBQA (Repro.)43.9534.5036.7833.2131.4529.2957.9338.16
InterpretableOurs (Level 4)42.7834.5036.9332.6531.7429.4158.2438.04
InterpretableOurs (Level 3)41.0133.6435.6831.8031.0228.7156.6136.93
InterpretableOurs (Level 2)34.3232.3629.8727.3127.8725.7450.7832.61
InterpretableOurs (Level 1)28.7530.2123.5818.3623.8020.6038.6326.27

Table 5. Task performance by dataset on the text clustering task.

(d) τ\tau is 0.4 (e) τ\tau is 0.5 (f) τ\tau is 0.6

(g) τ\tau is 0.7 (h) τ\tau is 0.8 (i) τ\tau is 0.9

Figure 20. Task performance vs. cognitive load on the text clustering task when the number of questions is matched across levels by random sampling.

References

  1. [1]Charu C. Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. In Mining Text Data, pages 77–128. Springer US, Boston, MA.DOI
  2. [2]Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado. Association for Computational Linguistics.
  3. [3]Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland. Association for Computational Linguistics.
  4. [4]Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California. Association for Computational Linguistics.
  5. [5]Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393, Montréal, Canada. Association for Computational Linguistics.
  6. [6]Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. SemEval 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32–43, Atlanta, Georgia, USA. Association for Computational Linguistics.
  7. [7]Vishal Anand, Milad Alshomary, and Kathleen McKeown. 2026. iBERT: Interpretable embeddings via sense decomposition. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1413–1429, Rabat, Morocco. Association for Computational Linguistics.
  8. [8]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset. Preprint, arXiv:1611.09268.arxiv.org/abs/1611.09268
  9. [9]Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling.arxiv.org/abs/2404.05961
  10. [10]Vinamra Benara, Chandan Singh, John Xavier Morris, Richard Antonello, Ion Stoica, Alexander Huth, and Jianfeng Gao. 2024. Crafting interpretable embeddings for language neuroscience by asking LLMs questions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.pubmed.ncbi.nlm.nih.gov/40276238
  11. [11]Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Proceedings of the 38th European Conference on Information Retrieval. ECIR.DOI
  12. [12]Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.DOI
  13. [13]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR.
  14. [14]Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282. Online. Association for Computational Linguistics.arxiv.org/abs/2004.07180
  15. [15]Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.arxiv.org/abs/1705.02364
  16. [16]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.arxiv.org/abs/1810.04805
  17. [17]Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.arxiv.org/abs/2104.08821
  18. [18]John Hewitt, John Thickstun, Christopher Manning, and Percy Liang. 2023. Backpack language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9103–9125, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2305.16765
  19. [19]Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. Scaling sentence embeddings with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3182–3196, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2307.16645
  20. [20]Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. Prompt-BERT: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8826–8837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.arxiv.org/abs/2201.04337
  21. [21]Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2405.17428
  22. [23]Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. 2024. Meta-task prompting elicits embeddings from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10141–10157, Bangkok, Thailand. Association for Computational Linguistics.
  23. [24]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Preprint, arXiv:2005.11401.arxiv.org/abs/2005.11401
  24. [25]Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi, and Manabu Okumura. 2026. Reverse-sequel: Improving training-free text embeddings via text reversal in decoder-only llms. Preprint, arXiv:2606.05858.arxiv.org/abs/2606.05858
  25. [26]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.arxiv.org/abs/1907.11692
  26. [27]Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW ’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  27. [28]Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).DOI
  28. [29]Denis McInerney, Geoffrey Young, Jan-Willem van de Meent, and Byron Wallace. 2023. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8477–8494, Singapore. Association for Computational Linguistics.
  29. [30]John Xavier Morris and Alexander M Rush. 2025. Contextual document embeddings. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.02525
  30. [32]Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.
  31. [33]Juri Opitz and Anette Frank. 2022. SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 625–638, Online only. Association for Computational Linguistics.DOI
  32. [34]Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKeown, and Chris Callison-Burch. 2023. Learning interpretable style embeddings via prompting LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15270–15290, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.12696
  33. [35]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1908.10084
  34. [36]Wonseok Shin and Songkuk Kim. 2026. Aligning sentence embeddings to human concepts via sparse autoencoders. In ICLR 2026 Workshop on Representational Alignment (Re^4-Align).arxiv.org/abs/2607.00023
  35. [37]Yiqun Sun, Qiang Huang, Yixuan Tang, Anthony Kum Hoe Tung, and Jun Yu. 2025. A general framework for producing interpretable semantic text embeddings. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.03435
  36. [38]Yixuan Tang, Zhenghong Lin, Yandong Sun, Wynne Hsu, Mong Li Lee, and Anthony K. H. Tung. 2026. Asking the right questions: Ontology-grounded interpretable embeddings for biomedical text. Preprint, arXiv:2603.01690.arxiv.org/abs/2603.01690
  37. [39]Matthieu Tehenan, Vikram Natarajan, Jonathan Michala, Milton Lin, and Juri Opitz. 2025. Mechanistic decomposition of sentence representations. Preprint, arXiv:2506.04373.arxiv.org/abs/2506.04373
  38. [40]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  39. [41]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  40. [42]Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251, Melbourne, Australia. Association for Computational Linguistics.DOI
  41. [43]David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hanneke Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.arxiv.org/abs/2004.14974
  42. [44]Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text embeddings by weakly-supervised contrastive pre-training. Preprint, arXiv:2212.03533.arxiv.org/abs/2212.03533
  43. [45]Yile Wang, Zhanyu Shen, and Hui Huang. 2025. LDIR: Low-dimensional dense and interpretable text embeddings with relative representations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14397–14409, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2505.10354

Paper details

vlnc:W_8fbjxkz8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents