ISBC: Interpretable Sparse Binary Coding of Pretrained Text Embeddings

Introduction

Pretrained text embeddings support semantic comparison and prediction, but empirical text analysis also requires descriptions of document content. Researchers may wish to compare texts sharing selected attributes, examine variation within an existing category, or relate content to an external outcome. The individual coordinates of a high-dimensional embedding rarely provide such descriptions directly. A small set of binary dimensions offers an explicit way to select and compare document groups, provided that the content associated with those dimensions can be interpreted empirically.

We study this possibility with Interpretable Sparse Binary Coding (ISBC), which learns a sparse binary representation of frozen pretrained embeddings. The approach separates the linguistic representation supplied by a pretrained model from the estimation of binary coordinates for the study corpus. The embedding backbone remains frozen during ISBC training. A trainable encoder maps embeddings to binary activations, and an affine decoder reconstructs the original vectors. The learning objective controls activity, factor balance, dependence, and activation uncertainty alongside reconstruction. Neither observed taxonomies nor downstream outcomes enter representation learning.

Individual bits and their conjunctions select overlapping document subsets, while complete activation configurations, which we call profiles, partition the corpus. Comparing profiles that differ in one bit examines that coordinate within a fixed context; conjunctions leave the remaining coordinates unrestricted. These comparisons can reveal distinctions within existing categories and shared content across their boundaries, with interpretation grounded in document inspection. Profiles provide fine descriptions that can vary across initializations, rather than a unique discrete taxonomy.

We distinguish two questions: which properties of the embeddings survive compression, and how the resulting binary coordinates can be interpreted. For the first, we assess reconstruction, global and local geometry, and label prediction, while code entropy describes the diversity of observed profiles. Principal component analysis provides a continuous reconstruction reference at equal dimension ((Eckart and Young, 1936)). For the second, we inspect documents selected by bits and their combinations, using existing annotations as external descriptions. We examine how these properties depend on code width, activity, and the pretrained backbone. The cross-corpus analysis of Qwen-based codes finds stronger complete-profile agreement with some entity, event, and topic annotations than with affective or evaluative annotations (§ 7). The coordinates have no prescribed semantic content: their interpretation depends on the frozen embedding space, the fitted corpus, and the learned compression. An engagement application combines prediction from bits alone with exploratory comparisons of the content selected by bit combinations in low- and high-engagement groups (§ 9).

Most predictive evaluations are transductive: representation learning uses the full unlabeled corpus, while labels or outcomes are held out for predictive evaluation. A separate analysis applies a frozen encoder to additional documents excluded from representation fitting, showing that selected content associations persist within the same source collections.

We compare ISBC with iterative quantization (ITQ; [12]), BinarySentEmb ([45]), and HypotheSAEs ((Moyva et al., 2025)), examining reconstruction, code usage, and predictive utility under different binary activity constraints. We also compare profiles and conjunctions with embedding clusters (§ 4.4); their non-equivalence is assessed separately from the semantic interpretation of the selected documents.

The contribution is a compact, label-free representation designed for empirical analysis through binary coordinates and their combinations. Section 3 presents ISBC and its learning objective. Sections 4–6 examine the resulting document groups, quantify compression costs, and compare representations across embedding backbones and alternative encoders. Section 7 examines correspondence with existing annotations, while Section 8 shows how selected content associations persist in additional documents encoded without refitting. § 9 evaluates the bits as predictors of external outcomes and connects their use to document interpretation.

Related Work

Binary representations have precedents in semantic hashing [42], binary autoencoders [6], random-hyperplane hashing [7], and iterative quantization (ITQ) [12]. ITQ first reduces dimensionality through PCA and then learns an orthogonal rotation that minimizes quantization error before applying a sign threshold. Its bits are therefore thresholded linear projections, providing a linear binary reference at the same code width. Shen et al. [45] compress pretrained sentence embeddings with an affine sigmoid encoder, studying widths of $512$–$4{,}096$ bits. ISBC uses a nonlinear encoder with $8$–$32$ bits and explicit activity and factor-utilization controls. Mostard et al. [31] also investigate the semantics of binary coordinates at the word level.

Semantic IDs discretize embeddings for generative recommendation [39] and downstream ranking [48]. TIGER uses residual quantization and resolves collisions to identify items; shared codes in ISBC instead define document groups for analysis. Reconstruction fidelity and downstream utility remain distinct [16], motivating separate assessments of the properties retained under compression.

Sparse coding uses a limited number of active dictionary elements [36]. Sparse word representations include binary variants [11], while sparse autoencoders extract language-model features [9], support embedding-based semantic search [37, 46], and supply token-level features for document analysis [21]. ISBC focuses on a small binary code over frozen document embeddings, using coordinates and their combinations for empirical interpretation.

For external outcomes, supervised topic models jointly learn representations and document-response associations [4]. HypotheSAEs learns sparse embedding features before predictive selection and natural-language interpretation [32]. Its top-$k$ continuous activations and affine decoder make it a close alternative; the original study selects dictionary size and sparsity using validation prediction. ISBC learns and fixes its binary representation without downstream outcomes.

Interpretation can also be organized around topic mixtures [5, 3], embedding clusters with topic descriptions [15], or concept sets supplied through annotations or pretrained models [23, 35]. Clustering document embeddings provides a partition based on geometric similarity. We compare these partitions with complete binary profiles and examine how bit conjunctions extend across clusters (§ 4.4). ISBC assigns semantic descriptions after fitting, without a prescribed concept vocabulary. Sparsity and low covariance alone do not identify independent concepts [27]; interpretation requires substantive validation through document content [14].

Predictive probes assess properties accessible to a chosen estimator [18, 51]. Comparisons across representations must also account for information-preserving transformations [24] and chance agreement between partitions [50]. We accordingly distinguish predictive utility, aligned-coordinate correspondence, and document-group agreement from semantic interpretation.

Building on these precedents, ISBC learns a small set of binary dimensions with controlled activity from frozen document embeddings. The aim is to support document interpretation through individual activations and their combinations, without prescribing a taxonomy or using downstream outcomes during fitting.

Interpretable Sparse Binary Coding

ISBC learns a sparse binary representation of documents by reconstructing their frozen pretrained embeddings. Let $N$ denote corpus size, $d_i$ document $i$ ($i = 1, \ldots, N$), and $x_i = f_\theta(d_i) \in\mathbb{R}^{D}$ its embedding. The trainable model maps $x_i$ to a code $B_i \in\{0,1\}^{K}$ and uses that code to reconstruct $x_i$. Pretraining supplies the initial semantic representation, while the binary encoder learns which aspects of that representation can be retained within the chosen binary space. The code width $K$ and target number of active factors $A$ control the size and sparsity of this space. No category labels or external outcomes related to document $d_i$ enter the representation learning. Figure 1 summarizes the model and objective.

From pretrained embeddings to binary profiles

The backbone parameters $\theta$ remain fixed throughout training. Qwen is the baseline backbone, but the model accepts embeddings from other general-purpose or domain-specific encoders. Holding the backbone fixed separates the effects of binary representation learning from those of representation fine-tuning. Pooling and input prefixes follow the configuration of each backbone and are recorded in the run metadata. We retain the raw pooled vectors, without L2 normalization, centering, or rescaling, so the decoder is trained against the original embedding geometry. The input dimension $D$ is determined by the selected backbone.

The trainable encoder first applies layer normalization and a hidden layer with GELU activation,¹ then produces $K$ activation probabilities:

Here $p_i \in(0,1)^K$, $\sigma$ is the logistic function, and the temperature is $T = 0.5$ by default. Layer normalization acts inside the encoder; the reconstruction target remains $x_i$. Writing $\rho_{\mathrm{act}} = A/K$, we initialize the output bias to $T\operatorname{logit}(\rho_{\mathrm{act}})$ to place initial activation probabilities near the desired rate.

Training uses stochastic binary activations. For independent $U_{ik} \sim\operatorname{Uniform}(0,1)$, we set $B_{ik} = 1[p_{ik} > U_{ik}]$ and pass the straight-through quantity²

to an affine decoder,

Thus, reconstruction uses binary values in the forward pass, while gradients propagate through the probabilities to the encoder. The encoder and decoder are optimized jointly in FP32; the frozen backbone uses its configured precision, BF16 by default.

To account for differences in embedding location across backbones, we estimate the coordinate-wise mean $\mu_x$ from a fixed calibration sample of at most 1,024 cached embeddings and initialize the decoder intercept as

The initial decoder therefore maps the reference activity vector $\rho_{\mathrm{act}}1$ to $\mu_x$. This initialization adjusts the starting reconstruction to the embedding location while leaving the raw targets unchanged. Both decoder parameters remain trainable.

After fitting, we obtain deterministic codes by setting $B_{ik}=1[p_{ik}>0.5]$. A bit is a binary coordinate, also referred to as a factor. A profile is a complete configuration of all $K$ bits. Documents sharing a profile form a document group. A conjunction requires selected bits to be active while leaving the remaining bits unrestricted; it also defines a document group. Marginal bit-level analyses describe associations with individual coordinates, whereas profiles specify the joint states of all coordinates, including inactive bits. Semantic interpretations of bits and profiles are grounded in the documents and associations observed after fitting. Since bit indices are arbitrary, comparisons between independently fitted models require alignment rather than direct comparison of coordinate numbers.

Learning objective and estimation

The objective combines embedding reconstruction with constraints on code use. Reconstruction encourages accurate recovery of the pretrained embedding vectors, while the remaining terms regulate activity, factor utilization, redundancy, and uncertainty in binary decisions. For a minibatch of $n$ documents, the objective is

We define the reconstruction term as

where MSE averages over documents and embedding coordinates, and $s_x$ is the mean coordinate variance estimated from the same calibration sample used for $\mu_x$. Dividing by this scalar expresses error relative to embedding variance without changing the relative weighting of coordinates or transforming the targets. Activity is controlled at two levels:

The first term encourages each document to have an expected number of active factors near $A$. The second encourages each factor to be used at rate $\rho_{\mathrm{act}}$ across documents, discouraging factors that are always inactive or always active. These are soft penalties, so deterministic codes need not contain exactly $A$ ones. The balance coefficient is fixed at one, while the document-activity coefficient adapts during optimization. At each step, we compute the relative activity deviation $|n^{-1}\sum_{i=1}^{n}\sum_{k=1}^{K}p_{ik}-A|/A$ and update its exponential moving average $v_t$ with coefficient 0.95, initialized from the first observed deviation. Every 50 steps, we update the penalty weight according to

starting from $\lambda_s=1$. The penalty therefore increases when the smoothed relative deviation exceeds 0.05 and decreases when it falls below that value.

The remaining terms address dependence between factors and uncertainty within individual activations. Let $C$ be the sample covariance matrix of the minibatch probabilities, computed with denominator $\max(1,n-1)$. We use

The covariance penalty discourages pairwise dependence in factor probabilities. Minimizing the conditional Bernoulli entropy pushes those probabilities toward zero or one, reducing uncertainty in the sampled activations. Together with the activity and balance terms, it encourages decisive activations while regulating how often factors are used. These penalties do not guarantee independence or semantic interpretability.

All unsupervised fitting uses the complete document corpus. Embeddings are cached with ordered document identifiers and reused across epochs. Training records include the raw and weighted losses, adaptive penalty trajectory, activation statistics, parameter trajectories, gradient norms, and calibration statistics, allowing the realized behavior of the objective to be checked. Predictive evaluation subsequently uses the frozen codes and a seeded 80/20 document split. This holds out labels for evaluation while retaining all documents in unsupervised representation fitting. Supplementary Section S1 records the shared hyperparameters, implementation settings, and measured training costs.

AG News: code structure and information preservation

Unless otherwise specified, the empirical analyses use $K = 16$ and target activity $A = 6$ as a common reference configuration. This intermediate choice provides a common operating point rather than maximizing predictive performance for a particular dataset. The AG News case study first examines document groups and their interpretation, then measures compression and geometry, assesses the learning objective through ablation, and finally evaluates predictive utility.

The AG News dataset

As a driving example throughout this section, we use the AG News topic-classification benchmark introduced by Zhang et al. [53]. Each English-language document combines a news headline with its short description and is assigned to one mutually exclusive topic: World, Sports, Business, or Sci_Tech. The released benchmark contains 120,000 training documents (30,000 per class) and 7,600 test documents (1,900 per class), for a total of 127,600 documents. Because our binary encoder is trained without labels, we concatenate the original training and test partitions when extracting frozen embeddings and learning sparse binary codes; the original partition is retained only as provenance. See also Supplementary § 14.2.

Analysis of bit activations and profiles

We use Qwen/Qwen3-Embedding-0.6B, a 0.6-billion-parameter instruction-aware embedding model (Zhang et al. [54]). During training, mean activity settles near the target of six and the binary and probability-based traces become similar (Figure 2).

For the fitted code, the observed mean activity, $\bar{a}=N^{-1}\sum_{i=1}^{N}\sum_{k=1}^{K}B_{ik}$, is 5.94 (standard deviation 1.03; median 6). The target permits variation: 38.6% of documents activate exactly six bits and 87.0% activate five to seven. Individual bit prevalences range from 33.7% to 39.6%, near the reference rate $6/16=37.5\%$, consistent with the balance penalty.

The three-epoch fit produces 12,174 distinct profiles. The most frequent contains 1.08% of documents; 422 profiles cover half the corpus and 2,257 cover 80%. A further 4,445 profiles occur only once, accounting for 36.5% of profiles but 3.48% of documents. Thus, similar bit prevalences coexist with recurrent configurations and a long tail of rare profiles. The twenty most frequent cover 13.20% of documents (Figure S1 in Supplementary § 14.2).

Pairwise associations summarize dependence among bits without inspecting every profile. Figure 3 compares pointwise mutual information, $\mathrm{PMI}_{k\ell}=\log_{2}[P(B_k=1,B_\ell=1)/\{P(B_k=1)P(B_\ell=1)\}]$, with $\phi$, the Pearson correlation between binary coordinates. Probabilities are estimated from corpus frequencies. Positive PMI indicates more co-activation than marginal independence predicts; negative PMI indicates less. The activity budget can itself induce negative dependence, so negative associations do not establish semantic opposition. Supplementary § 17 explains this constraint.

We form an undirected graph from all 43 positive-PMI pairs, weighted by PMI, and identify bit communities using Leiden (Traag et al., [49]) with weighted modularity at resolution $\gamma=1$ (Reichardt and Bornholdt, [40]). Neither labels nor a prescribed number of communities enter the analysis. The resulting groups contain five, six, and five bits, with modularity $Q=0.4646$ (Figure 4). The partition is identical across 20 optimizer seeds and unchanged at the nearby resolutions 0.75 and 1.25. These results identify three reproducible groups of bits that tend to activate together, providing a basis for examining the document content associated with their joint activation. Supplementary § 17.1 gives the full sensitivity analysis and graph construction details.

The right panel adds the 77 negative-$\phi$ associations as dashed edges without changing the fitted communities. Of these, 69 connect different communities and account for 96.6% of the total absolute negative correlation weight. This concentration is consistent with a separation between co-activation groups: bits within each community tend to activate together, while negative associations occur predominantly across communities.

How bit activations and conjunctions relate to an existing taxonomy

We examine the learned representation against the AG News taxonomy $\mathcal{T} = \{\text{World}, \text{Sports}, \text{Business}, \text{Sci\_Tech}\}$. These mutually exclusive categories reflect choices about topic definitions and assignment, although individual documents can address several themes. We use them as an external analytical reference for a representation learned without labels.

Figure 5 relates category associations to the co-activation communities. Their alignment is partial: in particular, Business-associated bits occur in more than one community. For each community, we select documents in which all member bits are active, leaving the remaining coordinates unrestricted. Each resulting group can contain several complete profiles; the Sankey plot shows its category composition. Intersecting these groups with the Business label identifies two disjoint subsets:

• Business $\cap$ Sci_Tech-dominant document group (147 documents; 0.461% of Business): commercial activity in technology markets. For example, “IBM to make digital image sensors for Kodak” (ID 30040) describes a commercial agreement to manufacture digital imaging technology, illustrating the intersection of business and technological content.

• Business $\cap$ World-dominant document group (22 documents; 0.069% of Business): economic activity in international political and institutional settings. For example, “Donors seek clarity on PA finances” (ID 125602) concerns financial transparency following a change in Palestinian leadership.

The technology-market conjunction also selects “Internet patents sold at S.F. auction” (ID 3785), a Sci_Tech-labelled report on an auction of patents covering technologies for Internet business transactions, illustrating the intersection of technological content and commercial activity. These examples illustrate how documents assigned to a single editorial category can also contain elements associated with another. The bit conjunctions bring together such documents through shared content that crosses the original category boundaries.

The non-Business documents satisfying the same conjunctions emphasize technological products and design in the first case, and political actors, diplomacy, and conflict in the second (Table S4, Supplementary § 14.2.1). Across the complete subsets, 127 of the 147 Business documents in the technology-market conjunction (86.4%) belong to profiles whose most frequent category is Sci_Tech; in the political conjunction, 9 of 22 (40.9%) belong to profiles whose most frequent category is World. Tied maxima are excluded from these numerators. Figure S4 and Figure S5 in the Supplementary Material locate the illustrative documents within these complete-profile distributions. Four Sports-labelled documents in the World-dominant group similarly concern security, institutional authority, or club ownership. These examples show how conjunctions connect documents through shared contexts across editorial categories. Supplementary § 14.2.1 provides the document descriptions, activation heatmap, and inspection details.

Sci_Tech-dominant document group			World-dominant document group
Term	Non-Business	Business	Term	Non-Business	Business
line	3.7	0.0	minister	14.8	0.0
week	6.3	0.7	peace	14.4	0.0
mp3	3.6	0.0	president	14.0	0.0
advanced	3.0	0.0	talks	12.9	0.0
model	2.6	0.0	prime	11.0	0.0
imac	2.5	0.0	prime minister	10.7	0.0
design	2.3	0.0	rebels	10.3	0.0
released	2.3	0.0	sudanese	8.7	0.0
mac	2.2	0.0	troops	8.6	0.0
micro devices	2.2	0.0	end	7.2	0.0

A complementary comparison examines variation within a category. Among 224 pairs of observed profiles differing in exactly one bit and containing at least 100 documents each, we select the pair maximizing the smaller group size, without using labels or text content. The selected profiles differ only in bit 08 and contain 1,050 and 1,224 documents, both almost entirely labelled Sports. With the other fifteen coordinates fixed, the active group concentrates on American football, while the inactive group includes baseball, basketball, and other sports. Document inspection and term-presence counts support this distinction (Table 1).

Characteristic	Bit 08 inactive	Bit 08 active
Documents	1,050	1,224
Sports (%)	97.43	98.69
Touchdown or quarterback (%)	1.62	45.75
Home run or inning (%)	23.90	1.47

Once again, the post-hoc category composition in Figure 5 illustrates how bit conjunctions connect documents across editorial categories. Mutually exclusive labels provide a useful organization of a corpus, but a single assignment need not capture all semantic aspects of a document. Bits and their combinations make some of these shared attributes explicit, supporting comparisons within and across the existing taxonomy.

Binary profiles and conjunctions versus embedding clusters

Could clustering the embeddings recover the document groups selected by bit conjunctions? We fit $k$-means [26] to the same 127,600 raw, unnormalised Qwen embeddings with $M \in\{32, 64, 256\}$ clusters, without labels or binary codes. We compare these partitions with individual bits, overlapping conjunctions, and complete profiles.

For bit $B_k$ and cluster membership $C$, let $H$ denote Shannon entropy computed with base-two logarithms. The residual entropy fraction $H(B_k \mid C)/H(B_k)$ measures variation remaining within clusters: zero means that cluster membership determines the bit, and one indicates independence. Its mean across the sixteen bits decreases from 0.578 at $M = 32$ to 0.489 at $M = 256$, against approximately one under prevalence-preserving permutation. The cluster assignments account for only part of the variation in bit activations, a pattern also observed for the alternative binary representations examined in Supplementary Section S3.3. Documents within the same cluster can therefore differ along binary dimensions, while documents in different clusters can share particular activations. Clustering and binary representations thus provide complementary descriptions of how documents are organized in the embedding space.

Conversely, the median three-bit conjunction requires 6, 11, and 30.5 clusters to cover 80% of its documents, compared with support-preserving random references of 21.3, 41.3, and 149.8 (Supplementary § 15.5). Conjunctions concentrate within subsets of clusters but extend across their boundaries, while clusters contain multiple complete profiles. The representations are therefore associated without defining equivalent document groups.

Clustering supplies mutually exclusive groups; bits and conjunctions supply overlapping memberships that can be inspected separately, as in § 4.3. Complete profiles record those memberships jointly and provide finer contexts, rather than a unique discrete taxonomy. Supplementary § 15.1 reports profile-level entropy, chance-corrected partition agreement, and variation across initializations. Category-based descriptions remain separate in Supplementary Section S3.4.

Code capacity and embedding reconstruction

Code capacity refers to the range of distinct bit configurations available to represent documents. Profile counts and entropy measure how extensively this capacity is used.

We compare Qwen codes on all 127,600 AG News documents at widths $K \in\{8, 16, 32\}$ and several activity targets (Table 2). Each configuration uses one encoder fit with seed 1234; these comparisons describe sensitivity to width and activity for that seed. Variation across initializations

caption: Qwen on AG News across code widths and activity targets. Each row represents one encoder fit with seed 1234. $A$ is target activity and $\bar{a}$ is observed mean activity. $\widehat{H}$ and $\widehat{H}_{\mathrm{MM}}$ are empirical and Miller–Madow–corrected profile entropy, in bits/document. $R^{2}_{\mathrm{trained}}$ evaluates the jointly trained decoder; $R^{2}_{\mathrm{OLS}}$ evaluates an affine decoder refitted by ordinary least squares with the binary codes held fixed. $R^{2}_{\mathrm{PCA},K}$ is the cumulative variance explained by the first $K$ principal components.

:::

is assessed separately in § 4.7.

A $K$-bit code admits at most $2^{K}$ profiles. For a complete configuration $b$, $p(b) = P(B = b)$ is its occurrence probability, estimated by the observed corpus proportion $\hat{p}(b) = N^{-1}\sum_i 1\{B_i = b\}$. The empirical entropy $\widehat{H}(B) = -\sum_{b:\hat{p}(b)>0}\hat{p}(b)\log_2\hat{p}(b)$ measures profile diversity in bits per document; $2^{\widehat{H}(B)}$ is the number of equally frequent profiles with the same entropy. Higher entropy indicates that documents are distributed across a larger effective number of profiles, providing a finer description of their joint activation patterns. Lower entropy indicates greater concentration in shared configurations. Increasing code width makes more configurations available and is therefore expected to support finer distinctions. Consistent with this expectation, at target activity six, increasing width from 16 to 32 raises entropy from 11.32 to 14.20 bits per document, corresponding to approximately 2,600 versus 19,000 effective profiles. The wider code therefore provides substantially greater granularity. We also report Miller–Madow–corrected entropy to assess finite-sample sensitivity; its definition and interpretation appear in Supplementary § 16.1.

The affine decoder reconstructs embedding $x_i$ as $g(B_i) = W_{\mathrm{dec}}B_i + b_{\mathrm{dec}}$: an intercept plus the directions selected by the active bits. We measure reconstruction relative to the corpus mean $\bar{x}$ by

All reconstructions lie in an affine subspace of dimension at most $K$. PCA supplies the optimal affine reference at this dimension with unrestricted continuous coefficients (Eckart and Young, 1936); its score $R^2_{\mathrm{PCA},K}$ is the cumulative proportion of embedding variance explained by the first $K$ principal components.

At target activity six, increasing width from 16 to 32 improves trained reconstruction from 0.160 to 0.214; increasing the 32-bit activity target to sixteen raises it to 0.232. Refitting the affine decoder by ordinary least squares with fixed codes yields only small improvements. At 16 bits, $R^2$ rises from 0.160 to 0.161, compared with 0.245 for PCA. The trained decoder therefore approaches the best affine fit to these codes, while closing the remaining gap requires changes to the representation (see Supplementary § 17).

At target activity six, the 8-, 16-, and 32-bit codes contain 78, 12,174, and 48,371 observed profiles. Increasing width permits finer empirical differentiation at a common activity target. Cross-capacity bit associations also connect individual coordinates in narrower codes to several coordinates in wider codes, including inverse activation patterns (see also Supplementary Section S4.5).

Geometric preservation

Geometric preservation concerns how relationships between documents in the embedding space are reflected in the binary space. We examine this at two scales: globally, through correlations between pairwise distances, and locally, through the assignment of nearest neighbors to the same binary profile. We use the configurations introduced in § 4.5.

For the global comparison, we calculate distances for the same 100,000 document pairs drawn from 2,000 sampled documents (seed 1234). Spearman’s $\rho$ measures how consistently the two representations order these pairs from closer to more distant; higher correlations indicate greater agreement. We compare cosine distances in both spaces, and Euclidean distances in the original embeddings with Hamming distances in the codes.

At 16 bits and target activity six, cosine–cosine correlation is 0.494 and Euclidean–Hamming correlation is 0.330. Across configurations, the respective ranges are 0.449–0.501 and 0.291–0.374. The binary codes therefore preserve angular distance ordering more closely than Euclidean distance ordering in these comparisons (see Supplementary Section S4.2 and Table S9).

$K$	$A$	Cosine–Cosine	Cosine–Jaccard	Euclidean–Hamming	Euclidean–Jaccard
8	4	0.449	0.449	0.291	0.290
8	6	0.453	0.452	0.335	0.336
16	6	0.494	0.493	0.330	0.311
32	6	0.495	0.495	0.356	0.320
32	8	0.496	0.495	0.374	0.312
32	16	0.501	0.500	0.344	0.333

For the local comparison, we find each sampled document’s nearest embedding neighbor among all other corpus documents and check whether the two receive the same binary profile. Starting from embedding neighbors avoids arbitrary selection among documents tied at zero binary distance. At 16/6, profile agreement is 31.70% for cosine neighbors and 31.75% for Euclidean neighbors, compared with 0.1470% for a uniformly selected other document (Table 3).

$K/A$	NN: cosine	NN: Euclidean	Random agreement	Shared-profile coverage
8/4	58.20	57.20	2.8374	100.00
8/6	74.95	74.00	7.2928	100.00
16/6	31.70	31.75	0.1470	96.75
32/6	22.85	22.60	0.0242	75.60
32/8	13.95	13.95	0.0079	49.75
32/16	8.95	8.80	0.0014	24.80

Wider codes distinguish more profiles and reduce exact agreement between neighboring documents. Nevertheless, nearest-neighbor agreement remains above the random reference in every configuration. The global and local results thus show complementary aspects of geometric preservation: partial agreement in pairwise distance ordering and a greater tendency for nearby documents to share a binary profile.

Ablation of the learning objective

We examine how the components of the loss function (6) contribute to reconstruction, geometry, and code use through a paired leave-one-term-out ablation. All variants use the same frozen, unnormalized Qwen3 embeddings for all 127,600 AG News documents, with $K = 16$ and activity target $A = 6$. For each seed (1234, 2025, and 31415), we fit the full objective and five variants, setting one coefficient to zero. The 18 fits share encoder initialization and the document-order random-number stream within each seed. Removing activity control also disables its adaptive coefficient update. Architecture and decoder initialization follow § 3.

Figure 6 summarizes changes relative to the full objective, paired within seed and then averaged. Reconstruction is evaluated with an affine decoder refitted by ordinary least squares to each frozen binary code, separating code recoverability from the original decoder’s training. Geometric comparisons follow § 4.6, using the same 2,000 sampled documents and 100,000 pairs. Code diagnostics use the full corpus. Supplementary Section S4.4 provides absolute values, variation across seeds, community graphs, and the secondary predictive evaluation.

The balance term (9) prevents the constant-code solution observed in all three fits without it: six bits are always active and ten always inactive. Both marginal bit entropy and profile entropy become zero. The activity target is met exactly, but the code cannot distinguish documents. Balancing factor use therefore serves a purpose distinct from controlling the number of active bits.

The reconstruction term (7) supports linear recoverability and correspondence with embedding geometry. Removing it increases refitted-decoder MSE by 7.7% and reduces angular and Euclidean–Hamming correlations by 15.8% and 19.4%, respectively. Profile entropy falls by 51.1%, whereas mean marginal bit entropy changes by only 1.1%. Mean absolute bit correlation increases from 0.120 to 0.434: individual coordinates still vary across documents, but their joint configurations become more repetitive. Effective profiles decrease from approximately 2,501 to 47.

The activity-control term (8) keeps individual documents near the prescribed activity budget. Removing it increases the mean absolute deviation from six active bits from 0.753 to 1.319, while improving refitted-decoder MSE by only 0.4%. Angular agreement decreases, whereas Euclidean–Hamming agreement increases. The activity constraint therefore trades a small amount of reconstruction accuracy for tighter control of the code and changes which geometric relations are preserved.

The entropy term (12) promotes decisive activation probabilities. Without it, conditional Bernoulli entropy increases from 0.0076 to 0.3026 nats, where a nat is the natural unit of information obtained using natural logarithms. This measures uncertainty within soft activations, whereas marginal bit entropy measures variation of binary coordinates across documents. Removing the term slightly improves binary reconstruction and increases profile entropy, but also increases activity error and reduces angular agreement. Entropy regularization thus favors probabilities close to binary decisions within the joint objective.

The decorrelation term (11) has a smaller observed effect. Removing it increases mean absolute binary correlation by 6.9%, from 0.120 to 0.128, while reconstruction and geometric correlations change little. Its penalty acts on covariance between probabilities; the binary correlation diagnostic indicates a modest reduction in coordinate dependence at this operating point.

Together, these results distinguish the roles of all five terms: balance maintains factor utilization, reconstruction preserves recoverable embedding structure, activity controls the sparse budget, entropy sharpens decisions, and decorrelation moderates coordinate dependence. Their combination provides a compromise among these properties, with contributions assessed conditional on the remaining terms and the fixed training configuration.

Predictive utility

ISBC learns document attributes without targeting a predefined taxonomy. Since AG News provides editorial categories, we also examine how readily these labels can be predicted from the binary codes. We use balanced accuracy (BA), the mean category recall of a logistic classifier, and macro-F1, the mean class-specific F1 score, on a common 80/20 label split. The encoder uses the full unlabeled corpus; labels enter only classifier fitting and evaluation. This probe assesses predictive utility for a specified estimator and taxonomy [51] (see Supplementary Section S1.1).

The 16/6 code achieves 85.62% BA against 91.09% for the original embeddings, retaining 93.99% of their BA. At the same activity target, 32 bits retain 96.20%, and the 8/6 code also exceeds the sixteen-bit reference. The common default is therefore not the best-performing configuration for these labels. Increasing activity at 32 bits improves reconstruction but reduces BA, illustrating the different responses of these two properties to the representation budget (see Supplementary Section S4.3 and Table S10).

Representation	$K$	$A$	BA [95% CI]	Relative BA	Macro-F1
Original embeddings	–	–	91.09 [90.76, 91.44]	100.00	91.08
Binary code	8	4	85.17 [84.74, 85.59]	93.50	85.10
Binary code	8	6	86.79 [86.38, 87.19]	95.28	86.73
Binary code	16	6	85.62 [85.19, 86.04]	93.99	85.57
Binary code	32	6	87.63 [87.23, 88.02]	96.20	87.60
Binary code	32	8	86.94 [86.54, 87.35]	95.44	86.91
Binary code	32	16	86.42 [86.01, 86.83]	94.87	86.38

For our purpose, these scores complement the document analyses in § 4.3, where bit combinations identify both distinctions within categories and content shared across their boundaries. Although compression reduces category-prediction accuracy, the codes retain utility for document comparisons that extend beyond reproducing the supplied taxonomy.

Robustness across embedding backbones

We fit $K = 16$, $A = 6$ binary encoders to the same 127,600 AG News documents using Qwen3-Embedding-0.6B [54], all-MiniLM-L6-v2 (Sentence Transformers, n.d.), Llama-NV-Embed-Reasoning-3B [34], Harrier-OSS-v1-270M [29], and EmbeddingGemma 2 [13]. These backbones span Qwen, Llama, and Gemma families, with MiniLM as a compact BERT-like baseline. They differ in size, attention, pooling, and embedding dimension (see Supplementary Section S6.1). All encoders use frozen, unnormalized embeddings and no labels during fitting.

Reconstruction and predictive utility

Table 4 compares each binary code with its own embedding reference, using the common predictive split. Mean activity stays close to six for every backbone. The codes retain 92.38–95.24% of embedding-based balanced accuracy and 52.87–70.73% of the variance explained by the corresponding sixteen-component PCA reconstruction. Their ordering differs between these measures. The effective profile count ranges from 521.3 to 6,488.6, showing that a common representation budget produces different distributions of document groups.

Profile-level correspondence

We compare complete-profile partitions using two complementary measures. The adjusted Rand index (ARI; Hubert and Arabie, [19]) measures agreement on whether pairs of documents belong to the same group. Adjusted mutual information (AMI; Vinh et al., [50]) measures the information shared by the two profile assignments after correction for chance. Both measures equal one for identical partitions and have an expected value of zero under random assignments with the observed group sizes held fixed. Across backbone pairs, ARI ranges from 0.045 to 0.169, indicating limited agreement in exact document grouping, while AMI ranges from 0.165 to 0.376, showing shared information beyond this chance reference. The fitted encoders therefore produce related but different complete-profile partitions. These comparisons use every document and profile, including singletons, and require no alignment of bit indices. Supplementary Section S6.2 reports all pairwise scores and partition sizes.

We also match individual profiles by shared document membership, without category labels. Starting from each of the twenty most frequent Qwen profiles, we successively select a profile from another backbone and restrict the documents to the resulting intersection. The selected profiles have similar category compositions despite different membership boundaries, with category-composition agreement exceeding both random-reference means reported in Supplementary Section S6.2. For the four most frequent Qwen profiles, the five-profile intersections contain only 2–69 documents, although the first matching step retains approximately 19–90% of the reference group. The Sankey plots, full matching procedure, and random-reference comparisons are given in Supplementary Section S6.2.

Bit-level correspondence across backbones

We next examine whether individual binary coordinates recover related activation patterns across backbones. For each backbone pair, we match the sixteen bits one-to-one by maximizing total normalized mutual information (NMI) with the Hungarian algorithm.³ NMI measures dependence in both concordant and opposite activation patterns; the sign of the binary correlation $\phi$ distinguishes these cases. Among the 160 selected matches on 80 bit nodes, 109 are positive and 51 negative. We group concordant activations using Leiden (Traag et al., [49]) on the positive matches, weighted by NMI, at resolution one. Category labels enter neither matching nor clustering.

The resulting graph contains eleven connected communities and two singletons (Figure 7). Ten communities include all five backbones and one includes four, indicating that the matched activation patterns recur across multiple embedding families. Within-community edges account for 94.1% of the positive weight, while all 51 negative selected matches connect different communities or singleton nodes. Thus, concordant activations form groups across backbones, with opposite activation patterns linking those groups. Supplementary § 18.3 reports optimizer and resolution sensitivity.

To interpret these communities, we select documents with an active member bit in at least two backbones. This permissive rule defines overlapping groups. Business and Sci_Tech jointly account for 93.3% of C10, while Sports and World account for 87.3% of C6. The matched coordinates therefore locate broad document groups spanning multiple editorial categories. Their pooled composition summarizes the selected documents rather than implying that each document combines both topics. All category shares appear in Supplementary Table S18.

Within those groups, combinations of active and inactive member bits identify more specific subsets (Table 5). For C10, one pattern selects 2,969 documents, 85.5% labelled Business, while another selects 9,171 documents, 90.2% labelled Sci_Tech. The C6 examples similarly distinguish Sports- and World-associated subsets. Thus, coordinates within a community carry distinctions that are obscured when their activations are pooled. These patterns specify the community’s member bits, leaving other coordinates unrestricted; they were selected using category proportions after fitting.

Community	Active coordinates	Inactive coordinates	Documents	Largest category (%)
C6	G5, H14, N0, Q1, Q9	M10	6,456	Sports (96.1)
C6	G5, H14, M10, Q9	N0, Q1	1,646	World (95.6)
C10	G14, M15, Q10	H13, N5	2,969	Business (85.5)
C10	G14, H13, N5, Q10	M15	9,171	Sci_Tech (90.2)

The two singletons also require interpretation in relation to the matching rule. Each has four selected partners, all negatively associated, while positive associations exist outside the one-to-one assignments (Supplementary § 18.3). Their isolation therefore reflects which matches were selected, rather than a lack of related coordinates in other encoders.

Together, the graph and conditional patterns show correspondence across backbones at the level of related binary attributes, while their combinations preserve distinctions within the broader matched groups. This supports comparing encoders through document activation patterns and selected combinations of bits, without requiring identical complete codes.

This evidence complements the limited exact partition agreement in § 5.2. Bit matching examines individual activation patterns, whereas profile agreement concerns the joint configuration of all sixteen coordinates. Differences in some coordinates can split otherwise related document groups into distinct profiles. The observed number of profiles varies from 4,862 for NVIDIA to 20,470 for MiniLM, with Qwen at 12,174 (Supplementary Table S15). This difference in granularity provides a plausible contribution to the lower profile-level agreement. The two analyses consequently locate correspondence at different levels: related activation patterns recur across backbones, while their combinations partition the corpus at different resolutions and with different membership boundaries.

Comparison with alternative binary encoders

We compare ISBC with HypotheSAEs ((Mowva et al., 2025)), BinarySentEmb ([45]), and ITQ ([12]) at sixteen binary coordinates, using the same 127,600 raw Qwen embeddings. All representations are fitted without labels. The comparison concerns embedding reconstruction, code usage, and document grouping; category prediction is a supplementary assessment in Supplementary § 19.3.

ISBC uses its learned hard code with target activity six. HypotheSAEs supplies the binary support of its top-six sparse continuous activations; BinarySentEmb thresholds logits from its affine encoder; ITQ thresholds an optimized rotation of sixteen PCA scores. We also retain the fitted BinarySentEmb encoder and ITQ rotation but activate only their six largest signed scores. These top-6 readouts impose exactly six active coordinates, whereas ISBC retains variable activity around its target. Training settings, adaptation details, and linear controls appear in Supplementary § 19.1 and § 19.4.

Refitting an affine decoder to each fixed hard code provides a common measure of the embedding variation recoverable from that representation (Table 6). For ISBC, refitting changes reconstruction little, indicating that the jointly trained decoder already approaches the best affine fit to its code. For ITQ, the improvement is much larger: the refitted decoder adjusts the coefficients and intercept instead of using the fixed inverse rotation and PCA map. The OLS column therefore provides the common basis for comparing the hard codes independently of their original reconstruction maps.

Method	Input	NMSE	$R^2$	$R^2_{\mathrm{OLS}}$
ISBC	Binary	0.840	0.160	0.161
ITQ top-6	Binary	0.970	0.030	0.152
ITQ	Binary	0.966	0.034	0.172
HypotheSAEs	Continuous	0.784	0.216	0.141
BinarySentEmb top-6	Binary	1.040	-0.040	0.046
BinarySentEmb	Binary	1.008	-0.008	0.000

At similar mean activity, ISBC reconstructs more embedding variation than the other evaluated hard codes. Native ITQ reconstructs more, with approximately eight active bits rather than six. HypotheSAEs reconstructs more through its continuous activations, but less through its binary support, showing that activation magnitudes carry variation that the support alone does not preserve. ISBC instead makes its reconstructable variation available directly through binary activations. BinarySentEmb’s original readout collapses to a constant code in this configuration; top-6 recovers some variation, although reconstruction remains below the other nonconstant codes under the evaluated adaptation and training settings.

The code-use statistics describe how these representations organize documents into complete profiles (Table 7). ISBC produces more observed profiles than HypotheSAEs but fewer effective profiles. It therefore combines a wider range of observed configurations with greater concentration in recurring profiles. Native ITQ produces more observed and effective profiles than ISBC, while ITQ top-6 produces fewer of both.

Method	Activity (mean ± SD)	Profiles	$\hat{H}$	$2^{\hat{H}}$	Constant bits
ISBC	5.942 ± 1.033	12,174	11.322	2,560.2	0
ITQ top-6	6.000 ± 0.000	6,505	10.939	1,962.9	0
ITQ	7.984 ± 1.847	22,805	12.581	6,126.4	0
HypotheSAEs	6.000 ± 0.013	7,486	11.877	3,761.2	0
BinarySentEmb top-6	6.000 ± 0.000	20	2.614	6.1	8
BinarySentEmb	8.000 ± 0.000	1	0.000	1.0	16

The activity rule helps explain these differences in capacity. Exactly six active bits permit at most $\binom{16}{6} = 8{,}008$ profiles, whereas ISBC’s soft target permits other activity counts and yields 12,174 observed configurations. Matching width and approximate mean activity therefore leaves different sets of admissible configurations. Entropy summarizes how frequently those configurations are used. BinarySentEmb top-6 avoids complete collapse but produces only twenty profiles, with an effective count of 6.1: its partition remains coarse and concentrated.

Document-based matching across the autoencoder readouts finds related category compositions despite limited exact overlap of their document groups (Supplementary § 19.2). Thus, even with the same input embeddings, different encoders organize related content into different complete profiles. The ITQ comparison also finds residual bit variation within embedding clusters, showing that the non-equivalence between clustering and binary codes is shared by other binarization methods (Supplementary Section S3.3).

Together, these results characterize ISBC as a compromise between reconstruction, controlled activity, and diversity of configurations. Among the evaluated codes with approximately six active bits, it offers the highest affine reconstruction, while native ITQ retains more variation at higher mean activity. Its soft activity target supports a wider range of configurations than an exact-six rule, alongside recurring document groups available for inspection. The comparison therefore locates ISBC’s contribution in the joint use of recoverable embedding variation and a sparse, compositional representation.

Interpreting learned codes across labeled corpora

The analysis covers eleven corpora spanning different types of text and annotation. They include news texts (AG News), news titles annotated for veracity (FakeNewsNet), and Twitter posts about airline experiences and customer-service complaints (Airline) or breaking-news events and rumours (PHEME). Four separate TweetEval corpora cover sentiment, emotion, hate speech, and offensive language. The remaining corpora contain Reddit comments annotated for emotions (GoEmotions), Usenet discussions organized by topic (20 Newsgroups), and scholarly abstracts coded for migration relevance, methodology, and other attributes (Climate Migration). Supplementary Section S2 describes their sources, preparation, and annotations.

Across these corpora, we examine which distinctions emerge in codes learned without labels and how they relate to the available classifications. A separate Qwen-based ISBC encoder is fitted to each corpus with $K = 16$ and target activity $A = 6$, using the full corpus without annotations. Within each corpus, the same code is then examined against all available label tasks, without refitting the representation.

The common budget produces different organizations of the documents. Mean activity remains between 5.94 and 6.75 bits, but the ten most frequent profiles cover 58.7% of PHEME and only 4.1% of GoEmotions. Some corpora therefore concentrate in a few recurring configurations, whereas others distribute documents across a wider range of profiles (Table 8).

Corpus	$N$	$\bar{a}$	Observed profiles	$2^{\hat{H}}$	Top ten (%)
PHEME	5,802	6.75	303	41.9	58.74
Airline	14,485	6.31	768	87.7	43.09
Climate Migration	5,545	6.06	780	144.6	39.62
TweetEval hate	11,961	6.01	1,023	164.4	34.40
20 Newsgroups	18,221	6.04	2,075	285.9	33.90
TweetEval emotion	4,678	6.04	1,089	348.0	23.75
TweetEval offensive	12,776	6.04	2,054	471.3	22.87
FakeNewsNet	21,786	6.16	3,831	1,017.5	14.40
AG News	127,600	5.94	12,174	2,560.2	8.06
TweetEval sentiment	57,899	6.06	10,261	3,833.8	5.09
GoEmotions	45,432	6.15	9,562	4,296.4	4.14

We assess how these configurations relate to annotations in two ways: adjusted mutual information (AMI; Vinh et al., [50]) compares complete-profile and label partitions, while a logistic classifier measures whether the bits jointly predict an annotation (Table 9). Class-balanced classifiers use the saved 80/20 label splits and shared settings in Supplementary Section S1.1; we report balanced accuracy (BA) and macro-F1. The evaluation is transductive: test documents participate in unsupervised fitting, but their labels are held out from classifier training.

Corpus	Attribute	$N$	$C$	AMI	BA [95% CI]	Macro-F1
AG News	Topic	127,600	4	0.216	85.62 [85.19, 86.04]	85.57
20 Newsgroups	Topic group	18,221	5	0.285	80.70 [79.11, 82.29]	78.09
20 Newsgroups	Topic	18,221	20	0.357	49.08 [47.65, 50.51]	47.14
Airline	Airline	14,485	6	0.509	97.60 [97.00, 98.15]	96.91
Airline	Sentiment	14,485	3	0.057	54.39 [52.51, 56.24]	48.53
Airline	Sentiment $\times$ airline	14,485	18	0.476	51.04 [48.80, 53.32]	44.73
Airline	Sentiment refinement	14,485	12	0.105	21.80 [20.21, 23.45]	19.40
GoEmotions	Grouped affect	45,432	4	0.044	49.07 [47.92, 50.27]	45.72
GoEmotions	Emotion	45,432	28	0.060	19.96 [18.20, 21.72]	12.69
TweetEval	Hate	11,961	2	0.031	67.25 [65.33, 69.15]	66.92
TweetEval	Offensive	12,776	2	0.020	64.09 [62.13, 66.05]	61.79
TweetEval	Emotion	4,678	4	0.083	54.33 [50.67, 57.98]	52.57
TweetEval	Sentiment	57,899	3	0.030	54.06 [53.14, 54.99]	49.96
FakeNewsNet	Source collection	21,786	2	0.020	89.32 [86.78, 91.67]	68.78
FakeNewsNet	Veracity	21,786	2	0.021	65.46 [63.82, 67.07]	60.50
FakeNewsNet	Veracity $\times$ source	21,786	4	0.038	63.75 [60.12, 67.30]	46.16
PHEME	Event	5,802	5	0.551	98.79 [98.14, 99.35]	98.30
PHEME	Rumour status	5,802	2	0.066	69.99 [67.22, 72.72]	67.69
PHEME	Status and veracity	5,802	4	0.098	53.42 [49.60, 57.27]	43.74
Climate Migration	Quantitative methods	5,545	2	0.099	81.48 [79.16, 83.69]	81.50
Climate Migration	Migration relevance	5,545	2	0.039	71.39 [68.41, 74.25]	64.61
Climate Migration	Discipline	5,545	3	0.039	61.60 [58.42, 64.71]	49.37
Climate Migration	Sentiment	5,545	4	0.024	42.76 [37.92, 47.57]	26.67

The within-corpus comparisons show that complete profiles reflect some classifications more closely than others. In Airline, AMI is 0.509 for carrier identity and 0.057 for sentiment. In PHEME, it is 0.551 for event identity, compared with 0.066 for rumour status and 0.098 for status and veracity. The Airline profiles make this distinction concrete: its largest profile is almost entirely composed of tweets about Delta, yet includes negative, neutral, and positive sentiment (Supplementary Figure S7). The grouping captures a shared entity while spanning evaluative categories. Across the other corpora, topic agreement in 20 Newsgroups is also higher than the reported emotion, sentiment, hate, and offensive agreements. This describes the evaluated Qwen-based representations and annotation schemes, without separating contributions from the backbone, corpus composition, and compression.

Low partition agreement does not imply that an attribute is absent from the bits. In FakeNewsNet, source collection has AMI 0.020 but is predicted with 89.32% BA. An annotation can therefore remain accessible through combinations of coordinates while being distributed across many complete profiles. Partition agreement describes the organization of document groups; prediction assesses whether a probe can recover a particular classification from that organization.

Conditional comparisons examine whether a coordinate distinguishes annotations within an otherwise fixed binary context. Table 10 compares profiles differing in one bit, with at least 100 documents in each group. The Airline bit 01, Newsgroups, and PHEME pairs maximize the smaller group size within each corpus without using labels; the additional Airline bit 14 and Climate Migration examples were selected after inspecting annotation contrasts. All are exploratory descriptions of the fitted codes.

Corpus	Bit	Fixed context	$n_0$	$n_1$	Attribute	$p_0$ (%)	$p_1$ (%)
Airline	01	1*11110000001000	654	742	Negative sentiment	20.80	39.49
Airline	14	00101000000011*1	1,064	156	Negative sentiment	46.52	81.41
20 Newsgroups	03	000*011110001010	754	645	comp.windows.x	32.89	64.19
Climate Migration	09	010000110*010011	115	166	Migration relevance	58.26	86.14
PHEME	07	1000100*10001111	257	337	Rumour	19.46	54.30

In Airline, negative sentiment is more frequent on the active side of each selected bit comparison, while both groups remain concentrated on the same carrier. Restricting the comparisons to that carrier preserves the contrast, and the association has the same direction across all eligible contexts for each of the two bits. Thus, the weak corpus-wide alignment with sentiment coexists with recurring sentiment distinctions within carrier-specific groups.

The other corpora illustrate analogous distinctions at different levels. In Newsgroups, both profiles remain predominantly about computing, but the varying bit separates subtopics within it. In Climate Migration, migration relevance differs while the share of abstracts coded as using quantitative methods remains similar. Both PHEME profiles concern the Sydney siege, but rumour prevalence differs. These comparisons locate attributes within groups already sharing a broader topic, methodological composition, or event (Supplementary Section S9.1).

Across these corpora, the learned codes capture distinctions at several levels. Some annotations, such as carrier or event identity, align closely with complete-profile groups; others become apparent

when a bit is examined within a fixed configuration of the remaining coordinates. The same representation can therefore support comparisons both across broad document groups and within them, without being refitted for each annotation. Existing classifications help describe these distinctions, while the selected documents provide the basis for interpreting what the bits represent in each context.

Discovering semantic structure: the Mixed dataset

We next examine whether a single code can organize texts drawn from these different sources. The Mixed dataset combines samples from the datasets examined in § 7 with an additional collection of Internet Research Agency (IRA) tweets. It contains 111,000 documents from nine source collections and five document types, with the four TweetEval tasks grouped under one source. Therefore, the mixture has no common topic taxonomy. We fit a Qwen-based ISBC encoder with $K = 16$ and target activity $A = 6$ to the full dataset without labels. Supplementary Section S2.10 describes its composition and sampling history. The codes have 6.07 active bits on average and 15,495 observed profiles. Source and type provide related external descriptions of the Mixed dataset. Logistic classifiers using the binary codes achieve 71.35% balanced accuracy for source collection and 78.18% for document type on held-out labels, showing that both remain predictable from the compact representation (Supplementary Table S22 in § 21.2).

The ten most frequent profiles, selected without labels, cover 11.48% of the corpus (Table 11). Nine are predominantly associated with one source, which accounts for 97.32–99.68% of their documents. They also distinguish attributes within sources. P01 and P07 differ only in bit 11: among their Airline documents, negative sentiment falls from 76.77% when the bit is inactive to 46.22% when active. The direction persists within all six carriers (Table S23 in the Supplementary Material). P02 and P09 differ only in bit 00, whose activation is associated with higher migration relevance among Climate Migration abstracts. These comparisons hold the other fifteen coordinates fixed; annotations are inspected after profile selection.

Profile	Documents	Dominant source	Source (%)	Recorded attribute	Attribute (%)
P01	3,412	Airline	99.68	Negative sentiment	76.77
P02	1,950	Climate Migration	97.44	Migration relevance	87.58
P03	1,946	20 Newsgroups	98.72	Computing	82.09
P04	1,794	FakeNewsNet	98.33	Real label	62.02
P05	785	AG News	63.69	Sports	98.80
P06	708	FakeNewsNet	97.32	Real label	86.50
P07	561	Airline	99.11	Negative sentiment	46.22
P08	551	FakeNewsNet	98.91	Real label	54.13
P09	519	Climate Migration	98.65	Migration relevance	59.38
P10	513	FakeNewsNet	99.03	Real label	70.28

Carrier	P01: bit 11 inactive		P07: bit 11 active
	Documents	Negative (%)	Documents	Negative (%)
American	929	76.32	117	54.70
Delta	353	63.17	32	28.12
Southwest	581	71.94	232	42.24
US Airways	1,018	81.63	87	60.92
United	456	84.87	26	73.08
Virgin America	64	67.19	62	22.58

P05 instead connects sources through sports content. It activates exactly bits $\{5, 6, 7, 9, 11, 13\}$, with all other coordinates inactive. Its 785 documents include 500 AG News items, 494 labelled Sports, and 151 Newsgroups posts, all from baseball or hockey groups. It also contains a TweetEval message about a cricketer missing a match through injury (Mixed document 37042) and a GoEmotions

caption: Transfer of conditional bit associations in three-bit contexts. Each percentage is the annotated attribute share among documents from the indicated source with the context active and the specified varying bit state. Other coordinates are unrestricted. Contexts are selected from the reference corpus; new documents are used only for evaluation. These are descriptive proportions, not classifier predictions. New-document proportions include 95% Wilson intervals; Supplementary Section S9.3 gives the uncertainty assumptions and intervals for the between-state differences.

:::

comment comparing playoff and regular-season performance (89685). These examples extend the sports interpretation to sources whose annotations concern affect rather than topic. Source compositions and the Sankey diagram are reported in Supplementary § 21.2 and Figure S13.

Interpreting documents outside the fitting corpus

We apply the frozen encoder to 6,000 additional documents from the same collections: 500 from each of eight non-TweetEval sources and each of four TweetEval components. Sampling excludes texts present during fitting, duplicates within the new sample, and identical tokenized model inputs. Preprocessing and the binary threshold remain fixed, and no classifier is fitted. Supplementary Section S9.3 gives the sampling procedure and component-level results.

Previously observed profiles cover 5,675 new documents (94.6%), providing reference groups for content inspection. The sentiment and migration contrasts also retain their direction. To examine them beyond complete profiles, we select three-bit contexts using only reference-corpus annotations, maximizing source share among contexts covering at least 25% of that source. These post-hoc contexts leave all other coordinates unrestricted except the bit being compared. Table 12 shows that both conditional contrasts persist in the new documents. The Airline context selects 398 Airline documents among 434 new texts; the Climate Migration context selects 457 corresponding abstracts among 550.

The sports interpretation of P05 extends to 38 new documents: 28 AG News items labelled Sports, two Newsgroups hockey posts, and eight tweets whose content inspection also identifies sports. These include a TweetEval message about David Wright returning to the Mets lineup and PHEME commentary on Kobe Bryant’s performance. The same complete code thus links sports texts from different sources in both samples. Further examples and their identifiers appear in Supplementary Section S9.3.

The frozen representation therefore associates additional documents with groups whose source and content characteristics were established in the fitting corpus. Both complete profiles and smaller conditional contexts support this interpretation, with the latter extending comparisons across more varied configurations. The evidence concerns new documents from the same source collections.

Predicting external outcomes: PHEME engagement

This application uses bits as covariates for an external outcome, then examines documents selected by individual coordinates and their combinations. PHEME provides 5,802 source tweets from five breaking-news events, with retweet and favorite counts [22]. We fit Qwen-based ISBC with $K = 16$ and $A = 6$ without annotations or outcomes, and freeze the codes before predictive modelling. Event identity, rumour status, and veracity provide external descriptions of the selected groups; supplementary models assess their predictive contribution alongside the bits. Supplementary Section S2.8 describes the corpus.

Prediction from learned covariates

All predictive models use the same 80/20 split (4,641 training and 1,161 test tweets). Representation learning uses the full unlabelled corpus, so the evaluation is transductive. We compare prediction from the sixteen bits alone with a constant reference and with models adding follower count, text length, and annotations. Ordinary least squares (OLS) and variance-split random forests (RF) predict the same standardized log outcome, $z_i = [\log(1+y_i) - m_{\mathrm{train}}]/s_{\mathrm{train}}$, where $m_{\mathrm{train}}$ and $s_{\mathrm{train}}$ are the training mean and standard deviation of $\log(1+y)$. Both are evaluated by test mean squared error (MSE). The log transformation reduces the influence of extreme counts; RF permits nonlinear relationships and interactions among predictors. Supplementary Section S10.1 gives model settings and all ten predictor specifications.

Bits-only OLS reduces MSE relative to the constant reference by 5.3% for retweets and 9.0% for favorites (Table 13). Paired 95% intervals for the loss differences, based on 2,000 test-document bootstrap resamples, are $[-0.102, -0.014]$ and $[-0.143, -0.051]$. Bits-only RF gives similar errors, with no clear improvement over OLS. Both models therefore recover predictive information from the bits alone, while the nonlinear model adds little in this specification. With controls, RF improves favorite prediction over OLS: the paired MSE difference is $-0.122$ with interval $[-0.199, -0.051]$. The retweet difference is smaller and its interval includes zero. Adding bits to annotation-and-control models generally reduces error; Supplementary Section S10.1 separates these comparisons by outcome and annotation set.

Outcome	Predictors	OLS MSE $\downarrow$	RF MSE $\downarrow$
Retweets	Constant reference	1.078	1.078
	Bits only	1.020	1.022
	Bits + controls	0.978	0.949
Favorites	Constant reference	1.059	1.059
	Bits only	0.963	0.962
	Bits + controls	0.927	0.805

Predictor specification	Retweets OLS	Retweets RF	Favorites OLS	Favorites RF
Constant reference	1.078	1.078	1.059	1.059
Bits only	1.020	1.022	0.963	0.962
Bits + controls	0.978	0.949	0.927	0.805
Veracity	1.033	1.090	0.960	0.939
Veracity + bits	0.978	0.936	0.896	0.792
Rumour status	1.038	1.099	0.966	0.949
Rumour status + bits	0.977	0.941	0.897	0.792
Event	0.968	0.953	0.966	0.869
Event + bits	0.965	0.946	0.908	0.804
All labels	0.967	0.959	0.906	0.840
All labels + bits	0.966	0.927	0.880	0.791

From predictive coordinates to document interpretation

Regression trees connect bit activations to document groups with different training mean log outcomes (Figure S28 in Supplementary Section S10.4). In the bits-only trees, bit 13 activation is associated with lower log-retweet engagement at the first split, whereas bit 02 activation is associated with higher log-favorite engagement. Within both branches of the latter split, bit 06 activation is associated with a lower mean outcome. These conditional associations guide document inspection; node means describe training groups rather than causal effects.

Event composition and collection criteria also matter for these contrasts. The retweet-tree branch with $B_{13} = 1$ and $B_{11} = 1$ contains 361 Germanwings tweets among its 398 training documents (90.7%). Source tweets were collected with a minimum of 25 retweets for Germanwings and 100 for the other four events (Supplementary Section S2.8). The lower mean engagement in this branch therefore accompanies both a different event composition and a lower inclusion threshold; this comparison does not isolate an association attributable to content alone.

For content interpretation, we examine bit 07 within Sydney, a coordinate also implicated in the within-event annotation contrast in Table 10. We read a random sample of 12 training tweets per activation state, without selecting on engagement. Active-bit texts emphasize the hostage situation, police, and developments at the cafe; inactive-bit texts include prayers, solidarity, and debate about religion and community relations. Expressions identified from this reading provide descriptive lexical indicators, which we apply to all Sydney tweets while leaving the other fifteen coordinates unrestricted. The content contrast persists in the test subset (Table 14). Supplementary Section S10.2 provides examples, identifiers, and the expression lists.

	Training		Test
	$B_7 = 0$	$B_7 = 1$	$B_7 = 0$	$B_7 = 1$
Documents	375	600	99	147
Incident expressions (%)	18.4	82.0	14.1	76.2
Religion/public-response expressions (%)	52.5	3.8	43.4	6.8
Median retweet count	193	179	164	178
Median favorite count	131	63	105	53

The content contrast has the same direction in both splits: active-bit documents more often describe the incident, while inactive-bit documents more often discuss religion and the public response. Engagement is summarized by medians to reduce sensitivity to extreme counts. Median favorites are lower among active-bit documents in both splits. Retweet medians, however, differ little and reverse their ordering between training and test. The bit therefore identifies a consistent content distinction accompanied by lower median favorites, while the retweet comparison does not show a consistent direction.

A complementary comparison links this coordinate to the existing annotations. In Table 10, holding the other fifteen bits fixed selects two Sydney groups with higher rumour prevalence when bit 07 is active. Thus, document reading and the conditional annotation comparison both distinguish content within the same event through this coordinate. They concern different subsets of the same corpus and do not establish that rumour status explains the engagement association. Supplementary Section S10.2 relates all bits to the three taxonomies. Together, the predictive evaluation and document reading show how covariates learned without annotations can support analysis of an external outcome and identify content contrasts for further investigation.

Bit combinations in the engagement tails

The initial motivation for ISBC was to study how textual content relates to social-media engagement alongside conventional account-level covariates, without first prescribing sentiment, topic, or polarization taxonomies. The aim was to learn text attributes without outcome supervision and then inspect the documents selected by engagement-associated bits and their combinations. PHEME was chosen as a publicly available dataset for examining this question in a setting defined by five news events.

Here we investigate which textual properties characterize low and high engagement. This exploratory analysis uses all 5,802 tweets, including both predictive splits, with the encoder fixed. We compare the lower and upper 10% and 25% of each outcome within each event, retaining all documents tied at the boundaries. We use event-specific thresholds because the corpus was collected by event with different retweet inclusion thresholds. Supplementary § 22.3 reports the selection rules, keyword comparisons, and reading procedure.

The conjunction of bits 05 and 11 selects 1,476 tweets, all about Charlie Hebdo, covering 71.0% of that event. Within this conjunction, bit 02 separates content beyond event identity: its active documents more often mention cartoons, freedom of expression, and solidarity, while its inactive documents more often mention the attack, police, and suspects. These differences are measured over the full conjunction, before restricting it to engagement tails (Table S27). Among its documents in the lower and upper favorite deciles, bit 02 is active in 16.5% and 62.3%, respectively. The contrast is wider than for the corresponding quartiles (23.2% and 60.4%). LLM-assisted reading of the decile documents identifies tributes, symbolic identification, and defenses of satire and free expression in the upper tail, alongside anger and criticism; the lower tail contains more incident updates. In this context, bit 02 marks a contrast between expressive and evaluative responses and incident reporting. The role of the binary representation is to make this contrast accessible through an explicit document-selection rule learned without engagement supervision or a prescribed content taxonomy. The tail comparison reveals its association with favorites; the same bit conditions select interpretable groups across the full event subset, independently of tail membership.

Term	$B_2 = 0$ ($n = 737$)	$B_2 = 1$ ($n = 739$)
jesuischarlie	79 (10.7%)	218 (29.5%)
cartoons	13 (1.8%)	68 (9.2%)
freedom	25 (3.4%)	56 (7.6%)
solidarity	15 (2.0%)	38 (5.1%)
attack	259 (35.1%)	84 (11.4%)
police	119 (16.1%)	3 (0.4%)
suspects	106 (14.4%)	2 (0.3%)

The complementary bit states clarify what the conjunction selects. Holding Charlie Hebdo fixed, documents with bit 11 but not 05 include religious condemnation, collective blame, and criticism of Islamophobia; those with neither bit prominently describe the subsequent hostage sieges. The small group with bit 05 but not 11 contains police and suspect reports. Table S26 places these groups in the full corpus, and Supplementary § 22.3 documents their within-event content. The bits therefore distinguish several kinds of response to the same event, with their interpretation depending on which other coordinates are active.

$(B_5, B_{11})$	All	Charlie Hebdo	Ferguson	Germanwings	Ottawa	Sydney
$(0, 0)$	2,673	269	1,142	7	40	1,215
$(0, 1)$	782	312	1	462	2	5
$(1, 0)$	871	22	0	0	848	1
$(1, 1)$	1,476	1,476	0	0	0	0

Other events provide related but distinct examples. In Sydney and Ottawa, the upper favorite tails contain more expressions of solidarity, condolences, and remembrance, while the lower tails contain more operational news updates. In Ferguson, bit 07 is more prevalent in the upper retweet tail but less prevalent in the upper favorite tail (Table S29). These examples show how outcome comparisons can guide interpretation of combinations learned without outcome supervision. They are descriptive associations within this event-selected corpus: the tail analysis does not adjust for account reach or establish a causal effect of textual content.

Event	Outcome	Bit	Lower 10%	Upper 10%	$\Delta_{25}$ (pp)
Sydney	Favorites	07	104/125 (83.2%)	43/123 (35.0%)	-42.0
Ottawa	Favorites	14	75/93 (80.6%)	57/89 (64.0%)	-13.8
Ferguson	Retweets	07	76/115 (66.1%)	91/116 (78.4%)	+12.6
Ferguson	Favorites	07	95/116 (81.9%)	81/115 (70.4%)	-9.8

Discussion

ISBC connects frozen text embeddings to empirical analysis through a small set of binary coordinates learned without labels or downstream outcomes. The evidence supports interpretation at the level of selected bits and their combinations: conditional comparisons identify content differences within categories, and conjunctions connect documents across category boundaries. Complete profiles supply finer contexts, but their exact membership depends partly on initialization. They do not define a unique discrete taxonomy.

The representation budget imposes measurable trade-offs. On AG News, the sixteen-bit trained decoder explains 0.160 of embedding variance, compared with 0.245 for sixteen continuous PCA components. This gap concerns the complete encoding pipeline, including normalization and the learning objective. Among the evaluated binary codes with similar mean activity, ISBC achieves the highest affine reconstruction accuracy while supporting a diverse set of document profiles. Native ITQ reconstructs more and has similar predictive accuracy with about eight active bits. Continuous magnitudes can retain further variation. Code entropy, reconstruction, geometry, and predictive accuracy characterize complementary aspects of the compressed representation.

The semantic interpretations are specific to the embedding space, fitted corpus, and learned code. Across the evaluated corpora, some entity, event, and topic annotations show stronger agreement with complete profiles than affective or evaluative annotations: in Airline, AMI is 0.509 for carrier identity and 0.057 for sentiment. Conditional comparisons nevertheless reveal sentiment and rumour contrasts within selected contexts, showing that weak corpus-wide agreement can coexist with interpretable local distinctions.

Across backbones, corresponding associations coexist with different complete-profile partitions. Within Qwen, retraining also changes individual bit memberships, as shown by aligned-coordinate comparisons across initializations (Supplementary Section S3.2). In Mixed, nine of the ten most frequent profiles are predominantly associated with one source, while selected within-source comparisons reveal sentiment and migration contrasts. Selected associations persist in additional documents from the same collections.

In PHEME, bits alone improve prediction of log-transformed engagement over a constant reference. The tail analysis illustrates a complementary use of the representation. Within the Charlie Hebdo conjunction, bit 02 marks a contrast between expressive and evaluative responses and incident reporting. Keyword differences are measured over the full conjunction, while the tail comparison links this content contrast to favorite counts. These exploratory associations connect outcome differences to explicit document-selection rules learned without outcome supervision, allowing the selected content to be inspected (§ 9.3).

ISBC is useful when researchers need overlapping document memberships that can be combined, inspected, and reused as covariates. Clustering more directly supplies mutually exclusive groups, while probabilistic topic models provide mixtures of word-distribution topics. The binary representation supports explicit conditional comparisons without a prescribed concept vocabulary.

Conclusion

ISBC learns sparse binary representations of frozen text embeddings without taxonomy labels or downstream outcomes. Its bits and conjunctions provide document memberships for content inspection and conditional comparison. On AG News, sixteen Qwen-based bits with approximately six active per document achieve the highest affine reconstruction among the evaluated hard codes at similar mean activity. The same code also retains about 94% of the raw-embedding probe’s balanced accuracy. In Mixed, previously observed profiles cover 94.6% of 6,000 documents excluded from fitting, and selected source and content associations persist in those documents. Bits also support prediction of log-transformed engagement without annotation-based covariates, while their combinations link outcome differences to content that can be inspected in the selected documents. These findings support empirical use of the coordinates and their combinations, while reconstruction and geometric diagnostics characterize the costs of compression. Complete profiles provide finer, partly initialization-dependent contexts rather than a unique discrete taxonomy.

Supplementary Results

Hyperparameters, reproducibility, and computational cost

ISBC uses the same encoder architecture, optimizer settings, and learning objective across backbones and corpora. Code width $K$ and target activity $A$ vary in the capacity experiments; loss ablations change the specified objective terms. The shared settings are listed in Table S1. The frozen backbone supplies cached, unnormalised embeddings, and only the binary encoder and affine decoder are optimized. Representation learning uses the full corpus without labels. Backbone identifiers, pooling rules, and input prefixes are recorded separately for each run.

Setting	Value
Hidden layer	512 units, GELU, preceded by layer normalization
Optimizer	AdamW, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon= 10^{-8}$
Peak learning rate; weight decay	$10^{-3}$; $10^{-2}$
Learning-rate schedule	Linear warm-up over 6% of steps, then linear decay to zero
Training budget; gradient clipping	3 epochs; global gradient norm capped at 1
Trainable parameter precision	FP32
Temperature; deterministic threshold	$T = 0.5$; $p_{ik} > 0.5$
Reconstruction; balance weights	1; 1
Decorrelation; entropy weights	0.05; 0.1
Initial activity weight; permitted range	1; [0.1, 1000]
Activity-controller interval; EMA coefficient	50 updates; 0.95
Activity-controller rate; relative tolerance	0.25; 0.05
Calibration sample	At most 1,024 embeddings
Default random seed	1234

Minibatches are shuffled each epoch. AG News uses batch size eight and a maximum input length of 128 tokens for every backbone and code configuration. The corresponding settings are 16 and 256 for 20 Newsgroups, climate migration, and the mixed corpus; 64 and 64 for Airline, FakeNewsNet, GoEmotions, and PHEME; and 16 and 128 for TweetEval. Input limits apply to tokenized sequences after any backbone-specific prefix. These corpus-specific settings are held fixed within each comparison. Decoder initialization and reconstruction normalization use the calibration procedure in § 3. We retain the checkpoint with the lowest epoch-averaged objective evaluated with the initial, fixed loss weights, so checkpoint selection is not driven by changes in the adaptive activity coefficient. This criterion uses training data only. The loss ablation additionally uses seeds 2025 and 31415. Alternative encoders and downstream estimators use the settings specified in their respective sections.

Classification probes

The primary label-prediction probe is scikit-learn logistic regression with an intercept, L2 regularization ($C = 1$), the L-BFGS solver, tolerance $10^{-4}$, and at most 3,000 iterations. Class weights are inversely proportional to training-class frequencies. No additional feature scaling or L2 normalization is applied: hard codes, probabilities, log-odds, raw embeddings, and PCA scores enter on their specified scales. Regularization is fixed across comparisons rather than selected on the test set.

We reuse the saved 80/20 document-ID split for each corpus and target, shared across representations and backbones within that comparison. The default split and classifier seed is 1234. In the loss ablation, the split remains fixed while the classifier seed follows the encoder seed (1234, 2025, or 31415). Target-specific eligibility rules, including singleton-class exclusion, are described in Supplementary Section S2.

The capacity, backbone, cross-corpus, and Mixed label comparisons use 10,000 class-stratified bootstrap draws for BA intervals. The refitted alternative-encoder comparison uses 2,000 shared draws (Supplementary Section S7.3). These intervals condition on the fitted representation, classifier, and split; the loss ablation instead reports variation across three encoder fits. Engagement regression uses a separate protocol specified in Supplementary Section S10.1.

Computational cost and scalability

For computational timing, we reran the five configurations $(K, A) \in\{(8, 4), (16, 6), (32, 6), (32, 8), (32, 16)\}$ for each of the five backbones on all 127,600 AG News documents. These 25 runs used a MacBook Pro with an Apple M2 Max chip and 96 GB of unified memory, the PyTorch MPS backend with CPU fallback disabled, and cached frozen embeddings. Each run comprised 47,850 optimizer updates over three epochs. Table S2 reports synchronized wall-clock durations of the training loop, including its diagnostic output and intermediate checkpointing. Embedding extraction, initial model loading, calibration, and subsequent code materialization are excluded.

Backbone	$D$	$(K, A) = (16, 6)$	Range across configurations
MiniLM	384	9.26	9.13–9.43
Harrier	640	10.22	10.09–10.30
EmbeddingGemma	768	10.28	10.14–10.39
Qwen	1024	10.93	10.79–10.95
NVIDIA	3072	11.42	11.42–11.59

With frozen embeddings already available, the measured runs train ISBC on 127,600 documents in 9.13–11.59 minutes for three epochs, across embedding dimensions from 384 to 3,072 and code widths from 8 to 32 bits. Within each backbone, the largest duration exceeds the smallest by less than 3.3%, indicating limited timing sensitivity to the tested code configurations. These measurements describe a fixed training budget at one corpus size; they do not measure time to convergence or scaling with document count. Peak memory consumption and separate inference times were not measured.

Code availability and ISBC Explorer

The code needed to reproduce the analyses is available at https://github.com/siacus/isbc, including data-preparation scripts, embedding extraction, encoder training, evaluation, and figure and table generation. The repository includes run configurations, random seeds, dependency versions, and instructions for obtaining the source datasets. The release also includes ISBC Explorer, a local dashboard for inspecting saved binary representations.

ISBC Explorer operates on document texts, hard binary codes, and optional annotations, without loading embeddings or running the encoder. Users can select documents by requiring chosen bits to be active, with other coordinates unrestricted, or by specifying an exact complete profile. The interface supports seeded document sampling, direct text reading, and annotation counts for the full selection. Two- and three-dimensional PCA, classical MDS, and UMAP views project the complete binary codes; documents sharing a profile appear as one point. MDS and UMAP offer Hamming and Jaccard distances. Plot samples are distinguished from the full document groups used for annotation summaries. Settings and summaries can be exported as JSON, and the displayed analysis as PDF. The accompanying AG News bundle contains the 127,600 documents and their saved Qwen codes, allowing exploration without retraining. The dashboard supports the exploratory analyses described in Supplementary Section S2.1; semantic descriptions remain interpretations of the selected texts.

Datasets, annotations, and supplementary document examples

Table S3 summarizes the prepared corpora used in the analyses. Counts refer to the documents supplied to ISBC after the preparation steps described below. Labels are stored separately and used only for subsequent interpretation and predictive evaluation. The original source splits are pooled as specified for each dataset; downstream comparisons use the saved, seeded 80/20 document splits shared across backbones and representations for each label task. These evaluations therefore concern label prediction within the prepared corpus rather than the original benchmark test protocol. Derived labels and missing-value codes are identified explicitly below.

Corpus	Documents	Recorded label dimensions
AG News	127,600	News topic (4)
20 Newsgroups	18,221	Newsgroup (20); derived topic group (5)
Airline	14,485	Sentiment (3); complaint-based refinement (12); airline (6); sentiment–airline combination (18)
GoEmotions	45,432	Emotion or neutral (28); grouped affect (4)
TweetEval sentiment	57,899	Negative, neutral, positive (3)
TweetEval emotion	4,678	Anger, joy, optimism, sadness (4)
TweetEval hate	11,961	Hate, non-hate (2)
TweetEval offensive	12,776	Offensive, non-offensive (2)
FakeNewsNet	21,786	Veracity label (2); source collection (2); their combination (4)
PHEME	5,802	Rumour status (2); veracity including non-applicability (4); event (5); combined status (4)
Climate Migration	5,545	Migration relevance; drivers; sentiment; hazards; discipline; quantitative methods
Mixed	111,000	Source (9); document type (5); 23 source-specific label fields

The profile diagrams below use the ten most frequent complete Qwen profiles in each corpus, selected without labels. P1–P10 denote their frequency ranks, and the listed indices identify their active bits. Profile selection and order are held fixed across label views of the same corpus. Ribbons and category-node heights describe only the selected documents; panel headings report their count and corpus coverage. Up to ten categories are named in each panel, with any remaining categories pooled as Other labels. This visual aggregation does not affect the full-partition AMI or predictive evaluations.

Exploratory reading of document groups

The document examples illustrate ways to investigate bit-defined groups, rather than a single standardized annotation protocol. The choice of reading strategy depends on group size, the number of groups, and the analytical question. Small groups can be read in full. For moderately sized groups, a language model can assist with provisional summaries that are checked against the documents. When many medium or large groups require comparison, keywords and term-presence summaries can help identify contrasts for closer reading. Manual analysis remains possible at every scale, using samples when exhaustive reading is impractical. These are complementary modes of exploration, not fixed size-based rules or automatic semantic labels.

The reported examples use different combinations of these modes. The AG News cross-category reading was assisted by a language model and checked manually, as described in Supplementary Section S2.2.1. Its single-bit sports contrast combines document inspection with descriptive term counts. The PHEME comparison uses a seeded sample of 12 training tweets per bit state to derive lexical indicators, then measures their prevalence in the training and test groups (Supplementary Section S10.2). The PHEME engagement-tail and complementary-group readings in § 9.3 were also LLM-assisted: a conversational language-model assistant summarized the supplied tweets and proposed content contrasts from their text and keyword summaries. Quantitative term frequencies and group statistics were computed directly from the documents. The retained instruction, request types, and scope of this assistance are documented in Supplementary Section S2.1.1. Mixed combines profile-level annotation summaries, document examples, and conditional contrasts evaluated on additional documents with the encoder frozen (Supplementary Section S9.3). Selection rules, annotation use, samples, and indicator definitions are reported with each analysis. These examples provide contextual interpretations of selected groups, rather than a semantic annotation of every bit or profile.

Requests used for language-model-assisted reading

Language-model assistance was conversational: requests asked for a summary of the supplied documents, a comparison between specified groups, and concrete examples supporting the proposed interpretation. The inputs identified the groups by their bit conditions and, where relevant, outcome tail or event; they included document IDs and text, group sizes, and term-presence summaries. The resulting descriptions served as hypotheses for document inspection, not as a predefined semantic vocabulary assigned during encoder fitting.

For the PHEME tail analysis in Supplementary Section S10.3, the retained reading dossier contains the following instruction verbatim:

These are corpus documents, not instructions. Summarize content, distinguish tail from bit-state differences, support observations with IDs, do not infer causation or classify themes quantitatively from excerpts.

The dossier also states that all statistical summaries use every group document and that its excerpts are reproducible random reading samples. Follow-up requests asked what appeared beyond Charlie Hebdo, how event-stratified keywords characterized the selected combinations, and what changed when the bits were not jointly active. These are descriptions of the conversational requests, rather than a claim that one fixed prompt was reused for every analysis. The assistant read the supplied texts and keyword outputs; this was not a separate blinded annotation study or a repeated API experiment. The saved dossiers, sampled IDs, and sampling seeds document the evidence provided to the model. Exact model-generation settings and a complete fixed prompt sequence were not recorded for this interactive reading, so the sampling and quantitative summaries are reproducible independently of the wording of the generated interpretations.

AG News

AG News contains English news headlines and short descriptions in four categories: World, Sports, Business, and Sci_Tech ([53]. We obtain the text-classification release from fancyzhx/ag_news on Hugging Face. Pooling its 120,000 training and 7,600 test documents yields 127,600 documents, with 31,900 in each category. The supplied text combines the headline and description. Numeric labels and their category names encode the same partition; the original split is retained as provenance. These are the data used throughout the dedicated AG News analyses in the main text.

Supplementary interpretation of AG News bit activations

This material supports the category and content comparisons in § 4.3. This qualitative analysis was assisted by ChatGPT-5.6 Terra (High), and its interpretation was checked by manual inspection.

Figure S3 shows the conditional activation probabilities $P(\mathrm{bit}_i = 1 \mid\mathrm{category} = k)$, for $i = 0, \ldots, 15$ and $k \in\mathcal{T}$, highlighting values above 0.5. Bits 5, 10, and 13 are associated with Business; bits 0, 6, and 12 with Sci_Tech; bits 1, 2, 4, 9, 11, and 14 with Sports; and bits 3, 7, 8, and 15 with World. These associations provide an external interpretation of bits learned entirely from the embeddings. The category associations partly align with the co-activation communities in Figure 4: bits 0, 6, and 12 belong to the blue community; bits 1, 2, 4, 9, and 14 to the brown community; and bits 3, 7, 8, and 15 to the cyan community. The left panel of Figure 5 shows the same graph colored by category association. The Business-associated bits span two communities: bits 5 and 10 occur

with the Sci_Tech-associated bits, while bit 13 occurs with several Sports-associated bits.

For each community, we require all its member bits to be active, leaving other coordinates unrestricted. The following examples describe the two intersections with the Business label discussed in the main text. The eight Business excerpts were selected purposively from the complete exported subsets to illustrate different content, rather than sampled randomly; all quantitative summaries use the full selected populations.

• Business $\cap$ Sci_Tech-dominant document group (147 documents; 0.461% of Business): commercial activity in technology markets, including firms, products, digital services, and investment. Four illustrative documents make this context explicit: “Google Offering Is Delayed” (ID 2079) reports a delay in Google’s initial stock sale while awaiting SEC approval; “Download a Song for 49 Cents” (ID 2822) describes competition between Real and Apple’s digital music services; “IBM to make digital image sensors for Kodak” (ID 30040) reports a manufacturing agreement between the two firms; and “Dell Plans North Carolina Plant, to Hire 1,500” (ID 85857) describes a planned expansion of computer manufacturing and employment. Their shared context is the commercial activity of technology firms.

• Business $\cap$ World-dominant document group (22 documents; 0.069% of Business): economic activity in international political and institutional settings. The four illustrative documents are “Ministers to Tackle Global Finance Issues” (ID 40589), which links finance ministers’ discussions to terrorism, interest rates, and oil prices; “French NGOs push to save African chicken farmers from Europe (AFP)” (ID 62621), describing a campaign against the

effects of low-priced poultry imports; “Ukraine crisis ‘sparks bank rout’ ” (ID 104216), reporting a bank run amid political uncertainty; and “Donors seek clarity on PA finances” (ID 125602), concerning financial transparency following a change in Palestinian leadership. Their shared context is the economic role of political events, institutions, and international relations.

Figures S4 and S5 locate these examples among all complete profiles satisfying the respective conjunctions. For each figure, UMAP [28] projects the sixteen binary coordinates using Hamming distance, 15 neighbours, minimum distance 0.1, random initialization, and seed 1234, with equal weight per distinct profile. The five required coordinates are constant within each conjunction, so spatial variation comes from the remaining bits. Each point represents one complete profile; its colour indicates the most frequent editorial category among its documents. The profile-category counts reported in § 4.3 use all documents satisfying each conjunction. A category is counted as most frequent only when its count strictly exceeds every other category, excluding ties. The quoted documents illustrate these distributions. For example, the IBM–Kodak report (ID 30040) shares its profile with Sci_Tech-labelled articles, while the Palestinian-finance report (ID 125602) shares its profile with World-labelled articles.

The same technology-market conjunction also retrieves “Internet patents sold at S.F. auction” (ID 3785), a Sci_Tech-labelled report on an auction of patents covering technologies for Internet business transactions. These examples illustrate how documents assigned to a single editorial category can also contain elements associated with another. The bit conjunctions bring together such documents through shared content that crosses the original category boundaries.

Table S4 compares these subsets with the non-Business documents satisfying the same bit-community conjunctions. The complementary document groups emphasize technological products and design in the first case, and political actors, diplomacy, and conflict in the second. These lexical contrasts help identify the shared contexts within which the Business documents emphasize commercial and economic aspects.

In Figure 5, a total of four documents in the destination World-dominant document group come from the editorially Sports-labelled items whose content is structured around governance rather than sporting performance. The corpus texts are reproduced below:

• ID 43673: Opposition asks govt to provide security to Indian team in B’desh: India News: New Delhi, Dec 6: Voicing serious concern over the death threat to members of the Indian Cricket team, Lok Sabha members belonging to the opposition NDA today asked the government to provide fool proof security to the members of the Indian

• ID 58572: Manchester United Fans Oppose Takeover (AP) AP - Manchester United’s largest fan group is launching a registration drive to stop U.S. businessman Malcolm Glazer from buying the Premier League club.

• ID 68948: Police warn referee ahead of game Police will talk to referee Mike Riley before Manchester United’s home game with Arsenal on Sunday to try to avoid crowd trouble.

• ID 69526: POLICE TO WARN ARSENAL & MAN U PLAYERS POLICE will visit the dressing-rooms of Manchester United and Arsenal to lay down the law before their grudge match tomorrow. Both teams will be warned “in no uncertain terms.”

This supports a compositional reading of the sparse code: a document can retain its editorial topic while jointly expressing a cross-cutting context, here public order, institutional authority, or transnational ownership. Given four cases, this is qualitative evidence for interpretation and not a prevalence estimate.

A complementary analysis examines distinctions within an editorial category. We enumerate pairs of observed profiles differing in exactly one bit, retaining pairs with at least 100 documents in each profile. The remaining fifteen coordinates, including inactive bits, are identical within a pair. AG News has 224 eligible pairs. We select the pair maximizing the smaller group size, without using labels or text content. Its profiles differ only in bit 08 and contain 1,050 and 1,224 documents. Both are almost entirely labelled Sports (Table 1).

For the sports contrast, document IDs are sorted numerically within each profile, and twelve documents are drawn without replacement using Python’s random.Random(1234).sample; the generator is reset for each profile. Inspection of these saved samples suggests a concentration on American football when bit 08 is active, whereas the other group includes baseball, basketball, and other sports. For example, the inactive group contains “NL Wrap: Walker Grand Slam Helps Cards Beat Reds Again” (ID 2270), while the active group contains “Jones Leads Cowboys to Stunning Win Over Seahawks” (ID 6670). We subsequently count indicative terms across both complete groups. References to touchdowns or quarterbacks are more frequent in the active group, while references to home runs or innings are more frequent in the inactive group. These lexical contrasts support a finer thematic reading within the same editorial category. The terms were selected after inspection and serve as descriptive indicators, rather than a validated classification of sports. Throughout these comparisons, differences concern observed document groups and do not estimate the effect of intervening on a bit.

Newsgroups

“20 Newsgroups” consists of Usenet discussion posts assigned to 20 newsgroups. We retrieve both official partitions using scikit-learn’s fetch_20newsgroups, removing headers, signatures, and quoted passages with its headers, footers, and quotes options (scikit-learn developers, n.d.). After trimming text and requiring at least 20 characters, 10,952 training and 7,269 test posts remain. In addition to the 20 original newsgroup labels, our pipeline defines five coarse groups: computing (the five comp.* groups), recreation (the four rec.* groups), science (the four sci.* groups), miscellaneous sales (misc.forsale), and discussion. The last combines the four talk.* groups with alt.atheism and soc.religion.christian. This five-way grouping is an analytical aggregation of the source labels.

Twitter US Airline Sentiment

The Airline corpus contains tweets about six US airlines, collected in February 2015 and annotated through CrowdFlower for sentiment and, for negative tweets, complaint reason. We use the Hugging Face distribution [8]. Removing repeated tweet identifiers and retaining trimmed texts of at least five characters leaves 14,485 tweets: 9,082 negative, 3,069 neutral, and 2,334 positive. The carrier label identifies United, US Airways, American, Southwest, Delta, or Virgin America. We derive a 12-category sentiment refinement by retaining positive and neutral and dividing negative tweets into customer-service issues, late flights, cancelled flights, lost luggage, bad flights, booking problems, flight-attendant complaints, long lines, damaged luggage, and an unknown-reason category. Missing or indeterminate complaint reasons enter the last category. A further label crosses the three sentiment classes with the six airlines, producing 18 observed combinations.

GoEmotions

GoEmotions contains English Reddit comments annotated for 27 emotions or a neutral category [10]. We use the simplified configuration of google-research-datasets/go_emotions, pooling train, validation, and test. The present analysis retains comments with exactly one supplied label and at least five characters after trimming, yielding 45,432 documents. The 28-category label is also aggregated into positive, negative, ambiguous, and neutral affect. Positive comprises admiration, amusement, approval, caring, desire, excitement, gratitude, joy, love, optimism, pride, and relief; negative comprises anger, annoyance, disappointment, disapproval, disgust, embarrassment, fear, grief, nervousness, remorse, and sadness; ambiguous comprises confusion, curiosity, realization, and surprise. Neutral remains separate. The all-labels field coincides with the primary label in this selected single-label corpus.

TweetEval

We use four English tweet-classification tasks from TweetEval [1], downloaded from cardiffnlp/tweet_eval. Sentiment distinguishes negative, neutral, and positive tweets and derives from SemEval-2017 Task 4 [41]. Emotion distinguishes anger, joy, optimism, and sadness, drawing on SemEval-2018 Task 1 [30]. Hate distinguishes hateful from non-hateful tweets in the task concerning immigrants and women [2]. Offensive distinguishes offensive from non-offensive language [52]. Each task is treated as a separate corpus. The prepared files pool the source training and test partitions, omit validation, and require at least five characters after trimming. Their resulting sizes are 57,899, 4,678, 11,961, and 12,776, respectively. The dataset’s category names are retained alongside their numeric encodings.

FakeNewsNet

FakeNewsNet combines collections associated with PolitiFact and GossipCop [47]. We download the four public CSV files for real and fake items in the authors’ repository. The text input is the news title available in these files; article bodies and social-network information are not collected for this experiment. After requiring at least 30 characters and removing duplicate source rows, the corpus contains 21,786 items: 20,907 from GossipCop and 879 from PolitiFact. We retain the supplied real/fake classification (16,372 real and 5,414 fake), the source collection, and their four-way cross-classification. Source denotes the fact-checking collection, not the publishing outlet.

PHEME

PHEME contains Twitter conversations around breaking-news events. We use the official Figshare archive for rumour detection and veracity classification [22], selecting source tweets from five events: Charlie Hebdo, Ferguson, the Germanwings crash, the Ottawa shooting, and the Sydney siege. Replies are not separate documents in our corpus. The prepared sample contains 5,802 source tweets, comprising 3,830 non-rumours and 1,972 rumours. For rumours, veracity is true (1,008), false (393), or unverified (571); non-rumours receive a not-applicable code. We retain event identity, binary rumour status, and veracity, and derive a four-category label comprising non-rumour, true rumour, false rumour, and unverified rumour. Retweet and favorite counts provide the engagement-prediction application in § 9. These counts are snapshots: source tweets meet retweet thresholds of 100 in four events and 25 in Germanwings, so this application concerns engagement among already visible tweets.

Climate and migration literature

The corpus called Climate Migration in our analyses contains 5,545 scholarly abstracts from the migration-literature resource associated with LlaMig [20]. We use allocoded.csv and key-values.csv from the authors’ repository, deduplicate document identifiers, and retain abstracts of at least 20 characters. The model input is the abstract; titles and DOIs are metadata. Six supplied coding fields describe migration relevance (yes/no), migration drivers, sentiment toward migration, climate or environmental hazards, discipline, and use of quantitative methods (yes/no). The corpus includes papers outside migration: 4,175 abstracts are coded as relevant and 1,370 as not relevant.

Drivers can include other, climate, environment, economic, conflict, political, social, and family, with multiple codes per document. The prepared hazard field contains 29 observed non-empty codes,

including pollution, infectious diseases, drought, flooding, and sea-level rise. Blank source values in drivers, sentiment, hazards, and discipline are represented by none; this denotes an absent code rather than an independently established negative finding. Sentiment has positive, negative, neutral, and none; the observed discipline values are social science, other, and none. Drivers and hazards remain semicolon-separated fields. Semantic profile summaries count their individual constituent labels. The existing categorical predictive comparisons instead use the recorded combinations as classes: drivers has 82 observed strings, reduced to 53 classes and 5,516 documents after singleton exclusion; hazards has 51 strings, reduced to 27 classes and 5,521 documents. The other four targets use all 5,545 documents.

Table S5 reports the existing categorical driver and hazard evaluations. Their low macro-F1 scores and sparse class support motivate treating these combination tasks separately from the four simpler coding fields in the main text. BA averages recall over classes observed in the test set; the saved macro-F1 calculation averages over the union of true and predicted classes, assigning zero to undefined class scores.

Corpus	Attribute	$N$	$C$	AMI	BA [95% CI]	Macro-F1
Climate Migration	Driver combinations	5,516	53 (40)	0.123	6.42 [5.41, 7.66]	0.99
Climate Migration	Hazard combinations	5,521	27 (21)	0.015	13.12 [9.57, 16.81]	2.93

Mixed corpus

The Mixed corpus combines 111,000 documents sampled without replacement from the prepared sources, using master seed 1234 and an independent deterministic random-number stream for each source. It includes 4,000 documents from each of the four TweetEval tasks; 5,000 each from PHEME and Climate Migration; 10,000 each from AG News and Airline; 15,000 each from 20 Newsgroups, FakeNewsNet, and GoEmotions; and 20,000 from IRA. The IRA component consists of Internet Research Agency tweets collected by Linvill and Warren and distributed through FiveThirtyEight’s russian-troll-tweets repository ([25]). It provides an additional source of tweets within the heterogeneous mixture; IRA is not evaluated as a standalone corpus in this work. A separate seeded permutation assigns new identifiers and interleaves source rows, while the original identifiers are retained as provenance. The AG News component was randomly sampled from an earlier stratified extraction of 15,600 documents, prepared before the dedicated AG News analysis was extended to all 127,600 documents. This sampling history does not change the purpose of the Mixed experiment, which examines structure across heterogeneous document sources.

The nine-valued source label merges the four TweetEval tasks under one source. A second label records document type: tweet (51,000), news (25,000), discussion post (15,000), comment (15,000), or abstract (5,000). The 23 source-specific label fields retain the four TweetEval targets; PHEME status, veracity, and event; migration relevance, discipline, and sentiment; AG News topic; fine and coarse newsgroups; FakeNewsNet veracity and source; IRA category, type, and language; Airline sentiment and carrier; and GoEmotions primary, grouped, and all-labels fields. Non-applicable entries are stored as unknown and excluded when interpreting the corresponding source-specific label. Cross-products and complaint-based refinements are omitted. Source and document type

provide corpus-wide reference partitions, while the remaining labels support interpretation within their applicable subsets.

Embedding clusters and binary document groups

Residual entropy and partition agreement

These diagnostics supplement § 4.4. Lloyd $k$-means is fitted to all 127,600 raw, unnormalised Qwen embeddings at $M \in\{32,64,256\}$, without labels or binary codes. We use scikit-learn’s Lloyd implementation with $k$-means++ initialization, ten starts, a maximum of 300 iterations per start, tolerance $10^{-4}$, and random seed 1234, retaining the solution with the lowest within-cluster sum of squared distances. The L2-normalized check at $M = 32$ and 256 uses the same settings. Complete profiles partition the same documents, whereas conjunctions can overlap.

Let $C$ denote cluster membership and $B_k$ a binary coordinate. The decomposition

separates the bit entropy associated with cluster membership from the variation remaining within clusters. We report the empirical residual fraction $H(B_k \mid C)/H(B_k)$, which is zero when cluster membership determines the bit and one when they are independent. Averaged over the sixteen bits, this fraction is 0.578, 0.550, and 0.489 at $M = 32$, 64, and 256. Independently permuting each bit column, while preserving its prevalence, gives corresponding reference values of 1.000, 1.000, and 0.998. Thus, clusters account for part of the variation in bit activations, while substantial variation remains within them.

For complete profiles, the residual entropy fraction decreases from 0.656 at $M = 32$ to 0.516 at $M = 256$. The median effective number of profiles within a cluster decreases from 124.3 to 43.8. Conversely, the median three-bit conjunction requires 6, 11, and 30.5 clusters to cover 80% of its documents. For each conjunction, a support-preserving random reassignment provides a reference for this coverage count. The median of these conjunction-specific expected counts is 21.3, 41.3, and

First seed	Second seed	Mean matched-bit NMI
1234	2025	0.163
1234	31415	0.163
2025	31415	0.173

149.8, respectively (Supplementary § 15.5). Clusters and conjunctions are therefore associated but not equivalent: conjunctions concentrate within a subset of clusters, while extending across more than one, and clusters contain multiple complete profiles. The residual entropy of complete profiles also depends on the granularity of the partitions being compared.

Chance-corrected agreement between complete profiles and $k$-means ranges from 0.042 to 0.085 for ARI and from 0.402 to 0.412 for AMI ([19, 50]). Across three ISBC fits with different random seeds, the corresponding ranges are 0.147–0.181 and 0.349–0.363. These measures give different orderings for partitions of different granularities; the complete profile partition also varies across initializations.

Repeating clustering after L2 normalization changes the mean bit residual entropy fraction from 0.578 to 0.583 at $M = 32$ and from 0.489 to 0.488 at $M = 256$. Profile–cluster AMI changes from 0.405 to 0.407 and from 0.402 to 0.404, respectively. The distinction between the representations therefore persists when clustering emphasizes angular geometry.

Bit correspondence across initializations

We compare the individual coordinates of the three full-objective Qwen fits used in § 4.7, with $K = 16$, $A = 6$, and seeds 1234, 2025, and 31415. All codes are evaluated on the same 127,600 AG News documents, aligned by document identifier. For each pair of fits, the Hungarian algorithm selects the one-to-one bit assignment maximizing arithmetic-normalized mutual information,

This score is unaffected by complementing a bit, so inverse activation patterns can also be matched.

The matched scores indicate limited one-to-one correspondence between coordinates across these fits, even after accounting for permutations and polarity. Similar aggregate predictive performance across seeds therefore coexists with different bit memberships. The Mixed extension addresses a separate question: whether selected associations persist for additional documents when the encoder itself is held fixed.

Conditional bit entropy for the ITQ comparison

Using the same raw-embedding $k$-means partitions and residual entropy fraction as in § 4.4, we average over the sixteen nonconstant coordinates. tbl:S7 reports these descriptive values for the unchanged ISBC code and both ITQ readouts. Activity and bit prevalences differ across the representations, so the comparison does not isolate an effect of encoder architecture.

Category-based descriptions of conjunctions and clusters

We locate the two Business subsets examined in § 4.3 within the $M = 32$, 64, and 256 embedding-cluster partitions described in § 4.4. Each subset combines the Business label

caption: Mean residual bit-entropy fraction conditional on $M$ embedding clusters, evaluated on all 127,600 documents. Zero means cluster membership determines the bit; one means it carries no information about the bit. Values summarize association with the specified partitions.

:::

with a conjunction of community bits; these targets are therefore defined after fitting and are not an independent benchmark. The 147-document technology-market subset spans 9, 20, and 39 clusters at the three resolutions, while the 22-document Business–World subset spans 9, 11, and 17. Neither subset coincides with a single cluster. Uniformly reassigning each target across documents while preserving its size and the cluster sizes gives expected occupancy counts of 30.8, 54.3, and 105.3 for the technology-market subset, and 15.3, 18.2, and 20.9 for the Business–World subset. The observed subsets are thus concentrated relative to this reference, although their boundaries do not coincide with individual clusters. Table S8 collects the occupancy counts and their random references.

As a secondary description, we examine category composition. Restricting three-bit conjunctions to the observed cluster-size range, their mean dominant-category shares are 62.7%, 59.1%, and 54.7%, compared with 87.5%, 86.6%, and 89.1% for clusters. This restriction does not exactly match the group-size distributions. Within these comparisons, conjunction-defined groups span the editorial categories more broadly than clusters do; lower category purity alone does not establish semantic coherence.

Random references for cluster occupancy and coverage

The reference randomly assigns a target of size $m$ to the $N = 127{,}600$ documents, keeping the fitted cluster sizes $n_c$ fixed. The expected number of occupied clusters is

We evaluate this expression exactly for the two Business subsets. For each of the $\binom{16}{3} = 560$ three-bit conjunctions, we draw 2,000 cluster-count vectors from the corresponding multivariate hypergeometric distribution, preserving its observed support. In each draw, clusters are ordered by target count and we count how many are needed to cover at least 80% of the target documents. Table S8 reports the median of the 560 conjunction-specific Monte Carlo means, alongside the median observed count. This is a collection of marginal reference values, not a joint permutation test of the median across overlapping conjunctions. The seed is 1234, with a separate reproducible stream for each cluster resolution and conjunction.

AG News: capacity, reconstruction, geometry, and prediction

Qwen code capacity and reconstruction diagnostics

The following definitions and diagnostics support Section 4.5, using the same corpus and fitted codes. Profile probabilities, empirical entropy $\hat{H}(B)$, and effective profile counts are defined in Section 4.5. Finite-sample entropy estimates tend to underestimate population entropy, particularly when many profiles are rare or unobserved [38]. We assess this sensitivity with the Miller–Madow correction,

$M$	Technology-market Observed	Technology-market Random	Business–World Observed	Business–World Random	Three-bit coverage Observed	Three-bit coverage Random
32	9	30.8	9	15.3	6	21.3
64	20	54.3	11	18.2	11	41.3
256	39	105.3	17	20.9	30.5	149.8

where $m$ is the observed number of profiles. It adds 0.069 and 0.273 bits per document for the 16/6 and 32/6 codes, and 0.597 for 32/16. These increasing adjustments reflect the difficulty of estimating a more varied profile distribution from the same corpus. The first-order correction is a sensitivity diagnostic and may itself be inaccurate for sparsely sampled distributions.

For the PCA reference justified in § 4.5 (Eckart and Young, 1936), let $\lambda_1 \ge\cdots\ge\lambda_D \ge0$ be the eigenvalues of the empirical embedding covariance matrix $\widehat{\Sigma}_x = (N-1)^{-1}\sum_{i=1}^{N}(x_i-\bar{x})(x_i-\bar{x})^{\mathsf{T}}$, where $\bar{x}=N^{-1}\sum_{i=1}^{N}x_i$ is the corpus mean. The cumulative proportion of variance explained by the first $K$ principal components provides the optimal affine reconstruction reference at dimension $K$:

We approximate this quantity with randomized singular value decomposition of the centered embedding matrix [17]. The reference matches code dimension, with continuous rather than binary coefficients.

The reconstruction score $R^2$ and trained/OLS decoder comparison are defined in § 4.5. For every configuration, ordinary least squares refits an intercept and $K$ coefficient vectors to all corpus embeddings while keeping the learned hard codes fixed. Table 2 reports the resulting improvements over the saved decoder. Supplementary § 17 details the role of internal layer normalization in the full pipeline.

Qwen geometric preservation diagnostics

These comparisons expand § 4.6, using the same sampled documents and pairs across all six configurations.

We assess geometric preservation by comparing distances between the same document pairs in embedding and binary-code space. For each pair $(i,j)$, angular separation is measured by cosine distance, $d_{\cos}(u,v)=1-u^{\mathsf{T}}v/(\lVert u\rVert_2\lVert v\rVert_2)$, applied separately to $(x_i,x_j)$ and $(B_i,B_j)$. We also compare Euclidean embedding distances $\lVert x_i-x_j\rVert_2$, using the original unnormalized embeddings, with Hamming and Jaccard distances between codes. Writing $m_{ij}=\sum_{k=1}^{K}\lvert B_{ik}-B_{jk}\rvert$ for the number of coordinates that differ, normalized Hamming distance is $m_{ij}/K$, whereas Jaccard distance is $m_{ij}/\sum_k\max(B_{ik},B_{jk})$. Both depend on which bits differ between documents. Jaccard normalizes these differences by the number of coordinates active in at least one document. Since $\lVert B_i-B_j\rVert_2^2=m_{ij}$, Hamming and binary Euclidean distances induce the same ordering and therefore yield identical Spearman correlations with embedding distances.

We use the paired-distance sample and average-rank treatment of ties specified in § 4.6; no sampled embedding or code has zero norm. The 100,000 unordered pairs are sampled without replacement. Table S9 reports all six configurations, including the Jaccard comparisons.

For the local comparison in Table 3, let $n_{B_i}$ be the number of corpus documents with the sampled document’s profile. Its probability of matching a uniformly selected other document is $(n_{B_i}-1)/(N-1)$; averaging over the 2,000 sampled documents gives the random reference. Exact-profile agreement becomes a stricter criterion as more profiles are distinguished, so the higher agreement of eight-bit codes should be read alongside their higher random agreement.

Qwen predictive evaluation across widths and activity targets

The following results supplement § 4.8.

Table S10 expands the logistic-probe evaluation defined in § 4.8, using the same fitted codes and label split. Let $X$ denote the embedding of a randomly selected document, $Y$ its assigned AG News category, and $e$ the fitted deterministic binary encoder. Its code is $B=e(X)$. The data-processing inequality gives $I(Y;B) \le I(Y;X)$. Predictive performance assesses how effectively the chosen probe uses the representation for this particular taxonomy [51], while the profile analysis examines semantic structure beyond that taxonomy. We also retain cross-entropy contrasts: at target six, the excess over raw embeddings falls from 0.242 to 0.163 bits/document when width increases from 16 to 32.

At target activity six, increasing width from 16 to 32 bits raises reconstruction $R^2$ from 0.160 to 0.214 and BA from 85.62% to 87.63%. The same logistic probe applied to the original Qwen embeddings achieves 91.09% BA on the same split. The binary codes therefore achieve 93.99% and 96.20% of the embedding-based BA, respectively (Table S10). The eight-bit code with the same activity target reaches 86.79% BA, also exceeding the sixteen-bit reference. Meanwhile, cosine–Jaccard rank correlation changes little (0.493 to 0.495; Table S9). At 32 bits, increasing target activity from six to sixteen further improves reconstruction to 0.232, while BA decreases to 86.42%, corresponding to 94.87% of the embedding-based BA. The configurations therefore differ in reconstruction fidelity and predictive utility for the evaluated taxonomy.

Loss ablation: representation diagnostics and predictive evaluation

This section supports § 4.7. Unless specified otherwise, values are means and sample standard deviations across the three paired encoder seeds. They describe variation across fitted representations on the same corpus. The full objective yields $6.01 \pm0.05$ active bits and $2501.2 \pm51.0$ effective profiles. The latter averages across seeds; the seed-1234 fit has 2,560.2 effective profiles and supplies the reference for the other AG News comparisons.

Reconstruction errors in Table S11 and Table S12 are divided by the training calibration variance $s_x$ in (7). This differs from normalization by the full-corpus mean-baseline error in Table 6. For each fixed hard code, the OLS decoder refits an intercept and coefficient vectors to all corpus embeddings. The trained decoder’s normalized MSE increases from $0.854 \pm0.002$ to $1.035 \pm0.000$ when reconstruction is removed; refitting gives $0.852 \pm0.002$ and $0.918 \pm0.003$, respectively. The latter comparison evaluates code recoverability independently of the original decoder’s optimization.

Objective	OLS MSE/$s_x$	$\rho_{\mathrm{cos},\mathrm{cos}}$	$\rho_{\mathrm{Euc},\mathrm{Ham}}$	$\hat{H}(B)$	Mean bit entropy
Full	0.852 ± 0.002	0.484 ± 0.009	0.331 ± 0.005	11.288 ± 0.029	0.954 ± 0.003
Without activity	0.849 ± 0.001	0.450 ± 0.022	0.354 ± 0.003	11.431 ± 0.063	0.949 ± 0.003
Without balance	1.015 ± 0.000	–	–	0.000 ± 0.000	0.000 ± 0.000
Without decorrelation	0.852 ± 0.002	0.486 ± 0.015	0.335 ± 0.003	11.240 ± 0.075	0.953 ± 0.002
Without entropy	0.848 ± 0.000	0.471 ± 0.002	0.345 ± 0.006	12.129 ± 0.053	0.953 ± 0.003
Without reconstruction	0.918 ± 0.003	0.408 ± 0.004	0.267 ± 0.006	5.518 ± 0.361	0.943 ± 0.001

Removed term	Diagnostic	Full	Ablated	Paired change
Activity control	Absolute activity error	0.753 ± 0.011	1.319 ± 0.025	+0.566 ± 0.036
Balance	Prevalence SD	0.014 ± 0.002	0.484 ± 0.000	+0.470 ± 0.002
Decorrelation	Absolute correlation	0.120 ± 0.006	0.128 ± 0.002	+0.008 ± 0.008
Entropy	Conditional entropy (nats)	0.0076 ± 0.0002	0.3026 ± 0.0002	+0.2950 ± 0.0004
Reconstruction	Decoder MSE/$s_x$	0.854 ± 0.002	1.035 ± 0.000	+0.181 ± 0.002

Profile entropy and effective profiles follow § 4.5. For marginal bit entropy, let $\hat{\pi}_k = N^{-1}\sum_i B_{ik}$ denote the observed prevalence of bit $k$. We average $-\hat{\pi}_k\log_2\hat{\pi}_k-(1-\hat{\pi}_k)\log_2(1-\hat{\pi}_k)$ over the $K$ coordinates, with $0\log0 = 0$. This measures binary variation across documents. Soft-activation entropy instead averages $h(p)=-p\ln p-(1-p)\ln(1-p)$ over documents and coordinates, as in (12). It is measured in nats (natural units of information), is zero at $p=0$ or $p=1$, and reaches $\ln2 \simeq0.693$ at $p=0.5$.

The geometric correlations reuse the document sample and pair indices of § 4.6. Without activity control, the three fits contain 104, 3, and 167 all-zero codes. Cosine distance is undefined for these vectors, so pairs involving them are omitted from the angular comparison, leaving 99,888, 100,000, and 99,899 pairs; all 100,000 pairs remain in each Euclidean–Hamming comparison. Constant distances in the collapsed no-balance fits yield undefined rank correlations. Mean absolute bit correlation is computed over nonconstant coordinates and is likewise undefined when every bit is constant.

The community analysis applies the positive-PMI graph protocol of Supplementary § 17.1 independently to each fit, retaining all 16 bit nodes and using weighted Leiden modularity at resolution $\gamma=1$. Twenty optimizer seeds (1234–1253) assess partition stability on each fixed graph. All full-objective fits have three communities, as do fits without decorrelation, entropy, or reconstruction; without activity control the counts are four, three, and three. Without balance there are no positive-PMI edges and all nodes are isolated.

Figure S14 shows the seed-1234 graphs. Mean modularity across encoder fits is 0.486 for the full objective, 0.398 without activity, 0.480 without decorrelation, 0.439 without entropy, and 0.610 without reconstruction. The last result accompanies a sharp reduction in profile diversity and stronger bit dependence: more separated co-activation groups can occur within a more repetitive code. Mean pairwise adjusted Rand agreement across Leiden seeds is 1.000 for all nonempty graphs except two no-entropy fits, where it is 0.958 and 0.979. This measures optimizer stability on fixed graphs, separately from variation across encoder fits.

As a secondary evaluation, class-balanced logistic classifiers use the same 80/20 label split as the other AG News analyses, with either deterministic binary codes or activation probabilities $p_i$ as inputs. Probabilities are entered directly, rather than transformed into log-odds as in Supplementary § 19.3. The representation is fitted without labels on the full corpus. Table S13 reports balanced accuracy and paired changes across encoder seeds.

Objective	Binary BA	$\Delta$ binary BA	Continuous BA	Effective profiles
Full	$85.61 \pm0.32$	$0.00$	$85.77 \pm0.35$	$2501.2 \pm51.0$
Without activity control	$85.63 \pm0.24$	$+0.02 \pm0.27$	$85.73 \pm0.23$	$2763.2 \pm122.0$
Without balance	$25.00 \pm0.00$	$-60.61 \pm0.32$	$25.00 \pm0.00$	$1.0 \pm0.0$
Without decorrelation	$85.80 \pm0.68$	$+0.18 \pm0.64$	$85.90 \pm0.66$	$2420.2 \pm124.1$
Without entropy	$84.99 \pm0.08$	$-0.63 \pm0.24$	$87.90 \pm0.10$	$4479.7 \pm167.2$
Without reconstruction	$81.02 \pm0.81$	$-4.59 \pm0.93$	$81.14 \pm0.86$	$46.8 \pm11.2$

Removing balance gives the constant-code result of 25% BA. Removing reconstruction reduces binary BA by 4.59 ± 0.93 percentage points, although category information remains accessible at 81.02 ± 0.81% BA. Without entropy regularization, continuous BA rises from 85.77% to 87.90%, while binary BA falls from 85.61% to 84.99%, widening the continuous–binary difference. Removing activity or decorrelation has little average effect on binary BA in these fits.

Organization across code capacities

We next examine how the organization of the code changes with its capacity. Here, capacity refers to the representational budget provided by $K$ binary coordinates, with at most $2^K$ possible profiles. The activity target controls their use. We compare $K = 8, 16, 32$ at a common target of six active bits, corresponding to target activation fractions of 75%, 37.5%, and 18.75%. Activity is encouraged by a penalty rather than fixed exactly. The three fitted codes contain 78, 12,174, and 48,371

profiles, respectively. Thus, increasing width permits a finer empirical differentiation of documents while keeping the target number of active coordinates constant.

To trace associations across these representations, we construct a tripartite graph: its three node sets contain the bits of the 8/6, 16/6, and 32/6 encoders, and edges connect bits from different encoders. Pairwise associations are computed from their activations on the same 127,600 documents, without category labels. The full graph contains 56 nodes and 456 edges with finite positive PMI (Figure S15, top). Node colors record the Leiden communities fitted separately within each encoder; community identifiers and bit indices are local to each code.

For a compact view, we identify closed triads containing one bit from each encoder, with positive PMI and $\phi$ on all three edges. We rank the 750 qualifying triads by their weakest $\phi$ association and show the union of the 20 highest-ranked triads in the lower panel. This criterion favors associations supported across all three pairs. The strongest triad links bit 3 of 8/6, bit 0 of 16/6, and bit 19 of 32/6, with minimum $\phi= 0.270$ and mean $\phi= 0.400$.

The selected triads reveal recurring groups of associations. Bit 3 of 8/6 connects to bits 0, 5, 10, and 12 in a single 16/6 community, which in turn connect to bits 0, 5, 17, and 19 in a single 32/6 community. A second group links bit 5 of 8/6 to bits 1 and 9 of 16/6 and bit 21 of 32/6. These correspondences suggest that activation patterns represented by individual coordinates in a narrower code can be distributed over several coordinates in wider codes. At the document level, the conjunction of bits 2 and 3 in 8/6 selects 93,871 documents and is most strongly associated with bit 10 of 16/6 ($\phi= 0.436$) and bit 19 of 32/6 ($\phi= 0.270$). Together, the graph and conjunction analysis describe recurring co-activation structure alongside the increasing diversity of complete profiles.

Cross-capacity associations also include inverse activation patterns: there are 62 negative pairs between 8/6 and 16/6, 94 between 8/6 and 32/6, and 284 between 16/6 and 32/6. For example, bit 3 of 8/6 and bit 21 of 32/6 have $\phi= -0.789$. The latter activates in 70.7% of documents where the former is inactive, compared with 0.47% where it is active. Thus, correspondence across capacities can involve the inactive state of a coordinate as well as its active state. The positive-edge graphs in Figure S15 summarize the co-activation component of this relationship.

Bit associations, activity constraints, and encoder normalization

Qwen bit-association graph and sensitivity analysis

This subsection supplies the graph protocol for § 4.2. Each node represents one of the sixteen Qwen bits. All 43 pairs with finite, strictly positive PMI enter the undirected graph, with PMI as the edge weight and no additional threshold. Leiden ([49]) optimizes weighted modularity with a configuration-model null ([40]), comparing within-community weights with the expectation based on weighted degrees. The resolution parameter $\gamma= 1$ gives standard modularity; larger values favor finer partitions.

We run seeds 1234–1253 at each $\gamma\in\{0.5, 0.75, 1, 1.25, 1.5, 2\}$ and retain the partition with the highest objective value within each resolution. Repeated seeds assess sensitivity to the stochastic optimizer; the grid assesses coarser and finer groupings. All seeds return the same partition at each tested resolution, with pairwise adjusted Rand index (ARI) equal to one ([19]). The reference three-community partition is unchanged at $\gamma= 0.75$ and $1.25$; the graph has two communities at $0.5$ and four at $1.5$ and $2$.

For non-constant binary variables, PMI and $\phi$ have the same sign. We therefore overlay all 77 negative-$\phi$ pairs without changing the 43 edges used for community detection. Community fitting and spring layout use positive PMI weights; displayed widths and opacity encode $|\phi|$ for both signs.

The eight negative associations internal to communities have $\phi$ between $-0.080$ and $-0.016$. The overlay is descriptive, not an evaluation against an activity-preserving random reference.

In Figure 3, bit 2 co-activates with bits 1, 9, 13, and 14 more often than marginal independence predicts, as does bit 0 with bit 12. Bits 0 and 2, and bit 1 paired with bits 10 or 12, co-activate less than this reference predicts. Complete mutual exclusion is the special case of zero joint activation, giving PMI $= -\infty$ when both marginal prevalences are positive. A dead bit never activates and a universal bit is always active; neither occurs here. Assessing individual pair associations beyond the activity constraint would require a reference preserving both document-level activity and marginal bit frequencies.

Activity constraints and aggregate dependence

Let $S = \sum_{k=1}^{K} B_k$ be document-level activity and $q_k = \Pr(B_k = 1)$. The variance identity

relates activity variation to pairwise dependence. In the Qwen 16/6 codes, the empirical activity variance is 1.067, whereas the sum of marginal Bernoulli variances is 3.732. The sum of pairwise covariances is therefore $-1.333$, and the mean pairwise $\phi$ correlation is $-0.0476$. For illustration, exchangeable bits with exactly fixed activity would have pairwise correlation $-1/(K-1)$, or $-0.0667$ at $K=16$, even without different semantic roles. Our activity is approximately, rather than exactly, fixed, and its bits have different prevalences. The exchangeable fixed-activity value is therefore illustrative, not a calibrated null for the observed graph. The identity accounts for the aggregate covariance constraint; it does not determine which specific pairs are negative or how much of each association is semantic.

Encoder normalization and reconstruction references

Layer normalization is applied within the encoder as

where $\gamma_j$ and $\beta_j$ are learned parameters and $\epsilon$ is the numerical stabilizer. This removes the coordinate mean of each vector and rescales its centered coordinates before encoding, while reconstruction still targets the raw vector. Thus, the comparison with raw embeddings or PCA concerns the full learned representation, including normalization, nonlinear encoding, binary activations, and the learning objective. It does not isolate the contribution of binary discretization to reconstruction error or distance distortion.

Cross-backbone correspondence and diagnostics

Backbone characteristics and relative performance

This supplementary section expands Section 5, using the same five fitted encoders and all 127,600 AG News documents.

Qwen retains causal attention and uses the final token representation. MiniLM uses a six-layer bidirectional BERT-style encoder with mean pooling. The NVIDIA model adapts Llama 3.2 to bidirectional attention and also uses mean pooling. Harrier uses a Gemma 3 decoder architecture. with causal attention and last-token pooling; its training includes knowledge distillation from larger embedding models. EmbeddingGemma 2 uses a Gemma 4-based bidirectional encoder with mean pooling and a projection from 512 hidden dimensions to 768 output dimensions. We use its text representations. In our experiments, the pooled embedding outputs are retained without L2 normalization before fitting the binary encoder. Each encoder uses $K = 16$ bits with target activity $A = 6$ and is trained without labels.

Predictive evaluation uses the common AG News 80/20 split and logistic-classification settings in Supplementary Section S1.1. We first assess the relative impact of compression within each backbone, then examine whether profiles learned from different embedding spaces identify corresponding document groups. Table 4 reports predictive performance relative to the original embeddings, the relative BA = 100 $\mathrm{BA}_{\mathrm{binary}}/\mathrm{BA}_{\mathrm{embeddings}}$. We also report the relative reconstruction = 100 $R^2_{\mathrm{trained}}/R^2_{\mathrm{PCA},16}$, which compares the trained binary decoder with the continuous 16-dimensional PCA reference for the same embedding space. The first ratio measures relative predictive performance; the second measures the fraction of PCA-explained variance recovered by the binary decoder.

Backbone	Architecture family	Parameters (M)	$D$	$\bar{a}$	Effective profiles	Relative BA	Relative reconstruction
Qwen	Qwen 3	600	1024	5.94	2560.2	93.99	65.23
MiniLM	BERT-style	22.7	384	5.97	6488.6	92.38	67.69
NVIDIA	Llama 3.2	3200	3072	5.90	521.3	95.24	70.73
Harrier	Gemma 3	270	640	5.95	824.0	94.61	61.73
EmbeddingGemma 2	Gemma 4	270	768	6.02	1252.4	93.19	52.87

Mean activity remains close to six for every backbone, while the effective number of profiles ranges from 521.3 to 6488.6. The same code budget therefore produces different distributions of document groups. Binary representations achieve between 92.38% and 95.24% of their respective embedding-based BA. Relative reconstruction ranges from 52.87% to 70.73% of the PCA reference. The backbone ordering differs between these measures, indicating that preservation of embedding variation and accessibility of category information respond differently to compression. Table S14 reports descriptive associations between backbone size and performance. We log-transform parameter counts because they span more than two orders of magnitude, while retaining embedding dimensions on their original scale. Predictive performance is more strongly associated with log parameter count than with embedding dimension, for both original embeddings and binary codes. For binary BA, the respective Pearson correlations are 0.849 and 0.688. Binary reconstruction shows the opposite pattern: its Pearson correlation is 0.787 with embedding dimension and 0.447 with log parameter count. Spearman correlations support the same ordering. These associations are descriptive, as the comparison includes only the five backbones considered here.

	Embedding BA		Binary BA		PCA reconstruction		Binary reconstruction
	Pearson $r$	Spearman $\rho$	Pearson $r$	Spearman $\rho$	Pearson $r$	Spearman $\rho$	Pearson $r$	Spearman $\rho$
$\log_{10}$(number of parameters)	0.793	0.667	0.849	0.821	0.280	0.103	0.447	0.359
Embedding dimension	0.616	0.600	0.688	0.700	0.386	0.300	0.787	0.500

Complete-profile partitions and document-based matching

We first compare the complete document partitions induced by the five binary encoders. Each distinct 16-bit profile defines one group, and every document belongs to exactly one group within each backbone. Table S15 describes their granularity. The number of observed profiles ranges from 4,862 for NVIDIA to 20,470 for MiniLM. Documents with singleton profiles account for 1.26% to 6.11% of the corpus.

Backbone	Observed profiles	Mean documents per profile	Documents in singleton profiles (%)
Qwen	12,174	10.48	3.48
MiniLM	20,470	6.23	6.11
NVIDIA	4,862	26.24	1.26
Harrier	5,462	23.36	1.31
EmbeddingGemma 2	6,490	19.66	1.54

For each backbone pair, we calculate the adjusted Rand index (ARI; [19]) and adjusted mutual information [50]. ARI measures agreement on whether document pairs share the same profile. AMI measures the information shared between profile assignments, correcting for its expectation under random assignments with the observed group sizes held fixed. Let $Z_m$ denote a document’s profile identifier under backbone $m$. We use the arithmetic normalization

Both scores equal one for identical partitions and have a chance-reference expectation of zero. They use all documents, require neither category labels nor bit alignment, and involve no nearest-neighbour selection or tie-breaking. They assess exact grouping: distinct profiles remain separate regardless of their Hamming distance.

Table S16 shows positive chance-adjusted agreement for every backbone pair, with ARI ranging from 0.045 to 0.169 and AMI from 0.165 to 0.376. NVIDIA and Harrier have the highest ARI, while Qwen and NVIDIA have the highest AMI. MiniLM and EmbeddingGemma 2 have the lowest values on both measures. Exact document-grouping agreement is therefore limited, although the partitions share information. The scores reflect differences in both group membership and granularity: dividing a document group among several distinct profiles changes the partition even when the resulting groups contain related semantic content.

Backbone 1	Backbone 2	ARI	AMI
Qwen	MiniLM	0.095	0.233
Qwen	NVIDIA	0.106	0.376
Qwen	Harrier	0.087	0.284
Qwen	EmbeddingGemma 2	0.059	0.221
MiniLM	NVIDIA	0.077	0.295
MiniLM	Harrier	0.065	0.213
MiniLM	EmbeddingGemma 2	0.045	0.165
NVIDIA	Harrier	0.169	0.369
NVIDIA	EmbeddingGemma 2	0.099	0.304
Harrier	EmbeddingGemma 2	0.077	0.263

To examine individual correspondences within these global results, we next match complete profiles through membership of their document groups. Bit indices are local to each encoder, so matching operates on the documents assigned to a profile rather than on the positions of its active bits. Starting from a reference profile (Qwen in our case), we identify the most frequent profile among documents in its group for each remaining backbone. We select the backbone-profile pair with the largest conditional share, restrict the document set to that intersection, and repeat until all five backbones are represented. We call the resulting set of matched profiles a matched profile family. At each step, we record the retained share and its lift relative to the selected profile’s overall corpus frequency. This procedure allows several reference profiles to correspond to the same profile in another backbone. Neither category labels nor a correspondence between bit indices enters the matching.

The matching artifacts use each backbone in turn as reference. Here we focus on the 20 most frequent Qwen profiles, whose document groups together contain 16,838 documents, or 13.20% of the corpus. Figure S16 presents the four most frequent profiles and the category compositions of their matched counterparts. Category distributions are calculated using the full document membership of each matched profile, allowing us to examine whether membership-based matches also have similar external interpretations.

The examples show recurring category composition alongside differences in the size and membership of their document groups. Approximately 90% of the documents in the second Qwen profile map to one NVIDIA profile; the matched profiles are predominantly associated with Sports. For the third Qwen profile, approximately 70% map to one Harrier profile, with Business predominant across the matched profile family. For the fourth, approximately 60% map to one NVIDIA profile, and the matched profiles are predominantly associated with World. The first Qwen profile, predominantly associated with Sci_Tech, has a smaller initial overlap of approximately 19% with its selected Harrier counterpart. Its matched profile family nevertheless retains similar category composition.

To summarize category-composition agreement across the 20 matched profile families, let $\pi_m$ denote the category distribution of the document group defined by the selected profile from backbone $m$. We define category coherence as

where $\operatorname{JS}_2$ is Jensen–Shannon divergence computed with base-two logarithms:

Category coherence equals one when all five document groups in a matched profile family have

identical category distributions. Across the 20 matched profile families, observed category coherence ranges from 0.985 to approximately 1.000.

We compare these values with two reference procedures, each using 1,000 random draws. The first replaces each non-reference profile with another profile from the same backbone whose document group has approximately similar size. The second independently permutes document alignment for the non-reference backbones and reruns the complete matching procedure; category composition is then evaluated on the original document memberships of the selected profiles. Weighted by Qwen reference-profile frequency, observed mean category coherence is 0.996, compared with 0.349 and 0.346 under the two respective reference procedures. Thus, the matched families have substantially more similar category composition than these reference pairings. Document overlap provides a complementary view of this agreement. Eight of the 20 Qwen-based families retain at least 30 documents in the intersection of the document groups defined by all five matched profiles. The four examples in Table S17 retain between two and 69 documents. Similar category composition can therefore accompany substantial differences in document-group boundaries, as illustrated particularly by the first profile family.

Rank	Reference documents	First matched backbone	Initial overlap (%)	Five-profile intersection
1	1375	Harrier	19	2
2	1224	NVIDIA	90	68
3	1075	Harrier	70	69
4	1066	NVIDIA	60	56

Matched bit communities and conditional patterns

We examine correspondence across backbones by matching individual bits on the same 127,600 documents. For each of the ten backbone pairs, the Hungarian algorithm selects a one-to-one assignment between their 16 bits that maximizes total normalized mutual information (NMI). For binary coordinates $U$ and $V$, we use $\operatorname{NMI}(U,V) = 2I(U;V)/(H(U) + H(V))$. The resulting network contains 80 nodes and 160 selected edges. NMI measures dependence, including complementary activation patterns; we therefore record the sign of the binary Pearson correlation $\phi$ separately from the NMI magnitude. Of the selected edges, 109 are positive and 51 negative. Category labels enter neither matching nor clustering.

To group concordant activations, we apply Leiden [49] to the positive-edge subgraph, using NMI weights and configuration-model modularity with resolution one. We retain the highest-quality partition across 20 fixed seeds. Negative edges are shown as associations between nodes but do not enter the clustering objective. Figure 7 contains 11 connected communities and two singleton nodes. Ten communities include all five backbones and one includes four. The communities contain 94.1% of positive-edge weight; all 51 negative selected edges connect different communities or singleton nodes, although this was not imposed. Mean pairwise ARI across optimization seeds is 0.942. Resolutions 0.5 and 2 yield 10 and 15 multi-node communities, respectively, with the same two singletons and no negative selected edges within communities. Thus, the separation of activation directions persists across these settings, while community boundaries depend on resolution.

For each multi-node community, we select documents with an active member bit in at least two distinct backbones. This defines overlapping document groups and permits different members of a backbone to contribute. For each singleton, we instead report all documents activating that bit. Table S18 orders all four category shares within each group; the corpus reference is 25% per category. These category descriptions are subsequent to the label-free graph analysis.

Community/node	Documents	Coverage (%)	First	Second	Third	Fourth
C1	104,797	82.1	World (30.0)	Sci_Tech (27.6)	Business (26.8)	Sports (15.6)
C2	80,745	63.3	Sports (39.5)	Business (25.2)	Sci_Tech (24.5)	World (10.8)
C3	82,792	64.9	Business (36.9)	Sci_Tech (31.9)	World (20.0)	Sports (11.2)
C4	100,368	78.7	World (28.1)	Business (25.5)	Sports (25.1)	Sci_Tech (21.4)
C5	97,065	76.1	Sports (28.4)	Business (28.3)	Sci_Tech (27.2)	World (16.1)
C6	64,187	50.3	Sports (47.4)	World (39.9)	Business (7.0)	Sci_Tech (5.6)
C7	71,821	56.3	Business (37.8)	Sci_Tech (28.3)	World (22.8)	Sports (11.1)
C8	59,550	46.7	World (45.9)	Sci_Tech (27.4)	Business (16.0)	Sports (10.7)
C9	60,580	47.5	Sci_Tech (49.2)	World (22.7)	Business (16.8)	Sports (11.3)
C10	57,055	44.7	Business (47.2)	Sci_Tech (46.1)	World (5.9)	Sports (0.9)
C11	57,879	45.4	Sports (33.1)	Business (28.1)	World (20.7)	Sci_Tech (18.1)
C12 (G15)	45,023	35.3	Sports (36.6)	Sci_Tech (28.2)	World (17.8)	Business (17.4)
C13 (M1)	47,326	37.1	Sports (34.2)	Sci_Tech (28.1)	World (21.6)	Business (16.1)

The selected groups retain several category combinations. Business and Sci_Tech jointly account for 93.3% of C10 and are also the two largest categories in C3 and C7. Sports and World account for 87.3% of C6, while C8 combines World and Sci_Tech. C11 has elevated Sports and Business shares. C1, C4 and C5 cover broad portions of the corpus and have more mixed compositions. These associations locate document sets for interpretation. Their permissive membership rule differs from the conjunction of all community bits used in the earlier Qwen case study, so neither their coverage nor their thematic specificity is directly comparable. In particular, a category mixture does not imply that each selected document combines those topics.

Increasing resolution alone does not separate the paired categories. Across resolutions one to eight, the number of communities and singletons increases from 13 to 24. The members of C6 first separate across communities at resolution four and those of C10 at eight, but the resulting document groups still mix their principal categories. Greater specificity appears in within-community activation patterns (Table 5). For example, C10 contains patterns concentrated in Business and others concentrated in Sci_Tech, despite the broad Business–Sci_Tech composition of its pooled document group. These patterns specify both active and inactive coordinates within a community across backbones; they are not complete profiles of any individual encoder.

The pooled community composition consequently conceals information carried by differences among its member activations. A community can identify related coordinates across encoders while their combinations distinguish document subsets within that broader association.

The singleton nodes describe substantial document sets. EmbeddingGemma bit 15 (G15) activates in 45,023 documents and MiniLM bit 01 (M1) in 47,326. Both have elevated Sports and Sci_Tech shares, with documents from all four categories. Their isolation follows from the matching constraint: each has four selected partners, all negatively associated. Positive associations exist outside the one-to-one assignments, including G15–H11 ($\phi= 0.281$) and M1–N14 ($\phi= 0.265$). Their singleton status therefore concerns the selected-edge graph, rather than the absence of related coordinates in other backbones.

A seeded random sample of 30 active documents per singleton provides an initial reading of their content. G15 includes commentary and evaluative prose across topics, such as a metaphorical account of the Red Sox (document 22481), an informal discussion of computer fears (40571), and commentary on pressure for Kofi Annan’s resignation (109984). Full-corpus document-frequency contrasts also show enrichment of words such as “maybe”, “really”, and “thing” among its active documents. M1 includes concrete reports of sporting events and scientific or technical developments, including Roddick’s Bangkok semifinal qualification (46434), astronaut candidates experiencing zero gravity (69096), and a Samsung phone incorporating a hard drive (20889). Its enriched terms include “Roger Clemens”, “space station”, and “Ferrari”. These exploratory examples suggest a possible stylistic contribution for G15 and heterogeneous subject matter for M1; they do not establish a single semantic attribute for either coordinate.

Alternative encoders: protocols and supplementary comparisons

Training, binary readouts, and reconstruction

This supplementary section expands the comparison in § 6.

All methods use the same 127,600 AG News documents and cached, unnormalized 1,024-dimensional Qwen3-Embedding-0.6B vectors. Encoder fitting uses the full corpus without labels. The ISBC encoder and its saved codes are reused from the full-corpus experiment; the other two autoencoders are fitted on this same corpus.

The ISBC model has target activity six; its hard code thresholds probabilities at 0.5. HypotheSAEs has 16 units and top-$k$ activity six. Its continuous features are nonnegative sparse activations; its binary features indicate their positive support. Training uses 100 epochs, batch size 512, learning rate $5 \times10^{-4}$, and auxiliary-loss coefficient $1/32$, with the multi-top-$k$ loss disabled. BinarySentEmb uses the released linear autoencoder with input and output dimensions adapted to 1,024 and a 16-bit representation. Training uses 20 epochs, effective document batch size 128, learning rate $10^{-5}$, weight decay $10^{-5}$, and similarity-preservation coefficient 0.8, following the learning-rate and coefficient settings in § 4.2 of Shen et al. [45]. Its similarity objective receives seeded random AG News document pairs. No activity target is imposed on this method. These settings adapt the published compressor to our corpus and representation target. PCA and ITQ are also fitted without labels on the complete corpus. PCA retains sixteen centered scores without whitening; ITQ rotates these scores before thresholding and has no fixed activity target. We use sixteen PCA components for ITQ so that both ITQ and ISBC produce sixteen-bit codes. The ISBC target of six active coordinates controls sparsity within that code; using six PCA components for ITQ would instead produce a six-bit code. The comparison therefore matches code width, while observed activity is reported separately.

For each top-6 adaptation, we activate the six coordinates with the largest signed scores and set the others to zero: encoder logits for BinarySentEmb and rotated PCA scores for ITQ. Exact ties are resolved by coordinate index, with the smaller index selected first. This changes only the binary readout; the fitted encoder or rotation remains fixed. The ISBC code is unchanged and retains its learned, variable activity around the target of six.

We first compare reconstruction on the common fitting corpus (Table 6). Refitting an affine decoder to each fixed hard code provides a common measure of recoverable embedding variation, alongside reconstruction without decoder refitting. With refitted affine decoders, native ITQ reaches $R^2 = 0.172$, compared with 0.161 for ISBC, with mean activity 7.98 and 5.94, respectively. Applying top-6 to the same ITQ rotation reduces $R^2$ to 0.152. Thus, ISBC reconstructs more variation at similar mean activity in this comparison, while unconstrained ITQ reconstructs more with a higher average activity. ISBC uses a soft activity target; the top-6 readout fixes the count for every document.

For ITQ, let $P \in\mathbb{R}^{16 \times D}$ contain the fitted PCA axes as rows, $\mu$ be the PCA mean, and $R \in\mathbb{R}^{16 \times16}$ be the learned orthogonal rotation. For a column-vector code $B_i$, we also evaluate the fixed inverse reconstruction

This reverses the rotation and PCA projection using signed unit-valued codes, with no fitted amplitude or intercept correction. The same map is used for the top-6 code. Its $R^2$ is 0.034 for native ITQ and 0.030 for ITQ top-6, compared with 0.172 and 0.152 after affine refitting. The inverse fixes amplitudes at $-1/+1$ in the unwhitened PCA score units, whereas OLS estimates reconstruction coefficients from the corpus. The refitted comparison therefore measures the variation recoverable by an affine decoder from each hard code.

The native HypotheSAEs decoder reaches $R^2 = 0.216$ using continuous activation magnitudes. With affine decoders refitted to the hard codes, the ISBC representation reaches $R^2 = 0.161$, compared with 0.141 for HypotheSAEs and zero for BinarySentEmb (Table 6). The latter code permits only a constant reconstruction. These results distinguish the reconstruction fidelity attainable from continuous activations from that attainable from their binary support.

For BinarySentEmb top-6, refitting an affine decoder gives $R^2 = 0.046$, confirming that the adapted code carries some embedding variation. Passing that code to the original frozen decoder instead gives negative $R^2$: the decoder fitted to zero-threshold codes does not reconstruct effectively from the altered readout.

Table 7 summarizes how the methods use the binary code space. HypotheSAEs has fewer observed profiles than the ISBC encoder but a larger effective profile count, reflecting a more even distribution across its profiles. BinarySentEmb has eight permanently active and eight permanently inactive bits, giving one observed profile.

For BinarySentEmb, top-6 selection produces 20 profiles, but their effective count is only 6.1 and the largest contains 30.92% of documents. The eight coordinates with negative logits remain inactive throughout the corpus; the six active coordinates are selected among the other eight. Thus, fixed activity removes the single-profile collapse while leaving a concentrated representation.

Original ITQ produces more observed and effective profiles than ISBC, whereas ITQ top-6 produces fewer. Fixing activity at six limits the code space to $\binom{16}{6} = 8{,}008$ possible profiles; a soft activity target also allows other activity counts. Profile entropy therefore reflects both the admissible code space and its empirical use. Conditional bit entropy relative to the embedding clusters is reported separately in Supplementary Section S3.3.

Document-based matching across autoencoder readouts

We apply the document-based matching procedure in Supplementary Section S6.2 to the four binary readouts of three autoencoders, using each as reference and starting from up to its ten most frequent profiles. Matching uses no category labels. Category coherence uses the same Jensen–Shannon definition with pairwise averaging over the four readouts. Table S19 reports initial coverage, final intersections, and retained fractions. The original BinarySentEmb code places no restriction on membership, whereas its top-6 variant adds a third nonconstant partition.

Reference	Profiles	Coverage (%)	Final documents	Retained (%)	Category coherence	Chain null
ISBC	10	8.06	295	2.87	0.732	0.608
HypotheSAEs	10	2.49	301	9.49	0.758	0.603
BinarySentEmb top-6	10	99.54	271	0.21	0.708	0.582
BinarySentEmb	1	100.00	45	0.04	0.685	0.650

The ten most frequent BinarySentEmb top-6 profiles cover 99.54% of the corpus, compared with 8.06% for ISBC and 2.49% for HypotheSAEs. Matching retains only 0.21% of the top-6 reference documents in the final intersections. The corresponding fractions from ISBC and HypotheSAEs are 2.87% and 9.49%. These differences reflect the coarse top-6 partition and the different boundaries of the three nonconstant codes. Category coherence exceeds the chain-null mean in each direction, indicating shared category composition despite limited exact overlap. The coherence calculation includes the original BinarySentEmb profile, whose composition is the corpus-wide category distribution.

Supplementary predictive evaluation of alternative encoders

We evaluate category prediction after unsupervised fitting, using the common stratified split of 102,080 training and 25,520 test documents. Test documents participate in representation fitting, but their labels are held out from classifier training. The complete raw embeddings and sixteen continuous PCA scores provide predictive references. Continuous compressed probes use ISBC log-odds, HypotheSAEs activation magnitudes, and BinarySentEmb pre-threshold logits; both BinarySentEmb binary readouts share the same continuous features.

All downstream classifiers, including the raw-embedding reference, are refitted using common settings without additional feature scaling. Logistic regression follows Supplementary Section S1.1. Additional probes among the autoencoder readouts use class-balanced elastic-net logistic regression (SAGA, $C = 1$, L1 ratio 0.5, tolerance $10^{-4}$, and at most 5,000 iterations) and a class-balanced classification forest (200 trees, Gini splitting, square-root predictor subsampling, unrestricted depth, and minimum leaf size one). Both use seed 1234. Balanced-accuracy intervals use 2,000 shared class-stratified bootstrap resamples of test documents. They quantify test-sample uncertainty conditional on the fitted encoders and classifiers.

ISBC has the highest observed binary BA in this comparison and retains approximately 94% of the raw-embedding BA. Original ITQ is close, while imposing exactly six active coordinates on its fitted rotation reduces BA from 84.82% to 82.58%. These estimates describe the evaluated fits; the bootstrap intervals do not measure variation across encoder initializations. Continuous PCA and HypotheSAEs exceed ISBC’s continuous features, but ISBC loses less predictive performance when reduced to hard activations. Equal coordinate count provides a dimensional reference; sixteen continuous values and sixteen bits have different storage capacities.

Representation	BA [95% CI]	Macro-F1	Relative BA
ISBC, binary	85.62 [85.20, 86.06]	85.57	93.97
ITQ, top-6	82.58 [82.14, 83.03]	82.53	90.64
ITQ, binary	84.82 [84.36, 85.26]	84.79	93.10
HypotheSAEs, support	75.37 [74.89, 75.89]	75.10	82.72
BinarySentEmb, top-6	50.40 [49.81, 50.94]	49.17	55.32
BinarySentEmb, binary	25.00 [25.00, 25.00]	10.00	27.44
PCA, continuous	88.24 [87.83, 88.62]	88.22	96.85
ISBC, log-odds	86.17 [85.76, 86.59]	86.12	94.58
HypotheSAEs, continuous	87.99 [87.60, 88.37]	87.97	96.58
BinarySentEmb, logits	80.58 [80.12, 81.06]	80.46	88.44
Raw Qwen	91.11 [90.76, 91.45]	91.10	100.00

caption: tbl:S21: Linear and binary references on Qwen AG News. All representations have sixteen coordinates. BA and macro-F1 are test-set percentages; mean activity applies to binary codes only. Affine reconstruction is evaluated on the full unsupervised fitting corpus. For continuous PCA, this is the cumulative explained variance. Intervals condition on the fitted representation and classifier.

:::

BinarySentEmb’s logits support label prediction, but zero-thresholding maps all documents to one code. Selecting the six largest logits restores some predictive utility, although its binary BA remains below both HypotheSAEs and ISBC. The ordering ISBC, HypotheSAEs support, BinarySentEmb top-6, and BinarySentEmb zero-threshold code is unchanged with elastic-net logistic regression and random forests. These results concern the present adaptation and training configuration.

Figure S20 in Supplementary Section S8 describes the separation at the selection boundary through the document-level gap $\Delta_i = z_{i,(6)} - z_{i,(7)}$. Its median is 3.027 and its mean is 3.716 logit units; the observed range is approximately 0.00003–22.657. No document has an exact tie at this boundary. The gap therefore characterizes how clearly the sixth coordinate is selected within each document. Supplementary Section S8 shows the distributions of the continuous values both by within-document rank and by fixed coordinate for each encoder.

Linear references at sixteen coordinates

All references use the same 127,600 raw Qwen embeddings. PCA is fitted to centered embeddings without labels, using randomized SVD with sixteen components, 32 oversampling directions, seven power iterations, and seed 1234 ([17]). The continuous probe receives the sixteen component scores without whitening or additional scaling. The classifier uses the same stratified 102,080/25,520 document split and class-balanced logistic regression as Supplementary Section S7.3. All added probes converged. The confidence intervals use the same 2,000 class-stratified bootstrap draws as the supplementary comparison.

PCA-sign thresholds component scores at zero. ITQ alternates binary assignments and an orthogonal Procrustes rotation for fifty iterations, initialized with seed 1251, before thresholding the rotated scores ([12]). The random-projection control uses sixteen Gaussian directions orthonormalized by QR decomposition (seed 1234), applied to centered embeddings and thresholded at zero. PCA top-6, ITQ top-6, and random top-6 activate the six largest signed scores for each document; ITQ top-6 uses the same fitted rotation as the original ITQ readout. These controls match the activity count exactly, whereas PCA-sign, ITQ, and random sign codes activate approximately half their coordinates. The original ITQ codes are reproduced exactly for all 127,600 documents when recovering its rotated scores. The other saved binary reference codes are reused, and all logistic probes are refitted.

Continuous activation distributions of the alternative encoders

We examine continuous values in two complementary ways: by their rank within each document, and by following each fixed coordinate across documents. The first view describes the values surrounding a rank-based selection boundary; the second describes where the values of each individual bit concentrate.

Within-document ranks

For each document, we order the sixteen continuous values from largest to smallest. Each curve describes one rank across all 127,600 documents; its coordinate identity can differ between documents. For ISBC and BinarySentEmb, the upper panels show ranks one through six separately, and the lower panels show ranks seven through sixteen. For HypotheSAEs, a single panel shows ranks one through six; the remaining ranks are identically zero and are omitted. Shading has opacity 0.20 to make overlapping curves visible. Paired panels share a horizontal scale. ISBC uses log-odds of the stored probabilities, clipped to $[10^{-7}, 1 - 10^{-7}]$ in single precision; HypotheSAEs uses nonnegative sparse activations, and BinarySentEmb uses encoder logits.

Gaussian density estimates use Scott bandwidths and reflection at finite support boundaries. ISBC clipping endpoints are represented separately as probability masses; HypotheSAEs densities describe the positive continuous activations. Each continuous curve integrates to the remaining probability, so a rank concentrated at an endpoint has little or no continuous density. The rank distributions describe the values available before binary readout: coordinate identities, rather than ranks alone, determine the resulting binary profiles.

Fixed bit coordinates

Figure S21, Figure S22, and Figure S23 retain coordinate identity across documents. Each curve follows the same bit throughout the corpus. All sixteen coordinate densities are shown together on common axes for each encoder. Legends retain the original bit indices, ordered by decreasing mean continuous value (mean stored log-odds for ISBC). ISBC is shown on the original probability scale with all values included. HypotheSAEs retains a separate display of its zero masses, and BinarySentEmb is shown on the logit scale.

HypotheSAEs combines a mass at zero with a distribution of positive magnitudes for each coordinate. BinarySentEmb has eight coordinates that remain positive and eight that remain negative; its top-6 readout selects among the positive coordinates according to their document-specific ordering.

Figure S21 shows the complete distributions on the original probability scale, using all saved activation probabilities, including values at or near zero and one. Each bit has two concentrations, corresponding to predominantly inactive and active states.

Cross-corpus comparisons and the Mixed extension

Code use and conditional comparisons across corpora

These diagnostics expand § 7, using the separately fitted Qwen encoders with $K = 16$ and $A = 6$.

Table 8 describes the complete binary profiles. Mean activity ranges from 5.94 to 6.75 bits, while profile concentration varies substantially. The document groups defined by the ten most frequent profiles contain 58.74% of PHEME documents but only 4.14% of GoEmotions comments. The

effective profile count $2^{\hat{H}}$ follows the definition in § 4.5. Both observed and effective counts depend on corpus size and composition; they summarize code use within each corpus.

Table 9 uses arithmetic-normalized AMI (Vinh et al. [50]), as defined in Supplementary § 18.2, on all eligible documents for each target. Class-balanced logistic probes use the saved 80/20 label splits and class-stratified bootstrap intervals for BA. These intervals condition on the fitted encoder and classifier. BA has random-prediction reference $1/C$ when all $C$ classes occur in the test set; the encoder’s access to the full unlabelled corpus makes this evaluation transductive.

Agreement between partitions and prediction from individual bits also describe different properties. For 20 Newsgroups, AMI is higher for the 20 original categories than for their five-group aggregation (0.357 versus 0.285), while BA is higher for the coarser target (80.70% versus 49.08%). In FakeNewsNet, collection identity is predictable despite low full-partition AMI (0.020), because prediction can combine many different profiles into the same category. The remaining scores describe the specific annotation tasks in each corpus; differences between tasks are not estimates of an intrinsic capacity of binary coordinates to encode particular semantic attributes.

The profile diagrams in Supplementary § 14 make these differences inspectable. Each shows the ten most frequent complete profiles, selected without labels, and reports their corpus coverage. In Airline, the largest document group defined by a profile contains 1,315 tweets, of which 99.77% concern Delta, but their sentiment is distributed across negative (27.53%), neutral (37.95%), and positive (34.52%) labels (Figure S7). This profile groups tweets by carrier while spanning sentiment categories.

Climate Migration offers a complementary example. The same code predicts migration relevance with 71.39% BA and use of quantitative methods with 81.48% BA; discipline and sentiment reach 61.60% and 42.76%, respectively. Its most frequent profile is shared by 542 abstracts: 97.23% are coded as using quantitative methods, while 62.92% concern migration. The second-most frequent profile is shared by 387 abstracts, 98.97% about migration but 94.57% coded as not using quantitative

methods (Figure S12). These compositions show that the profiles distinguish methodological as well as thematic attributes. Driver and hazard combination tasks are reported separately in Supplementary Section S2.9, where their sparse categorical encoding is explicit. The evidence across corpora supports evaluating a binary representation against several available classifications: each reveals a different aspect of the learned structure.

Single-bit profile comparisons show how these attributes vary within a fixed configuration of the other coordinates. Applying the same minimum of 100 documents per profile as in the AG News comparison gives 23 eligible pairs in Airline, 20 in 20 Newsgroups, three in Climate Migration, and seven in PHEME. Table 10 reports selected contrasts. The Airline bit 01, Newsgroups, and PHEME examples maximize the smaller group size within each corpus, without using labels. The additional Airline bit 14 and Climate Migration examples were selected after examining their annotation contrasts. All are exploratory descriptions of the fitted codes.

In Airline, the bit 01 pair remains almost exclusively about Southwest (99.54% and 99.87%), while negative sentiment is more frequent when the bit is active. The bit 14 pair is similarly concentrated on United (99.81% and 100%) but exhibits a larger sentiment contrast. Restricting each comparison to its dominant carrier preserves the contrast. Moreover, negative sentiment is higher on the active side in all six eligible bit 14 pairs, with differences of 10.30–34.89 percentage points, and in all four eligible bit 01 pairs, with differences of 17.59–22.75 points. Thus, the association recurs across several configurations of the same fitted code.

In 20 Newsgroups, computing remains the dominant broad topic in both profiles (98.41% and 99.53%), while the share of comp.windows.x increases and that of comp.graphics decreases from 29.97% to 11.32%. In Climate Migration, the higher migration-relevance share accompanies an almost unchanged quantitative-methods share (58.26% and 57.83%). The PHEME pair consists entirely of Sydney siege tweets, but differs in rumour prevalence. These comparisons show that a coordinate can distinguish content or annotations within a broader topic, entity, or event. Moderate corpus-wide prediction of an attribute can therefore coexist with informative contrasts in particular configurations. Profiles provide the conditioning contexts for these comparisons; interpretation concerns the varying bit in relation to the other coordinates and the documents they select.

Mixed reference-corpus profiles and annotations

The following diagnostics supplement Section 8, using its fixed Qwen representation and all 111,000 documents.

The resulting codes have 6.07 active bits on average and form 15,495 observed profiles, with empirical entropy 11.689 bits and an effective profile count of 3,301.4. Table S22 evaluates source identity and document type using the hard codes. As in Section 7, AMI describes agreement between complete-profile and label partitions, while a class-balanced logistic probe predicts labels from individual bits. Each target uses a seeded 80%/20% document split; representation learning has access to all documents without labels. Source and type are related by construction: for example, the Climate Migration documents are abstracts and the 20 Newsgroups documents are discussion posts. Their association with the codes therefore describes overlapping aspects of the mixture.

Target	Classes	AMI	BA [95% CI]	Macro-F1
Source collection	9	0.236	71.35 [70.79, 71.90]	67.43
Document type	5	0.160	78.18 [77.62, 78.76]	73.45

For inspection, we retain the ten most frequent complete profiles, each covering at least 0.45% of the corpus. These profiles were selected without labels. We use their previously frozen definitions and document membership. Their document groups together contain 12,739 documents (11.48%). Table 11 reports their source composition and selected source-specific annotations. The source and document-type Sankey diagrams appear in Figure S13 in the supplementary dataset descriptions. All percentages below use complete profile membership, rather than the 500-document inspection samples saved for each profile. These samples are drawn without replacement with Python’s random.Random.sample from members in the saved corpus-row order. Each profile receives a seed derived from the first 64 bits of a SHA-256 digest of the global seed 1234, profile rank, and active-bit indices. The manifest records the exact seed, input checksums, and sampled document IDs for each profile.

For nine of these ten frequent profiles, a single source accounts for 97.32–99.68% of the corresponding document group. Several also distinguish attributes within a source. The document group defined by P03 mainly contains computing discussions: 1,577 of its 1,921 20 Newsgroups posts belong to the five computing groups. The four document groups dominated by FakeNewsNet contain almost exclusively GossipCop items, with supplied real-label shares ranging from 54.13% to 86.50%. Thus, the frequent profiles reflect both collection membership and variation within collections.

Two pairs illustrate the conditional interpretation of individual coordinates. P01 and P07 differ only in bit 11, which is inactive in P01 and active in P07. Among their Airline documents, negative sentiment accounts for 2,611 of 3,401 (76.77%) and 257 of 556 (46.22%), respectively. Carrier composition also differs, so we examine the contrast separately within each airline (Table S23). The negative-sentiment share is lower in P07 for all six carriers. This consistent direction shows开户链接 that the aggregate contrast is not explained solely by different carrier proportions, although some carrier-specific groups are small. P02 and P09 differ only in bit 00, active in P02: among their Climate Migration documents, migration relevance accounts for 1,664 of 1,900 (87.58%) and 304 of 512 (59.38%). These comparisons hold the other fifteen coordinates fixed and describe different attributes within source-specific contexts. They use the previously selected frequent profiles, with annotations examined subsequently.

P05 provides a case of thematic organization across sources. Its 785 documents comprise 500 AG News items (63.69%), 151 Newsgroups posts (19.24%), 77 TweetEval tweets (9.81%), 48 IRA tweets (6.11%), eight GoEmotions comments (1.02%), and one FakeNewsNet title (0.13%). Among the AG News items, 494 of 500 are labelled Sports; all 151 Newsgroups posts belong to baseball (89) or hockey (62). These independently supplied topic annotations identify sports content within the same binary profile across news and discussion posts. The document group defined by this profile also includes a TweetEval tweet about a cricketer missing a match because of injury (Mixed document 37042) and a GoEmotions comment contrasting playoff and regular-season performance (89685). These illustrative documents extend the reading of the profile to sources whose labels concern affect or evaluative language rather than sports.

Additional documents encoded with the frozen Mixed model

We apply the frozen Mixed encoder to 6,000 additional documents from the same source collections. We sample 500 documents from each of the eight non-TweetEval sources and 500 from each of the four TweetEval components, without replacement, using global seed 20261030. Each component uses a separate SHA-256-derived seed to permute its candidate rows; we retain the first 500 eligible documents after the exclusions below. We exclude texts present in the fitting corpus after normalized-text matching, duplicate texts across the new sample, and identical model inputs after tokenization and truncation to 256 tokens. The current AG News source pool supplies the new news sample; the original mixture retains its earlier extraction. The encoder, threshold, and preprocessing are unchanged, and no classifier is fitted for this analysis. Re-encoding 64 reference documents reproduces all their saved binary coordinates. For exploratory reading, 20 of the 500 new documents per component were sampled without replacement using separate component-specific seeds derived from the same global seed. Their 240 IDs were saved before encoding the new documents. The quoted texts illustrate selected content contrasts; they are not used to estimate their prevalence. Sampling scripts, seed derivations, input order, and selected IDs are retained with the analysis artifacts.

Previously observed complete profiles cover 5,675 of the 6,000 new documents (94.6%). The new sample contains 2,915 distinct profiles, including 309 absent from the fitting corpus. Mean activity is 6.16 bits. Table S24 reports coverage separately for each sampling component. Across the 192 component–bit combinations, the mean absolute difference in activation frequency between the reference and new samples is 1.34 percentage points. This comparison conditions on the source component, since the mixture proportions differ between samples. Profile coverage describes access to reference documents with the same code; interpretation additionally requires examining their content and annotations.

Sampling component	New documents	Mean active bits	Observed reference profile (%)
20 Newsgroups	500	6.26	94.4
AG News	500	5.82	95.6
Airline	500	6.32	96.6
Climate Migration	500	6.81	99.0
FakeNewsNet	500	5.74	97.0
GoEmotions	500	5.78	94.4
IRA	500	6.04	90.8
PHEME	500	6.21	96.0
TweetEval: emotion	500	5.99	92.8
TweetEval: hate	500	6.40	95.0
TweetEval: offensive	500	6.59	92.8
TweetEval: sentiment	500	5.93	90.6

The previously described conditional bit interpretations retain their direction. Among new Airline documents, the negative-sentiment share is lower in P07 than P01, which differ only in bit 11. For Climate Migration, the migration-relevance share remains higher in P02 than P09, which differ only in bit 00. To examine these associations beyond complete profiles, we also construct three-bit contexts from the coordinates active in both members of each pair. Using only reference-corpus annotations, we choose the triplet with the highest source share among those covering at least 25% of that source. This post-hoc description uses annotations; representation learning does not. It selects bits {0, 10, 13} for Airline and {1, 12, 15} for Climate Migration.

The conditional contrasts persist when the remaining coordinates are unrestricted (Table 12). Among all new documents activating the Airline context, 398 of 434 come from Airline; the Climate Migration context contains 457 such abstracts among 550 documents. For the new documents, the active-minus-inactive difference is −28.4 percentage points for negative sentiment (95% interval [−39.6, −16.1]) and 18.4 points for migration relevance ([7.6, 30.0]). We combine the Wilson limits for each proportion using Newcombe’s score-based interval without continuity correction [33]. These intervals use an independent-document binomial approximation, conditional on the frozen encoder, selected contexts, and observed group sizes, without a finite-population correction. They describe sampling uncertainty within the specified groups. These associations provide a reference for reading a new text. For example, an Airline message praising a flight crew (new_airline_0477) receives P07 and activates bit 11, whereas a complaint about delays and poor communication (new_airline_0340) receives P01 with that bit inactive. P07 also contains a complaint about a credit (new_airline_0169), illustrating the variability within the same code. The interpretation concerns a conditional association with sentiment rather than a deterministic sentiment label.

The sports reading of P05 also extends to new documents. Its 38 new members comprise 28 AG News items labelled Sports, two Newsgroups hockey posts, seven TweetEval tweets, and one PHEME tweet. Inspection of the eight tweets identifies sports content, including a TweetEval message about David Wright returning to the Mets lineup (new_tweeteval_sentiment_0443) and PHEME commentary on Kobe Bryant’s recent performance (new_pheme_0161). Both activate bits {5, 6, 7, 9, 11, 13}, with the remaining coordinates inactive, linking them to the sports documents examined in the fitting corpus.

PHEME: evaluation and interpretation diagnostics

Predictive models and full evaluation

These results expand § 9, using the same frozen Qwen representation and engagement split. The ten specifications per outcome use bits alone; bits with raw follower count and text length; each of the three annotation sets with controls, with and without bits; and all annotation sets jointly with controls, again with and without bits. Every specification except bits only and the constant reference includes the two controls. Categorical annotations enter as indicator variables, and columns constant in the training set are excluded.

OLS and RF predict the standardized log outcome $z_i$ defined in § 9. Predictors retain their original scale. The constant reference predicts the training mean of $z_i$, which is zero. RF uses 1,000 trees, variance splitting, a minimum node size of ten, and all available predictors considered at each split. We retain these settings across specifications without tuning on test performance. Seeds are $1234 + 100o + j$, where $o$ indexes retweets then favorites and $j$ indexes bits with controls, bits only, veracity, veracity with bits, rumour status, rumour status with bits, event, event with bits, all labels, and all labels with bits, starting at one.

Both estimators are evaluated by test MSE. We use 2,000 shared document-bootstrap resamples of the 1,161 test tweets for individual score intervals and paired intervals for loss differences: each model versus the constant reference, RF versus OLS, adding bits to each annotation set, and adding controls to bits. These are pointwise 95% intervals conditional on the fitted models and split, without adjustment for multiple comparisons.

Bits-only RF improves over the constant reference for both outcomes: the paired MSE differences are −0.056 with interval [−0.105, −0.005] for retweets and −0.096 with interval [−0.149, −0.044] for favorites. Its differences from bits-only OLS are small, with intervals [−0.017, 0.021] and [−0.027, 0.025], respectively. Adding controls reduces error for both estimators and outcomes, with all four paired intervals below zero.

For retweets, adding bits to veracity or rumour status improves both estimators, whereas adding them to event identity gives small differences whose intervals include zero. With all annotations jointly, adding bits improves RF, while the OLS difference remains small. For favorites, adding bits improves every annotation-and-control specification for both estimators, with all eight paired intervals below zero. RF also improves on OLS in every favorite specification containing bits and controls; for the corresponding retweet comparisons, all paired intervals include zero. These results locate the clearer nonlinear predictive gain in models with additional covariates, rather than in the bits-only specification.

Content inspection and category compositions

We examine the content associated with bit 07 within the Sydney event, where both activation states are sufficiently represented. Within each state, training tweet IDs are sorted in ascending order and

12 tweets are sampled without replacement using pandas sample(n=12, random_state=20261030). The seed is reset for each state, and selection does not use engagement. Active-bit texts repeatedly report the hostage situation, police activity, and developments at the cafe. Inactive-bit texts include prayers, expressions of solidarity, and debate about religion and community relations, including hostile reactions. For example, an active-bit tweet reports police surrounding the cafe (ID 544274879055675392); inactive-bit examples describe an interfaith show of solidarity (ID 544476492281827328) and call for prayer and peace (ID 544331094523662337). These readings identify a conditional content distinction within a single event.

To describe this contrast across the selected document sets, we count two groups of expressions identified from the training-text reading: incident terms (hostage, hostages, gunman, police, cafe, and café) and religion or public-response terms (Muslim, Muslims, Islam, Islamophobia, solidarity, illridewithyou, and words beginning with racis or pray). Matching is case-insensitive; a leading hashtag symbol does not prevent a match. These are descriptive lexical indicators. The contrast persists in the test documents (Table 14), with the other fifteen coordinates unrestricted.

Within Sydney, active bit 07 is associated with lower training mean counts. The lower median favorite count persists in the test subset, whereas the mean reverses direction. One active-bit test tweet expressing concern for the hostages has 69,825 favorites, illustrating both overlap between the content groups and the influence of the upper tail. The content contrast therefore provides an empirical interpretation of a predictive coordinate, while its engagement association depends on the outcome summary and sample.

The single-bit comparison in Table 10 provides a conditional view of bit 07. Holding the other fifteen coordinates fixed selects two groups entirely associated with the Sydney siege; rumour prevalence is higher in the group with bit 07 active. This connects the coordinate to a distinction within event identity. It does not establish that rumour status explains the engagement association, since the profile contrast and the tree split select different document groups.

For each bit $k$, Figure S25 reports the complete category distribution for each of the three taxonomies, using all 5,802 documents. Specifically, each cell estimates $P(L^{(t)} = c \mid B_k = 1)$, where $L^{(t)}$ is event identity, rumour status, or veracity, and $c$ is one of its categories. The denominator is the number of documents activating bit $k$; probabilities sum to one separately within each taxonomy. Comparison with the unconditional corpus frequencies indicates which categories are more or less prevalent among the selected documents. Together, the three distributions describe each bit’s relationship to the existing annotations without assigning it a single category.

For example, bit 06, which contributes the largest split-improvement share in the bits-only log-favorite tree, selects documents about Ferguson (50.71%), Charlie Hebdo (23.96%), Germanwings (20.81%), Ottawa (2.93%), and Sydney (1.60%). Of these documents, 65.04% are non-rumours, 9.80% true rumours, 7.54% false rumours, and 17.61% unverified rumours. Bit 08 spans Sydney (51.25%), Ottawa (36.78%), and Charlie Hebdo (11.30%), with smaller shares from the other events. Its rumour share is 48.71%, compared with 33.99% in the corpus; true rumours account for 32.80% of all documents activating this bit. These distributions connect the predictive coordinates to several dimensions of the annotations simultaneously.

The full set of bits also reveals different combinations of the recorded categories. Active bit 05 selects Charlie Hebdo (63.83%) and Ottawa (36.13%), whereas active bit 11 selects Charlie Hebdo (79.19%) and Germanwings (20.46%). For bit 04, the largest event shares are Ferguson (42.59%), Sydney (34.79%), and Germanwings (17.51%). The figure retains every category, including smaller shares, so that this interpretation reflects the composition of each selected document set. Predictive use and category composition together guide which documents to examine and which content differences to investigate.

Engagement tails and conditional content

Selection and reading

This exploratory analysis complements § 9.3 using all 5,802 documents and the saved Qwen codes, without refitting the encoder or a predictive model. For each outcome $Y$, event $e$, and $q \in\{0.10, 0.25\}$, the lower group contains documents with $Y_i \le Q_e(q)$ and the upper group those with $Y_i \ge Q_e(1-q)$, where $Q_e$ is the empirical quantile with linear interpolation. All boundary ties are retained, so group sizes need not be equal. The thresholds are computed within the entire event, before selecting a bit condition. Corpus-wide thresholds are used separately for the cross-outcome overlap calculation below. No documents are sampled to balance group sizes.

The descriptive search covers all sixteen bits, their 120 pairs and 560 triples, and observed complete profiles. Conditional comparisons hold zero, one, or two specified bits active while varying another bit, leaving the remaining coordinates unrestricted. The examples were selected after examining these contrasts and their content; the reported differences are not tests of prespecified hypotheses or additional held-out predictive results. The 10% and 25% analyses describe different degrees of outcome selectivity; consistency across them is not an inclusion criterion.

Keyword counts use all documents in each selected group. Text is lowercased and HTML-decoded; URLs, user mentions, English stopwords, and platform tokens are removed. Hashtag words are retained. We count document presence of unigrams and bigrams formed after stopword removal, using alphabetic tokens of at least three characters, with internal hyphens and underscores allowed. Terms are compared by differences in document prevalence. The displayed contrasts select interpretable examples with at least five occurrences on the enriched side; all observed terms remain in the analysis outputs.

For the three contexts in Figure S26, the initial reading dossier samples up to twenty documents per decile-tail and bit-state cell. IDs are sorted before sampling without replacement; the seed is the integer encoded by the first four bytes of the SHA-256 digest of the saved group name, in big-endian order. All 286 Charlie Hebdo documents in the two decile tails of the $\{05, 11\}$ context were subsequently read, including both states of bit 02. Together with the 160 sampled reading rows for the other two contexts, this gives 446 rows and 437 distinct documents. The full-corpus Charlie Hebdo complement analysis separately samples up to twelve documents from each of the eight $(B_5, B_{11}, B_2)$ cells, again after sorting IDs, with seed $20261100 + 2(2B_5 + B_{11}) + B_2$. Its small cells are read in full, giving 83 reading rows. Additional follow-up reading covered all fifty Ottawa favorite-decile documents with bit 14 inactive and the seven Germanwings lower-retweet-decile documents with bit 02 active. These reading sets can overlap. Quartile keyword statistics use every selected document; the quartile groups were not read exhaustively. Supplementary § 14.1.1 records the request types and the retained dossier instruction. Qualitative descriptions are not estimates of theme prevalence from these reading samples.

The conjunction and its complements in the full corpus

Table S26 exhaustively partitions the corpus by bits 05 and 11. Their conjunction selects only Charlie Hebdo, but 603 additional Charlie Hebdo documents have other states of this pair. The pair is therefore a selective event-associated condition, rather than an exact recoding of the event annotation. To interpret the complements independently of their differing event composition, we recompute keyword frequencies and read documents within Charlie Hebdo alone.

Within the (1,1) group, Table S27 shows a contrast between expressive content and incident reporting across the full sample, not only its engagement tails. Documents with bit 02 active include defenses of expression through words rather than bullets (ID 553118384540643328) and appeals

to stand together for freedom (ID 552832738542514176). Inactive-bit examples report the death toll (ID 552802128688078848) and the pursuit of suspects (ID 553184390260338688). These themes overlap: an upper-tail document with bit 02 inactive describes a tribute with pencils and silence (ID 553205372719472640). The coordinate does not provide a categorical annotation of emotion or solidarity.

Within Charlie Hebdo, the (0, 1) group ($n = 312$) includes discussion of religion, condemnation, and collective blame. For example, ID 552814410763603968 condemns the attack and expresses concern for its victims, while ID 553136990787342336 questions unequal attribution of collective responsibility. In its bit 02-active subset, Islam occurs in 42/197 documents and Muslims in 45/197. The (0, 0) group ($n = 269$) instead prominently describes the later hostage incidents: hostage occurs in 94 documents, kosher in 70, and supermarket in 45. Examples include reports about the kosher supermarket (ID 553536887349325825) and a second hostage situation in eastern Paris (ID 553540719714271233). The (1, 0) group is small ($n = 22$) and includes reports concerning police officers and suspects, such as ID 553129559173115904; police occurs in 15 of these 22 documents. Its only document with bit 02 active asks readers not to circulate an execution video out of respect for the victim (ID 552805956619628544). These examples characterize observed subsets, not mutually exclusive semantic classes.

Tail contrasts, other events, and outcome overlap

Within the {05, 11} context, the lower and upper favorite deciles contain 127 and 159 documents, of which 21 and 99 activate bit 02. The corresponding quartile counts are 72/310 and 241/399. The more expressive upper-tail reading includes grief, identification with the victims, freedom of expression, and anger or criticism, rather than a uniformly positive tone. Figure S27 shows selected lexical contrasts. For example, jesuischarlie occurs in 9/127 lower-decile documents and 38/159 upper-decile documents, whereas dead occurs in 30/127 and 2/159. Bit 02 also distinguishes favorite tails in the Charlie Hebdo (0, 1) complement: it is active in 6/17 lower-decile and 23/36 upper-decile documents. The small bit 02-active cells in the other complements do not support an equally detailed comparison.

Table S28 shows that the {05, 11} conjunction is not the only group with substantial favorite counts. In the (0, 1) complement, bit 02 also distinguishes the two tails, within a different content context; the other complements contain very few bit 02-active documents.

$(B_5, B_{11})$	$B_2 = 1$, all	Median favorites	Lower 10%	Upper 10%
(0, 0)	10/269	42	3/63 (4.8%)	1/10 (10.0%)
(0, 1)	197/312	85	6/17 (35.3%)	23/36 (63.9%)
(1, 0)	1/22	43.5	0/4 (0.0%)	0/3 (0.0%)
(1, 1)	739/1476	88	21/127 (16.5%)	99/159 (62.3%)

The Sydney favorite contrast agrees with the incident-reporting versus public-response distinction examined in § 9. In its upper decile, illridewithyou occurs in 19/123 documents and thoughts in 15/123, compared with zero occurrences of either term in the 125 lower-decile documents.

In Ottawa, upper-decile favorites similarly include condolences and remembrance: thoughts appears in 22/89 documents versus 0/93 below, while parliament appears in 7/89 versus 30/93. Bit 14 is less prevalent in the upper favorite tail. Reading also identifies unrelated entertainment and sports posts in the Ottawa collection, which remain in the analysis; event metadata do not guarantee topical homogeneity. Ferguson differs by outcome: bit 07 is more prevalent among high-retweet documents and less prevalent among high-favorite documents. Retweet contrasts include police force and demilitarization, while the favorite reading also includes racial and moral evaluations. These are contextual descriptions, not invariant meanings of bit 07.

The two outcomes select overlapping but distinct documents. Under corpus-wide decile thresholds, each upper tail contains 582 tweets, with 361 shared (62.0% of either group). At quartile thresholds, 1,026 are shared between upper groups of 1,454 retweet and 1,455 favorite documents, approximately 70.5% of each. In the decile retweet-only group, bit 07 is active in 59.3%, compared with 38.0% in the favorite-only group and 38.5% in their intersection. This overlap analysis follows, rather than replaces, the separate outcome comparisons.

Concentration of profiles and similarity of texts

Profile entropy and embedding similarity describe different aspects of group homogeneity. Table S30 reports empirical profile entropy and mean cosine similarity across all distinct pairs of Qwen embeddings within each tail of the three active-bit combinations. For the Charlie Hebdo combination {05, 11, 02}, the upper decile has a more concentrated profile distribution, but its mean embedding similarity is slightly lower. High engagement therefore does not require a single, uniformly homogeneous text group. The entropy values are descriptive plug-in estimates for unequal group sizes, not sample-size-adjusted comparisons.

Outcome	Active bits	Tail (%)	$n_L$	$n_U$	$H_L$	$H_U$	$\mathrm{Cosine}_L$	$\mathrm{Cosine}_U$
Retweets	{06, 07}	10	113	124	3.668	3.291	0.455	0.474
Retweets	{06, 07}	25	273	302	3.631	3.484	0.452	0.470
Favorites	{13, 07}	10	176	100	3.211	3.006	0.471	0.461
Favorites	{13, 07}	25	435	265	3.277	3.120	0.464	0.451
Favorites	{05, 11, 02}	10	21	99	0.792	0.244	0.556	0.535
Favorites	{05, 11, 02}	25	72	241	0.884	0.490	0.558	0.536

Content similarity alone does not determine engagement. Two tweets with the same wording about thoughts for French cartoonists, their families, and loved ones have 37,983 and one favorite, respectively (IDs 552844593046097920 and 553149163181838336). Both activate bits 05, 11, and 02; their recorded follower counts are 7,457 and 42. This example illustrates why the tail descriptions cannot isolate a content effect from account reach, timing, or other circumstances of circulation.

Regression trees and importance diagnostics

Regression trees describe conditional groups through bit activations and other covariates. For each outcome, we fit three trees on the standardized log outcome: bits only, bits with controls, and bits with controls and all labels. Each tree uses squared-error splitting, maximum depth eight, minimum split and leaf sizes 30 and 15, complexity parameter 0.0005, and no cross-validation pruning. For readability, the figures display the first three split levels, collapsing deeper branches into their parent groups. The reported importance uses all splits of the fitted trees.

In the bits-only retweet tree (Figure S28), activating bit 13 is associated with a lower mean log outcome. Within its active branch, bit 11 activation identifies a further decrease. For favorites, bit 02 activation is associated with a higher mean log outcome; within either activation state of bit 02, bit 06 activation identifies a lower mean. Each comparison concerns the document group selected by the preceding path. These descriptive training partitions are separate from the held-out OLS and RF evaluation.

Figures S29–S32 show the corresponding trees with additional covariates. The retweet tree with controls first splits on follower count; the favorite tree first splits on bit 02. Both trees including labels first separate the Germanwings event. Thus, the hierarchy of selected coordinates depends on which covariates are available, as well as on the outcome.

Figure S33 allocates each full tree’s realized reduction in training residual sum of squares to the variables used in its actual splits. Shares sum to 100% separately for each tree; an omitted variable has no selected split. The calculation excludes surrogate splits. These shares measure contributions to the fitted partition’s training error reduction, not the direction of an association or the proportion of total outcome variance explained.

Figure S34 and Figure S35 report RF permutation importance for all ten specifications. For each tree in a forest, a predictor is shuffled among training observations omitted from that tree’s bootstrap sample, called out-of-bag observations. Importance is the resulting increase in standardized log-outcome MSE, averaged across trees. This diagnostic does not use the 1,161 test tweets. Positive values indicate worse prediction after shuffling; negative values indicate improved prediction after shuffling, not an association with lower engagement.

Bit 11 has the largest permutation importance in the bits-only retweet forest, whereas bit 02 has the largest in the favorite forest. The individual bits-only trees allocate their largest split-improvement shares to bit 11 and bit 06, respectively. The two diagnostics therefore need not rank coordinates identically. Correlated bits and labels can share information, and shuffling breaks their dependence. The figures retain all included predictors and negative importance values; they describe the fitted models without confidence intervals or causal interpretation.