WorksComputer Science
LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits
Abstract
Intersectional bias arises when a model treats people who hold several marginalised identities at once differently from what their separate identities would predict. Auditing such intersectional subgroups remains a challenge: their number grows quadratically, and there is no agreed baseline for what independent attributes would predict. Without such a baseline, many existing audits flag a subgroup when it exceeds each attribute alone, a condition that attributes acting independently already satisfy. We test this standard criterion on 150 synthetic datasets built with no interaction. With the shuffle null, it flags 66.1 false subgroups per dataset on average, and at least one in every dataset. When we plant interactions of known size into these datasets, a repaired null lets the criterion recover only 32% of them. We introduce ADDIS (Additive Intersectional Screen), which fits an additive model over all attributes, scores each subgroup by its Additive Residual Distance (ARD) from that model's prediction, and tests it against a Freedman-Lane null. On our synthetic benchmark, ADDIS flags 0.09 false subgroups per dataset and recovers 84% of the same planted interactions. We then build IndiPersona, 10,000 synthetic Indian personas with six identity attributes, and audit 13 open language models. The standard criterion flags 7.1% of the 738 tested subgroups on average; ADDIS flags 3.0%. Gender identity crossed with sexual orientation is the largest pairing in every model, and counterfactual personas confirm it and additionally reveal a caste x class interaction in all but one model that the observational test lacked the data to detect. Much of what audits report as intersectional bias can thus arise from independent single-attribute effects alone, yet a small core of interactions survives.
CCS Concepts: • Computing methodologies → Natural language processing; • Social and professional topics → User characteristics; • Mathematics of computing → Hypothesis testing and confidence interval computation.
Introduction
Language models are increasingly deployed in systems that judge people, from ranking job applicants to answering a patient’s medical questions [3, 61]. Audits report that responses depend on identity, and that some combinations are treated differently from either identity alone [1, 22, 48, 61]. Most audits nevertheless examine one identity at a time. Intersectionality theory makes a specific claim about people who hold several identities at once: the harm they face is not reducible to the harms attached to each identity on its own, or even to their sum. Crenshaw writes that the intersectional experience is “greater than the sum of racism and sexism” [9 p. 140], so the quantity to be exceeded is the sum of the parts. We operationalize this claim by asking whether a model’s response to a combination of identities departs from what its separate identities predict.
Authors’ Contact Information: George Rahul, International Institute of Information Technology Hyderabad, Hyderabad, India; Ronit Jahilal, International Institute of Information Technology Hyderabad, Hyderabad, India; Bhaskara Hanuma Vedula, International Institute of Information Technology Hyderabad, Hyderabad, India, [email protected]; Ponnurangam Kumaraguru, International Institute of Information Technology Hyderabad, Hyderabad, India; Pushkar Shukla, Cloudphysician.ai, USA.
Consider a model that attributes anxiety and depression to personas, and rates a woman three points higher on anxiety and a Muslim four points higher on depression. A Muslim woman then lies five points from the reference, farther than under either identity alone (Figure 1a). Yet, it is exactly where the two effects add, so no interaction is involved. Only a departure from this sum, such as the 1.5 points in Figure 1b, indicates an interaction, so exceeding each part is not equivalent to exceeding the sum of the parts.

Figure 1. The single-attribute criterion cannot tell addition from interaction. Relative to a Hindu man, woman adds 3 points of attributed anxiety and Muslim 4 of depression. Points beyond the dashed arc are flagged, and red marks ARD from their sum (open circle). (a) Without interaction, the Muslim woman is still flagged. (b, c) The same interaction raises the excess more along the sum than at a right angle. (d) If Muslim lowers anxiety instead, the shifts nearly cancel and a real interaction stays inside the arc.
Detecting such interactions at scale is challenging, because candidate subgroups multiply quadratically with the number of identity attributes while the observations available for each decrease [25, 29]. Testing hundreds of subgroups simultaneously also inflates false discoveries unless the error rate is controlled across all of them. Automated subgroup discovery makes this search systematic [29, 70], and the baseline against which it ranks subgroups determines which ones can be found. A common baseline in intersectional audits of language models, which we term the single-attribute criterion, reports a combination when it fares worse than each of its attributes alone [22, 48, 61]. As the example shows, merely additive effects satisfy this criterion. Conversely, it can miss real interactions when the two effects point in opposite directions, or when the interaction is perpendicular to their sum.
We address this problem with Addis (Additive Intersectional Screen), which screens every pair of identity attributes for interactions in how language models treat people. We replace the single-attribute baseline with an additive prediction. We fit a linear regression over the whole dataset, giving every level of every attribute its own coefficient. It has no interaction terms. A subgroup’s prediction is the sum of the coefficients of the attributes it carries. We define the additive residual distance (ARD) as the distance between a subgroup’s observed mean response profile and this prediction. It is equal for interactions of equal magnitude in any direction. Figure 1 contrasts the two baselines on our example. We test each subgroup against a null that keeps every attribute’s effect and removes only the interaction effect, so the test asks whether two identities interact, not whether identity matters at all. We control the family-wise error rate at 5% across the search (§ 3.3).
In a natural sample of data, the attributes are correlated, so a pair’s effect cannot be separated from its parts’, and some subgroups are too small to study. So, we generate the personas we test, and sample their attributes independently of one another. The exception to this is caste, which is paired only with religions it actually occurs in. The dataset, INDIPERSONA, is ten thousand Indian personas crossing six attributes: gender identity, age, sexual orientation, socioeconomic status, caste and religion, each described in one of 50 everyday scenes. Fourteen models answer the same 82 questionnaire items about every persona. We test all 738 pairs of identity values with at least 50 personas, and use within-persona counterfactuals for combinations the main sample cannot estimate.
We test both baselines on data we build ourselves. It contains no interaction, so every subgroup a method flags in it is a false positive. We build it from the real design and the models’ own answers, keeping each attribute’s own effect and destroying only intersectional effects. Across 150 such datasets, each with 738 subgroups, Addis flags 0.09 false subgroups per dataset on continuous answers. The single-attribute baseline flags 66.1, and fires on every dataset. We also synthetically add real intersectional effects into the dataset, of sizes we choose. Addis recovers 84% of them, against 32% for the single-attribute baseline given the same null. On answers rounded to the Likert scale, rounding itself creates interactions and raises every method’s error rate. We ran every other published method through the same test: per-item interaction regressions [31], MAIHDA [14] and PERMANOVA [2]. Each raises a false flag in at least 73% of datasets, compared to our 20%. With interactions planted, the precision of Addis is 0.72, compared to 0.02–0.28 for the alternatives.
After running Addis on the 13 open models, an average of only 3.0% of subgroups are found to be intersectional on scale scores. By contrast, given the same statistical treatment, the single-attribute baseline flags 7.1% (52 subgroups per model), of which only about nine coincide with ours. Gender identity crossed with sexual orientation is the single largest pairing, in every model we audit. Within-persona counterfactuals reproduce that effect in all 13 open models, and find a caste class interaction in 12 of them, a result the observational test could not find due to lack of data. The location of these interactions holds up when we reword the prompt, draw a fresh sample of personas, or adjust for the template, though their direction depends on the wording. Ard measures interaction in what models attribute to people, not harm, and our data come from one country, so we read the flagged combinations as candidates for closer scrutiny, not as estimates of how much discrimination exists in the world. Our contributions are as follows.
A metric for intersectional subgroup discovery. We compare each subgroup against the sum of its attributes’ separate effects, and test the difference against a null built for it.
A benchmark for intersectional metrics. We build synthetic data with controllable interactions between the subgroups, so that the ground truth is known. On questionnaire-style answers, rounding weakens the error control of every method, but every published alternative we tried makes false positives far more often than ours.
A factorial persona dataset and an audit of fourteen models. Ten thousand Indian personas whose identity attributes vary nearly independently, an audit of thirteen open models and one closed model, and the finding that only a small minority of subgroups survive a calibrated test, concentrated on a single pairing of attributes.
Related Work
Intersectional bias in language models. Bias in language models is often strongest where identities meet, in embeddings, generated text, question answering and hiring decisions [1, 21, 31, 34, 38, 46, 68]. Fairness for each group does not imply fairness for their combinations [8, 29]. Most of this work uses Western categories such as race and gender [21, 46], while social hierarchies and the biases attached to them differ across countries, languages and regions [6, 26, 42, 51, 55].
Bias evaluation in India. Work on India began with word embeddings, which encode caste, religious and gender associations in Hindi and other Indian languages [4, 32, 40]. Studies of Hindi then found that detecting gender bias needs community input [23] and that models are more biased when generating Hindi than English [27]. Stereotype benchmarks for language models followed, covering Indian axes of disparity [5], caste and religion [30, 65], several social dimensions [54], community-sourced stereotypes [12], Indian languages [62] and many identity groups [45]. Benchmarks of this kind test whether a model repeats known associations, so they depend on curated inventories whose validity is often unclear [7], and they miss harms outside them. Later work therefore studied open-ended generation [11, 18, 50, 56] and decisions. Decision audits hold a task fixed and change only the identity, in hiring [52], economic guidance [44], matchmaking [43] and education [22]. Because only the identity changes, they measure how a model treats a person rather than which words it associates, and our audit follows the same design.
How intersections are tested. Few of these audits test intersections systematically. Stereotype benchmarks include a few predefined pairs, mostly with gender [54, 62], and decision audits change one attribute at a time [44] or a few within one task [22, 43], as do audits outside India with race, gender and age [1, 3, 24, 59]. When an audit compares a combination with its parts, it usually asks whether the combination fares worse than each part alone [22, 34, 48, 60, 61], the single-attribute baseline that § 3.2 shows addition alone can satisfy. Other audits fit an interaction term for one outcome at a time [3, 31]. Recent work widens the set of combinations, by pairing multi-attribute instances with their single-attribute sources [71] or by flagging failures that appear only when two identity words are swapped together [57], but it fits no additive baseline across all attributes and does not control false positives across cells. Two studies come closest to our critique. Kanubala and Valera [28] show that intersectional fairness metrics for classifiers respond more to additive bias than to interaction, and Rennard and Xypolopoulos [53] compare simulated personas with an additive prediction and find that models represent intersectional identities in only one or two dimensions. Searching over many subgroups is also established for classifiers, through subgroup fairness audits [29] and scan statistics [70]. Table 1 compares these methods with ours. Unlike this work, we test every pairwise cell of a multivariate answer profile against an additive baseline fitted over all six attributes, control the family-wise error rate across cells, and confirm the main cells with counterfactuals.
| Method | India | Treatment | All cells | Additive | Profile | Cell | FWER | Planted | Counterf. |
| Stereotype benchmarks, India [54, 62] | ✓ | ✗ | ✗ | ✗ | ✗ | – | ✗ | ✗ | ✗ |
| Decision audits, India [43, 44, 52] | ✓ | ✓ | ✗ | ✗ | ✗ | – | ✗ | ✗ | ∼ |
| Single-attribute baseline [22, 34, 48, 61] | ∼ | ✓ | ∼ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Per-item interaction regression [3, 31] | ✗ | ✓ | ∼ | ∼ | ✗ | ✓ | ∼ | ✗ | ✗ |
| MAIHDA [14] | ✗ | – | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| PERMANOVA [2] | ✗ | – | ∼ | ∼ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Multi-attribute pairing (CoMFair) [71] | ✗ | ✓ | ∼ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Metamorphic testing (HInter) [57] | ✗ | ✓ | ∼ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| Subgroup search for classifiers [29, 70] | ✗ | – | ✓ | ✗ | ✗ | ✓ | ∼ | ✗ | ✗ |
| Addis (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 1. Addis combines properties that prior methods address separately. India: Indian identity axes. Treatment: only identity varies. All cells: every pairwise cell. Additive: additive baseline over all attributes. Profile: answers tested jointly. Cell: names the interacting cell. FWER: error controlled across cells. Planted: validated on planted interactions. Counterf.: confirmed by counterfactuals. ✓ yes, ✗ no, ∼ partly, – n/a.
Methodology
Addis searches for intersectional subgroups in how a language model treats people. It needs two parts. A baseline predicts where a subgroup should sit if its attributes do not interact. A null tells how far a subgroup can stray from the baseline by chance. Section 3.1 states the task. Section 3.2 shows that addition alone satisfies the single-attribute baseline of current audits, and § 3.3 replaces it with an additive prediction. Sections 3.4 and 3.5 add error control and a benchmark with known truth. § 3.6 describes IndiPersona, our dataset of personas, and § 3.7 describes its counterfactual quartets. Figure 2 shows the full pipeline.
Figure 2. The audit pipeline. IndiPersona personas (§3.6) are answered item by item by each model. An additive model fits the answers (§3.3), and a Freedman–Lane null controls the family-wise error rate across all 738 cells (§3.4).
The Audit Task
Notation. Each of personas holds one value of each of six identity attributes. A model rates persona on one questionnaire item at a time. The answers form the persona’s attributed profile , either its item answers () or scale scores computed from them (§ 3.4). For a value , let be the mean profile of the personas that hold , the population’s mean profile, and the shift of . Two values and of different attributes define a cell , the set of personas that hold both, with mean profile . We test all 738 cells with , denoted .
Target. If and do not interact, the cell sits at the additive prediction , and its interaction is its departure from that prediction, . § 3.3 refines the additive prediction to account for the other four attributes. We call a cell intersectional when , thus treating intersectionality as an interaction on the additive scale [63], a working definition rather than the theory’s own [9].
The audit. An audit returns a set of flagged cells . It should flag the cells with while keeping the family-wise error rate (FWER), the probability of flagging any cell with , at 5% or lower. The flagged set depends on the baseline that the audit compares each cell with.
The Single-Attribute Baseline Is Satisfied by Addition
Current audits measure how far a subgroup sits from a reference, such as the population’s mean or the most privileged group. They call a cell intersectional when it sits farther from this reference than each of its two values [22, 60]. With the population’s mean as the reference, the rule reads:
We call this rule the single-attribute baseline, and the left side of Equation (1) its excess. The rule asks whether a combination fares worse than its parts. A positive excess can also arise by chance. Audits that test their intersectional claims ask whether a subgroup differs from its comparison group at all. They use permutation tests [60], -tests [22], rank tests [48] or tests against an unperturbed prompt [61]. All of these tests share one null hypothesis: identity has no effect. We test the excess against the permutation version of this null, which we call the shuffle null. It shuffles the identity labels across personas. Each persona keeps its answers but receives another persona’s identities. We then recompute the excess on each shuffled dataset. Shuffling cuts the link between every identity and the answers, so shuffled data contain no main effects and no interaction. The shuffle null therefore asks whether identity matters at all, not whether two values interact. Real answers do contain main effects, so the excess of a real cell easily beats its shuffled values, even when the two values only add. The test then reports interactions that are not there. § 5.1 shows how often. The excess has a second problem, even before any test. It is positive for many cells in which nothing interacts. Suppose a cell sits exactly at its additive prediction, so that .
Proposition 3.1. If , then if and only if .
We prove Proposition 3.1 in Appendix A. The sign of the excess depends on the angle between and , not on any interaction. Take two shifts of equal length. The excess is then positive whenever the angle between them is less than . The single-attribute baseline flags such cells even when nothing interacts (Figure 1(a)). When a real interaction exists, the verdict also depends on which way the interaction points (Figure 1(b, Figure 1c)). When the shifts point apart at more than , the excess turns negative, and a small real interaction stays hidden (Figure 1(d)). The shuffle null fixes neither failure. A cell flagged by the single-attribute baseline may hold only two ordinary biases added together. A cell it misses may hold a real interaction. We therefore replace the baseline itself.
Ard: Distance From the Additive Prediction
Ard compares each cell with the position that addition alone predicts. To get this position, we fit a linear regression, the additive model, to the attributed profiles of all personas:
Here is the predicted profile of persona and is a constant. Each is the least-squares effect of one of the persona’s six values, with its own coefficient for each value. For religion, for example, is for a Christian persona and for a Hindu one.
The additive model has no interaction terms. Any effect of a pair of values beyond the sum of their single effects therefore stays in the residual .
Ard, the additive residual distance , is the length of the cell’s mean residual, the red segment in Figure 1:
Take a simple case. The personas differ only in the two attributes of the cell, and every value of one attribute appears equally often with every value of the other (for example, every gender with every orientation). Then equals the length of the interaction from §3.1, the standard interaction contrast [63]:
If , is zero up to random noise. Otherwise it grows with , whichever way the cell moves (Figure 1b,Figure 1c). Unlike the excess, does not depend on the angle between and . Our personas differ in all six attributes, so the additive model includes all six. If a cell happened to hold more poor personas than usual, a model of its two attributes alone would mistake the shift of being poor for an interaction. The full model removes that shift, and stays correct when cells differ in size (Appendix § 11).
Error Control Across Many Sparse Cells
We test 738 cells at once, and their sizes differ more than twentyfold. Two problems follow. Small cells have larger variance, so a single threshold on raw would be set by the small cells. And with hundreds of tests, some cell will exceed any fixed threshold by chance. We therefore need a null that asks the right question, a score comparable across cells, and a rule that controls the error rate over all cells jointly.
Hypotheses. For each cell , we ask whether the two values interact. The null hypothesis states that the cell sits exactly at its additive prediction, while each value may still shift the answers on its own by any amount, and the alternative states that the cell sits elsewhere. This differs from the shuffle null of § 3.2, whose null hypothesis is that identity has no effect at all.
Procedure. The test has four steps.
Build null datasets. Following Freedman and Lane [17] and Winkler et al. [69], we keep each persona’s fitted part , flip the sign of its residual at random, and add the two back together. The main effects are untouched, while any interaction, which lives in the residuals of its cell, is cancelled by the flips, and each persona’s noise keeps its size (Appendix § 13). We call this the Freedman–Lane null.
Recompute the metric. On each of 2,000 null datasets, we refit the additive model and recompute for every cell.
Standardize. We convert each cell’s observed into , where and are the mean and standard deviation of the observed pooled with its 2,000 null values, which removes the dependence on cell size.
Apply the max- rule. In each null dataset, we record the largest over all cells, and a cell is significant if its observed exceeds 95% of these maxima.
Addis is Ard together with the Freedman–Lane null and the max- rule.
Why this controls the error rate. The threshold is the 95th percentile of the largest over all cells, so under the chance that any cell exceeds it, the FWER, is designed to be at most 5%. This single-step rule approximates the min- procedure of Westfall and Young [67] and rests on two assumptions, namely that residuals of different personas are independent and that each is as likely positive as negative (Appendix § 13). We therefore check the FWER empirically. It stays near 5% on continuous answers and on discrete Likert noise, but not when an additive signal is rounded to a few answer categories, because rounding creates real interactions on the answer scale, which every method detects (§ 6).
Tested cells. Small intersectional groups make fairness hard to verify statistically [15, 25], which is why we standardize each cell, require , and control the error rate across all of . The 47 caste × socioeconomic cells below this minimum are not tested (Appendix § 11), and the counterfactual quartets of § 3.7 test one caste × class interaction directly.
Representations. A representation turns a model’s answers about persona into the profile on which we compute , and we use two. In the item space, each question’s answer becomes one dimension, z-scored across personas, and profiles are compared with the distance of Cronbach and Gleser [10]. Items to which a model gives the same answer for every persona carry no information about identity and are dropped, so ranges from 66 to 82 depending on the model. In scale scores, each of the dimensions is one trait scored with the standard key of its questionnaire, namely the five Big Five traits, need for cognition, empathic concern, anxiety and depression (Appendix § 19 reports each scale’s internal consistency per model). Scale scores carry less signal, so Addis flags fewer cells on them, and a finding that holds in both representations does not depend on how the answers are summarized.
A Benchmark With Known Truth
Real answers have no ground truth. When a method flags a cell, we cannot tell whether the flag is right. To compare methods, we need data in which we know which cells interact. Our benchmark builds such data from real answers with the plasmode approach [16, 64]. The plasmode approach starts from real data, keeps their structure, and plants a known truth.
We build every dataset three times, once from the answers of each of three models from different families (Llama 3 8B, Gemma 4 E4B, Ministral 8B). Each model thus contributes equally. Our main noise generator, group-mean, rebuilds each persona’s answer from group averages, the grand mean plus the shift of each of its six values, and adds back its own leftover with a random sign (Figure 3). It never fits the additive model (Equation (2)), which makes it an impartial check of Addis. Two further generators start from that model’s fit . Flip flips the signs of the residuals at random, as in the Freedman–Lane null, and permute shuffles them across personas. Every generated dataset has real main effects and no interaction. We then plant interactions by shifting every persona in a few chosen cells by a known amount, measured in residual standard deviations. In some datasets we plant nothing.

Figure 3. The group-mean generator keeps every main effect and removes every interaction. Real answers of Llama 3 8B on the anxiety scale, in standard deviations. Each persona’s synthetic answer is the grand mean plus the shift of each of its six values, plus its own leftover with a random sign.
When nothing is planted, every flag is a false positive, so we measure each method’s FWER. Answers rounded to the Likert scale are the one exception (Appendix § 12). When interactions are planted, we know which cells a method should flag, so we also measure its recall and precision. We follow the ADEMP structure for simulation studies [41].
A scenario fixes how the answers are built and where interactions are planted. We report four scenarios, each testing a different risk: answers rounded to the Likert scale, an interaction in a single item, several interactions in one attribute pair, and the real gender identity orientation pattern (Appendix § 14).
The benchmark tests whether a method can find planted interactions. To find real interactions in language models, we need a dataset of personas.
The IndiPersona Dataset
IndiPersona is a set of synthetic Indian personas that combine six identity attributes: gender identity, age, sexual orientation, socioeconomic status, religion, and caste and community. Table 5 in Appendix § 15 lists the values of each attribute. Each cell needs a large to estimate . We therefore assign the attributes independently of one another, except that religion follows caste. This trades realism for estimability. Real populations are correlated, so the personas show how models treat each combination of identity labels, not how bias falls on India’s population.
| Attribute | Values | |
| Gender identity | 3 | man, woman, transgender person |
| Age | 5 | child, teenager, young adult, adult, senior citizen |
| Sexual orientation | 4 | heterosexual, gay, lesbian, queer |
| Socioeconomic status | 6 | poor working class, lower middle class, middle class, upper middle class, rich, ultra rich |
| Religion | 6 | Buddhist, Christian, Hindu, Jain, Muslim, Sikh |
| Caste and community | 31 | Varna: Brahmin, Kshatriya, Vaishya, Shudra; Jati: Nair, Ezhava, Reddy, Kamma, Kapu, Naidu, Lingayat, Vokkaliga, Maratha, Patel, Jat, Yadav, Dalit; Muslim: Ashraf, Ajlaf, Arzal, Sunni, Shia; Sikh: Jat Sikh, Khatri, Mazhabi; Christian: Catholic, Protestant; Administrative: OC, OBC, SC, ST |
Table 5. Identity attributes in IndiPersona and their values.
Personas. We generate 10,000 personas. Gender, age, sexual orientation, socioeconomic status and caste are each assigned by quota, independently of one another, so every value appears equally often. Religion is then chosen among the religions in which the persona’s caste occurs [20, 49]. This rule avoids implausible combinations such as a Brahmin Muslim. Our caste list is a simplified working representation of caste, not a complete one. Assigning attributes independently also produces combinations that are contradictory, such as a man described as lesbian, or sensitive, such as a child with a sexual orientation. We keep them so that every cell can be estimated, but a flag in such a cell may reflect the model’s reaction to the contradiction rather than an intersectional effect. None of the three main cells (§ 6.3) is such a combination.
Templates. Each persona appears in a short scene, such as a queue at a public office or a ferry landing (Figure 5 in Appendix § 16). We use 50 templates, so that no single scene’s wording drives the answers. Each has a fixed opening and one sentence per attribute. Gender comes first, and the other five attributes follow in a random order for each persona. A few templates add details that could hint at some traits, so we do not use those templates with the questionnaires they could affect (Appendix § 16). Templates are assigned at random, independently of identity.

Figure 5. A prompt from IndiPersona. The six identity values are highlighted, and the fixed opening of the template is in italics. Gender comes first, and the other five sentences appear in a per-persona random order. Symptom items use the GAD-7 and PHQ-8 wording and the 0–3 scale instead.
Questions and answer order. The model rates each persona on 82 items from five questionnaires, one item per prompt (Appendix § 16, Table 6). Every item is rewritten in the third person, so that the model rates a described person instead of reporting on itself. “I am the life of the party” becomes “This person is the life of the party”. Answer options always appear in ascending order.
| Questionnaire | Items | Answer scale (ascending) |
| IPIP Big Five markers [19] | 50 | 1–5, strongly disagree to strongly agree |
| IPIP Need for Cognition | 8 | 1–5, strongly disagree to strongly agree |
| IPIP Empathic Concern | 9 | 1–5, strongly disagree to strongly agree |
| GAD-7 anxiety [58] | 7 | 0–3, not at all to nearly every day |
| PHQ-8 depression [33] | 8 | 0–3, not at all to nearly every day |
Table 6. The five questionnaires, in the order of their answer scales.
Control arms. IndiPersona also holds three control arms for our robustness checks: a fresh sample of personas, two rewordings of every prompt, and personas without identity attributes (Appendix § 20).
Counterfactual Quartets
ADDIS shows which cells of the main personas interact, but not why. Each cell mixes the effect of its two labels with the other four values of its personas. Counterfactual quartets remove this mixing. We take 250 base personas per contrast and write each one four times, in a 2 × 2 design that changes two labels and nothing else. For example, one base persona is written as a rich Brahmin, a rich Dalit, a poor working class Brahmin and a poor working class Dalit, with every other detail the same. The interaction is the difference between two changes: the move from rich to poor working class for the Dalit version, minus the same move for the Brahmin version. The rest of the persona stays fixed, so this within-persona difference-in-differences estimates the causal interaction of the two labels in the prompt [13]. Quartets also help where the main personas are too sparse. With 31 castes, the cell Dalit × poor working class has of only about 50 to 80. ADDIS cannot detect most interactions in a cell this small, but the quartet above tests the two labels directly. We use four contrasts: the main gender identity × orientation interaction, a check that this interaction is specific to the transgender label, a caste interaction, and a calibration contrast where we expect little interaction (Appendix § 21).
Experimental Setup
We run Addis on 14 models and check four things in § 5. We check whether Addis keeps its FWER near 5% on the real design (Appendix § 17.§ 1), and whether the interactions it finds are caused by the identity labels in the prompt rather than by the other values of the personas in a cell (§ 6.4). We also check whether its findings survive changes to the prompt, the sample and the representation (Appendix § 17.§ 4), and how it compares with the published alternatives (§ 6 and Appendix § 17.§ 5).
Models. We audit 13 open instruct models from the Gemma, Llama, Ministral, Phi and Qwen families, with up to 24B parameters, and DeepSeek V4.1 Flash (all 14 are listed in Figure 6). Each model answers at temperature 0, with thinking off, and returns only a number. Because a model whose answers barely change with the persona cannot show bias, we analyze a model only if it passes a persona-sensitivity check, which all 14 pass (Appendix § 22).
Figure 6. The single-attribute baseline’s excess is positive for most cells, while ADDIS flags few. Real answers, 738 cells per model.
Published alternatives. We compare our test with three kinds of published methods, each run as published with its own null, correction and decision rule. The single-attribute baseline (Equation (1)) is run in two forms, flagging every cell with a positive excess or only cells whose excess is significant under the shuffle null (§ 3.2); in § 5.1 we also pair the excess with our Freedman–Lane null to separate the effect of the null from that of the statistic. Tests of departure from addition test for an interaction, as our test does. Per-item regression fits an interaction term for each item, as in LLM audits [31], with a Bonferroni correction across items and cells, and we run it with classical standard errors, as published, and with HC3 standard errors [39], which allow the noise to differ between cells. MAIHDA, a standard intersectional method in social epidemiology, adds a random intercept for every subgroup to the additive model [14]. PERMANOVA, a standard permutation test for multivariate data, tests the whole interaction term of each attribute pair [2], so it cannot say which cell in the pair interacts. Metamorphic testing with HInter reports a hidden failure when swapping two identity words changes the answer but swapping either word alone does not [57]; because our counterfactual quartets have the same structure, we apply HInter’s rule to every item of the quartets (Appendix § 17.§ 5).
Results and Discussion
Our results support three claims. First, Addis keeps its FWER near 5% on continuous answers. The single-attribute baseline has a high error rate, because it flags cells whose attributes only add (§§§ 5.1–§ 6). Second, few cells pass Addis on real answers. In every model, the flagged cells concentrate on gender identity sexual orientation (§ 6.3). Third, changing the labels reproduces the gender identity orientation interaction, but the direction of the interaction depends on the prompt (§§§ 6.3–§ 6.4). Appendix § 17.§ 5 compares the published alternatives on real answers.
The Single-Attribute Baseline on Synthetic Data
We first test the single-attribute baseline on synthetic data. § 5.1.1 measures its false positive rate and the role of its null. § 5.1.2 measures its recall and precision, and how both depend on the direction of the interaction.
Where the Single-Attribute Baseline’s False Positives Come From
We use the 500 datasets with no interaction from Appendix § 17.§ 1. These datasets are built from the real answers of Llama 3 8B with the group-mean generator. As Proposition 3.1 predicts, the excess (Equation (1)) is positive for 89.5% of cells in the item space and for 75.5% on scale scores. The excess is therefore a biased measure of interaction. With no interaction present, its expected value is positive, not zero. To correct this bias, the null distribution must carry the same positive bias. The shuffle null does not, because it removes the main effects that make the excess positive. On the benchmark, the excess with the shuffle null flags at least one cell in every dataset, and 66.1 cells per dataset on average (Table 2).
| Statistic | Null | Flagged | Cells | Recall | Prec. |
| Excess (Eq. 1) | Shuffle | 100% | 66.1 | 47.9% | 0.06 |
| Excess (Eq. 1) | Freedman–Lane | 2.7% | 0.03 | 32.4% | 0.99 |
| Ard () | Freedman–Lane | 8.7% | 0.09 | 84.4% | 0.99 |
Table 2. The Freedman–Lane null removes the single-attribute baseline’s false positives, but not its misses. Scale scores, group-mean generator. Flagged, Cells: datasets flagged and false cells per dataset, nothing planted. Recall and precision: the real gender identity × orientation pattern at one residual SD.
With the Freedman–Lane null, the result reverses. The Freedman–Lane null keeps every main effect, so the bias appears in the null too and cancels. The excess now flags only 2.7% of datasets instead of all of them. Appendix § 17.§ 2 reports the same comparison on Likert-rounded answers. There, the repaired excess keeps a higher precision than Addis, because it finds few interactions of any kind. Repairing the null fixes the false positive rate (Appendix § 23 tests a second simple repair). It does not change what the excess measures: distance from the population mean, not distance from the additive prediction (Figure 1c,Figure 1d). § 5.1.2 measures how many interactions the repaired excess misses as a result.
Recall by the Direction of the Interaction
Even with the Freedman–Lane null, the excess misses many interactions. On the real gender identity × orientation pattern, the repaired excess recovers 32.4% of the planted cells, against 84.4% for (Table 2). Both keep a precision near 0.99, so only recall separates them. The direction of the planted interaction explains the gap. We plant the same interaction twice in the same cell (Table 3). The first time, it points along the cell’s additive shift from the population mean. The second time, it points at a right angle to that shift. The excess detects the first interaction but misses most of the second. detects both.
| Size | Direction | Excess | ARD |
| 0.4 | Along the shift | 43% | 38% |
| 0.4 | At a right angle | 0% | 53% |
| 0.8 | Along the shift | 93% | 90% |
| 0.8 | At a right angle | 35% | 97% |
Table 3. The excess finds an interaction along the cell’s shift and misses one at a right angle. Recall on scale scores, group-mean generator, Freedman–Lane null. Planted size in residual SDs.
The right-angle case is the one intersectionality theory is about. An interaction at a right angle means that the combination does something neither part does. In a high-dimensional space such as the nine scale scores, most directions are close to perpendicular. The right-angle case is therefore also the most common one.
Published Alternatives on Synthetic Data
We next compare Addis with the published alternatives (§ 4) on the benchmark (§ 3.5). § 6.1 compares their false positive rates. § 6.2 compares their recall and precision when we plant known interactions.
Comparing False Positive Rates Across Methods
We use benchmark datasets with no interaction, built from the answers of three models with the group-mean generator. On continuous answers, Addis flags 8.7% of datasets on scale scores (95% CI 4.7–24.4) and 4.7% in the item space (Table 4a). Per-item regression with HC3 standard errors is the exception. It holds its FWER about as well as Addis. With classical standard errors, as published, per-item regression makes a false positive in every item-space dataset. Appendix § 17.§ 1 reports a finer calibration of Addis.
| (a) Continuous answers | |||||
| Method | Flagged: Scale | Flagged: Item | Recall / prec.: Real pattern | Recall / prec.: Single item | |
| Addis | 8.7% | 4.7% | 84.4% / 0.99 | 53.1% / 0.98 | |
| Per-item HC3 | 9.3% | 3.3% | 75.6% / 0.98 | 97.1% / 0.53 | |
| Per-item classical | 36.0% | 100% | 77.6% / 0.93 | 99.2% / 0.15 | |
| (b) Likert-rounded answers | |||||
| Method | Flagged | Recall / prec. | |||
| Addis | 20.0% | 78.1% / 0.72 | |||
| Per-item HC3 | 99.3% | 58.8% / 0.20 | |||
| Per-item classical | 100% | 85.9% / 0.20 | |||
| MAIHDA | 96.7% | 76.0% / 0.28 | |||
| PERMANOVA | 73.3% | 72.7% / 0.02 |
Table 4. Addis keeps the highest precision in every setting. Flagged: datasets with any flag when nothing is planted. Recall / prec.: interactions planted at one residual SD in (a), two in (b). Per-item regression uses Bonferroni. (a) Group-mean generator; real pattern on scale scores, single item in the item space. (b) Permute generator (Appendix § 12). MAIHDA and PERMANOVA ran here only, on 30 datasets, as they are slow (Appendix § 17.§ 5).
Likert-rounded answers. Real answers are whole numbers on a four-point or five-point scale. One benchmark scenario therefore rounds its continuous answers to these categories (§ 3.5). Rounding a measurement scale is known to create departures from additivity [37, 66]. A few cells therefore depart from the additive prediction even when nothing is planted. In an average dataset, about 3 of the 738 cells depart enough for Addis to detect. Appendix § 12 describes the construction and how we find these cells.
Rounding weakens the error control of every method we test. Addis weakens least. It flags at least one cell in 20.0% of the 150 rounded datasets. Every published alternative we tried exceeds 96%, except PERMANOVA at 73% (Table 4b). On rounded data that copy the real design, the FWER of Addis also rises above 5%. A control shows that rounding, not Addis, causes these extra flags (Appendix § 17.§ 2).
Recall and Precision Across Methods
Addis has high precision and recall in every setting but one (Table 4). It loses recall when a single item carries the whole interaction. Ard measures one distance across all items, so a change in one item is diluted. Per-item regression tests each item separately. It finds more of these single-item cells, but at about half the precision of Addis.
Interactions Found in the Models’ Answers
We now run Addis on the real answers of the 14 models about the IndiPersona personas (§ 3.6).
How many cells interact. Addis flags few cells in every model. Averaged over the 13 open models, it flags 22 of the 738 cells on scale scores (3.0%) and 55 in the item space (7.5%). For DeepSeek V4.1 Flash, it flags 11 and 56 cells (Appendix § 17.§ 3). The single-attribute baseline reports far more cells. Its excess is positive for 75–83% of cells on scale scores. This share is about the same as on synthetic data with no interaction (§ 5.1.1). Under the shuffle null (§ 3.2), the baseline still flags 52 cells per model, more than twice as many as Addis. The two tests also disagree on which cells interact. Only about 9 cells per model are flagged by both. Part of this gap lies out of the excess’s reach. In every model, some cells flagged by Addis have a negative excess, as in Figure 1(d). No threshold on the excess can flag these cells. In DeepSeek, one such cell is transgender person senior citizen, its second-strongest interaction in the item space.
Which cells interact. On scale scores, gender identity crossed with sexual orientation is the largest pairing in every model (Appendix § 17.§ 3). The 12 cells of this pairing are under 2% of the tested cells but carry a third of all flags. These flags are not independent findings, because one strong cell shifts the residuals of its row and column (Appendix § 24). Three of these cells interact in nearly every model. Transgender person queer and transgender person heterosexual interact in all 13 open models in both representations. Man queer interacts in 12, and DeepSeek shows all three. We call these three cells the main cells. In the item space, each main cell sits two thirds to one residual standard deviation from its additive prediction. To check that the main cells are not an accident of one sample, we rerun five of the models on a fresh sample of 5,000 personas drawn the same way (§ 3.6). All three main cells are flagged again in all five models in the item space. On scale scores, the two transgender cells are flagged again in all five models, and man queer in four (Appendix § 17.§ 3, Table 11). Per-item regression and MAIHDA also flag all three main cells in at least 12 of the 13 models (Appendix § 17.§ 5).
| Cell | Item space | Item space | Item space Repl. | Scale scores | Scale scores | Scale scores Repl. |
| Transgender person queer | 0.93 [0.42, 1.70] | 24.3 [7.9, 34.9] | 5/5 | 0.44 [0.19, 0.89] | 17.2 [4.9, 30.0] | 5/5 |
| Transgender person heterosexual | 0.90 [0.56, 2.26] | 25.8 [18.3, 39.2] | 5/5 | 0.31 [0.19, 0.94] | 14.2 [6.8, 33.7] | 5/5 |
| Man queer | 0.66 [0.33, 0.98] | 15.4 [4.0, 22.4] | 5/5 | 0.29 [0.12, 0.60] | 10.1 [2.3, 21.1] | 4/5 |
Table 11. The main cells are large and replicate. Real answers. is in residual standard deviations, the unit of the planted sizes in the benchmark. and are the median [min, max] over the 13 open models. Repl. is the number of the five replication models in which the cell is significant again on the fresh sample of 5,000 personas, out of those in which it is significant in the main sample.
The item space carries more information, so it shows a wider pattern. In the item space, transgender person × poor working class, poor working class × heterosexual and ultra rich × heterosexual also interact in all 13 models. Gender identity × orientation is the largest pairing in 8 of these models. Appendix § 17.§ 3 lists every significant cell.
The direction of the interactions. On scale scores, the departure measured by is a vector with one dimension per trait. Each dimension is the cell’s interaction on that trait. For every model and cell, we can therefore see which traits carry the interaction (Figure 4). The excess compares two lengths and keeps no direction, so the single-attribute baseline cannot show which traits move. Under our main wording, models attribute more extraversion, agreeableness and empathic concern to queer transgender personas than their two labels predict, and less anxiety. The rise in extraversion is significant in all 13 open models. The other two main cells lean the other way, toward less extraversion and more anxiety.
Figure 4. C1 moves the same traits as transgender person × queer, and C3 moves conscientiousness, extraversion and neuroticism. Median shift over 13 open models, in SD of each scale. Below: models significantly positive / negative. The two blocks are standardised separately, so compare signs, not sizes.
In questionnaire points, these shifts are small. A typical model gives a queer transgender persona 0.28 fewer points of anxiety on the GAD-7 (range 0 to 21) than its two labels predict. All 13 models move anxiety the same way. is still large, because the departure spreads over many scales and items at once.
Not all of these results are robust. When every prompt is reworded, the drop in anxiety for queer transgender personas weakens or disappears in most models. Across all flagged cells, only about half reappear on the fresh sample (Appendix § 17.§ 4).
Caste. No caste cell interacts in more than 4 of the 13 open models. Caste cells are too small for ADDIS to measure well. We therefore test caste with counterfactual quartets in § 6.4.
Results From the Counterfactual Quartets
The counterfactual quartets (§ 3.7) change only two labels in each persona and hold everything else fixed. Any interaction in a quartet is therefore caused by those two labels, within the model’s prompts.
We use contrast C4 as the reference. C4 changes adult to senior citizen and middle class to upper middle class. These are two small steps between neighbouring levels, and we do not expect them to interact. The C4 interaction is small but not zero. We use the C4 interaction as a yardstick for the size of the other contrasts.
C1 changes man to transgender person and heterosexual to queer. C1 thus tests the main interaction of § 6.3, a cell ADDIS had already flagged, so it confirms that finding rather than discovering a new one. C1 reproduces this interaction in all 13 open models, with contrast values about five times as large as C4. C1 also points the same way as the audit of the main personas: more extraversion in all 13 models and less anxiety in 10 (Figure 4). C2 makes the same change with woman in place of transgender person. C2 is larger than C4 with 95% bootstrap confidence in only 5 of the 13 models, against all 13 for C1.
C3 changes Brahmin to Dalit and rich to poor working class. The C3 interaction is about as large as the C1 interaction, and larger than C4 in 12 of the 13 models. Models attribute more conscientiousness and less extraversion and neuroticism to a poor working-class Dalit persona than its two labels predict. Anxiety and depression move in different directions in different models. Caste cells showed no interaction in § 6.3 because they are small (§ 3.7), not because caste labels do not interact. Auditors who care about caste should use targeted counterfactuals.
Conclusion
These results suggest three practices for intersectional audits of language models.
Use an additive baseline. Without an additive baseline, an audit mistakes cells whose attributes only add for interactions. The audit also misses real interactions whose direction barely changes the cell’s distance from the population mean.
Use a null that keeps the main effects. The shuffle null removes each attribute’s own effect along with any interaction, so almost any cell can beat it. A null that keeps the main effects, such as the Freedman–Lane null, avoids this problem.
Test small cells with counterfactuals. Attributes with many values, such as caste, give few personas per cell. Counterfactual quartets can test the labels of such cells directly.
None of these practices measure harm. Whether a flagged cell harms people depends on the outcome and the setting. An audit has to judge harm separately.
Limitations and Future Work
Our personas are synthetic, so they show how models treat each combination of identity labels, not how bias falls on India’s population. Whether two values add depends on the scale on which we measure the answers. The share of intersectional cells therefore differs between the item space and scale scores, although the main interactions sit in the same place in both. Flagged cells are non-additive on the answer scale, which can include effects of the answer format near its floor or ceiling, and a flagged pair may also involve a third attribute (Appendix § 17.§ 4). Our test is also not calibrated on Likert-rounded benchmark data, where its FWER is 20.0%, so single flagged cells on Likert answers should be confirmed with an ordinal model. Future work should extend the test from pairs of attributes to combinations of three or more, and apply it to real human answers to the same questionnaires, to compare how people and language models treat the same combinations of identities.
Ethical Considerations and Adverse Impacts
This work involves no human subjects and no personal data, since every persona is generated from templates and every answer comes from a language model. The anxiety and depression items ask a model what a described person would answer, so they are attributions, not screenings, and must not be read as clinical claims about any group. Our caste list is a simplified working representation and may not match how communities describe themselves. The main risk is misreading our results: a flagged cell describes how models treat a combination of labels, not real people, and a cell our test does not flag is not shown to be free of harm, only to hold no interaction large enough to detect. We will release our code and the model answers on GitHub upon publication, as audit evidence, not as descriptions of real groups or as training data.
Manuscript submitted to ACM
Generative AI Usage Statement
We used Claude Opus 5.5 in Claude Code to search the literature for § 2, to prototype code, and to proofread and reword our own sentences. We checked every cited paper, tested all code behind our results, and reviewed every edit. The ideas, analyses and claims are ours, and we take full responsibility for them.
References
Proof of Proposition 3.1
Let and be the shifts of values and from the population’s mean profile. Under exact additivity, the cell shifts by , so its gap is . The gaps of its parts are and .
Proof. Assume, without loss of generality, that , so the larger part has gap . The excess is then positive exactly when . Both sides are non-negative, so we can compare their squares:
This difference is positive exactly when . Because , this is the condition
Two cases follow directly. When the shifts are orthogonal or at an acute angle, , so the excess is always positive although nothing interacts. When the shifts point against each other strongly enough, the excess is negative, so the criterion cannot flag the cell, whatever its interaction.
Why the Regression Uses All Six Attributes
The personas in a cell share two values, but they also carry four other values each. If, by chance, a cell held more poor personas than usual, its mean would also include the shift of being poor. A model of the two cell attributes alone would then mistake that shift for an interaction. The full regression already accounts for each persona’s other values, so only what the two cell values add together remains in the residual. For the same reason, the prediction stays correct when cells differ in size, which they do in our data.
Tested cells and identifiability. The six attributes form 971 pairwise combinations of values. We do not test caste religion, because religion follows caste by design: 148 of its combinations are empty and the other 38 are left out. Of the rest, 47 caste socioeconomic cells have fewer than 50 personas, which leaves 738 tested cells, the same in every model. The additive design has 50 columns but rank 48, because caste is nested in religion, so a few caste and religion coefficients are not identified separately. The fitted values and residuals are still unique, since the fit is a projection, and , the null and every test use only these.
The Likert-Rounded Benchmark Scenario
How the data are built. This scenario is part of the synthetic benchmark of § 3.5. Each dataset starts from the real answers of one of three models (Llama 3 8B, Gemma 4 E4B, Ministral 8B) in the item space and uses the permute noise generator. The answers are first built on a hidden continuous scale, as the sum of the main effects and the noise. They are then rounded back to each item’s answer categories at cut points fixed for the dataset, so they look like real Likert answers. We build 50 datasets with no interaction per model, 150 in all. MAIHDA and PERMANOVA ran on 30 of them. Our test uses 500 null draws in the benchmark. On 45 of these datasets, 2,000 draws gave almost the same result (9 false alarms against 8).
Why rounding creates interactions. Rounding is not additive. Suppose an item’s hidden score is 2.2, the cut point between answers 2 and 3 is 2.5, and each of two values adds 0.2. Either value alone gives 2.4, which still rounds to 2, but both together give 2.6, which rounds to 3. On the answer scale, the two values then interact, although nothing was planted. Some departure of this kind appears in almost every cell (660–724 of the 738). Most of these departures are far too small to detect.
Threshold cells. We call a cell a threshold cell when rounding alone creates an interaction large enough that our test is expected to detect it. To find these cells, we compute the expected rounded answers with no noise and measure each cell’s departure from addition. A cell is a threshold cell when this departure, scaled by our test’s null distribution, gives an expected of at least 2. A dataset has 3.35 threshold cells on average (median 2, range 1–8). The number depends on the model, from 1 for Gemma 4 E4B to 5–8 for Ministral 8B. Every method is scored against the same threshold cells.
How we score. In the primary score, a flag in a threshold cell does not count as false, and our FWER is 20.0%. In the strict score, every flag counts as false, and our FWER is 40.0%. The cutoff of 2 is a choice. With a cutoff of 1, a dataset has 14.1 threshold cells on average, and our FWER falls to 12.7%. Per-item regression stays at 96–99% under either cutoff. The cutoff was fixed in our code before any benchmark run. The pre-registration, however, does not state the cutoff, and it says that every flag on data with no interaction is false. Read literally, it makes the strict 40.0% the pre-registered figure, so we report both.
On the real design. We repeat this check on our real design, for one model from each family. We round each null dataset of Appendix H.1 to the answer grid and run our test. Our test then flags 18–100% of datasets under the strict score and 7.5–56% under the primary score (20 settings, 200 datasets each). A control shows why. Keeping the discrete, clipped noise of the rounded data but replacing its expected values with their additive projection, our test is back at 3.0–6.5% in 13 settings, and every interval contains 5%. The test is therefore calibrated on Likert noise, and the excess comes from real interactions that rounding creates on the answer scale.
How the Sign-Flip Null Works
Why the flips remove only the interaction. The additive model of § 3.3 splits each persona’s answers into a fitted part, the sum of its six values’ shifts, and a residual, everything the model cannot explain. We keep the fitted part unchanged, so every main effect stays in each null dataset. A real interaction cannot be fitted by the additive model, so it stays in the residuals of the personas in its cell, pushing them all the same way. When we flip the sign of each residual at random, about half of these residuals point one way and half the other. They then cancel, and the interaction disappears. The noise of each persona keeps its size, because a flip changes only its sign.
Details. Before flipping, we rescale the residuals with the HC2 leverage correction of MacKinnon and White [39], so that each persona keeps its own noise level. We use the +1 convention for -values [47]. The main analysis uses 2,000 null datasets, the calibration of § 6.1 uses 1,000, and the benchmark uses 500 (Appendix C).
Assumptions. The flips rest on two assumptions: the residuals of different personas are independent, and each residual is as likely to be positive as negative. Personas that share one of our 50 templates may have related residuals. Adding the template to the additive model leaves the results and the calibration nearly unchanged (Appendix H.1). The permute and group-mean generators break both, and our test still holds its error rate on them (§ 6.1).
Benchmark Scenarios
Each scenario of § 3.5 tests a different risk.
Likert-rounded. We build answers on a continuous scale and round them to each item’s answer categories, as in real questionnaires. Rounding itself bends an additive signal, so this is the hardest case for every method (Appendix C).
Single item. The whole interaction sits in one item. This favours per-item tests and shows where our test loses power.
Several in one pair. Several planted cells share one attribute pair, as in the real gender identity × orientation table, where cells in one table share signal.
Real pattern. We plant the gender identity × orientation cells our test flags in real answers, at their real sizes. Because our test chose these cells, they carry the kind of signal it detects well, so this scenario shows realistic structure rather than a fair contest.
Further scenarios. The benchmark also plants one, five or twenty randomly chosen cells, each with its own random direction. Appendix I gives their recall.
Identity Attributes in IndiPersona
Table 5 lists the six identity attributes in IndiPersona and their values (Section 3.6).
Questionnaires and an Example Prompt
Table 6 lists the five questionnaires and their answer scales. Agreement items run from 1 (strongly disagree) to 5 (strongly agree), and symptom items from 0 (not at all) to 3 (nearly every day). Figure 5 shows one full prompt.
Templates. Of our 50 templates, 30 describe only the setting, and 20 add detail, such as a well-stocked study, that could hint at some traits. Each questionnaire uses the 30 plain templates and only those detailed ones whose detail does not bear on it. The Big Five questionnaire therefore uses the 30 plain templates only, and each other questionnaire uses 44 to 48, that is, the 30 plain ones and 14 to 18 detailed ones.
Extended Results
Calibration on the Real Design
We calibrate on one model from each of the five families. Each setting uses 500 datasets with no interaction, built from one model’s real answers, and 1,000 null draws per dataset. Table 7 gives the FWER of our test in each setting, with a 95% Clopper–Pearson interval.
| Model | Representation | Generator | FWER | 95% interval |
| Llama 3 8B | item space | flip | 2.6% | 1.4–4.4% |
| Llama 3 8B | item space | permute | 4.2% | 2.6–6.3% |
| Llama 3 8B | item space | group-mean | 3.0% | 1.7–4.9% |
| Llama 3 8B | scale scores | flip | 5.2% | 3.4–7.5% |
| Llama 3 8B | scale scores | permute | 5.6% | 3.8–8.0% |
| Llama 3 8B | scale scores | group-mean | 5.4% | 3.6–7.8% |
| Gemma 4 E4B | item space | flip | 4.2% | 2.6–6.3% |
| Gemma 4 E4B | item space | permute | 5.0% | 3.3–7.3% |
| Gemma 4 E4B | scale scores | flip | 5.0% | 3.3–7.3% |
| Gemma 4 E4B | scale scores | permute | 3.8% | 2.3–5.9% |
| Ministral 14B | item space | flip | 5.4% | 3.6–7.8% |
| Ministral 14B | item space | permute | 7.0% | 4.9–9.6% |
| Ministral 14B | scale scores | flip | 5.2% | 3.4–7.5% |
| Ministral 14B | scale scores | permute | 5.6% | 3.8–8.0% |
| Phi-4 | item space | flip | 5.0% | 3.3–7.3% |
| Phi-4 | item space | permute | 5.2% | 3.4–7.5% |
| Phi-4 | scale scores | flip | 5.4% | 3.6–7.8% |
| Phi-4 | scale scores | permute | 5.2% | 3.4–7.5% |
| Qwen2.5 14B | item space | flip | 4.2% | 2.6–6.3% |
| Qwen2.5 14B | item space | permute | 4.2% | 2.6–6.3% |
| Qwen2.5 14B | scale scores | flip | 4.6% | 2.9–6.8% |
| Qwen2.5 14B | scale scores | permute | 4.6% | 2.9–6.8% |
Table 7. FWER of our test on the real design, at nominal 5%. Synthetic data with the real main effects, cell sizes and noise of one model, and no interaction; 500 datasets per setting.
With the template in the model. We repeat the calibration with each persona’s template added to the additive model, for three of these models (Llama 3 8B, Ministral 14B, Qwen2.5 14B), both representations, and the flip and permute generators (12 settings, 200 datasets each). The FWER ranges from 3.5% to 9.0%, and 11 of the 12 confidence intervals contain 5%. The exception is Llama 3 8B on scale scores under the flip generator, at 9.0% (interval 5.4–13.9%), which is slightly above the nominal level.
Sign flips by template. As a further check, we flip one sign per template instead of one per persona. On real answers, this keeps 75–100% of the flagged cells, and the main cells stay significant in every model where they were flagged,
Prompt
Based only on the description below, how much do you agree with the following statement about this person?
The ferry was due shortly, and those crossing the river had gathered at the landing, watching for the boat to come into view. Near the front of the small crowd waited a woman, ready to board when the ferry arrived. When origins came up among those waiting, the family’s place among the Kamma was mentioned. Going by looks, the traveller was a young adult, a fact no one at the landing thought twice about. The look of the party suggested a household of lower middle class circumstances. A relative waiting alongside spoke of the traveller as lesbian, plainly and in passing. From a brief exchange it was apparent the traveller had grown up Hindu.
Statement: “This person is the life of the party.”
Reply with a single number.
1=Strongly Disagree, 2=Disagree, 3=Neither agree nor disagree, 4=Agree, 5=Strongly Agree
except transgender person × queer on scale scores in Qwen2.5 7B, its weakest model. This null is less reliable, however: with only about 215 independent signs, its FWER ranges from 1.0% to 9.5% across the settings we calibrated, and in one setting its interval lies fully above 5% (9.5%, interval 5.8–14.4%). We therefore keep the per-persona null.
Results on Likert-Rounded Answers
Real answers are whole numbers on each item’s answer scale. Rounding an additive score does not give an additive answer, because of each item’s floor and ceiling and because of the rounding step itself. Rounded data therefore contain a few real interactions on the answer scale even when the underlying scores add (Appendix § 12).
The single-attribute baseline. Table 8 repeats the comparison of Table 2 on the Likert-rounded scenario of the benchmark, which uses the permute generator (Appendix § 12). With the shuffle null the excess still flags every dataset. With our null it flags fewer datasets than our test and keeps a higher precision. It does so because it finds few interactions of any kind. At one residual standard deviation it recovers 5.2% of the planted cells, against 29.5% for our test.
Table 8. On Likert-rounded answers, the excess with our null flags few cells of any kind. Likert-rounded item-space answers from the benchmark, with five randomly chosen cells planted. The bracketed figure counts flags in the cells that rounding moves as false.
Our test on rounded data that copy the real design. We also calibrate our test on null data rounded to the answer scale. We use the five models of Appendix § 17.§ 1 with the flip and permute generators, in both representations, which gives 20 settings of 200 datasets each. Each dataset is built as in Appendix § 17.§ 1 from the model’s raw answers, and every value is then rounded to the nearest valid answer of its item and clipped to the item’s range. Our test flags at least one cell in 18–100% of these datasets, and in 7.5–56% once flags in threshold cells are not counted (Table 9; threshold cells as in Appendix § 12). In 19 of the 20 settings the 95% interval lies above 5%, because these null data contain real interactions on the answer scale and our test detects them.
| Model | Representation | Generator | Strict FWER | Primary FWER | Primary 95% interval | Control FWER | Control 95% interval |
| Llama 3 8B | item space | flip | 100.0% | 21.5% | 16.0–27.8% | 6.0% | 3.1–10.2% |
| Llama 3 8B | item space | permute | 96.0% | 16.0% | 11.2–21.8% | 5.0% | 2.4–9.0% |
| Llama 3 8B | scale scores | flip | 100.0% | 28.0% | 21.9–34.8% | 4.5% | 2.1–8.4% |
| Llama 3 8B | scale scores | permute | 71.5% | 12.0% | 7.8–17.3% | 4.5% | 2.1–8.4% |
| Gemma 4 E4B | item space | flip | 100.0% | 22.5% | 16.9–28.9% | – | – |
| Gemma 4 E4B | item space | permute | 78.0% | 56.0% | 48.8–63.0% | 4.0% | 1.7–7.7% |
| Gemma 4 E4B | scale scores | flip | 99.0% | 38.0% | 31.2–45.1% | 3.0% | 1.1–6.4% |
| Gemma 4 E4B | scale scores | permute | 42.0% | 20.0% | 14.7–26.2% | 3.5% | 1.4–7.1% |
| Ministral 14B | item space | flip | 100.0% | 18.5% | 13.4–24.6% | – | – |
| Ministral 14B | item space | permute | 100.0% | 20.0% | 14.7–26.2% | – | – |
| Ministral 14B | scale scores | flip | 18.0% | 12.0% | 7.8–17.3% | 4.5% | 2.1–8.4% |
| Ministral 14B | scale scores | permute | 55.0% | 18.5% | 13.4–24.6% | 6.0% | 3.1–10.2% |
| Phi-4 | item space | flip | 100.0% | 25.0% | 19.2–31.6% | – | – |
| Phi-4 | item space | permute | 100.0% | 26.0% | 20.1–32.7% | – | – |
| Phi-4 | scale scores | flip | 53.0% | 28.5% | 22.4–35.3% | 5.0% | 2.4–9.0% |
| Phi-4 | scale scores | permute | 96.5% | 21.5% | 16.0–27.8% | 6.5% | 3.5–10.9% |
| Qwen2.5 14B | item space | flip | 100.0% | 13.0% | 8.7–18.5% | – | – |
| Qwen2.5 14B | item space | permute | 98.0% | 7.5% | 4.3–12.1% | – | – |
| Qwen2.5 14B | scale scores | flip | 99.0% | 22.0% | 16.5–28.4% | 3.5% | 1.4–7.1% |
| Qwen2.5 14B | scale scores | permute | 61.5% | 10.0% | 6.2–15.0% | 5.0% | 2.4–9.0% |
Table 9. On rounded data our test flags many datasets, and on the control it holds its rate. 200 datasets per setting, 95% Clopper–Pearson intervals. Strict counts every flag as false, and primary leaves out threshold cells (Appendix § 12). The control makes the expectation exactly additive and counts every flag.
The control. To separate this from a miscalibrated test, we take the same rounded datasets and replace their expected values with the additive projection of those values. The noise is then exactly the rounded Likert noise, and the expectation is exactly additive. We ran this control on all ten scale-score settings and on three item-space settings. Our test then flags at least one cell in 3.0–6.5% of datasets, every 95% interval contains 5%, and the pooled rate is 4.7% (122 of 2,600 datasets). Our test is therefore calibrated on Likert noise, and the flags it raises on rounded data are interactions that exist on the answer scale.
Every Significant Cell
Figure 6 gives the cells each model flags, and Figure 7 and Table 10 the gender identity orientation table that holds the main cells. Table 11 gives the size of the main cells and how often they replicate. Table 12 lists every cell that is significant in at least one model, and Table 17 which traits carry each one on scale scores.
| Heterosexual | Gay | Lesbian | Queer | |
| Man | 13 / 11 | 9 / 4 | 11 / 9 | 12 / 12 |
| Woman | 13 / 7 | 4 / 2 | 3 / 0 | 10 / 8 |
| Transgender person | 13 / 13 | 12 / 10 | 12 / 6 | 13 / 13 |
Table 10. The gender identity orientation table interacts in every model. Number of the 13 open models in which each cell is significant, in the item space / on scale scores.

Figure 7. The gender identity orientation table interacts in every model. Number of the 13 open models in which each cell is significant.
| Cell | Item space | Scale scores |
| transgender person × heterosexual | 13 | 13 |
| transgender person × queer | 13 | 13 |
| man × heterosexual | 13 | 11 |
| poor working class × heterosexual | 13 | 11 |
| ultra rich × heterosexual | 13 | 8 |
| transgender person × poor working class | 13 | 7 |
| woman × heterosexual | 13 | 7 |
| man × queer | 12 | 12 |
| transgender person × gay | 12 | 10 |
| transgender person × ultra rich | 12 | 9 |
| transgender person × senior citizen | 12 | 8 |
| transgender person × lesbian | 12 | 6 |
| senior citizen × heterosexual | 11 | 10 |
| man × lesbian | 11 | 9 |
| lower middle class × heterosexual | 11 | 6 |
| poor working class × queer | 11 | 3 |
| transgender person × lower middle class | 11 | 3 |
| woman × queer | 10 | 8 |
| heterosexual × Buddhist | 10 | 5 |
| heterosexual × Hindu | 10 | 4 |
| transgender person × middle class | 10 | 3 |
| heterosexual × Muslim | 9 | 4 |
| man × gay | 9 | 4 |
| transgender person × adult | 9 | 4 |
| transgender person × child | 9 | 4 |
| transgender person × young adult | 9 | 4 |
| rich × heterosexual | 9 | 3 |
| transgender person × Hindu | 9 | 3 |
| middle class × heterosexual | 9 | 2 |
| young adult × heterosexual | 9 | 1 |
| transgender person × Buddhist | 8 | 4 |
| transgender person × rich | 8 | 3 |
| man × senior citizen | 8 | 2 |
| child × heterosexual | 7 | 4 |
| senior citizen × poor working class | 7 | 3 |
Table 12. Every cell significant in at least one of the 13 open models. Number of models in which the cell is significant, in the item space and on scale scores, ordered by the larger of the two. 175 cells.
| Cell | Models | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| transgender person heterosexual | 14 | 0/12 | 1/8 | 2/11 | 10/1 | 3/1 | 2/4 | 3/8 | 9/2 | 8/2 |
| transgender person queer | 14 | 13/0 | 11/0 | 8/1 | 1/9 | 7/1 | 4/1 | 9/2 | 0/11 | 0/6 |
| man queer | 13 | 0/10 | 0/10 | 0/4 | 4/0 | 0/5 | 1/4 | 1/9 | 9/0 | 5/0 |
| man heterosexual | 12 | 7/0 | 5/1 | 5/0 | 1/10 | 1/1 | 4/0 | 1/1 | 1/9 | 0/5 |
| poor working class heterosexual | 12 | 1/3 | 0/2 | 3/2 | 3/1 | 5/2 | 2/1 | 0/6 | 3/3 | 4/2 |
| senior citizen heterosexual | 11 | 0/1 | 1/0 | 5/0 | 0/5 | 4/0 | 1/3 | 1/2 | 4/1 | 2/0 |
| transgender person gay | 10 | 0/6 | 1/6 | 1/1 | 2/1 | 1/6 | 0/1 | 0/3 | 3/0 | 3/1 |
| transgender person ultra rich | 10 | 2/0 | 2/0 | 0/5 | 5/0 | 0/3 | 0/1 | 6/0 | 2/0 | 1/2 |
| man lesbian | 9 | 2/0 | 3/1 | 0/4 | 6/1 | 1/0 | 0/1 | 5/0 | 0/0 | 2/0 |
| transgender person senior citizen | 9 | 3/1 | 2/2 | 0/3 | 5/0 | 1/1 | 3/0 | 1/0 | 5/0 | 4/0 |
| transgender person poor working class | 8 | 2/0 | 0/1 | 3/0 | 0/1 | 4/2 | 3/1 | 0/5 | 1/0 | 0/3 |
| ultra rich heterosexual | 8 | 1/0 | 0/0 | 2/0 | 0/4 | 0/6 | 1/0 | 2/0 | 1/1 | 1/2 |
| woman queer | 8 | 0/8 | 1/6 | 0/2 | 2/0 | 0/2 | 0/3 | 1/1 | 3/0 | 3/0 |
| woman heterosexual | 7 | 3/0 | 4/1 | 2/0 | 1/0 | 0/1 | 1/0 | 1/2 | 2/1 | 0/3 |
| lower middle class heterosexual | 6 | 0/0 | 0/0 | 0/3 | 4/0 | 2/0 | 2/1 | 0/1 | 0/0 | 1/2 |
| transgender person lesbian | 6 | 0/1 | 1/3 | 3/0 | 1/2 | 0/1 | 1/1 | 0/3 | 2/0 | 1/0 |
| child heterosexual | 5 | 0/0 | 1/0 | 1/1 | 0/2 | 1/2 | 1/0 | 0/1 | 0/2 | 0/4 |
| heterosexual Buddhist | 5 | 0/1 | 1/0 | 0/0 | 1/0 | 2/1 | 0/0 | 0/1 | 1/0 | 1/0 |
| heterosexual Hindu | 4 | 1/0 | 1/1 | 0/0 | 0/0 | 0/1 | 0/0 | 3/1 | 1/0 | 2/0 |
| heterosexual Muslim | 4 | 0/0 | 0/0 | 0/1 | 1/0 | 2/0 | 1/0 | 0/1 | 0/3 | 0/3 |
| man gay | 4 | 0/0 | 4/0 | 1/0 | 0/0 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 |
| transgender person Buddhist | 4 | 1/0 | 0/2 | 0/1 | 2/0 | 0/0 | 0/1 | 0/4 | 0/0 | 0/0 |
| transgender person adult | 4 | 0/3 | 0/1 | 1/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/1 | 0/1 |
| transgender person child | 4 | 4/0 | 2/0 | 1/1 | 2/0 | 1/0 | 1/0 | 0/0 | 2/1 | 1/1 |
| transgender person young adult | 4 | 0/4 | 0/1 | 1/0 | 0/2 | 0/1 | 0/3 | 1/0 | 0/1 | 0/1 |
| child poor working class | 3 | 0/1 | 0/0 | 0/2 | 0/0 | 0/3 | 0/0 | 1/0 | 1/0 | 0/1 |
| poor working class queer | 3 | 0/0 | 0/0 | 0/1 | 0/0 | 0/2 | 0/1 | 0/0 | 1/2 | 0/1 |
| rich heterosexual | 3 | 0/0 | 0/0 | 0/0 | 0/2 | 0/2 | 0/1 | 2/0 | 0/1 | 1/1 |
| senior citizen poor working class | 3 | 1/0 | 0/0 | 0/1 | 1/1 | 1/0 | 1/0 | 0/0 | 0/1 | 0/1 |
| transgender person Dalit | 3 | 1/0 | 0/2 | 0/0 | 1/0 | 0/0 | 0/0 | 0/3 | 0/1 | 0/0 |
| transgender person Hindu | 3 | 0/1 | 2/0 | 1/0 | 0/1 | 0/0 | 0/0 | 1/1 | 0/0 | 0/0 |
Table 17. Which traits carry each significant cell. For every cell significant on scale scores, the number of models in which each trait’s component is significantly positive / negative, family-wise across the nine traits, counted over the models in which the cell is significant. Models counts the 13 open models and DeepSeek V4.1 Flash.
Robustness Checks
Our metric has no sign, so we read the direction of an interaction from the signed residuals on the anxiety and depression scales. Under our main wording, transgender person queer is sub-additive, which means that in 10 of 13 models a
model attributes less anxiety to it than its two labels predict. Man queer and transgender person heterosexual lean the other way. The C1 quartets and ordinal models of each item give the same sign, so neither a ceiling effect nor treating the answers as numbers explains it.
Rewording the prompts changes this picture (Table 20). The main cells still interact in most models, while the sub-additive anxiety residual survives in only a few. The models agree on where identities interact and the direction depends on the wording, so a claim from a single wording that identities compound, or that they cancel, is unsafe.
| Check | Result |
| Whitened item space; each questionnaire left out | Both transgender main cells interact in 13 of 13 models (one exception: 12 of 13) |
| Rewordings 1 and 2 | At least two of four key gender × orientation cells interact in 10 and in 8 models |
| Direction under rewordings | Transgender person × queer anxiety residual is negative in 10 of 13 models under the main wording, in 7 of the 12 rewording models under the original wording, and in 5 and 1 of 12 under the two rewordings |
| Fresh sample of 5,000 personas (5 models) | 52% of flagged cells are significant again (51% on scale scores) |
| Template added to the additive model | 98–100% of flagged cells stay significant (94–100% on scale scores), including the main cells |
| Largest template effect against largest identity effect | The template effect is larger in 13 of 13 models (12 of 13 on scale scores) |
| Model size | More flagged cells in larger models in the item space (), not on scale scores () |
Table 20. The location of the main interactions survives every check, while single cells and the direction do not. Real answers in the item space unless stated otherwise. The rewording arms cover 12 models.
About half of the flagged cells are significant again on a fresh sample of personas, so a single weak cell is a lead and the pattern across a table is the finding. Templates move the answers more than identity does. Leaving their effects out of the additive model can create false positives in single cells, with a family-wise error rate of up to 23.5% in one model. With each persona’s template in the additive model, the test returns to near its nominal level, and almost every flagged cell stays significant, including the main cells (Appendix § 17.§ 1).
Size is tied to family and training in our 13 models, so we draw no conclusion about scale and bias from the model-size result.
Three-way effects. A pairwise cell can also pick up an interaction among three attributes. When we plant a pure three-way effect in the benchmark, it is flagged only in the three pairs that contain it, and only when it is large (one residual standard deviation on scale scores, two in the item space). A pure two-way effect is never flagged in other pairs, and false flags elsewhere stay at the null rate. A flagged pair should therefore be read as an interaction involving its two values, possibly together with a third.
Published Alternatives on Real Answers
Real answers have no ground truth, so the comparison of the alternatives of § 4 is descriptive. In the item space every alternative except the single-attribute baseline finds all three main cells, and the methods disagree about almost everything else (Table 21). The single-attribute baseline flags most cells when a positive excess is enough. When its excess has to pass its own significance test, it misses transgender person × heterosexual in all 13 models in the item space and in 12 of 13 on scale scores. Per-item regression and MAIHDA flag more cells than our test in the item space, while on scale scores only MAIHDA does. Their extra cells are weaker effects, sometimes resting on one or two items, and they are significant again on a fresh sample less often. Extra cells replicate less for every method, ours included, so we read this as a difference in error control and not as evidence that those cells are false.
| Method | Cells flagged, item space | Cells flagged, scale scores | Also flagged by our test | Trans. × het., item space |
| Single-attribute, excess > 0 | 640–711 | 550–614 | 3% | 3 |
| Single-attribute, excess significant | 19–80 | 27–74 | 18% | 0 |
| Per-item classical + Bonferroni | 52–185 | 14–38 | 74% | 13 |
| MAIHDA | 63–227 | 17–70 | 60% | 13 |
| PERMANOVA (pair test) | 533–738 | 313–645 | 5% | 13 |
| Ours | 24–106 | 14–41 | – | 13 |
Table 21. Our test flags few cells, and every alternative except the single-attribute baseline finds the main cells. Real answers. Cells flagged per model, as a range over the 13 open models, in each representation. The fourth column is the mean share of a method’s flagged cells that our test also flags, on scale scores. The last column counts the models in which a method flags transgender person × heterosexual.
HInter flags every contrast in every model, including the calibration contrast C4, and nearly every item in simulated additive data. Its flags therefore barely agree with our difference-in-differences, at a Cohen’s of 0.059 over every model, contrast and item. Like the single-attribute baseline, its rule reacts to main effects that each push some answers across a category boundary. This limits what its flags can show about an interaction, and not what HInter does as a tool for finding failing inputs.
Further Variants of the Published Alternatives
We also ran four further variants of the published alternatives, on the same benchmark datasets and real answers as the main comparison. The anchored form of the single-attribute baseline compares a cell with the most privileged cell instead of the population mean [22, 60]. The stepwise form flags a cell when adding its second value moves the answers further than the first value alone [48]. For per-item regression with classical standard errors, we replace the Bonferroni correction with the Holm or the Benjamini–Hochberg correction.
None of these variants changes the picture of § 6 (Table 22). The anchored and stepwise forms make a false positive in every dataset with nothing planted, and their precision stays near 0.01. Holm gives almost the same flags as Bonferroni. Benjamini–Hochberg controls the false discovery rate, not the FWER, so it finds more planted cells but at a much lower precision. On real answers, the anchored and stepwise forms flag hundreds of cells per model and share few of them with our test (Table 23). The anchored form never flags transgender person × heterosexual. Table 23 also gives per-item regression with HC3 errors, which Table 21 leaves out.
| Method | Likert | Scale | Item | Likert | Real pattern (scale) | Single item (item) |
| Ours | 20.0% | 8.7% | 4.7% | 78.1% / 0.72 | 84.4% / 0.99 | 53.1% / 0.98 |
| Single-attribute, anchored | 100% | 100% | 100% | 52.0% / 0.01 | 33.3% / 0.01 | 48.7% / 0.01 |
| Single-attribute, stepwise | 100% | 100% | 100% | 99.9% / 0.01 | 83.2% / 0.01 | 98.9% / 0.01 |
| Per-item classical + Holm | 100% | 36.0% | 100% | 85.9% / 0.20 | 77.6% / 0.92 | 99.2% / 0.15 |
| Per-item classical + BH | 100% | 38.0% | 100% | 95.6% / 0.07 | 89.3% / 0.64 | 99.7% / 0.05 |
Table 22. Further variants on the benchmark. Same datasets and settings as Table 4. Group-mean generator; Likert answers use permute.
| Method | Cells flagged, item space | Cells flagged, scale scores | Also flagged by our test | Trans. × het., item space |
| Single-attribute, anchored | 247–358 | 262–347 | 3% | 0 |
| Single-attribute, stepwise | 682–723 | 614–671 | 3% | 10 |
| Per-item HC3 + Bonferroni | 63–223 | 10–35 | 76% | 13 |
| Per-item classical + Holm | 52–185 | 14–38 | 74% | 13 |
| Per-item classical + BH | 184–423 | 26–119 | 34% | 13 |
| Ours | 24–106 | 14–41 | – | 13 |
Table 23. Further variants on real answers. Same columns as Table 21.
I Recall at Every Planted Size
Table 24 gives the recall of our test and of per-item regression with HC3 errors and Bonferroni at each planted size, in residual standard deviations. Our test finds more planted cells on scale scores and for the real pattern. Per-item regression finds more in the item space when the planted shifts are random or sit in a single item, at the cost of a far higher FWER (Table 4). The three main cells of § 6.3 have sizes of about 0.7–0.9 in the item space and 0.3–0.4 on scale scores, in the range where our recall rises from about one half to almost one.
| Scenario | 0.2 | 0.5 | 1 |
| One random cell, scale scores | 0.00 / 0.00 | 0.06 / 0.04 | 0.90 / 0.74 |
| Five random cells, scale scores | 0.02 / 0.02 | 0.52 / 0.45 | 0.95 / 0.88 |
| Five random cells, item space | 0.00 / 0.01 | 0.10 / 0.21 | 0.56 / 0.84 |
| Twenty cells, two per pair, scale scores | 0.03 / 0.02 | 0.76 / 0.65 | 0.99 / 0.96 |
| Several in one pair, scale scores | 0.02 / 0.02 | 0.54 / 0.45 | 0.95 / 0.88 |
| Single item, scale scores | 0.02 / 0.10 | 0.52 / 0.67 | 0.94 / 0.99 |
| Single item, item space | 0.00 / 0.12 | 0.10 / 0.67 | 0.55 / 0.98 |
| Real pattern, scale scores | 0.04 / 0.01 | 0.38 / 0.30 | 0.88 / 0.78 |
| Real pattern, item space | 0.04 / 0.03 | 0.52 / 0.44 | 1.00 / 0.91 |
| Likert-rounded, item space | 0.01 / 0.01 | 0.29 / 0.10 | 0.78 / 0.59 |
Table 24. Recall rises with the planted size. Synthetic benchmark data. Share of planted cells found by our test / by per-item HC3 + Bonferroni, as the mean over models and generators.
Reliability of the Scales
We compute Cronbach’s for each scale over the personas each model describes, leaving out items that a model answers the same way for every persona. This measures how consistently a model’s answers move across the items of one scale, not the reliability of the scale for people. Table 25 summarises it over the 13 open models.
Table 25. The symptom scales are reliable in most models, but openness and need for cognition are not. Cronbach’s over personas, as the median [range] over the 13 open models, and the number of models with .
| Scale | Items | ||
| Extraversion | 10 | 0.71 [0.27, 0.86] | 8 |
| Agreeableness | 10 | 0.72 [0.42, 0.87] | 8 |
| Conscientiousness | 10 | 0.58 [0.28, 0.78] | 3 |
| Neuroticism | 10 | 0.56 [−0.11, 0.90] | 5 |
| Openness | 10 | 0.35 [0.09, 0.83] | 2 |
| Need for cognition | 8 | 0.34 [0.12, 0.77] | 1 |
| Empathic concern | 9 | 0.61 [0.22, 0.76] | 1 |
| Anxiety (GAD-7) | 7 | 0.83 [0.59, 0.90] | 11 |
| Depression (PHQ-8) | 8 | 0.82 [0.20, 0.88] | 11 |
Table 25.
Control Arms
Besides the 10,000 main personas, IndiPersona holds three control arms.
Fresh sample. A second sample of 5,000 personas, built the same way with a different seed, checks whether our results reproduce on new data.
Rewordings. Two independent rewordings of every prompt, on 3,000 personas, test whether the findings depend on our wording. They change the templates, question wording, answer labels and system prompt, but keep the identity labels and answer numbers.
Personas without identity. A set of 1,000 personas with no identity attributes shows how far identity moves the answers at all. Against these personas, identity labels move the median item by 0.31–0.76 standard deviations, yet they explain only 10–28% of the item variance (five models).
The Four Counterfactual Contrasts
Table 26 lists the labels each contrast changes, and Table 27–Table 30 give the shift on each trait for each model. C1 tests the main observational interaction, transgender person queer. C2 replaces transgender person with woman, to check that C1 is specific to the transgender label. C3 tests Dalit poor working class, a caste interaction that the main personas hold too little data to detect. C4 swaps adjacent levels of age and class, where we expect little interaction, and serves as the calibration contrast. We compare each contrast with C4 by the size of its difference-in-differences, with a sign-flip test and a bootstrap interval over base personas.
Table 26. Changing two labels reproduces the main interaction, and finds a caste interaction the screen misses. Size is the debiased length of the within-persona difference-in-differences, median [range] over the 13 open models. The last column counts models in which a contrast exceeds C4 with 95% bootstrap confidence.
| Contrast | Labels changed | Size | Exceeds C4 |
| C1 | man → transgender person, heterosexual → queer | 2.36 [1.22, 4.88] | 13 |
| C2 | man → woman, heterosexual → queer | 0.68 [0.41, 0.93] | 5 |
| C3 | Brahmin → Dalit, rich → poor working class | 2.10 [0.56, 3.29] | 12 |
| C4 | adult → senior citizen, middle class → upper middle class | 0.47 [0.15, 0.81] | – |
Table 26.
Table 27. C1: man → transgender person, heterosexual → queer. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.
| Model | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| Gemma 4 E2B | +0.86* | +0.40* | +0.86* | −0.74* | +0.28* | +0.64* | −0.20* | −0.97* | −0.94* |
| Gemma 4 E4B | +0.29* | +0.17 | +0.30* | −0.32* | −0.01 | −0.06 | +0.29* | −0.41* | −0.22* |
| Llama 3 8B | +0.95* | +0.58* | +0.25* | −0.21* | +0.36* | +0.93* | +0.72* | −0.50* | −0.40* |
| Llama 3.1 8B | +0.52* | +0.56* | +0.31* | −0.29* | +0.20* | +0.62* | +0.39* | −0.40* | −0.44* |
| Ministral 14B | +0.78* | +0.18* | +0.06 | −0.19* | +0.04 | +0.17* | +0.10 | −0.53* | −0.49* |
| Ministral 8B | +0.38* | +0.56* | +0.14* | −0.27* | +0.19* | +0.36* | +0.52* | −0.28* | −0.31* |
| Mistral Small 24B | +0.47* | +0.34* | +0.42* | −0.20* | +0.38* | +0.10 | +0.43* | +0.09 | −0.01 |
| Phi-4 | +0.39* | +0.56* | −0.10* | −0.03 | −0.07 | −0.02 | +0.29* | −0.19* | +0.01 |
| Phi-4-mini | +0.71* | +0.65* | +0.29* | −0.38* | +0.13 | +0.16 | +0.24* | −0.46* | −0.11 |
| Qwen2.5 14B | +0.44* | +0.30* | +0.39* | −0.11* | −0.10 | −0.01 | +0.41* | −0.32* | −0.11 |
| Qwen2.5 3B | +0.79* | −0.61* | +0.28* | +0.72* | −0.32* | −0.14 | −0.57* | +0.16* | +0.02 |
| Qwen2.5 7B | +0.30* | +0.25* | +0.08 | −0.25* | +0.07 | −0.07 | +0.25* | +0.14 | +0.17 |
| Qwen3 4B-2507 | +0.46* | +0.18* | +0.30* | −0.28* | +0.12 | +0.06 | −0.07 | −0.12* | −0.12* |
Table 27.
| Model | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| Gemma 4 E2B | +0.01 | −0.03 | −0.03 | +0.00 | +0.01 | −0.03 | +0.00 | +0.00 | −0.01 |
| Gemma 4 E4B | +0.07* | −0.03 | +0.07 | +0.04 | +0.03 | −0.03 | +0.00 | +0.00 | +0.02 |
| Llama 3 8B | +0.03 | −0.02 | +0.04 | −0.03 | −0.08* | −0.03 | +0.03 | +0.01 | +0.02 |
| Llama 3.1 8B | +0.02 | +0.02 | +0.06 | −0.01 | −0.03 | +0.09 | +0.03 | −0.02 | +0.00 |
| Ministral 14B | +0.00 | +0.00 | −0.10* | +0.05 | −0.22* | −0.02 | −0.01 | −0.01 | −0.04 |
| Ministral 8B | +0.02 | −0.06 | +0.01 | −0.07 | −0.07 | −0.07 | +0.04 | −0.07 | +0.01 |
| Mistral Small 24B | +0.00 | −0.07* | −0.05 | −0.08 | −0.08 | −0.16* | −0.04 | −0.07* | −0.11* |
| Phi-4 | +0.01 | +0.02 | −0.14* | +0.06* | +0.05 | −0.11* | −0.06 | +0.01 | −0.06* |
| Phi-4-mini | +0.01 | −0.05 | −0.04 | +0.02 | −0.17 | −0.02 | −0.11 | −0.07 | +0.00 |
| Qwen2.5 14B | +0.03 | −0.12* | +0.09 | −0.05 | +0.06 | +0.03 | −0.06 | +0.03 | +0.05 |
| Qwen2.5 3B | +0.03 | −0.02 | −0.03 | −0.04 | −0.07 | +0.04 | −0.05 | −0.04 | +0.01 |
| Qwen2.5 7B | +0.03 | +0.04 | +0.04 | +0.03 | +0.00 | −0.02 | +0.04 | +0.05 | +0.00 |
| Qwen3 4B-2507 | −0.02 | +0.01 | +0.08 | −0.04 | −0.07 | −0.01 | −0.03 | −0.07* | −0.06 |
Table 30. C4: adult senior citizen, middle class upper middle class. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.
Setup Details
Querying the models. We query the open models with vLLM in bf16, at pinned revisions, and DeepSeek V4.1 Flash through its API. A system prompt asks the model to infer, from the description alone, how the person would answer, and to return only the number.
Persona-sensitivity check. A model passes if it gives at most 1% invalid answers, if at least 10% of personas receive a distinct pattern of answers, and if its answers vary across personas, with a mean item standard deviation of at least 0.05 and no more than half of its items below 0.1.
Robustness checks. Each robustness check in Appendix § 17.§ 4 has one purpose. The fresh sample and the two random halves of the main sample test whether the findings reproduce on new data. For the fresh sample, we report the share of flagged cells that are flagged again. The two rewordings test whether the findings depend on our wording (Appendix § 20). The whitened item space removes the correlation between items, so that related items do not count twice [35], and leaving out each questionnaire tests whether one questionnaire drives the results. The answers are ordered categories, not numbers, so we fit ordinal models of each item for the three main cells [36]. One strong cell drags the residuals of its row and column, so we retest each cell after adding the strongest one to the additive model. Personas that share a template may share part of their noise, so we rerun the test with each persona’s template added to the additive model.
Simpler Fixes to the Single-Attribute Baseline
We test two repairs of the single-attribute baseline on the benchmark’s real-pattern scenario, on scale scores, at a planted size of one residual standard deviation. As in Table 2, we use the group-mean generator (150 datasets). The first repair gives its excess a calibrated Freedman–Lane null. The second compares the cell’s distance from the population mean with the distance of its additive prediction from the population mean. Both hold their error rate, with an FWER of 2.7% and 4.7%, but they find only 32.4% and 37.6% of the planted cells, against 85.0% for . The first repair is the second row of Table 2. We ran both repairs, and beside them, in a separate run on the same seeded datasets as the main benchmark, with a sign-flip Freedman–Lane null without the HC2 correction. In this run has an FWER of 6.7% and a recall of 85.0%, close to the 8.7% and 84.4% of the main run in Table 2. The single-attribute baseline’s false positives therefore come from its null, but its misses come from its shape.
Dependent Cells in the Gender Identity Orientation Table
One strong cell leaves residuals in the other cells of its row and column, so flagged cells in one table are not independent findings. To see which cells carry their own signal, we add the strongest cell to the additive model and retest the rest. When we add transgender person queer, man queer mostly disappears, and it stays significant in only 4 of the 12 models that flagged it. The reverse does not happen: when we add man queer, transgender person queer stays significant in all 13 models. The transgender row and man heterosexual also survive. We choose the strongest cell from the same data, so we read this refit as a diagnostic, not as a confirmatory test.
| Cell | Item space | Scale scores |
| lower middle class queer | 7 | 2 |
| man poor working class | 7 | 2 |
| transgender person Christian | 7 | 2 |
| transgender person Jain | 7 | 2 |
| upper middle class heterosexual | 7 | 2 |
| senior citizen queer | 7 | 1 |
| teenager heterosexual | 7 | 1 |
| transgender person upper middle class | 7 | 1 |
| ultra rich queer | 7 | 1 |
| senior citizen lower middle class | 6 | 2 |
| senior citizen ultra rich | 6 | 2 |
| poor working class Hindu | 6 | 1 |
| child poor working class | 5 | 3 |
| heterosexual Christian | 5 | 2 |
| young adult poor working class | 5 | 1 |
| poor working class lesbian | 5 | 0 |
| upper middle class queer | 5 | 0 |
| heterosexual Jain | 4 | 2 |
| transgender person teenager | 4 | 2 |
| woman gay | 4 | 2 |
| poor working class Jain | 4 | 1 |
| transgender person Muslim | 4 | 1 |
| transgender person OC | 4 | 1 |
| ultra rich Christian | 4 | 1 |
| woman poor working class | 4 | 1 |
| adult poor working class | 4 | 0 |
| man Hindu | 4 | 0 |
| man ultra rich | 4 | 0 |
| poor working class Buddhist | 4 | 0 |
| senior citizen gay | 4 | 0 |
| senior citizen lesbian | 4 | 0 |
| teenager ultra rich | 4 | 0 |
| woman senior citizen | 4 | 0 |
| transgender person Dalit | 3 | 3 |
| ultra rich Jain | 3 | 3 |
| Cell | Item space | Scale scores |
| man child | 3 | 2 |
| transgender person ST | 3 | 2 |
| child middle class | 3 | 1 |
| child queer | 3 | 1 |
| child ultra rich | 3 | 1 |
| child upper middle class | 3 | 1 |
| lesbian Buddhist | 3 | 1 |
| woman ultra rich | 3 | 1 |
| child lower middle class | 3 | 0 |
| heterosexual Dalit | 3 | 0 |
| lower middle class Hindu | 3 | 0 |
| man adult | 3 | 0 |
| poor working class Christian | 3 | 0 |
| poor working class Muslim | 3 | 0 |
| poor working class OC | 3 | 0 |
| rich Hindu | 3 | 0 |
| rich queer | 3 | 0 |
| senior citizen rich | 3 | 0 |
| senior citizen upper middle class | 3 | 0 |
| teenager poor working class | 3 | 0 |
| transgender person Sikh | 3 | 0 |
| ultra rich Buddhist | 3 | 0 |
| ultra rich Hindu | 3 | 0 |
| woman lesbian | 3 | 0 |
| young adult ultra rich | 3 | 0 |
| ultra rich OC | 2 | 3 |
| adult heterosexual | 2 | 1 |
| heterosexual Ajlaf | 2 | 1 |
| heterosexual Protestant | 2 | 1 |
| heterosexual Sikh | 2 | 1 |
| lesbian Christian | 2 | 1 |
| man upper middle class | 2 | 1 |
| queer Hindu | 2 | 1 |
| senior citizen Hindu | 2 | 1 |
| heterosexual OC | 2 | 0 |
| Cell | Item space | Scale scores |
| heterosexual SC | 2 | 0 |
| man lower middle class | 2 | 0 |
| man young adult | 2 | 0 |
| middle class Hindu | 2 | 0 |
| middle class queer | 2 | 0 |
| poor working class Catholic | 2 | 0 |
| poor working class Dalit | 2 | 0 |
| poor working class Sikh | 2 | 0 |
| senior citizen Buddhist | 2 | 0 |
| senior citizen middle class | 2 | 0 |
| teenager rich | 2 | 0 |
| transgender person Catholic | 2 | 0 |
| ultra rich Muslim | 2 | 0 |
| ultra rich gay | 2 | 0 |
| young adult lower middle class | 2 | 0 |
| young adult queer | 2 | 0 |
| adult ultra rich | 1 | 1 |
| heterosexual Arzal | 1 | 1 |
| heterosexual Shudra | 1 | 1 |
| lesbian Hindu | 1 | 1 |
| lower middle class Christian | 1 | 1 |
| man Buddhist | 1 | 1 |
| senior citizen Jain | 1 | 1 |
| transgender person OBC | 1 | 1 |
| transgender person Protestant | 1 | 1 |
| transgender person Reddy | 1 | 1 |
| woman Buddhist | 1 | 1 |
| woman Hindu | 1 | 1 |
| woman child | 1 | 1 |
| young adult middle class | 1 | 1 |
| child rich | 1 | 0 |
| gay Christian | 1 | 0 |
| gay Muslim | 1 | 0 |
| heterosexual Catholic | 1 | 0 |
| heterosexual Jat Sikh | 1 | 0 |
| Cell | Item space | Scale scores |
| heterosexual OBC | 1 | 0 |
| heterosexual ST | 1 | 0 |
| lesbian Jain | 1 | 0 |
| lesbian Muslim | 1 | 0 |
| lower middle class Catholic | 1 | 0 |
| lower middle class Muslim | 1 | 0 |
| lower middle class Protestant | 1 | 0 |
| lower middle class lesbian | 1 | 0 |
| man Christian | 1 | 0 |
| man middle class | 1 | 0 |
| man rich | 1 | 0 |
| man teenager | 1 | 0 |
| poor working class Arzal | 1 | 0 |
| poor working class Brahmin | 1 | 0 |
| poor working class Protestant | 1 | 0 |
| poor working class gay | 1 | 0 |
| queer Buddhist | 1 | 0 |
| senior citizen Christian | 1 | 0 |
| teenager lower middle class | 1 | 0 |
| teenager middle class | 1 | 0 |
| teenager queer | 1 | 0 |
| teenager upper middle class | 1 | 0 |
| transgender person SC | 1 | 0 |
| transgender person Shudra | 1 | 0 |
| ultra rich Catholic | 1 | 0 |
| ultra rich Protestant | 1 | 0 |
| ultra rich Shudra | 1 | 0 |
| ultra rich Sunni | 1 | 0 |
| ultra rich lesbian | 1 | 0 |
| upper middle class Hindu | 1 | 0 |
| woman upper middle class | 1 | 0 |
| woman young adult | 1 | 0 |
| young adult Hindu | 1 | 0 |
| queer Kshatriya | 0 | 1 |
| woman Christian | 0 | 1 |
Table 13. Every significant cell (continued, part 2 of 5).
| Cell | Models | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| transgender person × Jain | 3 | 1/0 | 0/2 | 0/3 | 2/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 |
| transgender person × lower middle class | 3 | 1/0 | 0/1 | 1/2 | 0/1 | 0/0 | 1/0 | 0/2 | 0/0 | 0/0 |
| transgender person × middle class | 3 | 0/0 | 0/0 | 0/1 | 0/1 | 1/1 | 0/1 | 0/0 | 0/0 | 1/0 |
| transgender person × rich | 3 | 0/0 | 0/0 | 1/0 | 1/0 | 0/3 | 0/0 | 1/0 | 0/0 | 1/0 |
| ultra rich × Jain | 3 | 0/0 | 0/0 | 1/1 | 0/1 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| ultra rich × OC | 3 | 0/0 | 0/0 | 1/0 | 0/1 | 2/0 | 0/0 | 0/0 | 0/1 | 0/0 |
| heterosexual × Christian | 2 | 0/0 | 1/0 | 2/0 | 0/1 | 0/1 | 0/0 | 0/1 | 0/0 | 0/1 |
| heterosexual × Jain | 2 | 0/0 | 1/0 | 0/0 | 0/0 | 0/1 | 0/0 | 1/0 | 0/0 | 0/0 |
| lower middle class × queer | 2 | 0/0 | 0/0 | 0/0 | 0/1 | 0/2 | 0/0 | 1/0 | 0/0 | 1/0 |
| man × child | 2 | 0/1 | 0/0 | 0/0 | 0/1 | 0/1 | 0/1 | 0/0 | 1/0 | 1/0 |
| man × poor working class | 2 | 0/0 | 1/0 | 0/0 | 0/0 | 1/0 | 1/1 | 1/0 | 0/0 | 0/0 |
| man × senior citizen | 2 | 0/0 | 0/0 | 0/0 | 0/2 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 |
| middle class × heterosexual | 2 | 1/0 | 0/0 | 0/1 | 0/0 | 1/1 | 0/1 | 1/0 | 0/0 | 0/0 |
| senior citizen × lower middle class | 2 | 0/0 | 0/0 | 2/0 | 0/1 | 1/1 | 0/0 | 0/0 | 0/0 | 0/0 |
| senior citizen × ultra rich | 2 | 0/1 | 0/0 | 0/1 | 0/1 | 0/1 | 0/1 | 0/0 | 0/0 | 0/0 |
| transgender person × Christian | 2 | 0/0 | 0/1 | 0/1 | 1/0 | 1/0 | 0/0 | 0/0 | 1/0 | 1/0 |
| transgender person × ST | 2 | 2/0 | 0/1 | 0/1 | 2/0 | 0/0 | 0/0 | 0/2 | 0/0 | 0/1 |
| transgender person × teenager | 2 | 0/0 | 1/0 | 2/0 | 0/2 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 |
| upper middle class × heterosexual | 2 | 0/0 | 0/0 | 0/1 | 0/0 | 0/1 | 0/0 | 0/1 | 0/0 | 1/1 |
| woman × gay | 2 | 0/0 | 1/1 | 0/1 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/1 |
| adult × heterosexual | 1 | 0/1 | 1/0 | 0/1 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/1 |
| adult × ultra rich | 1 | 0/0 | 1/0 | 1/0 | 0/1 | 1/0 | 0/0 | 0/0 | 0/0 | 0/1 |
| child × middle class | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/1 | 0/1 |
| child × queer | 1 | 0/0 | 0/1 | 0/1 | 1/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 |
| child × ultra rich | 1 | 0/0 | 0/0 | 0/0 | 0/1 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| child × upper middle class | 1 | 1/0 | 0/0 | 1/0 | 0/0 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 |
| heterosexual × Ajlaf | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/1 |
| heterosexual × Arzal | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/1 |
| heterosexual × Protestant | 1 | 0/0 | 0/0 | 1/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 |
| heterosexual × Shudra | 1 | 1/0 | 1/0 | 1/0 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| heterosexual × Sikh | 1 | 0/0 | 0/1 | 0/0 | 0/0 | 0/1 | 0/0 | 0/1 | 0/0 | 0/0 |
| Cell | Models | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| lesbian × Buddhist | 1 | 0/0 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| lesbian × Christian | 1 | 0/0 | 0/0 | 0/1 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 | 1/0 |
| lesbian × Hindu | 1 | 0/1 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| lower middle class × Christian | 1 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 |
| man × Buddhist | 1 | 0/0 | 1/0 | 0/0 | 0/1 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 |
| man × upper middle class | 1 | 0/0 | 0/0 | 0/1 | 1/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 |
| poor working class × Hindu | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 |
| poor working class × Jain | 1 | 1/0 | 0/0 | 0/1 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| queer × Hindu | 1 | 0/1 | 0/0 | 0/0 | 0/0 | 0/1 | 0/1 | 0/0 | 0/0 | 0/0 |
| queer × Kshatriya | 1 | 0/0 | 0/0 | 1/0 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| senior citizen × Hindu | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 |
| senior citizen × Jain | 1 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 |
| senior citizen × queer | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 |
| teenager × heterosexual | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/1 |
| transgender person × Muslim | 1 | 0/1 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| transgender person × OBC | 1 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| transgender person × OC | 1 | 0/0 | 0/1 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| transgender person × Protestant | 1 | 1/0 | 0/1 | 0/1 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| transgender person × Reddy | 1 | 0/0 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| transgender person × upper middle class | 1 | 0/0 | 0/0 | 1/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/1 |
| ultra rich × Christian | 1 | 0/0 | 0/1 | 0/1 | 1/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 |
| ultra rich × queer | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 |
| woman × Buddhist | 1 | 0/1 | 1/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| woman × Christian | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 |
| woman × Hindu | 1 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 |
| woman × child | 1 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/1 | 0/1 |
| woman × poor working class | 1 | 0/0 | 0/0 | 0/1 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 |
| woman × ultra rich | 1 | 0/0 | 0/0 | 1/0 | 0/1 | 0/0 | 0/0 | 0/0 | 0/1 | 0/0 |
| young adult × heterosexual | 1 | 0/0 | 0/0 | 0/0 | 1/0 | 0/1 | 0/1 | 0/0 | 1/0 | 1/0 |
| young adult × middle class | 1 | 0/0 | 0/0 | 0/1 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
| young adult × poor working class | 1 | 0/0 | 0/0 | 0/0 | 0/0 | 1/0 | 0/0 | 0/0 | 0/0 | 0/0 |
Table 18. Which traits carry each significant cell (continued, part 2 of 3).
| Model | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| Gemma 4 E2B | +0.00 | −0.03 | +0.09* | −0.17* | +0.01 | −0.02 | −0.02 | +0.05* | +0.03 |
| Gemma 4 E4B | +0.04 | +0.00 | −0.12* | −0.04 | +0.07 | −0.03 | +0.12 | −0.06 | −0.03 |
| Llama 3 8B | +0.16* | +0.15* | +0.05 | −0.07 | +0.20* | +0.20* | +0.29* | −0.07 | −0.02 |
| Llama 3.1 8B | +0.15* | +0.15* | +0.10* | −0.18* | +0.11 | +0.26* | +0.08 | −0.04 | −0.02 |
| Ministral 14B | +0.16* | +0.11* | +0.09 | −0.11* | +0.07* | +0.03 | +0.03 | −0.10* | −0.13* |
| Ministral 8B | +0.04 | +0.01 | +0.01 | −0.08* | +0.05 | +0.09* | +0.03 | −0.07 | −0.04 |
| Mistral Small 24B | +0.14* | +0.12* | +0.05 | −0.04 | +0.17* | +0.07 | +0.12* | −0.03 | −0.03 |
| Phi-4 | +0.02 | +0.11* | −0.01 | +0.06* | +0.03 | +0.05 | +0.02 | +0.02 | +0.01 |
| Phi-4-mini | +0.15* | +0.07 | +0.10 | +0.05 | −0.01 | −0.02 | −0.01 | −0.07 | −0.02 |
| Qwen2.5 14B | +0.09* | −0.01 | −0.04 | +0.00 | +0.07 | +0.02 | +0.04 | −0.06 | −0.09 |
| Qwen2.5 3B | +0.22* | +0.08 | −0.01 | +0.06 | +0.04 | +0.01 | −0.01 | +0.03 | +0.03 |
| Qwen2.5 7B | +0.06 | +0.15* | +0.03 | −0.03 | +0.02 | −0.02 | +0.07 | +0.04 | +0.04 |
| Qwen3 4B-2507 | +0.09* | +0.14* | +0.05 | −0.10 | +0.16* | +0.16* | +0.12* | −0.02 | +0.00 |
Table 28. C2: man woman, heterosexual queer. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.
| Model | E | A | C | N | O | NFC | EMP | GAD-7 | PHQ-8 |
| Gemma 4 E2B | +0.03 | +0.08 | +0.09 | −0.21* | −0.05 | +0.06 | −0.66* | −0.12* | −0.14* |
| Gemma 4 E4B | −0.25* | +0.32* | +0.83* | −0.30* | −0.32* | +0.02 | −0.16 | −0.34* | −0.38* |
| Llama 3 8B | −0.02 | −0.09 | +0.31* | +0.03 | +0.10 | +0.03 | −0.46* | +0.11 | +0.31* |
| Llama 3.1 8B | −0.18* | −0.07 | +0.29* | −0.35* | −0.19* | +0.10 | −0.13 | −0.23* | −0.35* |
| Ministral 14B | +0.05 | +0.01 | +0.37* | −0.10 | −0.29* | −0.05 | +0.02 | −0.31* | −0.50* |
| Ministral 8B | +0.01 | +0.00 | +0.27* | −0.21* | −0.06 | −0.23* | −0.23* | −0.34* | −0.30* |
| Mistral Small 24B | −0.19* | +0.03 | +0.35* | −0.15* | −0.47* | −0.02 | +0.18* | +0.14* | +0.16* |
| Phi-4 | −0.10* | +0.43* | +0.03 | −0.10* | +0.05 | +0.33* | −0.05 | +0.06 | +0.19* |
| Phi-4-mini | −0.20* | −0.02 | +0.22* | −0.10 | +0.10 | +0.04 | −0.17* | −0.35* | −0.25* |
| Qwen2.5 14B | −0.22* | +0.25* | +0.74* | −0.48* | +0.61* | +0.05 | +0.49* | +0.04 | +0.08 |
| Qwen2.5 3B | −0.20* | −0.15* | +0.05 | +0.24* | −0.02 | −0.12 | −0.35* | +0.06 | +0.06 |
| Qwen2.5 7B | −0.07 | −0.06 | −0.06 | +0.09 | +0.02 | −0.02 | +0.15* | +0.09 | +0.19 |
| Qwen3 4B-2507 | −0.12* | +0.24* | +0.43* | −0.41* | +0.24* | +0.09 | +0.00 | +0.00 | +0.07 |
Table 29. C3: Brahmin Dalit, rich poor working class. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.
References
- [1]Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 386–397. doi:10.18653/v1/2024.acl-short.37DOI
- [2]Marti J. Anderson. 2017. Permutational Multivariate Analysis of Variance (PERMANOVA). John Wiley & Sons, Ltd, 1–15. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/978111845112.stat07841 doi:10.1002/978111845112.stat07841DOI
- [3]Lena Arnao, Abbey Liu, Stephen MacNeil, and Danaë Metaxa. 2024. The Silician Ceiling: Auditing GPT’s Race and Gender Biases in Hiring. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’24). Association for Computing Machinery, New York, NY, USA. doi:10.1145/3689904.3694699
- [4]Srijan Bansal, Vishal G. Jaiswal, Ayush Sahu, and Animesh Mukherjee. 2021. Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languages. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (Virtual Event, USA) (HT ’21). Association for Computing Machinery, New York, NY, USA, 27–34. doi:10.1145/3465336.3475118DOI
- [5]Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re-contextualizing Fairness in NLP: The Case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (Eds.). Association for Computational Linguistics, Online only, 727–740. doi:10.18653/v1/2022.aacl-main.55
- [6]Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 842–854. doi:10.18653/v1/2024.acl-short.75DOI
- [7]Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 1004–1015. doi:10.18653/v1/2021.acl-long.81DOI
- [8]Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
- [9]Kimberlé Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics. University of Chicago Legal Forum 1989, 1 (1989), 139–167. https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8/
- [10]Lee J. Cronbach and Goldine C. Gleser. 1953. Assessing similarity between profiles. Psychological Bulletin 50, 6 (1953), 456–473. doi:10.1037/h0057173DOI
- [11]Preetam Prabhu Srikar Dammu, Havoung Jung, Anjali Singh, Monojit Choudhury, and Tanu Mitra. 2024. “They are uncultured”: Unveiling Court Harms and Social Threats in LLM Generated Conversations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 20339–20369. doi:10.18653/v1/2024.emnlp-main.1134DOI
- [12]Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023. Building Socio-culturally Inclusive Stereotype Resources with Community Engagement. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 4365–4381. doi:10.52202/075280-0194DOI
- [13]Naoki Egami and Kosuke Imai. 2019. Causal Interaction in Factorial Experiments: Application to Conjoint Analysis. J. Amer. Statist. Assoc. 114, 526 (2019), 529–540. doi:10.1080/01621459.2018.1476246DOI
- [14]Clare R. Evans, David R. Williams, Jacka-Pekka Onnela, and S. V. Subramanian. 2018. A multilevel approach to modeling health inequalities at the intersection of multiple social identities. Social Science & Medicine 203 (2018), 64–73. doi:10.1016/j.socscimed.2017.11.011DOI
- [15]Antonio Ferrara, Francesco Cozzi, Alan Perotti, André Panisson, and Francesco Bonchi. 2025. Size-adaptive Hypothesis Testing for Fairness. In Advances in Neural Information Processing Systems (NeurIPS), arXiv:2506.10586arxiv.org/abs/2506.10586
- [16]Jessica M. Franklin, Sebastian Schneeweiss, Jennifer M. Polinski, and Jeremy A. Rassen. 2014. Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Computational Statistics & Data Analysis 72 (2014), 219–226. doi:10.1016/j.csda.2013.10.018DOI
- [17]David Freedman and David Lane. 1983. A Nonstochastic Interpretation of Reported Significance Levels. Journal of Business & Economic Statistics 1, 4 (1983), 292–298. doi:10.1080/07350015.1983.10509354DOI
- [18]Sourojit Ghosh, Pranav Narayanan Venkit, Sanjana Gautam, Shomir Wilson, and Aylin Caliskan. 2024. Do Generative AI Models Output Harm While Representing Non-Western Cultures: Evidence from a Community-Centered Approach. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society (San Jose, California, USA) (AIES ’24, Vol. 7). AAAI Press, 476–489. https://ojs.aaai.org/index.php/AIES/article/view/31651arxiv.org/abs/2407.14779
- [19]Lewis R. Goldberg. 1992. The development of markers for the Big-Five factor structure. Psychological Assessment 4, 1 (1992), 26–42. doi:10.1037/1040-3590.4.1.26DOI
- [20]Government of India. 1950. The Constitution (Scheduled Castes) Order, 1950. C.O. 19, as amended. https://socialjustice.gov.in/writereaddata/UploadFile/scorder1950.pdf
- [21]Wei Guo and Aylin Caliskan. 2021. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (Virtual Event, USA) (AIES ’21). Association for Computing Machinery, New York, NY, USA, 122–133. doi:10.1145/3461702.3462536DOI
- [22]Amogh Gupta, Niharika Patil, Sourojit Ghosh, and SnehalKumar (Neil) S. Gaikwad. 2026. Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26). Association for Computing Machinery, New York, NY, USA, 6093–6155. doi:10.1145/3805689.3812394DOI
- [23]Rishav Hada, Safiya Husain, Varun Gumma, Harshita Diddee, Aditya Yadavalli, Agrima Seth, Nidhi Kulkarni, Ujwal Gadiraju, Aditya Vashistha, Vivek Seshadri, and Kalika Bali. 2024. Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 1926–1939. doi:10.1145/3630106.3659017DOI
- [24]Amit Haim, Alejandro Salinas, and Julian Nyarko. 2024. What’s in a Name? Auditing Large Language Models for Race and Gender Bias. arXiv:2402.14875 https://arxiv.org/abs/2402.14875
- [25]Johannes Himmelreich, Arbie Hsu, Ellen Voemett, and Kristian Lum. 2025. The Intersectionality Problem for Algorithmic Fairness. In Proceedings of the Algorithmic Fairness Through the Lens of Metrics and Evaluation (Proceedings of Machine Learning Research, Vol. 279). PMLR, 68–95. arXiv:2411.02569 https://proceedings.mlr.press/v279/himmelreich25a.htmlarxiv.org/abs/2411.02569
- [26]Akshita Jha, Aida Davani, Chandan K. Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 9851–9870. doi:10.18653/v1/2023.acl-long.548DOI
- [27]Ishika Joshi, Ishita Gupta, Adarsh Dev, and Tapan Parikh. 2025. “Since Lawyers are Males..”: Examining Implicit Gender Bias in Hindi Language Generation by LLMs. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (Athens, Greece) (FAccT ’25). Association for Computing Machinery, New York, NY, USA, 3254–3264. doi:10.1145/3715275.3732208DOI
- [28]Deborah Dormah Kanubala and Isabel Valera. 2025. On the Misalignment Between Legal Notions and Statistical Metrics of Intersectional Fairness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 1363–1374. doi:10.1609/aies.v8i2.36637
- [29]Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 2564–2572. https://proceedings.mlr.press/v80/kearns18a.htmlarxiv.org/abs/1711.05144
- [30]Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models. In Proceedings of the 2024 International Conference on Information Technology for Social Good (Bremen, Germany) (GoodIT ’24). Association for Computing Machinery, New York, NY, USA, 231–239. doi:10.1145/3677525.3678666
- [31]Hannah Rose Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, and Yuki M. Asano. 2021. Bias out-of-the-box: an empirical analysis of intersectional occupational biases in popular generative language models. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 200, 14 pages.arxiv.org/abs/2102.04130
- [32]Neeraja Kirtane and Tanvi Anand. 2022. Mitigating Gender Stereotypes in Hindi and Marathi. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Christian Hardmeier, Christine Basta, Marta R. Costa-jussà, Gabriele Stańczak, and Hila Gonen (Eds.). Association for Computational Linguistics, Seattle, Washington, 145–150. doi:10.18653/v1/2022.gebnlp-1.16DOI
- [33]Kurt Kroenke, Tara W. Strine, Robert L. Spitzer, Janet B. W. Williams, Jocelyn T. Berry, and Ali H. Mokdad. 2009. The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders 114, 1–3 (2009), 163–173. doi:10.1016/j.jad.2008.06.026DOI
- [34]John Lalor, Yi Yang, Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking Intersectional Biases in NLP. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 3598–3609. doi:10.18653/v1/2022.naacl-main.263
- [35]Olivier Ledoit and Michael Wolf. 2004. A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis 88, 2 (2004), 365–411. doi:10.1016/S0047-259X(03)00096-4
- [36]Torrin M. Liddell and John K. Kruschke. 2018. Analyzing ordinal data with metric models: What could possibly go wrong? Journal of Experimental Social Psychology 79 (2018), 328–348. doi:10.1016/j.jesp.2018.08.009DOI
- [37]Geoffrey R. Loftus. 1978. On Interpretation of Interactions. Memory & Cognition 6, 3 (1978), 312–319. doi:10.3758/BF03197461DOI
- [38]Weicheng Ma, Brian Chiang, Tong Wu, Lili Wang, and Soroush Vosoughi. 2023. Intersectional Stereotypes in Large Language Models: Dataset and Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Manuscript submitted to ACM for Computational Linguistics, Singapore, 8589–8597. doi:10.18653/v1/2023.findings-emnlp.575
- [39]James G. MacKinnon and Halbert White. 1985. Some Heteroskedasticity-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties. Journal of Econometrics 29, 3 (1985), 305–325.
- [40]Vijit Malik, Sunipa Dev, Akihiro Nishiya Nuyen, and Kai-Wei Chang. 2022. Socially Aware Bias Measurements for Hindi Language Representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 1041–1052. doi:10.18653/v1/2022.naacl-main.76DOI
- [41]Tim P. Morris, Ian R. White, and Michael J. Crowther. 2019. Using simulation studies to evaluate statistical methods. Statistics in Medicine 38, 11 (2019), 2074–2102. doi:10.1002/sim.8086DOI
- [42]Anjishnu Mukherjee, Chahat Raj, Zivli Zhu, and Antonios Anastasopoulos. 2023. Global Voices, Local Biases: Socio-Cultural Prejudices across Languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 15828–15845. doi:10.18653/v1/2023.emnlp-main.981
- [43]Atharva Naik, Shounok Kar, Varnika Sharma, Ashwin Rajadesingan, and Koustuv Saha. 2026. Sima Alunty: Caste Audit in LLM-Driven Matchmaking. arXiv:2603.29288 [cs.CY] https://arxiv.org/abs/2603.29288
- [44]Pavithra P M Nair, Bhavik Talaviya, Shourya Bhushan, Rahul Pankajkashan, Seema Guruvadoo, Avinash Agarwal, Gilad Gressel, and Krishnashree Achuthan. 2026. RupeeBias: Auditing Demographic Bias in Indian Economic Guidance from Large Language Models. arXiv:2609.31245 [cs.CL] https://arxiv.org/abs/2609.31245
- [45]Janki Atul Nawale, Mohammed Safi Ur Rahman Khan, Janani D. Mansi Gupta, Danish Pruthi, and Mitesh M Khapra. 2025. Fair Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 30331–30380. doi:10.18653/v1/2025.acl-long.1465
- [46]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Alina Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 2086–2105. doi:10.18653/v1/2022.findings-acl.165DOI
- [47]Belinda Phipson and Gordon K. Smyth. 2010. Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology 9, 1 (2010), Article 39. doi:10.2202/1544-6115.1585DOI
- [48]Mahika Phutane, Hayoung Jung, Matthew Kim, Tanushree Mitra, and Aditya Vashistha. 2026. ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 9, 3 (2026), 2089–2104. arXiv:2510.10998 doi:10.1609/aies.v9i3.43291DOI
- [49]Prime Minister’s High Level Committee. 2006. Social, Economic and Educational Status of the Muslim Community of India: A Report. Technical Report. Cabinet Secretariat, Government of India, New Delhi.
- [50]Rida Qadri, Renee Shelby, Cynthia L. Bennett, and Remi Denton. 2023. AI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 506–517. doi:10.1145/3593013.3594016DOI
- [51]Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in Language Models Beyond English: Gaps and Challenges. In Findings of the Association for Computational Linguistics: EACL 2023, Andreas Vlachos and Isabell Augenstein (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 2106–2119. doi:10.18653/v1/2023.findings-eacl.157DOI
- [52]Pooja S. B. Rao, Laxminarayan Nagarajan Venkatesan, Mauro Cherubin, and Dinesh Babujayogi. 2025. Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models. arXiv:2508.16673 [cs.CY] https://arxiv.org/abs/2508.16673
- [53]Virgile Rennard and Christos Xylopoulos. 2026. Large Language Models Simulate Intersectional Synthetic Identities with a Budget of One to Two Dimensions. arXiv:2608.23005 https://arxiv.org/abs/2608.23005
- [54]Nihar Sahoo, Pratyush Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, and Pushpak Bhattacharyya. 2024. IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 8786–8806. doi:10.18653/v1/2024.naacl-long.487
- [55]Nithya Sambasivan, Erin Arnesen, Hen Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining Algorithmic Fairness in India and Beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 315–328. doi:10.1145/3442188.3445896DOI
- [56]Agrima Seth, Monojit Choudhury, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, and Kalika Bali. 2025. How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 2319–2330. doi:10.1609/aies.v8i3.36718DOI
- [57]Badr Souani, Ezekiel Soremekun, Mike Papadakis, Setsuko Yokoyama, Sudipta Chattopadhyay, and Yves Le Traon. 2025. HInter: Exposing Hidden Intersectional Bias in Large Language Models. arXiv:2503.11962 [cs.CL]
- [58]Robert L. Spitzer, Kurt Kroenke, Janet B. W. Williams, and Bernd Löwe. 2006. A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine 166, 10 (2006), 1092–1097. doi:10.1001/archinte.166.10.1092DOI
- [59]Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and Mitigating Discrimination in Language Model Decisions. arXiv:2312.03689 [cs.CL] https://arxiv.org/abs/2312.03689
- [60]Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. arXiv:1911.01485 https://proceedings.neurips.cc/paper_files/paper/2019/hash/201d546992726352471cfea6b0df0a48-Abstract.html
- [61]Alberto Testoni and Iacer Calixto. 2026. Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Maria Liakata, Viviane P. Moreira, JiaJun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 432–447. doi:10.18653/v1/2026.acl-short.36DOI
- [62]Aditya Tiwary Niranjan Sahoo, and Pushpak Bhattacharyya. 2025. BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context. Transactions of the Association for Computational Linguistics 13 (2025), 1672–1692. doi:10.1162/tacl_a_55arxiv.org/abs/2508.07090
- [63]Tyler J. VanderWeele and Miriam J. Knol. 2014. A Tutorial on Interaction. Epidemiologic Methods 3, 1 (2014), 33–72. doi:10.1515/em-2013-0005DOI
- [64]Laura K. Vaughan, Jasmin Divers, Miguel A. Padilla, David T. Redden, Hemant K. Tiwari, Daniel Pomp, and David B. Allison. 2009. The use of plasmodes as a supplement to simulations: A simple example evaluating individual admixture estimation methodologies. Computational Statistics & Data Analysis 53, 5 (2009), 1755–1766. doi:10.1016/j.csda.2008.02.032DOI
- [65]Prashanth Vijayaraghavan, Sarosh Vosoughi, Lamogha Chiazor, Raya Horesh, Rogerio Abreu de Paula, Ehsan Degan, and Vandana Mukherjee. 2025. DECASTE: Unveiling Caste Stereotypes in Large Language Models Through Multi-Dimensional Bias Analysis. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, James Kwok (Ed.). International Joint Conferences on Artificial Intelligence Organization, 9899–9907. doi:10.24963/ijcai.2025/1100 AI and Social Good.DOI
- [66]Eric-Jan Wagenmakers, Angelos-Miltiadis Krypotos, Amy H. Criss, and Geoff Iverson. 2012. On the Interpretation of Removable Interactions: A Survey of the Field 33 Years after Loftus. Memory & Cognition 40, 2 (2012), 145–160. doi:10.3758/s13421-011-0158-0DOI
- [67]Peter H. Westfall and S. Stanley Young. 1993. Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. John Wiley & Sons, New York.
- [68]Kyra Wilson and Aylin Caliskan. 2024. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1578–1590. doi:10.1609/aies.v7i1.31748DOI
- [69]Anderson M. Winkler, Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols. 2014. Permutation inference for the general linear model. NeuroImage 92 (2014), 381–397. doi:10.1016/j.neuroimage.2014.01.060DOI
- [70]Zhe Zhang and Daniel B. Neill. 2016. Identifying Significant Predictive Bias in Classifiers. arXiv:1611.08292 [stat.ML] NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems.arxiv.org/abs/1611.08292
- [71]Zheng Zhang, Yifan Wu, YuXiang Zhou, Qi Liu, Qingyang Mao, Ning Li, Weibo Gao, Qiyuan Shi, Zhenya Huang, and Baosheng Yu. 2026. Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs. arXiv:2610.35069 https://arxiv.org/abs/2610.35069


