betaFeedback

WorksComputer Science

LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits

Abstract

Intersectional bias arises when a model treats people who hold several marginalised identities at once differently from what their separate identities would predict. Auditing such intersectional subgroups remains a challenge: their number grows quadratically, and there is no agreed baseline for what independent attributes would predict. Without such a baseline, many existing audits flag a subgroup when it exceeds each attribute alone, a condition that attributes acting independently already satisfy. We test this standard criterion on 150 synthetic datasets built with no interaction. With the shuffle null, it flags 66.1 false subgroups per dataset on average, and at least one in every dataset. When we plant interactions of known size into these datasets, a repaired null lets the criterion recover only 32% of them. We introduce ADDIS (Additive Intersectional Screen), which fits an additive model over all attributes, scores each subgroup by its Additive Residual Distance (ARD) from that model's prediction, and tests it against a Freedman-Lane null. On our synthetic benchmark, ADDIS flags 0.09 false subgroups per dataset and recovers 84% of the same planted interactions. We then build IndiPersona, 10,000 synthetic Indian personas with six identity attributes, and audit 13 open language models. The standard criterion flags 7.1% of the 738 tested subgroups on average; ADDIS flags 3.0%. Gender identity crossed with sexual orientation is the largest pairing in every model, and counterfactual personas confirm it and additionally reveal a caste x class interaction in all but one model that the observational test lacked the data to detect. Much of what audits report as intersectional bias can thus arise from independent single-attribute effects alone, yet a small core of interactions survives.

CCS Concepts: • Computing methodologies → Natural language processing; • Social and professional topics → User characteristics; • Mathematics of computing → Hypothesis testing and confidence interval computation.

Introduction

Language models are increasingly deployed in systems that judge people, from ranking job applicants to answering a patient’s medical questions [3, 61]. Audits report that responses depend on identity, and that some combinations are treated differently from either identity alone [1, 22, 48, 61]. Most audits nevertheless examine one identity at a time. Intersectionality theory makes a specific claim about people who hold several identities at once: the harm they face is not reducible to the harms attached to each identity on its own, or even to their sum. Crenshaw writes that the intersectional experience is “greater than the sum of racism and sexism” [9 p. 140], so the quantity to be exceeded is the sum of the parts. We operationalize this claim by asking whether a model’s response to a combination of identities departs from what its separate identities predict.

Authors’ Contact Information: George Rahul, International Institute of Information Technology Hyderabad, Hyderabad, India; Ronit Jahilal, International Institute of Information Technology Hyderabad, Hyderabad, India; Bhaskara Hanuma Vedula, International Institute of Information Technology Hyderabad, Hyderabad, India, [email protected]; Ponnurangam Kumaraguru, International Institute of Information Technology Hyderabad, Hyderabad, India; Pushkar Shukla, Cloudphysician.ai, USA.

Consider a model that attributes anxiety and depression to personas, and rates a woman three points higher on anxiety and a Muslim four points higher on depression. A Muslim woman then lies five points from the reference, farther than under either identity alone (Figure 1a). Yet, it is exactly where the two effects add, so no interaction is involved. Only a departure from this sum, such as the 1.5 points in Figure 1b, indicates an interaction, so exceeding each part is not equivalent to exceeding the sum of the parts.

Four-panel schematic showing additive, aligned, perpendicular, and opposed attribute shifts

Figure 1. The single-attribute criterion cannot tell addition from interaction. Relative to a Hindu man, woman adds 3 points of attributed anxiety and Muslim 4 of depression. Points beyond the dashed arc are flagged, and red marks ARD δ\delta from their sum (open circle). (a) Without interaction, the Muslim woman is still flagged. (b, c) The same interaction raises the excess more along the sum than at a right angle. (d) If Muslim lowers anxiety instead, the shifts nearly cancel and a real interaction stays inside the arc.

Detecting such interactions at scale is challenging, because candidate subgroups multiply quadratically with the number of identity attributes while the observations available for each decrease [25, 29]. Testing hundreds of subgroups simultaneously also inflates false discoveries unless the error rate is controlled across all of them. Automated subgroup discovery makes this search systematic [29, 70], and the baseline against which it ranks subgroups determines which ones can be found. A common baseline in intersectional audits of language models, which we term the single-attribute criterion, reports a combination when it fares worse than each of its attributes alone [22, 48, 61]. As the example shows, merely additive effects satisfy this criterion. Conversely, it can miss real interactions when the two effects point in opposite directions, or when the interaction is perpendicular to their sum.

We address this problem with Addis (Additive Intersectional Screen), which screens every pair of identity attributes for interactions in how language models treat people. We replace the single-attribute baseline with an additive prediction. We fit a linear regression over the whole dataset, giving every level of every attribute its own coefficient. It has no interaction terms. A subgroup’s prediction is the sum of the coefficients of the attributes it carries. We define the additive residual distance (ARD) δ\delta as the distance between a subgroup’s observed mean response profile and this prediction. It is equal for interactions of equal magnitude in any direction. Figure 1 contrasts the two baselines on our example. We test each subgroup against a null that keeps every attribute’s effect and removes only the interaction effect, so the test asks whether two identities interact, not whether identity matters at all. We control the family-wise error rate at 5% across the search (§ 3.3).

In a natural sample of data, the attributes are correlated, so a pair’s effect cannot be separated from its parts’, and some subgroups are too small to study. So, we generate the personas we test, and sample their attributes independently of one another. The exception to this is caste, which is paired only with religions it actually occurs in. The dataset, INDIPERSONA, is ten thousand Indian personas crossing six attributes: gender identity, age, sexual orientation, socioeconomic status, caste and religion, each described in one of 50 everyday scenes. Fourteen models answer the same 82 questionnaire items about every persona. We test all 738 pairs of identity values with at least 50 personas, and use within-persona counterfactuals for combinations the main sample cannot estimate.

We test both baselines on data we build ourselves. It contains no interaction, so every subgroup a method flags in it is a false positive. We build it from the real design and the models’ own answers, keeping each attribute’s own effect and destroying only intersectional effects. Across 150 such datasets, each with 738 subgroups, Addis flags 0.09 false subgroups per dataset on continuous answers. The single-attribute baseline flags 66.1, and fires on every dataset. We also synthetically add real intersectional effects into the dataset, of sizes we choose. Addis recovers 84% of them, against 32% for the single-attribute baseline given the same null. On answers rounded to the Likert scale, rounding itself creates interactions and raises every method’s error rate. We ran every other published method through the same test: per-item interaction regressions [31], MAIHDA [14] and PERMANOVA [2]. Each raises a false flag in at least 73% of datasets, compared to our 20%. With interactions planted, the precision of Addis is 0.72, compared to 0.02–0.28 for the alternatives.

After running Addis on the 13 open models, an average of only 3.0% of subgroups are found to be intersectional on scale scores. By contrast, given the same statistical treatment, the single-attribute baseline flags 7.1% (52 subgroups per model), of which only about nine coincide with ours. Gender identity crossed with sexual orientation is the single largest pairing, in every model we audit. Within-persona counterfactuals reproduce that effect in all 13 open models, and find a caste ×\times class interaction in 12 of them, a result the observational test could not find due to lack of data. The location of these interactions holds up when we reword the prompt, draw a fresh sample of personas, or adjust for the template, though their direction depends on the wording. Ard measures interaction in what models attribute to people, not harm, and our data come from one country, so we read the flagged combinations as candidates for closer scrutiny, not as estimates of how much discrimination exists in the world. Our contributions are as follows.

  1. A metric for intersectional subgroup discovery. We compare each subgroup against the sum of its attributes’ separate effects, and test the difference against a null built for it.

  1. A benchmark for intersectional metrics. We build synthetic data with controllable interactions between the subgroups, so that the ground truth is known. On questionnaire-style answers, rounding weakens the error control of every method, but every published alternative we tried makes false positives far more often than ours.

  1. A factorial persona dataset and an audit of fourteen models. Ten thousand Indian personas whose identity attributes vary nearly independently, an audit of thirteen open models and one closed model, and the finding that only a small minority of subgroups survive a calibrated test, concentrated on a single pairing of attributes.

Related Work

Intersectional bias in language models. Bias in language models is often strongest where identities meet, in embeddings, generated text, question answering and hiring decisions [1, 21, 31, 34, 38, 46, 68]. Fairness for each group does not imply fairness for their combinations [8, 29]. Most of this work uses Western categories such as race and gender [21, 46], while social hierarchies and the biases attached to them differ across countries, languages and regions [6, 26, 42, 51, 55].

Bias evaluation in India. Work on India began with word embeddings, which encode caste, religious and gender associations in Hindi and other Indian languages [4, 32, 40]. Studies of Hindi then found that detecting gender bias needs community input [23] and that models are more biased when generating Hindi than English [27]. Stereotype benchmarks for language models followed, covering Indian axes of disparity [5], caste and religion [30, 65], several social dimensions [54], community-sourced stereotypes [12], Indian languages [62] and many identity groups [45]. Benchmarks of this kind test whether a model repeats known associations, so they depend on curated inventories whose validity is often unclear [7], and they miss harms outside them. Later work therefore studied open-ended generation [11, 18, 50, 56] and decisions. Decision audits hold a task fixed and change only the identity, in hiring [52], economic guidance [44], matchmaking [43] and education [22]. Because only the identity changes, they measure how a model treats a person rather than which words it associates, and our audit follows the same design.

How intersections are tested. Few of these audits test intersections systematically. Stereotype benchmarks include a few predefined pairs, mostly with gender [54, 62], and decision audits change one attribute at a time [44] or a few within one task [22, 43], as do audits outside India with race, gender and age [1, 3, 24, 59]. When an audit compares a combination with its parts, it usually asks whether the combination fares worse than each part alone [22, 34, 48, 60, 61], the single-attribute baseline that § 3.2 shows addition alone can satisfy. Other audits fit an interaction term for one outcome at a time [3, 31]. Recent work widens the set of combinations, by pairing multi-attribute instances with their single-attribute sources [71] or by flagging failures that appear only when two identity words are swapped together [57], but it fits no additive baseline across all attributes and does not control false positives across cells. Two studies come closest to our critique. Kanubala and Valera [28] show that intersectional fairness metrics for classifiers respond more to additive bias than to interaction, and Rennard and Xypolopoulos [53] compare simulated personas with an additive prediction and find that models represent intersectional identities in only one or two dimensions. Searching over many subgroups is also established for classifiers, through subgroup fairness audits [29] and scan statistics [70]. Table 1 compares these methods with ours. Unlike this work, we test every pairwise cell of a multivariate answer profile against an additive baseline fitted over all six attributes, control the family-wise error rate across cells, and confirm the main cells with counterfactuals.

MethodIndiaTreatmentAll cellsAdditiveProfileCellFWERPlantedCounterf.
Stereotype benchmarks, India [54, 62]✓✗✗✗✗–✗✗✗
Decision audits, India [43, 44, 52]✓✓✗✗✗–✗✗∼
Single-attribute baseline [22, 34, 48, 61]∼✓∼✗✗✓✗✗✗
Per-item interaction regression [3, 31]✗✓∼∼✗✓∼✗✗
MAIHDA [14]✗–✓✓✗✓✗✗✗
PERMANOVA [2]✗–∼∼✓✗✗✗✗
Multi-attribute pairing (CoMFair) [71]✗✓∼✗✗✓✗✗✗
Metamorphic testing (HInter) [57]✗✓∼✗✗✓✗✗✓
Subgroup search for classifiers [29, 70]✗–✓✗✗✓∼✗✗
Addis (ours)✓✓✓✓✓✓✓✓✓

Table 1. Addis combines properties that prior methods address separately. India: Indian identity axes. Treatment: only identity varies. All cells: every pairwise cell. Additive: additive baseline over all attributes. Profile: answers tested jointly. Cell: names the interacting cell. FWER: error controlled across cells. Planted: validated on planted interactions. Counterf.: confirmed by counterfactuals. ✓ yes, ✗ no, ∼ partly, – n/a.

Methodology

Addis searches for intersectional subgroups in how a language model treats people. It needs two parts. A baseline predicts where a subgroup should sit if its attributes do not interact. A null tells how far a subgroup can stray from the baseline by chance. Section 3.1 states the task. Section 3.2 shows that addition alone satisfies the single-attribute baseline of current audits, and § 3.3 replaces it with an additive prediction. Sections 3.4 and 3.5 add error control and a benchmark with known truth. § 3.6 describes IndiPersona, our dataset of personas, and § 3.7 describes its counterfactual quartets. Figure 2 shows the full pipeline.

Figure 2. The audit pipeline. IndiPersona personas (§3.6) are answered item by item by each model. An additive model fits the answers (§3.3), and a Freedman–Lane null controls the family-wise error rate across all 738 cells (§3.4).

The Audit Task

Notation. Each of N=10,000N = 10{,}000 personas holds one value of each of six identity attributes. A model rates persona ii on one questionnaire item at a time. The answers form the persona’s attributed profile yi∈Rd\mathbf{y}_i \in\mathbb{R}^d, either its item answers (d≤82d \le82) or d=9d = 9 scale scores computed from them (§ 3.4). For a value aa, let yˉa\bar{\mathbf{y}}_a be the mean profile of the personas that hold aa, yˉ\bar{\mathbf{y}} the population’s mean profile, and ua=yˉa−yˉ\mathbf{u}_a = \bar{\mathbf{y}}_a - \bar{\mathbf{y}} the shift of aa. Two values aa and bb of different attributes define a cell a∧ba \wedge b, the set of nabn_{ab} personas that hold both, with mean profile yˉa∧b\bar{\mathbf{y}}_{a \wedge b}. We test all 738 cells with nab≥50n_{ab} \ge50, denoted C\mathcal{C}.

Target. If aa and bb do not interact, the cell sits at the additive prediction yˉ+ua+ub\bar{\mathbf{y}} + \mathbf{u}_a + \mathbf{u}_b, and its interaction is its departure from that prediction, Δab=yˉa∧b−(yˉ+ua+ub)\Delta_{ab} = \bar{\mathbf{y}}_{a \wedge b} - (\bar{\mathbf{y}} + \mathbf{u}_a + \mathbf{u}_b). § 3.3 refines the additive prediction to account for the other four attributes. We call a cell intersectional when Δab≠0\Delta_{ab} \ne0, thus treating intersectionality as an interaction on the additive scale [63], a working definition rather than the theory’s own [9].

The audit. An audit returns a set of flagged cells F⊆CF \subseteq\mathcal{C}. It should flag the cells with Δab≠0\Delta_{ab} \ne0 while keeping the family-wise error rate (FWER), the probability of flagging any cell with Δab=0\Delta_{ab} = 0, at 5% or lower. The flagged set depends on the baseline that the audit compares each cell with.

The Single-Attribute Baseline Is Satisfied by Addition

Current audits measure how far a subgroup sits from a reference, such as the population’s mean or the most privileged group. They call a cell a∧ba \wedge b intersectional when it sits farther from this reference than each of its two values [22, 60]. With the population’s mean yˉ\bar{\mathbf{y}} as the reference, the rule reads:

excess⁡(a∧b)=∥yˉa∧b−yˉ∥−max⁡(∥ua∥,∥ub∥)>0.(1)\operatorname{excess}(a \wedge b) = \lVert\bar{\mathbf{y}}_{a \wedge b} - \bar{\mathbf{y}} \rVert- \max(\lVert\mathbf{u}_a \rVert, \lVert\mathbf{u}_b \rVert) > 0. \tag*{(1)}

We call this rule the single-attribute baseline, and the left side of Equation (1) its excess. The rule asks whether a combination fares worse than its parts. A positive excess can also arise by chance. Audits that test their intersectional claims ask whether a subgroup differs from its comparison group at all. They use permutation tests [60], tt-tests [22], rank tests [48] or tests against an unperturbed prompt [61]. All of these tests share one null hypothesis: identity has no effect. We test the excess against the permutation version of this null, which we call the shuffle null. It shuffles the identity labels across personas. Each persona keeps its answers but receives another persona’s identities. We then recompute the excess on each shuffled dataset. Shuffling cuts the link between every identity and the answers, so shuffled data contain no main effects and no interaction. The shuffle null therefore asks whether identity matters at all, not whether two values interact. Real answers do contain main effects, so the excess of a real cell easily beats its shuffled values, even when the two values only add. The test then reports interactions that are not there. § 5.1 shows how often. The excess has a second problem, even before any test. It is positive for many cells in which nothing interacts. Suppose a cell sits exactly at its additive prediction, so that Δab=0\Delta_{ab}=0.

Proposition 3.1. If Δab=0\Delta_{ab}=0, then excess⁡(a∧b)>0\operatorname{excess}(a\wedge b)>0 if and only if ⟨ua,ub⟩>−12min⁡(∥ua∥2,∥ub∥2)\langle u_a,u_b\rangle>-\frac{1}{2}\min(\lVert u_a\rVert^2,\lVert u_b\rVert^2).

We prove Proposition 3.1 in Appendix A. The sign of the excess depends on the angle between uau_a and ubu_b, not on any interaction. Take two shifts of equal length. The excess is then positive whenever the angle between them is less than 120∘120^\circ. The single-attribute baseline flags such cells even when nothing interacts (Figure 1(a)). When a real interaction exists, the verdict also depends on which way the interaction points (Figure 1(b, Figure 1c)). When the shifts point apart at more than 120∘120^\circ, the excess turns negative, and a small real interaction stays hidden (Figure 1(d)). The shuffle null fixes neither failure. A cell flagged by the single-attribute baseline may hold only two ordinary biases added together. A cell it misses may hold a real interaction. We therefore replace the baseline itself.

Ard: Distance From the Additive Prediction

Ard compares each cell with the position that addition alone predicts. To get this position, we fit a linear regression, the additive model, to the attributed profiles of all personas:

y^i=c+βgender(i)+βage(i)+βorientation(i)+βSES(i)+βreligion(i)+βcaste(i).(2)\hat{y}_i=c+\beta_{\mathrm{gender}(i)}+\beta_{\mathrm{age}(i)}+\beta_{\mathrm{orientation}(i)}+\beta_{\mathrm{SES}(i)}+\beta_{\mathrm{religion}(i)}+\beta_{\mathrm{caste}(i)}. \tag*{(2)}

Here y^i\hat{y}_i is the predicted profile of persona ii and cc is a constant. Each β\beta is the least-squares effect of one of the persona’s six values, with its own coefficient for each value. For religion, for example, βreligion(i)\beta_{\mathrm{religion}(i)} is βChristian\beta_{\mathrm{Christian}} for a Christian persona and βHindu\beta_{\mathrm{Hindu}} for a Hindu one.

The additive model has no interaction terms. Any effect of a pair of values beyond the sum of their single effects therefore stays in the residual yi−y^iy_i-\hat{y}_i.

Ard, the additive residual distance δ\delta, is the length of the cell’s mean residual, the red segment in Figure 1:

δ(a∧b)=∥1nab∑i∈a∧b(yi−y^i)∥.(3)\delta(a\wedge b)=\left\lVert\frac{1}{n_{ab}}\sum_{i\in a\wedge b}(y_i-\hat{y}_i)\right\rVert. \tag*{(3)}

Take a simple case. The personas differ only in the two attributes of the cell, and every value of one attribute appears equally often with every value of the other (for example, every gender with every orientation). Then δ\delta equals the length of the interaction Δab\Delta_{ab} from §3.1, the standard interaction contrast [63]:

δ(a∧b)=∥Δab∥=∥yˉa∧b−yˉa−yˉb+yˉ∥.(4)\delta(a\wedge b)=\lVert\Delta_{ab}\rVert=\lVert\bar{y}_{a\wedge b}-\bar{y}_a-\bar{y}_b+\bar{y}\rVert. \tag*{(4)}

If Δab=0\Delta_{ab}=0, δ\delta is zero up to random noise. Otherwise it grows with ∥Δab∥\lVert\Delta_{ab}\rVert, whichever way the cell moves (Figure 1b,Figure 1c). Unlike the excess, δ\delta does not depend on the angle between uau_a and ubu_b. Our personas differ in all six attributes, so the additive model includes all six. If a cell happened to hold more poor personas than usual, a model of its two attributes alone would mistake the shift of being poor for an interaction. The full model removes that shift, and stays correct when cells differ in size (Appendix § 11).

Error Control Across Many Sparse Cells

We test 738 cells at once, and their sizes nabn_{ab} differ more than twentyfold. Two problems follow. Small cells have larger variance, so a single threshold on raw δ\delta would be set by the small cells. And with hundreds of tests, some cell will exceed any fixed threshold by chance. We therefore need a null that asks the right question, a score comparable across cells, and a rule that controls the error rate over all cells jointly.

Hypotheses. For each cell a∧ba \wedge b, we ask whether the two values interact. The null hypothesis H0:Δab=0H_0:\Delta_{ab}=0 states that the cell sits exactly at its additive prediction, while each value may still shift the answers on its own by any amount, and the alternative Δab≠0\Delta_{ab} \ne0 states that the cell sits elsewhere. This differs from the shuffle null of § 3.2, whose null hypothesis is that identity has no effect at all.

Procedure. The test has four steps.

  1. Build null datasets. Following Freedman and Lane [17] and Winkler et al. [69], we keep each persona’s fitted part y^i\hat{y}_i, flip the sign of its residual yi−y^iy_i-\hat{y}_i at random, and add the two back together. The main effects are untouched, while any interaction, which lives in the residuals of its cell, is cancelled by the flips, and each persona’s noise keeps its size (Appendix § 13). We call this the Freedman–Lane null.

  2. Recompute the metric. On each of 2,000 null datasets, we refit the additive model and recompute δ\delta for every cell.

  3. Standardize. We convert each cell’s observed δ\delta into zab=(δ(a∧b)−mab)/sabz_{ab}=(\delta(a\wedge b)-m_{ab})/s_{ab}, where mabm_{ab} and sabs_{ab} are the mean and standard deviation of the observed δ\delta pooled with its 2,000 null values, which removes the dependence on cell size.

  4. Apply the max-zz rule. In each null dataset, we record the largest zz over all cells, and a cell is significant if its observed zabz_{ab} exceeds 95% of these maxima.

Addis is Ard together with the Freedman–Lane null and the max-zz rule.

Why this controls the error rate. The threshold is the 95th percentile of the largest zz over all cells, so under H0H_0 the chance that any cell exceeds it, the FWER, is designed to be at most 5%. This single-step rule approximates the min-PP procedure of Westfall and Young [67] and rests on two assumptions, namely that residuals of different personas are independent and that each is as likely positive as negative (Appendix § 13). We therefore check the FWER empirically. It stays near 5% on continuous answers and on discrete Likert noise, but not when an additive signal is rounded to a few answer categories, because rounding creates real interactions on the answer scale, which every method detects (§ 6).

Tested cells. Small intersectional groups make fairness hard to verify statistically [15, 25], which is why we standardize each cell, require nab≥50n_{ab} \ge50, and control the error rate across all of CC. The 47 caste × socioeconomic cells below this minimum are not tested (Appendix § 11), and the counterfactual quartets of § 3.7 test one caste × class interaction directly.

Representations. A representation turns a model’s answers about persona ii into the profile yiy_i on which we compute δ\delta, and we use two. In the item space, each question’s answer becomes one dimension, z-scored across personas, and profiles are compared with the distance of Cronbach and Gleser [10]. Items to which a model gives the same answer for every persona carry no information about identity and are dropped, so dd ranges from 66 to 82 depending on the model. In scale scores, each of the d=9d=9 dimensions is one trait scored with the standard key of its questionnaire, namely the five Big Five traits, need for cognition, empathic concern, anxiety and depression (Appendix § 19 reports each scale’s internal consistency per model). Scale scores carry less signal, so Addis flags fewer cells on them, and a finding that holds in both representations does not depend on how the answers are summarized.

A Benchmark With Known Truth

Real answers have no ground truth. When a method flags a cell, we cannot tell whether the flag is right. To compare methods, we need data in which we know which cells interact. Our benchmark builds such data from real answers with the plasmode approach [16, 64]. The plasmode approach starts from real data, keeps their structure, and plants a known truth.

We build every dataset three times, once from the answers of each of three models from different families (Llama 3 8B, Gemma 4 E4B, Ministral 8B). Each model thus contributes equally. Our main noise generator, group-mean, rebuilds each persona’s answer from group averages, the grand mean plus the shift of each of its six values, and adds back its own leftover with a random sign (Figure 3). It never fits the additive model (Equation (2)), which makes it an impartial check of Addis. Two further generators start from that model’s fit y^i\hat{y}_i. Flip flips the signs of the residuals yi−y^iy_i-\hat{y}_i at random, as in the Freedman–Lane null, and permute shuffles them across personas. Every generated dataset has real main effects and no interaction. We then plant interactions by shifting every persona in a few chosen cells by a known amount, measured in residual standard deviations. In some datasets we plant nothing.

The group-mean generator keeps every main effect and removes every interaction

Figure 3. The group-mean generator keeps every main effect and removes every interaction. Real answers of Llama 3 8B on the anxiety scale, in standard deviations. Each persona’s synthetic answer is the grand mean plus the shift of each of its six values, plus its own leftover with a random sign.

When nothing is planted, every flag is a false positive, so we measure each method’s FWER. Answers rounded to the Likert scale are the one exception (Appendix § 12). When interactions are planted, we know which cells a method should flag, so we also measure its recall and precision. We follow the ADEMP structure for simulation studies [41].

A scenario fixes how the answers are built and where interactions are planted. We report four scenarios, each testing a different risk: answers rounded to the Likert scale, an interaction in a single item, several interactions in one attribute pair, and the real gender identity ×\times orientation pattern (Appendix § 14).

The benchmark tests whether a method can find planted interactions. To find real interactions in language models, we need a dataset of personas.

The IndiPersona Dataset

IndiPersona is a set of synthetic Indian personas that combine six identity attributes: gender identity, age, sexual orientation, socioeconomic status, religion, and caste and community. Table 5 in Appendix § 15 lists the values of each attribute. Each cell needs a large nabn_{ab} to estimate δ\delta. We therefore assign the attributes independently of one another, except that religion follows caste. This trades realism for estimability. Real populations are correlated, so the personas show how models treat each combination of identity labels, not how bias falls on India’s population.

AttributennValues
Gender identity3man, woman, transgender person
Age5child, teenager, young adult, adult, senior citizen
Sexual orientation4heterosexual, gay, lesbian, queer
Socioeconomic status6poor working class, lower middle class, middle class, upper middle class, rich, ultra rich
Religion6Buddhist, Christian, Hindu, Jain, Muslim, Sikh
Caste and community31Varna: Brahmin, Kshatriya, Vaishya, Shudra; Jati: Nair, Ezhava, Reddy, Kamma, Kapu, Naidu, Lingayat, Vokkaliga, Maratha, Patel, Jat, Yadav, Dalit; Muslim: Ashraf, Ajlaf, Arzal, Sunni, Shia; Sikh: Jat Sikh, Khatri, Mazhabi; Christian: Catholic, Protestant; Administrative: OC, OBC, SC, ST

Table 5. Identity attributes in IndiPersona and their values.

Personas. We generate 10,000 personas. Gender, age, sexual orientation, socioeconomic status and caste are each assigned by quota, independently of one another, so every value appears equally often. Religion is then chosen among the religions in which the persona’s caste occurs [20, 49]. This rule avoids implausible combinations such as a Brahmin Muslim. Our caste list is a simplified working representation of caste, not a complete one. Assigning attributes independently also produces combinations that are contradictory, such as a man described as lesbian, or sensitive, such as a child with a sexual orientation. We keep them so that every cell can be estimated, but a flag in such a cell may reflect the model’s reaction to the contradiction rather than an intersectional effect. None of the three main cells (§ 6.3) is such a combination.

Templates. Each persona appears in a short scene, such as a queue at a public office or a ferry landing (Figure 5 in Appendix § 16). We use 50 templates, so that no single scene’s wording drives the answers. Each has a fixed opening and one sentence per attribute. Gender comes first, and the other five attributes follow in a random order for each persona. A few templates add details that could hint at some traits, so we do not use those templates with the questionnaires they could affect (Appendix § 16). Templates are assigned at random, independently of identity.

Prompt from IndiPersona

Figure 5. A prompt from IndiPersona. The six identity values are highlighted, and the fixed opening of the template is in italics. Gender comes first, and the other five sentences appear in a per-persona random order. Symptom items use the GAD-7 and PHQ-8 wording and the 0–3 scale instead.

Questions and answer order. The model rates each persona on 82 items from five questionnaires, one item per prompt (Appendix § 16, Table 6). Every item is rewritten in the third person, so that the model rates a described person instead of reporting on itself. “I am the life of the party” becomes “This person is the life of the party”. Answer options always appear in ascending order.

QuestionnaireItemsAnswer scale (ascending)
IPIP Big Five markers [19]501–5, strongly disagree to strongly agree
IPIP Need for Cognition81–5, strongly disagree to strongly agree
IPIP Empathic Concern91–5, strongly disagree to strongly agree
GAD-7 anxiety [58]70–3, not at all to nearly every day
PHQ-8 depression [33]80–3, not at all to nearly every day

Table 6. The five questionnaires, in the order of their answer scales.

Control arms. IndiPersona also holds three control arms for our robustness checks: a fresh sample of personas, two rewordings of every prompt, and personas without identity attributes (Appendix § 20).

Counterfactual Quartets

ADDIS shows which cells of the main personas interact, but not why. Each cell mixes the effect of its two labels with the other four values of its personas. Counterfactual quartets remove this mixing. We take 250 base personas per contrast and write each one four times, in a 2 × 2 design that changes two labels and nothing else. For example, one base persona is written as a rich Brahmin, a rich Dalit, a poor working class Brahmin and a poor working class Dalit, with every other detail the same. The interaction is the difference between two changes: the move from rich to poor working class for the Dalit version, minus the same move for the Brahmin version. The rest of the persona stays fixed, so this within-persona difference-in-differences estimates the causal interaction of the two labels in the prompt [13]. Quartets also help where the main personas are too sparse. With 31 castes, the cell Dalit × poor working class has nabn_{ab} of only about 50 to 80. ADDIS cannot detect most interactions in a cell this small, but the quartet above tests the two labels directly. We use four contrasts: the main gender identity × orientation interaction, a check that this interaction is specific to the transgender label, a caste interaction, and a calibration contrast where we expect little interaction (Appendix § 21).

Experimental Setup

We run Addis on 14 models and check four things in § 5. We check whether Addis keeps its FWER near 5% on the real design (Appendix § 17.§ 1), and whether the interactions it finds are caused by the identity labels in the prompt rather than by the other values of the personas in a cell (§ 6.4). We also check whether its findings survive changes to the prompt, the sample and the representation (Appendix § 17.§ 4), and how it compares with the published alternatives (§ 6 and Appendix § 17.§ 5).

Models. We audit 13 open instruct models from the Gemma, Llama, Ministral, Phi and Qwen families, with up to 24B parameters, and DeepSeek V4.1 Flash (all 14 are listed in Figure 6). Each model answers at temperature 0, with thinking off, and returns only a number. Because a model whose answers barely change with the persona cannot show bias, we analyze a model only if it passes a persona-sensitivity check, which all 14 pass (Appendix § 22).

Figure 6. The single-attribute baseline’s excess is positive for most cells, while ADDIS flags few. Real answers, 738 cells per model.

Published alternatives. We compare our test with three kinds of published methods, each run as published with its own null, correction and decision rule. The single-attribute baseline (Equation (1)) is run in two forms, flagging every cell with a positive excess or only cells whose excess is significant under the shuffle null (§ 3.2); in § 5.1 we also pair the excess with our Freedman–Lane null to separate the effect of the null from that of the statistic. Tests of departure from addition test for an interaction, as our test does. Per-item regression fits an interaction term for each item, as in LLM audits [31], with a Bonferroni correction across items and cells, and we run it with classical standard errors, as published, and with HC3 standard errors [39], which allow the noise to differ between cells. MAIHDA, a standard intersectional method in social epidemiology, adds a random intercept for every subgroup to the additive model [14]. PERMANOVA, a standard permutation test for multivariate data, tests the whole interaction term of each attribute pair [2], so it cannot say which cell in the pair interacts. Metamorphic testing with HInter reports a hidden failure when swapping two identity words changes the answer but swapping either word alone does not [57]; because our counterfactual quartets have the same 2×22 \times2 structure, we apply HInter’s rule to every item of the quartets (Appendix § 17.§ 5).

Results and Discussion

Our results support three claims. First, Addis keeps its FWER near 5% on continuous answers. The single-attribute baseline has a high error rate, because it flags cells whose attributes only add (§§§ 5.1–§ 6). Second, few cells pass Addis on real answers. In every model, the flagged cells concentrate on gender identity ×\times sexual orientation (§ 6.3). Third, changing the labels reproduces the gender identity ×\times orientation interaction, but the direction of the interaction depends on the prompt (§§§ 6.3–§ 6.4). Appendix § 17.§ 5 compares the published alternatives on real answers.

The Single-Attribute Baseline on Synthetic Data

We first test the single-attribute baseline on synthetic data. § 5.1.1 measures its false positive rate and the role of its null. § 5.1.2 measures its recall and precision, and how both depend on the direction of the interaction.

Where the Single-Attribute Baseline’s False Positives Come From

We use the 500 datasets with no interaction from Appendix § 17.§ 1. These datasets are built from the real answers of Llama 3 8B with the group-mean generator. As Proposition 3.1 predicts, the excess (Equation (1)) is positive for 89.5% of cells in the item space and for 75.5% on scale scores. The excess is therefore a biased measure of interaction. With no interaction present, its expected value is positive, not zero. To correct this bias, the null distribution must carry the same positive bias. The shuffle null does not, because it removes the main effects that make the excess positive. On the benchmark, the excess with the shuffle null flags at least one cell in every dataset, and 66.1 cells per dataset on average (Table 2).

StatisticNullFlaggedCellsRecallPrec.
Excess (Eq. 1)Shuffle100%66.147.9%0.06
Excess (Eq. 1)Freedman–Lane2.7%0.0332.4%0.99
Ard (δ\delta)Freedman–Lane8.7%0.0984.4%0.99

Table 2. The Freedman–Lane null removes the single-attribute baseline’s false positives, but not its misses. Scale scores, group-mean generator. Flagged, Cells: datasets flagged and false cells per dataset, nothing planted. Recall and precision: the real gender identity × orientation pattern at one residual SD.

With the Freedman–Lane null, the result reverses. The Freedman–Lane null keeps every main effect, so the bias appears in the null too and cancels. The excess now flags only 2.7% of datasets instead of all of them. Appendix § 17.§ 2 reports the same comparison on Likert-rounded answers. There, the repaired excess keeps a higher precision than Addis, because it finds few interactions of any kind. Repairing the null fixes the false positive rate (Appendix § 23 tests a second simple repair). It does not change what the excess measures: distance from the population mean, not distance from the additive prediction (Figure 1c,Figure 1d). § 5.1.2 measures how many interactions the repaired excess misses as a result.

Recall by the Direction of the Interaction

Even with the Freedman–Lane null, the excess misses many interactions. On the real gender identity × orientation pattern, the repaired excess recovers 32.4% of the planted cells, against 84.4% for δ\delta (Table 2). Both keep a precision near 0.99, so only recall separates them. The direction of the planted interaction explains the gap. We plant the same interaction twice in the same cell (Table 3). The first time, it points along the cell’s additive shift from the population mean. The second time, it points at a right angle to that shift. The excess detects the first interaction but misses most of the second. δ\delta detects both.

SizeDirectionExcessARD
0.4Along the shift43%38%
0.4At a right angle0%53%
0.8Along the shift93%90%
0.8At a right angle35%97%

Table 3. The excess finds an interaction along the cell’s shift and misses one at a right angle. Recall on scale scores, group-mean generator, Freedman–Lane null. Planted size in residual SDs.

The right-angle case is the one intersectionality theory is about. An interaction at a right angle means that the combination does something neither part does. In a high-dimensional space such as the nine scale scores, most directions are close to perpendicular. The right-angle case is therefore also the most common one.

Published Alternatives on Synthetic Data

We next compare Addis with the published alternatives (§ 4) on the benchmark (§ 3.5). § 6.1 compares their false positive rates. § 6.2 compares their recall and precision when we plant known interactions.

Comparing False Positive Rates Across Methods

We use benchmark datasets with no interaction, built from the answers of three models with the group-mean generator. On continuous answers, Addis flags 8.7% of datasets on scale scores (95% CI 4.7–24.4) and 4.7% in the item space (Table 4a). Per-item regression with HC3 standard errors is the exception. It holds its FWER about as well as Addis. With classical standard errors, as published, per-item regression makes a false positive in every item-space dataset. Appendix § 17.§ 1 reports a finer calibration of Addis.

(a) Continuous answers
MethodFlagged: ScaleFlagged: ItemRecall / prec.: Real patternRecall / prec.: Single item
Addis8.7%4.7%84.4% / 0.9953.1% / 0.98
Per-item HC39.3%3.3%75.6% / 0.9897.1% / 0.53
Per-item classical36.0%100%77.6% / 0.9399.2% / 0.15
(b) Likert-rounded answers
MethodFlaggedRecall / prec.
Addis20.0%78.1% / 0.72
Per-item HC399.3%58.8% / 0.20
Per-item classical100%85.9% / 0.20
MAIHDA96.7%76.0% / 0.28
PERMANOVA73.3%72.7% / 0.02

Table 4. Addis keeps the highest precision in every setting. Flagged: datasets with any flag when nothing is planted. Recall / prec.: interactions planted at one residual SD in (a), two in (b). Per-item regression uses Bonferroni. (a) Group-mean generator; real pattern on scale scores, single item in the item space. (b) Permute generator (Appendix § 12). MAIHDA and PERMANOVA ran here only, on 30 datasets, as they are slow (Appendix § 17.§ 5).

Likert-rounded answers. Real answers are whole numbers on a four-point or five-point scale. One benchmark scenario therefore rounds its continuous answers to these categories (§ 3.5). Rounding a measurement scale is known to create departures from additivity [37, 66]. A few cells therefore depart from the additive prediction even when nothing is planted. In an average dataset, about 3 of the 738 cells depart enough for Addis to detect. Appendix § 12 describes the construction and how we find these cells.

Rounding weakens the error control of every method we test. Addis weakens least. It flags at least one cell in 20.0% of the 150 rounded datasets. Every published alternative we tried exceeds 96%, except PERMANOVA at 73% (Table 4b). On rounded data that copy the real design, the FWER of Addis also rises above 5%. A control shows that rounding, not Addis, causes these extra flags (Appendix § 17.§ 2).

Recall and Precision Across Methods

Addis has high precision and recall in every setting but one (Table 4). It loses recall when a single item carries the whole interaction. Ard measures one distance across all items, so a change in one item is diluted. Per-item regression tests each item separately. It finds more of these single-item cells, but at about half the precision of Addis.

Interactions Found in the Models’ Answers

We now run Addis on the real answers of the 14 models about the IndiPersona personas (§ 3.6).

How many cells interact. Addis flags few cells in every model. Averaged over the 13 open models, it flags 22 of the 738 cells on scale scores (3.0%) and 55 in the item space (7.5%). For DeepSeek V4.1 Flash, it flags 11 and 56 cells (Appendix § 17.§ 3). The single-attribute baseline reports far more cells. Its excess is positive for 75–83% of cells on scale scores. This share is about the same as on synthetic data with no interaction (§ 5.1.1). Under the shuffle null (§ 3.2), the baseline still flags 52 cells per model, more than twice as many as Addis. The two tests also disagree on which cells interact. Only about 9 cells per model are flagged by both. Part of this gap lies out of the excess’s reach. In every model, some cells flagged by Addis have a negative excess, as in Figure 1(d). No threshold on the excess can flag these cells. In DeepSeek, one such cell is transgender person ×\times senior citizen, its second-strongest interaction in the item space.

Which cells interact. On scale scores, gender identity crossed with sexual orientation is the largest pairing in every model (Appendix § 17.§ 3). The 12 cells of this pairing are under 2% of the tested cells but carry a third of all flags. These flags are not independent findings, because one strong cell shifts the residuals of its row and column (Appendix § 24). Three of these cells interact in nearly every model. Transgender person ×\times queer and transgender person ×\times heterosexual interact in all 13 open models in both representations. Man ×\times queer interacts in 12, and DeepSeek shows all three. We call these three cells the main cells. In the item space, each main cell sits two thirds to one residual standard deviation from its additive prediction. To check that the main cells are not an accident of one sample, we rerun five of the models on a fresh sample of 5,000 personas drawn the same way (§ 3.6). All three main cells are flagged again in all five models in the item space. On scale scores, the two transgender cells are flagged again in all five models, and man ×\times queer in four (Appendix § 17.§ 3, Table 11). Per-item regression and MAIHDA also flag all three main cells in at least 12 of the 13 models (Appendix § 17.§ 5).

CellItem space δsd\delta_{\mathrm{sd}}Item space zzItem space Repl.Scale scores δsd\delta_{\mathrm{sd}}Scale scores zzScale scores Repl.
Transgender person ×\times queer0.93 [0.42, 1.70]24.3 [7.9, 34.9]5/50.44 [0.19, 0.89]17.2 [4.9, 30.0]5/5
Transgender person ×\times heterosexual0.90 [0.56, 2.26]25.8 [18.3, 39.2]5/50.31 [0.19, 0.94]14.2 [6.8, 33.7]5/5
Man ×\times queer0.66 [0.33, 0.98]15.4 [4.0, 22.4]5/50.29 [0.12, 0.60]10.1 [2.3, 21.1]4/5

Table 11. The main cells are large and replicate. Real answers. δsd\delta_{\mathrm{sd}} is δ\delta in residual standard deviations, the unit of the planted sizes in the benchmark. δsd\delta_{\mathrm{sd}} and zz are the median [min, max] over the 13 open models. Repl. is the number of the five replication models in which the cell is significant again on the fresh sample of 5,000 personas, out of those in which it is significant in the main sample.

The item space carries more information, so it shows a wider pattern. In the item space, transgender person × poor working class, poor working class × heterosexual and ultra rich × heterosexual also interact in all 13 models. Gender identity × orientation is the largest pairing in 8 of these models. Appendix § 17.§ 3 lists every significant cell.

The direction of the interactions. On scale scores, the departure measured by δ\delta is a vector with one dimension per trait. Each dimension is the cell’s interaction on that trait. For every model and cell, we can therefore see which traits carry the interaction (Figure 4). The excess compares two lengths and keeps no direction, so the single-attribute baseline cannot show which traits move. Under our main wording, models attribute more extraversion, agreeableness and empathic concern to queer transgender personas than their two labels predict, and less anxiety. The rise in extraversion is significant in all 13 open models. The other two main cells lean the other way, toward less extraversion and more anxiety.

Figure 4. C1 moves the same traits as transgender person × queer, and C3 moves conscientiousness, extraversion and neuroticism. Median shift over 13 open models, in SD of each scale. Below: models significantly positive / negative. The two blocks are standardised separately, so compare signs, not sizes.

In questionnaire points, these shifts are small. A typical model gives a queer transgender persona 0.28 fewer points of anxiety on the GAD-7 (range 0 to 21) than its two labels predict. All 13 models move anxiety the same way. δ\delta is still large, because the departure spreads over many scales and items at once.

Not all of these results are robust. When every prompt is reworded, the drop in anxiety for queer transgender personas weakens or disappears in most models. Across all flagged cells, only about half reappear on the fresh sample (Appendix § 17.§ 4).

Caste. No caste cell interacts in more than 4 of the 13 open models. Caste cells are too small for ADDIS to measure well. We therefore test caste with counterfactual quartets in § 6.4.

Results From the Counterfactual Quartets

The counterfactual quartets (§ 3.7) change only two labels in each persona and hold everything else fixed. Any interaction in a quartet is therefore caused by those two labels, within the model’s prompts.

We use contrast C4 as the reference. C4 changes adult to senior citizen and middle class to upper middle class. These are two small steps between neighbouring levels, and we do not expect them to interact. The C4 interaction is small but not zero. We use the C4 interaction as a yardstick for the size of the other contrasts.

C1 changes man to transgender person and heterosexual to queer. C1 thus tests the main interaction of § 6.3, a cell ADDIS had already flagged, so it confirms that finding rather than discovering a new one. C1 reproduces this interaction in all 13 open models, with contrast values about five times as large as C4. C1 also points the same way as the audit of the main personas: more extraversion in all 13 models and less anxiety in 10 (Figure 4). C2 makes the same change with woman in place of transgender person. C2 is larger than C4 with 95% bootstrap confidence in only 5 of the 13 models, against all 13 for C1.

C3 changes Brahmin to Dalit and rich to poor working class. The C3 interaction is about as large as the C1 interaction, and larger than C4 in 12 of the 13 models. Models attribute more conscientiousness and less extraversion and neuroticism to a poor working-class Dalit persona than its two labels predict. Anxiety and depression move in different directions in different models. Caste cells showed no interaction in § 6.3 because they are small (§ 3.7), not because caste labels do not interact. Auditors who care about caste should use targeted counterfactuals.

Conclusion

These results suggest three practices for intersectional audits of language models.

Use an additive baseline. Without an additive baseline, an audit mistakes cells whose attributes only add for interactions. The audit also misses real interactions whose direction barely changes the cell’s distance from the population mean.

Use a null that keeps the main effects. The shuffle null removes each attribute’s own effect along with any interaction, so almost any cell can beat it. A null that keeps the main effects, such as the Freedman–Lane null, avoids this problem.

Test small cells with counterfactuals. Attributes with many values, such as caste, give few personas per cell. Counterfactual quartets can test the labels of such cells directly.

None of these practices measure harm. Whether a flagged cell harms people depends on the outcome and the setting. An audit has to judge harm separately.

Limitations and Future Work

Our personas are synthetic, so they show how models treat each combination of identity labels, not how bias falls on India’s population. Whether two values add depends on the scale on which we measure the answers. The share of intersectional cells therefore differs between the item space and scale scores, although the main interactions sit in the same place in both. Flagged cells are non-additive on the answer scale, which can include effects of the answer format near its floor or ceiling, and a flagged pair may also involve a third attribute (Appendix § 17.§ 4). Our test is also not calibrated on Likert-rounded benchmark data, where its FWER is 20.0%, so single flagged cells on Likert answers should be confirmed with an ordinal model. Future work should extend the test from pairs of attributes to combinations of three or more, and apply it to real human answers to the same questionnaires, to compare how people and language models treat the same combinations of identities.

Ethical Considerations and Adverse Impacts

This work involves no human subjects and no personal data, since every persona is generated from templates and every answer comes from a language model. The anxiety and depression items ask a model what a described person would answer, so they are attributions, not screenings, and must not be read as clinical claims about any group. Our caste list is a simplified working representation and may not match how communities describe themselves. The main risk is misreading our results: a flagged cell describes how models treat a combination of labels, not real people, and a cell our test does not flag is not shown to be free of harm, only to hold no interaction large enough to detect. We will release our code and the model answers on GitHub upon publication, as audit evidence, not as descriptions of real groups or as training data.

Manuscript submitted to ACM

Generative AI Usage Statement

We used Claude Opus 5.5 in Claude Code to search the literature for § 2, to prototype code, and to proofread and reword our own sentences. We checked every cited paper, tested all code behind our results, and reviewed every edit. The ideas, analyses and claims are ours, and we take full responsibility for them.

References

Proof of Proposition 3.1

Let uu and vv be the shifts of values aa and bb from the population’s mean profile. Under exact additivity, the cell a∧ba \wedge b shifts by u+vu+v, so its gap is ∥u+v∥\lVert u+v\rVert. The gaps of its parts are ∥u∥\lVert u\rVert and ∥v∥\lVert v\rVert.

Proof. Assume, without loss of generality, that ∥u∥≥∥v∥\lVert u\rVert\ge\lVert v\rVert, so the larger part has gap ∥u∥\lVert u\rVert. The excess is then positive exactly when ∥u+v∥>∥u∥\lVert u+v\rVert> \lVert u\rVert. Both sides are non-negative, so we can compare their squares:

∥u+v∥2−∥u∥2=∥v∥2+2⟨u,v⟩\lVert u+v\rVert^{2} - \lVert u\rVert^{2} = \lVert v\rVert^{2} + 2\langle u,v\rangle

This difference is positive exactly when ⟨u,v⟩>−∥v∥2/2\langle u,v\rangle> -\lVert v\rVert^{2}/2. Because ∥v∥≤∥u∥\lVert v\rVert\le\lVert u\rVert, this is the condition

⟨u,v⟩>−12min⁡(∥u∥2,∥v∥2)\langle u,v\rangle> -\frac{1}{2}\min(\lVert u\rVert^{2},\lVert v\rVert^{2})

□\square

Two cases follow directly. When the shifts are orthogonal or at an acute angle, ⟨u,v⟩≥0\langle u,v\rangle\ge0, so the excess is always positive although nothing interacts. When the shifts point against each other strongly enough, the excess is negative, so the criterion cannot flag the cell, whatever its interaction.

Why the Regression Uses All Six Attributes

The personas in a cell share two values, but they also carry four other values each. If, by chance, a cell held more poor personas than usual, its mean would also include the shift of being poor. A model of the two cell attributes alone would then mistake that shift for an interaction. The full regression already accounts for each persona’s other values, so only what the two cell values add together remains in the residual. For the same reason, the prediction stays correct when cells differ in size, which they do in our data.

Tested cells and identifiability. The six attributes form 971 pairwise combinations of values. We do not test caste ×\times religion, because religion follows caste by design: 148 of its combinations are empty and the other 38 are left out. Of the rest, 47 caste ×\times socioeconomic cells have fewer than 50 personas, which leaves 738 tested cells, the same in every model. The additive design has 50 columns but rank 48, because caste is nested in religion, so a few caste and religion coefficients are not identified separately. The fitted values and residuals are still unique, since the fit is a projection, and δ\delta, the null and every test use only these.

The Likert-Rounded Benchmark Scenario

How the data are built. This scenario is part of the synthetic benchmark of § 3.5. Each dataset starts from the real answers of one of three models (Llama 3 8B, Gemma 4 E4B, Ministral 8B) in the item space and uses the permute noise generator. The answers are first built on a hidden continuous scale, as the sum of the main effects and the noise. They are then rounded back to each item’s answer categories at cut points fixed for the dataset, so they look like real Likert answers. We build 50 datasets with no interaction per model, 150 in all. MAIHDA and PERMANOVA ran on 30 of them. Our test uses 500 null draws in the benchmark. On 45 of these datasets, 2,000 draws gave almost the same result (9 false alarms against 8).

Why rounding creates interactions. Rounding is not additive. Suppose an item’s hidden score is 2.2, the cut point between answers 2 and 3 is 2.5, and each of two values adds 0.2. Either value alone gives 2.4, which still rounds to 2, but both together give 2.6, which rounds to 3. On the answer scale, the two values then interact, although nothing was planted. Some departure of this kind appears in almost every cell (660–724 of the 738). Most of these departures are far too small to detect.

Threshold cells. We call a cell a threshold cell when rounding alone creates an interaction large enough that our test is expected to detect it. To find these cells, we compute the expected rounded answers with no noise and measure each cell’s departure from addition. A cell is a threshold cell when this departure, scaled by our test’s null distribution, gives an expected zz of at least 2. A dataset has 3.35 threshold cells on average (median 2, range 1–8). The number depends on the model, from 1 for Gemma 4 E4B to 5–8 for Ministral 8B. Every method is scored against the same threshold cells.

How we score. In the primary score, a flag in a threshold cell does not count as false, and our FWER is 20.0%. In the strict score, every flag counts as false, and our FWER is 40.0%. The cutoff of 2 is a choice. With a cutoff of 1, a dataset has 14.1 threshold cells on average, and our FWER falls to 12.7%. Per-item regression stays at 96–99% under either cutoff. The cutoff was fixed in our code before any benchmark run. The pre-registration, however, does not state the cutoff, and it says that every flag on data with no interaction is false. Read literally, it makes the strict 40.0% the pre-registered figure, so we report both.

On the real design. We repeat this check on our real design, for one model from each family. We round each null dataset of Appendix H.1 to the answer grid and run our test. Our test then flags 18–100% of datasets under the strict score and 7.5–56% under the primary score (20 settings, 200 datasets each). A control shows why. Keeping the discrete, clipped noise of the rounded data but replacing its expected values with their additive projection, our test is back at 3.0–6.5% in 13 settings, and every interval contains 5%. The test is therefore calibrated on Likert noise, and the excess comes from real interactions that rounding creates on the answer scale.

How the Sign-Flip Null Works

Why the flips remove only the interaction. The additive model of § 3.3 splits each persona’s answers into a fitted part, the sum of its six values’ shifts, and a residual, everything the model cannot explain. We keep the fitted part unchanged, so every main effect stays in each null dataset. A real interaction cannot be fitted by the additive model, so it stays in the residuals of the personas in its cell, pushing them all the same way. When we flip the sign of each residual at random, about half of these residuals point one way and half the other. They then cancel, and the interaction disappears. The noise of each persona keeps its size, because a flip changes only its sign.

Details. Before flipping, we rescale the residuals with the HC2 leverage correction of MacKinnon and White [39], so that each persona keeps its own noise level. We use the +1 convention for pp-values [47]. The main analysis uses 2,000 null datasets, the calibration of § 6.1 uses 1,000, and the benchmark uses 500 (Appendix C).

Assumptions. The flips rest on two assumptions: the residuals of different personas are independent, and each residual is as likely to be positive as negative. Personas that share one of our 50 templates may have related residuals. Adding the template to the additive model leaves the results and the calibration nearly unchanged (Appendix H.1). The permute and group-mean generators break both, and our test still holds its error rate on them (§ 6.1).

Benchmark Scenarios

Each scenario of § 3.5 tests a different risk.

Likert-rounded. We build answers on a continuous scale and round them to each item’s answer categories, as in real questionnaires. Rounding itself bends an additive signal, so this is the hardest case for every method (Appendix C).

Single item. The whole interaction sits in one item. This favours per-item tests and shows where our test loses power.

Several in one pair. Several planted cells share one attribute pair, as in the real gender identity × orientation table, where cells in one table share signal.

Real pattern. We plant the gender identity × orientation cells our test flags in real answers, at their real sizes. Because our test chose these cells, they carry the kind of signal it detects well, so this scenario shows realistic structure rather than a fair contest.

Further scenarios. The benchmark also plants one, five or twenty randomly chosen cells, each with its own random direction. Appendix I gives their recall.

Identity Attributes in IndiPersona

Table 5 lists the six identity attributes in IndiPersona and their values (Section 3.6).

Questionnaires and an Example Prompt

Table 6 lists the five questionnaires and their answer scales. Agreement items run from 1 (strongly disagree) to 5 (strongly agree), and symptom items from 0 (not at all) to 3 (nearly every day). Figure 5 shows one full prompt.

Templates. Of our 50 templates, 30 describe only the setting, and 20 add detail, such as a well-stocked study, that could hint at some traits. Each questionnaire uses the 30 plain templates and only those detailed ones whose detail does not bear on it. The Big Five questionnaire therefore uses the 30 plain templates only, and each other questionnaire uses 44 to 48, that is, the 30 plain ones and 14 to 18 detailed ones.

Extended Results

Calibration on the Real Design

We calibrate on one model from each of the five families. Each setting uses 500 datasets with no interaction, built from one model’s real answers, and 1,000 null draws per dataset. Table 7 gives the FWER of our test in each setting, with a 95% Clopper–Pearson interval.

ModelRepresentationGeneratorFWER95% interval
Llama 3 8Bitem spaceflip2.6%1.4–4.4%
Llama 3 8Bitem spacepermute4.2%2.6–6.3%
Llama 3 8Bitem spacegroup-mean3.0%1.7–4.9%
Llama 3 8Bscale scoresflip5.2%3.4–7.5%
Llama 3 8Bscale scorespermute5.6%3.8–8.0%
Llama 3 8Bscale scoresgroup-mean5.4%3.6–7.8%
Gemma 4 E4Bitem spaceflip4.2%2.6–6.3%
Gemma 4 E4Bitem spacepermute5.0%3.3–7.3%
Gemma 4 E4Bscale scoresflip5.0%3.3–7.3%
Gemma 4 E4Bscale scorespermute3.8%2.3–5.9%
Ministral 14Bitem spaceflip5.4%3.6–7.8%
Ministral 14Bitem spacepermute7.0%4.9–9.6%
Ministral 14Bscale scoresflip5.2%3.4–7.5%
Ministral 14Bscale scorespermute5.6%3.8–8.0%
Phi-4item spaceflip5.0%3.3–7.3%
Phi-4item spacepermute5.2%3.4–7.5%
Phi-4scale scoresflip5.4%3.6–7.8%
Phi-4scale scorespermute5.2%3.4–7.5%
Qwen2.5 14Bitem spaceflip4.2%2.6–6.3%
Qwen2.5 14Bitem spacepermute4.2%2.6–6.3%
Qwen2.5 14Bscale scoresflip4.6%2.9–6.8%
Qwen2.5 14Bscale scorespermute4.6%2.9–6.8%

Table 7. FWER of our test on the real design, at nominal 5%. Synthetic data with the real main effects, cell sizes and noise of one model, and no interaction; 500 datasets per setting.

With the template in the model. We repeat the calibration with each persona’s template added to the additive model, for three of these models (Llama 3 8B, Ministral 14B, Qwen2.5 14B), both representations, and the flip and permute generators (12 settings, 200 datasets each). The FWER ranges from 3.5% to 9.0%, and 11 of the 12 confidence intervals contain 5%. The exception is Llama 3 8B on scale scores under the flip generator, at 9.0% (interval 5.4–13.9%), which is slightly above the nominal level.

Sign flips by template. As a further check, we flip one sign per template instead of one per persona. On real answers, this keeps 75–100% of the flagged cells, and the main cells stay significant in every model where they were flagged,

Prompt

Based only on the description below, how much do you agree with the following statement about this person?

The ferry was due shortly, and those crossing the river had gathered at the landing, watching for the boat to come into view. Near the front of the small crowd waited a woman, ready to board when the ferry arrived. When origins came up among those waiting, the family’s place among the Kamma was mentioned. Going by looks, the traveller was a young adult, a fact no one at the landing thought twice about. The look of the party suggested a household of lower middle class circumstances. A relative waiting alongside spoke of the traveller as lesbian, plainly and in passing. From a brief exchange it was apparent the traveller had grown up Hindu.

Statement: “This person is the life of the party.”

Reply with a single number.

1=Strongly Disagree, 2=Disagree, 3=Neither agree nor disagree, 4=Agree, 5=Strongly Agree

except transgender person × queer on scale scores in Qwen2.5 7B, its weakest model. This null is less reliable, however: with only about 215 independent signs, its FWER ranges from 1.0% to 9.5% across the settings we calibrated, and in one setting its interval lies fully above 5% (9.5%, interval 5.8–14.4%). We therefore keep the per-persona null.

Results on Likert-Rounded Answers

Real answers are whole numbers on each item’s answer scale. Rounding an additive score does not give an additive answer, because of each item’s floor and ceiling and because of the rounding step itself. Rounded data therefore contain a few real interactions on the answer scale even when the underlying scores add (Appendix § 12).

The single-attribute baseline. Table 8 repeats the comparison of Table 2 on the Likert-rounded scenario of the benchmark, which uses the permute generator (Appendix § 12). With the shuffle null the excess still flags every dataset. With our null it flags fewer datasets than our test and keeps a higher precision. It does so because it finds few interactions of any kind. At one residual standard deviation it recovers 5.2% of the planted cells, against 29.5% for our test.

StatisticNullDatasets flagged, nothing plantedRecall, 1 SDRecall, 2 SDPrecision, 2 SD
Excess (Eq. (1))Shuffle100%15.6%66.0%0.18
Excess (Eq. (1))Freedman–Lane5.3%5.2%69.2%0.98
δ\delta (ours)Freedman–Lane20.0% [40.0%]29.5%78.1%0.72

Table 8. On Likert-rounded answers, the excess with our null flags few cells of any kind. Likert-rounded item-space answers from the benchmark, with five randomly chosen cells planted. The bracketed figure counts flags in the cells that rounding moves as false.

Our test on rounded data that copy the real design. We also calibrate our test on null data rounded to the answer scale. We use the five models of Appendix § 17.§ 1 with the flip and permute generators, in both representations, which gives 20 settings of 200 datasets each. Each dataset is built as in Appendix § 17.§ 1 from the model’s raw answers, and every value is then rounded to the nearest valid answer of its item and clipped to the item’s range. Our test flags at least one cell in 18–100% of these datasets, and in 7.5–56% once flags in threshold cells are not counted (Table 9; threshold cells as in Appendix § 12). In 19 of the 20 settings the 95% interval lies above 5%, because these null data contain real interactions on the answer scale and our test detects them.

ModelRepresentationGeneratorStrict FWERPrimary FWERPrimary 95% intervalControl FWERControl 95% interval
Llama 3 8Bitem spaceflip100.0%21.5%16.0–27.8%6.0%3.1–10.2%
Llama 3 8Bitem spacepermute96.0%16.0%11.2–21.8%5.0%2.4–9.0%
Llama 3 8Bscale scoresflip100.0%28.0%21.9–34.8%4.5%2.1–8.4%
Llama 3 8Bscale scorespermute71.5%12.0%7.8–17.3%4.5%2.1–8.4%
Gemma 4 E4Bitem spaceflip100.0%22.5%16.9–28.9%––
Gemma 4 E4Bitem spacepermute78.0%56.0%48.8–63.0%4.0%1.7–7.7%
Gemma 4 E4Bscale scoresflip99.0%38.0%31.2–45.1%3.0%1.1–6.4%
Gemma 4 E4Bscale scorespermute42.0%20.0%14.7–26.2%3.5%1.4–7.1%
Ministral 14Bitem spaceflip100.0%18.5%13.4–24.6%––
Ministral 14Bitem spacepermute100.0%20.0%14.7–26.2%––
Ministral 14Bscale scoresflip18.0%12.0%7.8–17.3%4.5%2.1–8.4%
Ministral 14Bscale scorespermute55.0%18.5%13.4–24.6%6.0%3.1–10.2%
Phi-4item spaceflip100.0%25.0%19.2–31.6%––
Phi-4item spacepermute100.0%26.0%20.1–32.7%––
Phi-4scale scoresflip53.0%28.5%22.4–35.3%5.0%2.4–9.0%
Phi-4scale scorespermute96.5%21.5%16.0–27.8%6.5%3.5–10.9%
Qwen2.5 14Bitem spaceflip100.0%13.0%8.7–18.5%––
Qwen2.5 14Bitem spacepermute98.0%7.5%4.3–12.1%––
Qwen2.5 14Bscale scoresflip99.0%22.0%16.5–28.4%3.5%1.4–7.1%
Qwen2.5 14Bscale scorespermute61.5%10.0%6.2–15.0%5.0%2.4–9.0%

Table 9. On rounded data our test flags many datasets, and on the control it holds its rate. 200 datasets per setting, 95% Clopper–Pearson intervals. Strict counts every flag as false, and primary leaves out threshold cells (Appendix § 12). The control makes the expectation exactly additive and counts every flag.

The control. To separate this from a miscalibrated test, we take the same rounded datasets and replace their expected values with the additive projection of those values. The noise is then exactly the rounded Likert noise, and the expectation is exactly additive. We ran this control on all ten scale-score settings and on three item-space settings. Our test then flags at least one cell in 3.0–6.5% of datasets, every 95% interval contains 5%, and the pooled rate is 4.7% (122 of 2,600 datasets). Our test is therefore calibrated on Likert noise, and the flags it raises on rounded data are interactions that exist on the answer scale.

Every Significant Cell

Figure 6 gives the cells each model flags, and Figure 7 and Table 10 the gender identity ×\times orientation table that holds the main cells. Table 11 gives the size of the main cells and how often they replicate. Table 12 lists every cell that is significant in at least one model, and Table 17 which traits carry each one on scale scores.

HeterosexualGayLesbianQueer
Man13 / 119 / 411 / 912 / 12
Woman13 / 74 / 23 / 010 / 8
Transgender person13 / 1312 / 1012 / 613 / 13

Table 10. The gender identity ×\times orientation table interacts in every model. Number of the 13 open models in which each cell is significant, in the item space / on scale scores.

Heatmap of the gender identity by orientation table, shown for item space and scale scores

Figure 7. The gender identity ×\times orientation table interacts in every model. Number of the 13 open models in which each cell is significant.

CellItem spaceScale scores
transgender person × heterosexual1313
transgender person × queer1313
man × heterosexual1311
poor working class × heterosexual1311
ultra rich × heterosexual138
transgender person × poor working class137
woman × heterosexual137
man × queer1212
transgender person × gay1210
transgender person × ultra rich129
transgender person × senior citizen128
transgender person × lesbian126
senior citizen × heterosexual1110
man × lesbian119
lower middle class × heterosexual116
poor working class × queer113
transgender person × lower middle class113
woman × queer108
heterosexual × Buddhist105
heterosexual × Hindu104
transgender person × middle class103
heterosexual × Muslim94
man × gay94
transgender person × adult94
transgender person × child94
transgender person × young adult94
rich × heterosexual93
transgender person × Hindu93
middle class × heterosexual92
young adult × heterosexual91
transgender person × Buddhist84
transgender person × rich83
man × senior citizen82
child × heterosexual74
senior citizen × poor working class73

Table 12. Every cell significant in at least one of the 13 open models. Number of models in which the cell is significant, in the item space and on scale scores, ordered by the larger of the two. 175 cells.

CellModelsEACNONFCEMPGAD-7PHQ-8
transgender person ×\times heterosexual140/121/82/1110/13/12/43/89/28/2
transgender person ×\times queer1413/011/08/11/97/14/19/20/110/6
man ×\times queer130/100/100/44/00/51/41/99/05/0
man ×\times heterosexual127/05/15/01/101/14/01/11/90/5
poor working class ×\times heterosexual121/30/23/23/15/22/10/63/34/2
senior citizen ×\times heterosexual110/11/05/00/54/01/31/24/12/0
transgender person ×\times gay100/61/61/12/11/60/10/33/03/1
transgender person ×\times ultra rich102/02/00/55/00/30/16/02/01/2
man ×\times lesbian92/03/10/46/11/00/15/00/02/0
transgender person ×\times senior citizen93/12/20/35/01/13/01/05/04/0
transgender person ×\times poor working class82/00/13/00/14/23/10/51/00/3
ultra rich ×\times heterosexual81/00/02/00/40/61/02/01/11/2
woman ×\times queer80/81/60/22/00/20/31/13/03/0
woman ×\times heterosexual73/04/12/01/00/11/01/22/10/3
lower middle class ×\times heterosexual60/00/00/34/02/02/10/10/01/2
transgender person ×\times lesbian60/11/33/01/20/11/10/32/01/0
child ×\times heterosexual50/01/01/10/21/21/00/10/20/4
heterosexual ×\times Buddhist50/11/00/01/02/10/00/11/01/0
heterosexual ×\times Hindu41/01/10/00/00/10/03/11/02/0
heterosexual ×\times Muslim40/00/00/11/02/01/00/10/30/3
man ×\times gay40/04/01/00/01/01/00/00/00/0
transgender person ×\times Buddhist41/00/20/12/00/00/10/40/00/0
transgender person ×\times adult40/30/11/00/10/00/00/00/10/1
transgender person ×\times child44/02/01/12/01/01/00/02/11/1
transgender person ×\times young adult40/40/11/00/20/10/31/00/10/1
child ×\times poor working class30/10/00/20/00/30/01/01/00/1
poor working class ×\times queer30/00/00/10/00/20/10/01/20/1
rich ×\times heterosexual30/00/00/00/20/20/12/00/11/1
senior citizen ×\times poor working class31/00/00/11/11/01/00/00/10/1
transgender person ×\times Dalit31/00/20/01/00/00/00/30/10/0
transgender person ×\times Hindu30/12/01/00/10/00/01/10/00/0

Table 17. Which traits carry each significant cell. For every cell significant on scale scores, the number of models in which each trait’s component is significantly positive / negative, family-wise across the nine traits, counted over the models in which the cell is significant. Models counts the 13 open models and DeepSeek V4.1 Flash.

Robustness Checks

Our metric has no sign, so we read the direction of an interaction from the signed residuals on the anxiety and depression scales. Under our main wording, transgender person ×\times queer is sub-additive, which means that in 10 of 13 models a

model attributes less anxiety to it than its two labels predict. Man ×\times queer and transgender person ×\times heterosexual lean the other way. The C1 quartets and ordinal models of each item give the same sign, so neither a ceiling effect nor treating the answers as numbers explains it.

Rewording the prompts changes this picture (Table 20). The main cells still interact in most models, while the sub-additive anxiety residual survives in only a few. The models agree on where identities interact and the direction depends on the wording, so a claim from a single wording that identities compound, or that they cancel, is unsafe.

CheckResult
Whitened item space; each questionnaire left outBoth transgender main cells interact in 13 of 13 models (one exception: 12 of 13)
Rewordings 1 and 2At least two of four key gender × orientation cells interact in 10 and in 8 models
Direction under rewordingsTransgender person × queer anxiety residual is negative in 10 of 13 models under the main wording, in 7 of the 12 rewording models under the original wording, and in 5 and 1 of 12 under the two rewordings
Fresh sample of 5,000 personas (5 models)52% of flagged cells are significant again (51% on scale scores)
Template added to the additive model98–100% of flagged cells stay significant (94–100% on scale scores), including the main cells
Largest template effect against largest identity effectThe template effect is larger in 13 of 13 models (12 of 13 on scale scores)
Model sizeMore flagged cells in larger models in the item space (ρ=0.68\rho= 0.68), not on scale scores (ρ=0.16\rho= 0.16)

Table 20. The location of the main interactions survives every check, while single cells and the direction do not. Real answers in the item space unless stated otherwise. The rewording arms cover 12 models.

About half of the flagged cells are significant again on a fresh sample of personas, so a single weak cell is a lead and the pattern across a table is the finding. Templates move the answers more than identity does. Leaving their effects out of the additive model can create false positives in single cells, with a family-wise error rate of up to 23.5% in one model. With each persona’s template in the additive model, the test returns to near its nominal level, and almost every flagged cell stays significant, including the main cells (Appendix § 17.§ 1).

Size is tied to family and training in our 13 models, so we draw no conclusion about scale and bias from the model-size result.

Three-way effects. A pairwise cell can also pick up an interaction among three attributes. When we plant a pure three-way effect in the benchmark, it is flagged only in the three pairs that contain it, and only when it is large (one residual standard deviation on scale scores, two in the item space). A pure two-way effect is never flagged in other pairs, and false flags elsewhere stay at the null rate. A flagged pair should therefore be read as an interaction involving its two values, possibly together with a third.

Published Alternatives on Real Answers

Real answers have no ground truth, so the comparison of the alternatives of § 4 is descriptive. In the item space every alternative except the single-attribute baseline finds all three main cells, and the methods disagree about almost everything else (Table 21). The single-attribute baseline flags most cells when a positive excess is enough. When its excess has to pass its own significance test, it misses transgender person × heterosexual in all 13 models in the item space and in 12 of 13 on scale scores. Per-item regression and MAIHDA flag more cells than our test in the item space, while on scale scores only MAIHDA does. Their extra cells are weaker effects, sometimes resting on one or two items, and they are significant again on a fresh sample less often. Extra cells replicate less for every method, ours included, so we read this as a difference in error control and not as evidence that those cells are false.

MethodCells flagged, item spaceCells flagged, scale scoresAlso flagged by our testTrans. × het., item space
Single-attribute, excess > 0640–711550–6143%3
Single-attribute, excess significant19–8027–7418%0
Per-item classical + Bonferroni52–18514–3874%13
MAIHDA63–22717–7060%13
PERMANOVA (pair test)533–738313–6455%13
Ours24–10614–41–13

Table 21. Our test flags few cells, and every alternative except the single-attribute baseline finds the main cells. Real answers. Cells flagged per model, as a range over the 13 open models, in each representation. The fourth column is the mean share of a method’s flagged cells that our test also flags, on scale scores. The last column counts the models in which a method flags transgender person × heterosexual.

HInter flags every contrast in every model, including the calibration contrast C4, and nearly every item in simulated additive data. Its flags therefore barely agree with our difference-in-differences, at a Cohen’s κ\kappa of 0.059 over every model, contrast and item. Like the single-attribute baseline, its rule reacts to main effects that each push some answers across a category boundary. This limits what its flags can show about an interaction, and not what HInter does as a tool for finding failing inputs.

Further Variants of the Published Alternatives

We also ran four further variants of the published alternatives, on the same benchmark datasets and real answers as the main comparison. The anchored form of the single-attribute baseline compares a cell with the most privileged cell instead of the population mean [22, 60]. The stepwise form flags a cell when adding its second value moves the answers further than the first value alone [48]. For per-item regression with classical standard errors, we replace the Bonferroni correction with the Holm or the Benjamini–Hochberg correction.

None of these variants changes the picture of § 6 (Table 22). The anchored and stepwise forms make a false positive in every dataset with nothing planted, and their precision stays near 0.01. Holm gives almost the same flags as Bonferroni. Benjamini–Hochberg controls the false discovery rate, not the FWER, so it finds more planted cells but at a much lower precision. On real answers, the anchored and stepwise forms flag hundreds of cells per model and share few of them with our test (Table 23). The anchored form never flags transgender person × heterosexual. Table 23 also gives per-item regression with HC3 errors, which Table 21 leaves out.

MethodLikertScaleItemLikertReal pattern (scale)Single item (item)
Ours20.0%8.7%4.7%78.1% / 0.7284.4% / 0.9953.1% / 0.98
Single-attribute, anchored100%100%100%52.0% / 0.0133.3% / 0.0148.7% / 0.01
Single-attribute, stepwise100%100%100%99.9% / 0.0183.2% / 0.0198.9% / 0.01
Per-item classical + Holm100%36.0%100%85.9% / 0.2077.6% / 0.9299.2% / 0.15
Per-item classical + BH100%38.0%100%95.6% / 0.0789.3% / 0.6499.7% / 0.05

Table 22. Further variants on the benchmark. Same datasets and settings as Table 4. Group-mean generator; Likert answers use permute.

MethodCells flagged, item spaceCells flagged, scale scoresAlso flagged by our testTrans. × het., item space
Single-attribute, anchored247–358262–3473%0
Single-attribute, stepwise682–723614–6713%10
Per-item HC3 + Bonferroni63–22310–3576%13
Per-item classical + Holm52–18514–3874%13
Per-item classical + BH184–42326–11934%13
Ours24–10614–41–13

Table 23. Further variants on real answers. Same columns as Table 21.

I Recall at Every Planted Size

Table 24 gives the recall of our test and of per-item regression with HC3 errors and Bonferroni at each planted size, in residual standard deviations. Our test finds more planted cells on scale scores and for the real pattern. Per-item regression finds more in the item space when the planted shifts are random or sit in a single item, at the cost of a far higher FWER (Table 4). The three main cells of § 6.3 have sizes of about 0.7–0.9 in the item space and 0.3–0.4 on scale scores, in the range where our recall rises from about one half to almost one.

Scenario0.20.51
One random cell, scale scores0.00 / 0.000.06 / 0.040.90 / 0.74
Five random cells, scale scores0.02 / 0.020.52 / 0.450.95 / 0.88
Five random cells, item space0.00 / 0.010.10 / 0.210.56 / 0.84
Twenty cells, two per pair, scale scores0.03 / 0.020.76 / 0.650.99 / 0.96
Several in one pair, scale scores0.02 / 0.020.54 / 0.450.95 / 0.88
Single item, scale scores0.02 / 0.100.52 / 0.670.94 / 0.99
Single item, item space0.00 / 0.120.10 / 0.670.55 / 0.98
Real pattern, scale scores0.04 / 0.010.38 / 0.300.88 / 0.78
Real pattern, item space0.04 / 0.030.52 / 0.441.00 / 0.91
Likert-rounded, item space0.01 / 0.010.29 / 0.100.78 / 0.59

Table 24. Recall rises with the planted size. Synthetic benchmark data. Share of planted cells found by our test / by per-item HC3 + Bonferroni, as the mean over models and generators.

Reliability of the Scales

We compute Cronbach’s α\alpha for each scale over the personas each model describes, leaving out items that a model answers the same way for every persona. This measures how consistently a model’s answers move across the items of one scale, not the reliability of the scale for people. Table 25 summarises it over the 13 open models.

Table 25. The symptom scales are reliable in most models, but openness and need for cognition are not. Cronbach’s α\alpha over personas, as the median [range] over the 13 open models, and the number of models with α≥0.70\alpha\ge0.70.

ScaleItemsα\alphaα≥0.70\alpha\ge0.70
Extraversion100.71 [0.27, 0.86]8
Agreeableness100.72 [0.42, 0.87]8
Conscientiousness100.58 [0.28, 0.78]3
Neuroticism100.56 [−0.11, 0.90]5
Openness100.35 [0.09, 0.83]2
Need for cognition80.34 [0.12, 0.77]1
Empathic concern90.61 [0.22, 0.76]1
Anxiety (GAD-7)70.83 [0.59, 0.90]11
Depression (PHQ-8)80.82 [0.20, 0.88]11

Table 25.

Control Arms

Besides the 10,000 main personas, IndiPersona holds three control arms.

Fresh sample. A second sample of 5,000 personas, built the same way with a different seed, checks whether our results reproduce on new data.

Rewordings. Two independent rewordings of every prompt, on 3,000 personas, test whether the findings depend on our wording. They change the templates, question wording, answer labels and system prompt, but keep the identity labels and answer numbers.

Personas without identity. A set of 1,000 personas with no identity attributes shows how far identity moves the answers at all. Against these personas, identity labels move the median item by 0.31–0.76 standard deviations, yet they explain only 10–28% of the item variance (five models).

The Four Counterfactual Contrasts

Table 26 lists the labels each contrast changes, and Table 27–Table 30 give the shift on each trait for each model. C1 tests the main observational interaction, transgender person ×\times queer. C2 replaces transgender person with woman, to check that C1 is specific to the transgender label. C3 tests Dalit ×\times poor working class, a caste interaction that the main personas hold too little data to detect. C4 swaps adjacent levels of age and class, where we expect little interaction, and serves as the calibration contrast. We compare each contrast with C4 by the size of its difference-in-differences, with a sign-flip test and a bootstrap interval over base personas.

Table 26. Changing two labels reproduces the main interaction, and finds a caste interaction the screen misses. Size is the debiased length of the within-persona difference-in-differences, median [range] over the 13 open models. The last column counts models in which a contrast exceeds C4 with 95% bootstrap confidence.

ContrastLabels changedSizeExceeds C4
C1man → transgender person, heterosexual → queer2.36 [1.22, 4.88]13
C2man → woman, heterosexual → queer0.68 [0.41, 0.93]5
C3Brahmin → Dalit, rich → poor working class2.10 [0.56, 3.29]12
C4adult → senior citizen, middle class → upper middle class0.47 [0.15, 0.81]–

Table 26.

Table 27. C1: man → transgender person, heterosexual → queer. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.

ModelEACNONFCEMPGAD-7PHQ-8
Gemma 4 E2B+0.86*+0.40*+0.86*−0.74*+0.28*+0.64*−0.20*−0.97*−0.94*
Gemma 4 E4B+0.29*+0.17+0.30*−0.32*−0.01−0.06+0.29*−0.41*−0.22*
Llama 3 8B+0.95*+0.58*+0.25*−0.21*+0.36*+0.93*+0.72*−0.50*−0.40*
Llama 3.1 8B+0.52*+0.56*+0.31*−0.29*+0.20*+0.62*+0.39*−0.40*−0.44*
Ministral 14B+0.78*+0.18*+0.06−0.19*+0.04+0.17*+0.10−0.53*−0.49*
Ministral 8B+0.38*+0.56*+0.14*−0.27*+0.19*+0.36*+0.52*−0.28*−0.31*
Mistral Small 24B+0.47*+0.34*+0.42*−0.20*+0.38*+0.10+0.43*+0.09−0.01
Phi-4+0.39*+0.56*−0.10*−0.03−0.07−0.02+0.29*−0.19*+0.01
Phi-4-mini+0.71*+0.65*+0.29*−0.38*+0.13+0.16+0.24*−0.46*−0.11
Qwen2.5 14B+0.44*+0.30*+0.39*−0.11*−0.10−0.01+0.41*−0.32*−0.11
Qwen2.5 3B+0.79*−0.61*+0.28*+0.72*−0.32*−0.14−0.57*+0.16*+0.02
Qwen2.5 7B+0.30*+0.25*+0.08−0.25*+0.07−0.07+0.25*+0.14+0.17
Qwen3 4B-2507+0.46*+0.18*+0.30*−0.28*+0.12+0.06−0.07−0.12*−0.12*

Table 27.

ModelEACNONFCEMPGAD-7PHQ-8
Gemma 4 E2B+0.01−0.03−0.03+0.00+0.01−0.03+0.00+0.00−0.01
Gemma 4 E4B+0.07*−0.03+0.07+0.04+0.03−0.03+0.00+0.00+0.02
Llama 3 8B+0.03−0.02+0.04−0.03−0.08*−0.03+0.03+0.01+0.02
Llama 3.1 8B+0.02+0.02+0.06−0.01−0.03+0.09+0.03−0.02+0.00
Ministral 14B+0.00+0.00−0.10*+0.05−0.22*−0.02−0.01−0.01−0.04
Ministral 8B+0.02−0.06+0.01−0.07−0.07−0.07+0.04−0.07+0.01
Mistral Small 24B+0.00−0.07*−0.05−0.08−0.08−0.16*−0.04−0.07*−0.11*
Phi-4+0.01+0.02−0.14*+0.06*+0.05−0.11*−0.06+0.01−0.06*
Phi-4-mini+0.01−0.05−0.04+0.02−0.17−0.02−0.11−0.07+0.00
Qwen2.5 14B+0.03−0.12*+0.09−0.05+0.06+0.03−0.06+0.03+0.05
Qwen2.5 3B+0.03−0.02−0.03−0.04−0.07+0.04−0.05−0.04+0.01
Qwen2.5 7B+0.03+0.04+0.04+0.03+0.00−0.02+0.04+0.05+0.00
Qwen3 4B-2507−0.02+0.01+0.08−0.04−0.07−0.01−0.03−0.07*−0.06

Table 30. C4: adult →\to senior citizen, middle class →\to upper middle class. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.

Setup Details

Querying the models. We query the open models with vLLM in bf16, at pinned revisions, and DeepSeek V4.1 Flash through its API. A system prompt asks the model to infer, from the description alone, how the person would answer, and to return only the number.

Persona-sensitivity check. A model passes if it gives at most 1% invalid answers, if at least 10% of personas receive a distinct pattern of answers, and if its answers vary across personas, with a mean item standard deviation of at least 0.05 and no more than half of its items below 0.1.

Robustness checks. Each robustness check in Appendix § 17.§ 4 has one purpose. The fresh sample and the two random halves of the main sample test whether the findings reproduce on new data. For the fresh sample, we report the share of flagged cells that are flagged again. The two rewordings test whether the findings depend on our wording (Appendix § 20). The whitened item space removes the correlation between items, so that related items do not count twice [35], and leaving out each questionnaire tests whether one questionnaire drives the results. The answers are ordered categories, not numbers, so we fit ordinal models of each item for the three main cells [36]. One strong cell drags the residuals of its row and column, so we retest each cell after adding the strongest one to the additive model. Personas that share a template may share part of their noise, so we rerun the test with each persona’s template added to the additive model.

Simpler Fixes to the Single-Attribute Baseline

We test two repairs of the single-attribute baseline on the benchmark’s real-pattern scenario, on scale scores, at a planted size of one residual standard deviation. As in Table 2, we use the group-mean generator (150 datasets). The first repair gives its excess a calibrated Freedman–Lane null. The second compares the cell’s distance from the population mean with the distance of its additive prediction from the population mean. Both hold their error rate, with an FWER of 2.7% and 4.7%, but they find only 32.4% and 37.6% of the planted cells, against 85.0% for δ\delta. The first repair is the second row of Table 2. We ran both repairs, and δ\delta beside them, in a separate run on the same seeded datasets as the main benchmark, with a sign-flip Freedman–Lane null without the HC2 correction. In this run δ\delta has an FWER of 6.7% and a recall of 85.0%, close to the 8.7% and 84.4% of the main run in Table 2. The single-attribute baseline’s false positives therefore come from its null, but its misses come from its shape.

Dependent Cells in the Gender Identity ×\times Orientation Table

One strong cell leaves residuals in the other cells of its row and column, so flagged cells in one table are not independent findings. To see which cells carry their own signal, we add the strongest cell to the additive model and retest the rest. When we add transgender person ×\times queer, man ×\times queer mostly disappears, and it stays significant in only 4 of the 12 models that flagged it. The reverse does not happen: when we add man ×\times queer, transgender person ×\times queer stays significant in all 13 models. The transgender row and man ×\times heterosexual also survive. We choose the strongest cell from the same data, so we read this refit as a diagnostic, not as a confirmatory test.

CellItem spaceScale scores
lower middle class ×\times queer72
man ×\times poor working class72
transgender person ×\times Christian72
transgender person ×\times Jain72
upper middle class ×\times heterosexual72
senior citizen ×\times queer71
teenager ×\times heterosexual71
transgender person ×\times upper middle class71
ultra rich ×\times queer71
senior citizen ×\times lower middle class62
senior citizen ×\times ultra rich62
poor working class ×\times Hindu61
child ×\times poor working class53
heterosexual ×\times Christian52
young adult ×\times poor working class51
poor working class ×\times lesbian50
upper middle class ×\times queer50
heterosexual ×\times Jain42
transgender person ×\times teenager42
woman ×\times gay42
poor working class ×\times Jain41
transgender person ×\times Muslim41
transgender person ×\times OC41
ultra rich ×\times Christian41
woman ×\times poor working class41
adult ×\times poor working class40
man ×\times Hindu40
man ×\times ultra rich40
poor working class ×\times Buddhist40
senior citizen ×\times gay40
senior citizen ×\times lesbian40
teenager ×\times ultra rich40
woman ×\times senior citizen40
transgender person ×\times Dalit33
ultra rich ×\times Jain33
CellItem spaceScale scores
man ×\times child32
transgender person ×\times ST32
child ×\times middle class31
child ×\times queer31
child ×\times ultra rich31
child ×\times upper middle class31
lesbian ×\times Buddhist31
woman ×\times ultra rich31
child ×\times lower middle class30
heterosexual ×\times Dalit30
lower middle class ×\times Hindu30
man ×\times adult30
poor working class ×\times Christian30
poor working class ×\times Muslim30
poor working class ×\times OC30
rich ×\times Hindu30
rich ×\times queer30
senior citizen ×\times rich30
senior citizen ×\times upper middle class30
teenager ×\times poor working class30
transgender person ×\times Sikh30
ultra rich ×\times Buddhist30
ultra rich ×\times Hindu30
woman ×\times lesbian30
young adult ×\times ultra rich30
ultra rich ×\times OC23
adult ×\times heterosexual21
heterosexual ×\times Ajlaf21
heterosexual ×\times Protestant21
heterosexual ×\times Sikh21
lesbian ×\times Christian21
man ×\times upper middle class21
queer ×\times Hindu21
senior citizen ×\times Hindu21
heterosexual ×\times OC20
CellItem spaceScale scores
heterosexual ×\times SC20
man ×\times lower middle class20
man ×\times young adult20
middle class ×\times Hindu20
middle class ×\times queer20
poor working class ×\times Catholic20
poor working class ×\times Dalit20
poor working class ×\times Sikh20
senior citizen ×\times Buddhist20
senior citizen ×\times middle class20
teenager ×\times rich20
transgender person ×\times Catholic20
ultra rich ×\times Muslim20
ultra rich ×\times gay20
young adult ×\times lower middle class20
young adult ×\times queer20
adult ×\times ultra rich11
heterosexual ×\times Arzal11
heterosexual ×\times Shudra11
lesbian ×\times Hindu11
lower middle class ×\times Christian11
man ×\times Buddhist11
senior citizen ×\times Jain11
transgender person ×\times OBC11
transgender person ×\times Protestant11
transgender person ×\times Reddy11
woman ×\times Buddhist11
woman ×\times Hindu11
woman ×\times child11
young adult ×\times middle class11
child ×\times rich10
gay ×\times Christian10
gay ×\times Muslim10
heterosexual ×\times Catholic10
heterosexual ×\times Jat Sikh10
CellItem spaceScale scores
heterosexual ×\times OBC10
heterosexual ×\times ST10
lesbian ×\times Jain10
lesbian ×\times Muslim10
lower middle class ×\times Catholic10
lower middle class ×\times Muslim10
lower middle class ×\times Protestant10
lower middle class ×\times lesbian10
man ×\times Christian10
man ×\times middle class10
man ×\times rich10
man ×\times teenager10
poor working class ×\times Arzal10
poor working class ×\times Brahmin10
poor working class ×\times Protestant10
poor working class ×\times gay10
queer ×\times Buddhist10
senior citizen ×\times Christian10
teenager ×\times lower middle class10
teenager ×\times middle class10
teenager ×\times queer10
teenager ×\times upper middle class10
transgender person ×\times SC10
transgender person ×\times Shudra10
ultra rich ×\times Catholic10
ultra rich ×\times Protestant10
ultra rich ×\times Shudra10
ultra rich ×\times Sunni10
ultra rich ×\times lesbian10
upper middle class ×\times Hindu10
woman ×\times upper middle class10
woman ×\times young adult10
young adult ×\times Hindu10
queer ×\times Kshatriya01
woman ×\times Christian01

Table 13. Every significant cell (continued, part 2 of 5).

CellModelsEACNONFCEMPGAD-7PHQ-8
transgender person × Jain31/00/20/32/00/00/00/10/00/0
transgender person × lower middle class31/00/11/20/10/01/00/20/00/0
transgender person × middle class30/00/00/10/11/10/10/00/01/0
transgender person × rich30/00/01/01/00/30/01/00/01/0
ultra rich × Jain30/00/01/10/11/00/00/00/00/0
ultra rich × OC30/00/01/00/12/00/00/00/10/0
heterosexual × Christian20/01/02/00/10/10/00/10/00/1
heterosexual × Jain20/01/00/00/00/10/01/00/00/0
lower middle class × queer20/00/00/00/10/20/01/00/01/0
man × child20/10/00/00/10/10/10/01/01/0
man × poor working class20/01/00/00/01/01/11/00/00/0
man × senior citizen20/00/00/00/20/00/10/00/00/0
middle class × heterosexual21/00/00/10/01/10/11/00/00/0
senior citizen × lower middle class20/00/02/00/11/10/00/00/00/0
senior citizen × ultra rich20/10/00/10/10/10/10/00/00/0
transgender person × Christian20/00/10/11/01/00/00/01/01/0
transgender person × ST22/00/10/12/00/00/00/20/00/1
transgender person × teenager20/01/02/00/20/00/01/00/00/0
upper middle class × heterosexual20/00/00/10/00/10/00/10/01/1
woman × gay20/01/10/10/01/00/00/00/00/1
adult × heterosexual10/11/00/10/00/00/01/00/00/1
adult × ultra rich10/01/01/00/11/00/00/00/00/1
child × middle class10/00/00/00/00/10/00/00/10/1
child × queer10/00/10/11/00/00/10/00/00/0
child × ultra rich10/00/00/00/11/00/00/00/00/0
child × upper middle class11/00/01/00/01/01/00/00/00/0
heterosexual × Ajlaf10/00/00/00/01/00/00/00/00/1
heterosexual × Arzal10/00/00/00/00/00/00/00/10/1
heterosexual × Protestant10/00/01/00/10/00/00/00/00/1
heterosexual × Shudra11/01/01/00/01/00/00/00/00/0
heterosexual × Sikh10/00/10/00/00/10/00/10/00/0
CellModelsEACNONFCEMPGAD-7PHQ-8
lesbian × Buddhist10/01/01/00/00/00/00/00/00/0
lesbian × Christian10/00/00/11/00/00/00/00/01/0
lesbian × Hindu10/10/10/00/00/00/00/00/00/0
lower middle class × Christian10/00/00/10/00/00/01/00/00/0
man × Buddhist10/01/00/00/10/01/00/00/00/0
man × upper middle class10/00/00/11/00/10/00/00/00/0
poor working class × Hindu10/00/00/00/00/00/00/10/00/0
poor working class × Jain11/00/00/11/01/00/00/00/00/0
queer × Hindu10/10/00/00/00/10/10/00/00/0
queer × Kshatriya10/00/01/00/01/00/00/00/00/0
senior citizen × Hindu10/00/00/00/00/00/01/00/00/0
senior citizen × Jain10/00/10/00/00/00/00/10/00/0
senior citizen × queer10/00/00/00/00/10/00/00/00/0
teenager × heterosexual10/00/00/00/00/00/00/00/10/1
transgender person × Muslim10/10/00/00/10/00/00/00/00/0
transgender person × OBC10/00/00/10/00/00/00/00/00/0
transgender person × OC10/00/10/10/00/00/00/00/00/0
transgender person × Protestant11/00/10/11/00/00/00/00/00/0
transgender person × Reddy10/01/01/00/00/00/00/00/00/0
transgender person × upper middle class10/00/01/00/00/10/00/00/00/1
ultra rich × Christian10/00/10/11/00/00/10/00/00/0
ultra rich × queer10/00/00/00/01/01/00/00/00/0
woman × Buddhist10/11/01/00/00/00/00/00/00/0
woman × Christian10/00/00/00/00/10/00/00/00/0
woman × Hindu10/00/10/00/00/00/00/10/00/0
woman × child10/00/00/00/10/00/00/00/10/1
woman × poor working class10/00/00/10/00/00/01/00/00/0
woman × ultra rich10/00/01/00/10/00/00/00/10/0
young adult × heterosexual10/00/00/01/00/10/10/01/01/0
young adult × middle class10/00/00/10/01/00/00/00/00/0
young adult × poor working class10/00/00/00/01/00/00/00/00/0

Table 18. Which traits carry each significant cell (continued, part 2 of 3).

ModelEACNONFCEMPGAD-7PHQ-8
Gemma 4 E2B+0.00−0.03+0.09*−0.17*+0.01−0.02−0.02+0.05*+0.03
Gemma 4 E4B+0.04+0.00−0.12*−0.04+0.07−0.03+0.12−0.06−0.03
Llama 3 8B+0.16*+0.15*+0.05−0.07+0.20*+0.20*+0.29*−0.07−0.02
Llama 3.1 8B+0.15*+0.15*+0.10*−0.18*+0.11+0.26*+0.08−0.04−0.02
Ministral 14B+0.16*+0.11*+0.09−0.11*+0.07*+0.03+0.03−0.10*−0.13*
Ministral 8B+0.04+0.01+0.01−0.08*+0.05+0.09*+0.03−0.07−0.04
Mistral Small 24B+0.14*+0.12*+0.05−0.04+0.17*+0.07+0.12*−0.03−0.03
Phi-4+0.02+0.11*−0.01+0.06*+0.03+0.05+0.02+0.02+0.01
Phi-4-mini+0.15*+0.07+0.10+0.05−0.01−0.02−0.01−0.07−0.02
Qwen2.5 14B+0.09*−0.01−0.04+0.00+0.07+0.02+0.04−0.06−0.09
Qwen2.5 3B+0.22*+0.08−0.01+0.06+0.04+0.01−0.01+0.03+0.03
Qwen2.5 7B+0.06+0.15*+0.03−0.03+0.02−0.02+0.07+0.04+0.04
Qwen3 4B-2507+0.09*+0.14*+0.05−0.10+0.16*+0.16*+0.12*−0.02+0.00

Table 28. C2: man →\to woman, heterosexual →\to queer. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.

ModelEACNONFCEMPGAD-7PHQ-8
Gemma 4 E2B+0.03+0.08+0.09−0.21*−0.05+0.06−0.66*−0.12*−0.14*
Gemma 4 E4B−0.25*+0.32*+0.83*−0.30*−0.32*+0.02−0.16−0.34*−0.38*
Llama 3 8B−0.02−0.09+0.31*+0.03+0.10+0.03−0.46*+0.11+0.31*
Llama 3.1 8B−0.18*−0.07+0.29*−0.35*−0.19*+0.10−0.13−0.23*−0.35*
Ministral 14B+0.05+0.01+0.37*−0.10−0.29*−0.05+0.02−0.31*−0.50*
Ministral 8B+0.01+0.00+0.27*−0.21*−0.06−0.23*−0.23*−0.34*−0.30*
Mistral Small 24B−0.19*+0.03+0.35*−0.15*−0.47*−0.02+0.18*+0.14*+0.16*
Phi-4−0.10*+0.43*+0.03−0.10*+0.05+0.33*−0.05+0.06+0.19*
Phi-4-mini−0.20*−0.02+0.22*−0.10+0.10+0.04−0.17*−0.35*−0.25*
Qwen2.5 14B−0.22*+0.25*+0.74*−0.48*+0.61*+0.05+0.49*+0.04+0.08
Qwen2.5 3B−0.20*−0.15*+0.05+0.24*−0.02−0.12−0.35*+0.06+0.06
Qwen2.5 7B−0.07−0.06−0.06+0.09+0.02−0.02+0.15*+0.09+0.19
Qwen3 4B-2507−0.12*+0.24*+0.43*−0.41*+0.24*+0.09+0.00+0.00+0.07

Table 29. C3: Brahmin →\to Dalit, rich →\to poor working class. Mean difference-in-differences per model in standard deviations of each scale, standardized within the counterfactual arm; * significant family-wise across the nine scales.

References

  1. [1]Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 386–397. doi:10.18653/v1/2024.acl-short.37DOI
  2. [2]Marti J. Anderson. 2017. Permutational Multivariate Analysis of Variance (PERMANOVA). John Wiley & Sons, Ltd, 1–15. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/978111845112.stat07841 doi:10.1002/978111845112.stat07841DOI
  3. [3]Lena Arnao, Abbey Liu, Stephen MacNeil, and Danaë Metaxa. 2024. The Silician Ceiling: Auditing GPT’s Race and Gender Biases in Hiring. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’24). Association for Computing Machinery, New York, NY, USA. doi:10.1145/3689904.3694699
  4. [4]Srijan Bansal, Vishal G. Jaiswal, Ayush Sahu, and Animesh Mukherjee. 2021. Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languages. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (Virtual Event, USA) (HT ’21). Association for Computing Machinery, New York, NY, USA, 27–34. doi:10.1145/3465336.3475118DOI
  5. [5]Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re-contextualizing Fairness in NLP: The Case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (Eds.). Association for Computational Linguistics, Online only, 727–740. doi:10.18653/v1/2022.aacl-main.55
  6. [6]Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 842–854. doi:10.18653/v1/2024.acl-short.75DOI
  7. [7]Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 1004–1015. doi:10.18653/v1/2021.acl-long.81DOI
  8. [8]Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
  9. [9]Kimberlé Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics. University of Chicago Legal Forum 1989, 1 (1989), 139–167. https://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8/
  10. [10]Lee J. Cronbach and Goldine C. Gleser. 1953. Assessing similarity between profiles. Psychological Bulletin 50, 6 (1953), 456–473. doi:10.1037/h0057173DOI
  11. [11]Preetam Prabhu Srikar Dammu, Havoung Jung, Anjali Singh, Monojit Choudhury, and Tanu Mitra. 2024. “They are uncultured”: Unveiling Court Harms and Social Threats in LLM Generated Conversations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 20339–20369. doi:10.18653/v1/2024.emnlp-main.1134DOI
  12. [12]Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023. Building Socio-culturally Inclusive Stereotype Resources with Community Engagement. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 4365–4381. doi:10.52202/075280-0194DOI
  13. [13]Naoki Egami and Kosuke Imai. 2019. Causal Interaction in Factorial Experiments: Application to Conjoint Analysis. J. Amer. Statist. Assoc. 114, 526 (2019), 529–540. doi:10.1080/01621459.2018.1476246DOI
  14. [14]Clare R. Evans, David R. Williams, Jacka-Pekka Onnela, and S. V. Subramanian. 2018. A multilevel approach to modeling health inequalities at the intersection of multiple social identities. Social Science & Medicine 203 (2018), 64–73. doi:10.1016/j.socscimed.2017.11.011DOI
  15. [15]Antonio Ferrara, Francesco Cozzi, Alan Perotti, André Panisson, and Francesco Bonchi. 2025. Size-adaptive Hypothesis Testing for Fairness. In Advances in Neural Information Processing Systems (NeurIPS), arXiv:2506.10586arxiv.org/abs/2506.10586
  16. [16]Jessica M. Franklin, Sebastian Schneeweiss, Jennifer M. Polinski, and Jeremy A. Rassen. 2014. Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Computational Statistics & Data Analysis 72 (2014), 219–226. doi:10.1016/j.csda.2013.10.018DOI
  17. [17]David Freedman and David Lane. 1983. A Nonstochastic Interpretation of Reported Significance Levels. Journal of Business & Economic Statistics 1, 4 (1983), 292–298. doi:10.1080/07350015.1983.10509354DOI
  18. [18]Sourojit Ghosh, Pranav Narayanan Venkit, Sanjana Gautam, Shomir Wilson, and Aylin Caliskan. 2024. Do Generative AI Models Output Harm While Representing Non-Western Cultures: Evidence from a Community-Centered Approach. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society (San Jose, California, USA) (AIES ’24, Vol. 7). AAAI Press, 476–489. https://ojs.aaai.org/index.php/AIES/article/view/31651arxiv.org/abs/2407.14779
  19. [19]Lewis R. Goldberg. 1992. The development of markers for the Big-Five factor structure. Psychological Assessment 4, 1 (1992), 26–42. doi:10.1037/1040-3590.4.1.26DOI
  20. [20]Government of India. 1950. The Constitution (Scheduled Castes) Order, 1950. C.O. 19, as amended. https://socialjustice.gov.in/writereaddata/UploadFile/scorder1950.pdf
  21. [21]Wei Guo and Aylin Caliskan. 2021. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (Virtual Event, USA) (AIES ’21). Association for Computing Machinery, New York, NY, USA, 122–133. doi:10.1145/3461702.3462536DOI
  22. [22]Amogh Gupta, Niharika Patil, Sourojit Ghosh, and SnehalKumar (Neil) S. Gaikwad. 2026. Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26). Association for Computing Machinery, New York, NY, USA, 6093–6155. doi:10.1145/3805689.3812394DOI
  23. [23]Rishav Hada, Safiya Husain, Varun Gumma, Harshita Diddee, Aditya Yadavalli, Agrima Seth, Nidhi Kulkarni, Ujwal Gadiraju, Aditya Vashistha, Vivek Seshadri, and Kalika Bali. 2024. Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 1926–1939. doi:10.1145/3630106.3659017DOI
  24. [24]Amit Haim, Alejandro Salinas, and Julian Nyarko. 2024. What’s in a Name? Auditing Large Language Models for Race and Gender Bias. arXiv:2402.14875 https://arxiv.org/abs/2402.14875
  25. [25]Johannes Himmelreich, Arbie Hsu, Ellen Voemett, and Kristian Lum. 2025. The Intersectionality Problem for Algorithmic Fairness. In Proceedings of the Algorithmic Fairness Through the Lens of Metrics and Evaluation (Proceedings of Machine Learning Research, Vol. 279). PMLR, 68–95. arXiv:2411.02569 https://proceedings.mlr.press/v279/himmelreich25a.htmlarxiv.org/abs/2411.02569
  26. [26]Akshita Jha, Aida Davani, Chandan K. Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 9851–9870. doi:10.18653/v1/2023.acl-long.548DOI
  27. [27]Ishika Joshi, Ishita Gupta, Adarsh Dev, and Tapan Parikh. 2025. “Since Lawyers are Males..”: Examining Implicit Gender Bias in Hindi Language Generation by LLMs. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (Athens, Greece) (FAccT ’25). Association for Computing Machinery, New York, NY, USA, 3254–3264. doi:10.1145/3715275.3732208DOI
  28. [28]Deborah Dormah Kanubala and Isabel Valera. 2025. On the Misalignment Between Legal Notions and Statistical Metrics of Intersectional Fairness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 1363–1374. doi:10.1609/aies.v8i2.36637
  29. [29]Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 2564–2572. https://proceedings.mlr.press/v80/kearns18a.htmlarxiv.org/abs/1711.05144
  30. [30]Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models. In Proceedings of the 2024 International Conference on Information Technology for Social Good (Bremen, Germany) (GoodIT ’24). Association for Computing Machinery, New York, NY, USA, 231–239. doi:10.1145/3677525.3678666
  31. [31]Hannah Rose Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, and Yuki M. Asano. 2021. Bias out-of-the-box: an empirical analysis of intersectional occupational biases in popular generative language models. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 200, 14 pages.arxiv.org/abs/2102.04130
  32. [32]Neeraja Kirtane and Tanvi Anand. 2022. Mitigating Gender Stereotypes in Hindi and Marathi. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Christian Hardmeier, Christine Basta, Marta R. Costa-jussà, Gabriele Stańczak, and Hila Gonen (Eds.). Association for Computational Linguistics, Seattle, Washington, 145–150. doi:10.18653/v1/2022.gebnlp-1.16DOI
  33. [33]Kurt Kroenke, Tara W. Strine, Robert L. Spitzer, Janet B. W. Williams, Jocelyn T. Berry, and Ali H. Mokdad. 2009. The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders 114, 1–3 (2009), 163–173. doi:10.1016/j.jad.2008.06.026DOI
  34. [34]John Lalor, Yi Yang, Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking Intersectional Biases in NLP. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 3598–3609. doi:10.18653/v1/2022.naacl-main.263
  35. [35]Olivier Ledoit and Michael Wolf. 2004. A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis 88, 2 (2004), 365–411. doi:10.1016/S0047-259X(03)00096-4
  36. [36]Torrin M. Liddell and John K. Kruschke. 2018. Analyzing ordinal data with metric models: What could possibly go wrong? Journal of Experimental Social Psychology 79 (2018), 328–348. doi:10.1016/j.jesp.2018.08.009DOI
  37. [37]Geoffrey R. Loftus. 1978. On Interpretation of Interactions. Memory & Cognition 6, 3 (1978), 312–319. doi:10.3758/BF03197461DOI
  38. [38]Weicheng Ma, Brian Chiang, Tong Wu, Lili Wang, and Soroush Vosoughi. 2023. Intersectional Stereotypes in Large Language Models: Dataset and Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Manuscript submitted to ACM for Computational Linguistics, Singapore, 8589–8597. doi:10.18653/v1/2023.findings-emnlp.575
  39. [39]James G. MacKinnon and Halbert White. 1985. Some Heteroskedasticity-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties. Journal of Econometrics 29, 3 (1985), 305–325.
  40. [40]Vijit Malik, Sunipa Dev, Akihiro Nishiya Nuyen, and Kai-Wei Chang. 2022. Socially Aware Bias Measurements for Hindi Language Representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 1041–1052. doi:10.18653/v1/2022.naacl-main.76DOI
  41. [41]Tim P. Morris, Ian R. White, and Michael J. Crowther. 2019. Using simulation studies to evaluate statistical methods. Statistics in Medicine 38, 11 (2019), 2074–2102. doi:10.1002/sim.8086DOI
  42. [42]Anjishnu Mukherjee, Chahat Raj, Zivli Zhu, and Antonios Anastasopoulos. 2023. Global Voices, Local Biases: Socio-Cultural Prejudices across Languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 15828–15845. doi:10.18653/v1/2023.emnlp-main.981
  43. [43]Atharva Naik, Shounok Kar, Varnika Sharma, Ashwin Rajadesingan, and Koustuv Saha. 2026. Sima Alunty: Caste Audit in LLM-Driven Matchmaking. arXiv:2603.29288 [cs.CY] https://arxiv.org/abs/2603.29288
  44. [44]Pavithra P M Nair, Bhavik Talaviya, Shourya Bhushan, Rahul Pankajkashan, Seema Guruvadoo, Avinash Agarwal, Gilad Gressel, and Krishnashree Achuthan. 2026. RupeeBias: Auditing Demographic Bias in Indian Economic Guidance from Large Language Models. arXiv:2609.31245 [cs.CL] https://arxiv.org/abs/2609.31245
  45. [45]Janki Atul Nawale, Mohammed Safi Ur Rahman Khan, Janani D. Mansi Gupta, Danish Pruthi, and Mitesh M Khapra. 2025. Fair Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 30331–30380. doi:10.18653/v1/2025.acl-long.1465
  46. [46]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Alina Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 2086–2105. doi:10.18653/v1/2022.findings-acl.165DOI
  47. [47]Belinda Phipson and Gordon K. Smyth. 2010. Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology 9, 1 (2010), Article 39. doi:10.2202/1544-6115.1585DOI
  48. [48]Mahika Phutane, Hayoung Jung, Matthew Kim, Tanushree Mitra, and Aditya Vashistha. 2026. ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 9, 3 (2026), 2089–2104. arXiv:2510.10998 doi:10.1609/aies.v9i3.43291DOI
  49. [49]Prime Minister’s High Level Committee. 2006. Social, Economic and Educational Status of the Muslim Community of India: A Report. Technical Report. Cabinet Secretariat, Government of India, New Delhi.
  50. [50]Rida Qadri, Renee Shelby, Cynthia L. Bennett, and Remi Denton. 2023. AI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 506–517. doi:10.1145/3593013.3594016DOI
  51. [51]Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in Language Models Beyond English: Gaps and Challenges. In Findings of the Association for Computational Linguistics: EACL 2023, Andreas Vlachos and Isabell Augenstein (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 2106–2119. doi:10.18653/v1/2023.findings-eacl.157DOI
  52. [52]Pooja S. B. Rao, Laxminarayan Nagarajan Venkatesan, Mauro Cherubin, and Dinesh Babujayogi. 2025. Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models. arXiv:2508.16673 [cs.CY] https://arxiv.org/abs/2508.16673
  53. [53]Virgile Rennard and Christos Xylopoulos. 2026. Large Language Models Simulate Intersectional Synthetic Identities with a Budget of One to Two Dimensions. arXiv:2608.23005 https://arxiv.org/abs/2608.23005
  54. [54]Nihar Sahoo, Pratyush Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, and Pushpak Bhattacharyya. 2024. IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 8786–8806. doi:10.18653/v1/2024.naacl-long.487
  55. [55]Nithya Sambasivan, Erin Arnesen, Hen Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining Algorithmic Fairness in India and Beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 315–328. doi:10.1145/3442188.3445896DOI
  56. [56]Agrima Seth, Monojit Choudhury, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, and Kalika Bali. 2025. How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 2319–2330. doi:10.1609/aies.v8i3.36718DOI
  57. [57]Badr Souani, Ezekiel Soremekun, Mike Papadakis, Setsuko Yokoyama, Sudipta Chattopadhyay, and Yves Le Traon. 2025. HInter: Exposing Hidden Intersectional Bias in Large Language Models. arXiv:2503.11962 [cs.CL]
  58. [58]Robert L. Spitzer, Kurt Kroenke, Janet B. W. Williams, and Bernd Löwe. 2006. A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine 166, 10 (2006), 1092–1097. doi:10.1001/archinte.166.10.1092DOI
  59. [59]Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and Mitigating Discrimination in Language Model Decisions. arXiv:2312.03689 [cs.CL] https://arxiv.org/abs/2312.03689
  60. [60]Yi Chern Tan and L. Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. arXiv:1911.01485 https://proceedings.neurips.cc/paper_files/paper/2019/hash/201d546992726352471cfea6b0df0a48-Abstract.html
  61. [61]Alberto Testoni and Iacer Calixto. 2026. Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Maria Liakata, Viviane P. Moreira, JiaJun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 432–447. doi:10.18653/v1/2026.acl-short.36DOI
  62. [62]Aditya Tiwary Niranjan Sahoo, and Pushpak Bhattacharyya. 2025. BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context. Transactions of the Association for Computational Linguistics 13 (2025), 1672–1692. doi:10.1162/tacl_a_55arxiv.org/abs/2508.07090
  63. [63]Tyler J. VanderWeele and Miriam J. Knol. 2014. A Tutorial on Interaction. Epidemiologic Methods 3, 1 (2014), 33–72. doi:10.1515/em-2013-0005DOI
  64. [64]Laura K. Vaughan, Jasmin Divers, Miguel A. Padilla, David T. Redden, Hemant K. Tiwari, Daniel Pomp, and David B. Allison. 2009. The use of plasmodes as a supplement to simulations: A simple example evaluating individual admixture estimation methodologies. Computational Statistics & Data Analysis 53, 5 (2009), 1755–1766. doi:10.1016/j.csda.2008.02.032DOI
  65. [65]Prashanth Vijayaraghavan, Sarosh Vosoughi, Lamogha Chiazor, Raya Horesh, Rogerio Abreu de Paula, Ehsan Degan, and Vandana Mukherjee. 2025. DECASTE: Unveiling Caste Stereotypes in Large Language Models Through Multi-Dimensional Bias Analysis. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, James Kwok (Ed.). International Joint Conferences on Artificial Intelligence Organization, 9899–9907. doi:10.24963/ijcai.2025/1100 AI and Social Good.DOI
  66. [66]Eric-Jan Wagenmakers, Angelos-Miltiadis Krypotos, Amy H. Criss, and Geoff Iverson. 2012. On the Interpretation of Removable Interactions: A Survey of the Field 33 Years after Loftus. Memory & Cognition 40, 2 (2012), 145–160. doi:10.3758/s13421-011-0158-0DOI
  67. [67]Peter H. Westfall and S. Stanley Young. 1993. Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. John Wiley & Sons, New York.
  68. [68]Kyra Wilson and Aylin Caliskan. 2024. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1578–1590. doi:10.1609/aies.v7i1.31748DOI
  69. [69]Anderson M. Winkler, Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols. 2014. Permutation inference for the general linear model. NeuroImage 92 (2014), 381–397. doi:10.1016/j.neuroimage.2014.01.060DOI
  70. [70]Zhe Zhang and Daniel B. Neill. 2016. Identifying Significant Predictive Bias in Classifiers. arXiv:1611.08292 [stat.ML] NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems.arxiv.org/abs/1611.08292
  71. [71]Zheng Zhang, Yifan Wu, YuXiang Zhou, Qi Liu, Qingyang Mao, Ning Li, Weibo Gao, Qiyuan Shi, Zhenya Huang, and Baosheng Yu. 2026. Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs. arXiv:2610.35069 https://arxiv.org/abs/2610.35069

Paper details

vlnc:W_gfhack9x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code

View statement
Looking for related papers…
Contents