betaFeedback

WorksComputer Science

A Scalable Multi Task Learning Framework for Modeling Demographic Disagreements in Annotation Tasks

Abstract

Annotators from different demographic backgrounds often disagree in systematic ways due to varying lived experiences, cultural contexts, and their relationship to the content. When such disagreement occurs, one group's perspective may sometimes warrant prioritization because they hold an epistemically advantaged standpoint or bear greater stake in the outcome. For example, when assessing toxicity of posts on social media, it may be important to prioritize a targeted group's viewpoint. We propose a Stakeholder-Aware Joint Multi-Task Learning (SAJ-MTL) framework to jointly model demographic disagreement both across- and within- annotator groups. The model uses conditional gradient routing to handle sparsity induced by group-conditional labeling, and parameterization supports customized weighting of groups in different task settings. SAJ-MTL improves group-level performance in toxicity detection through better context-aware understanding of demographic-sensitive content. Only annotator group-membership information is required, which is more practical than existing methods requiring more privacy-invasive individual-level annotator details. Our scalable architecture is also computationally efficient to support large-scale deployment.

Introduction

Across many machine learning tasks, annotator judgments vary by demographic group [63]. Such disagreements depend not only on annotator demographics, but also on the critical relationship between the annotator identity and the group for whom the content is most relevant or familiar [17]. In medical diagnosis, Black and Latinx physicians in the US are historically more likely to serve in predominantly Black and Latinx communities [35], and therefore may develop more expertise on the conditions that prevail in these communities and on how symptoms present. In a different domain of annotating online discourse, the sentiment of the phrase “quite good” depends on whether the speaker is British or American, as well as the annotator’s familiarity with the idioms [39]. From stove design [21] to content moderation [64], judgment decisions are often highly context-dependent and significantly influenced by an annotator’s relationship to the content. Thus, when annotating for such highly subjective tasks, one must not only be cognizant and intentional about the relation between annotators and content, but when faced with disagreement, prioritizing certain viewpoints often becomes particularly essential in practice.

Early crowdsourcing work predominantly treated disagreement as annotator noise to be minimized, collapsing multiple judgments into a single “ground truth” via majority vote [57, 34, 45]. However, there is growing recognition that subjective annotation tasks [62, 43] such as toxicity detection [53], stance identification [24], and sentiment analysis [6] are shaped by genuine differences in how demographic groups interpret content. Thus, a growing literature now argues that disagreement encodes meaningful social variation, and choices for how to handle disagreement are value-laden [20]. Fazelpour and De-Arteaga [17] adopt a standpoint theory lens [29] to posit that the relationship between annotator identity and the content being annotated is a crucial dimension through which sociocultural identity and diversity is relevant to the machine learning pipeline. Empirical evidence lends support to this perspective. For example, Davani et al. [11] empirically demonstrate that majority aggregation systematically erases minority viewpoints, a concern also raised on conceptual grounds by Noble [44], while [22, 54, 46] show that disagreement often reflects important structured group- and individual-level differences.

While some works have tied annotator identity directly into models, this is typically done broadly as an auxiliary signal, not as a principled source of epistemic variation [70, 50, 32, 65]. Most existing methods considering annotator identity do not fully account for how the annotator-content relationship influences subjective judgments. This gap has been acknowledged in the literature; Binns [4] highlights that although some perspectives carry greater epistemic authority in fairness-critical settings, current modeling pipelines lack capabilities to explicitly operationalize such prioritization. Fleisig et al. [19], a notable exception, attempts richer modeling of annotator variation. To do this, however, it requires fine-grained individual-level data or surveys, which raises significant privacy concerns and is often infeasible because such fine-grained data is rarely available. Overall, we perceive an unmet need for a framework that: (1) models both across-group and within-group disagreements explicitly; (2) enables dynamic, context-dependent viewpoint prioritization; and (3) remains computationally tractable without requiring privacy-invasive individual annotator-level features. This modeling space remains underexplored, lacking architectures that embed disagreement and socially grounded prioritization into the predictive pipeline itself.

To address these challenges, we propose a novel multi-task learning (MTL) architecture, Stakeholder-Aware Joint MTL (SAJ-MTL), that explicitly models interactions between annotator and target demographics to model context-dependent weights on socially-grounded perspectives. SAJ-MTL treats annotator–target relationships as primary signals rather than incidental metadata. In regard to challenge (1) above, instead of smoothing away disagreement, the model captures both across-group and within-group disagreements via two protocols: Across-Groups (AG) vs. Across-and-Within-Groups (AWG). By conditioning model predictions on annotator and target identities, the framework preserves multiple perspectives rather than collapsing them into a single label. As for challenge (2), SAJ-MTL enables dynamic viewpoint prioritization by allowing the model to integrate weighting functions that reflect epistemic authority, democratic considerations of representation, or task-specific priorities, rather than assuming a single canonical perspective. Finally, for challenge (3), SAJ-MTL achieves tractable computation by factorizing annotator and target representations into an additive MTL architecture. Avoiding the quadratic explosion of per-pair modeling enables the framework to handle many stakeholder groups without sacrificing predictive performance and memory stagnation.

We evaluate SAJ-MTL on online toxicity detection, a consequential setting wherein demographic disagreements directly shape annotation outcomes. Toxicity judgments are deeply subjective, tied to lived experience, historical context, and group-specific exposure to harm. As prior work shows, language that appears innocuous to one community may be offensive when directed toward another [53, 12]. Moreover, when annotators hold negative beliefs about a group, they are less likely to label speech targeting that group as hateful [10]. Toxicity datasets frequently employ majority-vote aggregation, which suppresses minority-group opinions and systematically biases models away from those most affected by harmful content [11]. By considering the relationships between annotator and target demographics, SAJ-MTL produces a more faithful representation of how different communities perceive harm and enables viewpoint prioritization that centers the perspective of those at risk of being harmed. Evaluation on two toxicity datasets show SAJ-MTL improves context-aware understanding of demographically sensitive content and improves prediction.

Contributions: 1) Joint Stakeholder Modeling. SAJ-MTL explicitly models demographic interactions between annotator, target and content, with generality allowing broad use across diverse subjective annotation tasks with multiple stakeholders. The model dynamically assesses and weighs different, socially-grounded perspectives given specific context rather than an averaged viewpoint. Conditional gradient routing supports sparse samples per task. 2) Disagreement-Aware Protocols. We describe two protocols to systematically model annotator disagreements and preserve diverse viewpoints erased by majority-vote aggregation: Across-Groups (AG) vs. Across-and-Within-Groups (AWG). 3) Scalable MTL. SAJ-MTL reduces scaling complexity from multiplicative to additive w.r.t. group cardinality of annotator and target, ensuring feasibility and cost-effectiveness for applications with many demographic groups. 4) Better Toxicity Detection. When applied to toxicity detection, SAJ-MTL models how different annotator–target groups perceive harmful language, yielding predictions that respect demographic variation, leading to more context-sensitive toxicity judgments and reduced fairness disparities across groups. 5) Reproducibility. We will share our source code via Github, including architecture code and detailed experimental protocols.

Related Work

We connect research in machine learning (ML) and natural language processing (NLP) on annotator disagreement, content relationships, and scalable MTL.

Tasks with Annotator Disagreements. Crowdsourcing and annotation work has repeatedly found structured heterogeneity in labels that correlates with annotator attributes [63]. Early work explicitly modeled annotator reliability and systematic biases [13, 67] while later work extended this to jointly learn annotator confusion matrices with classifiers [49]. Even tasks with seemingly objective ground truth are labeled differently depending on the background, expertise, familiarity and lived experiences of the annotators, which is well-documented across domains. Judgments about stove design are highly subjective, with ideal forms varying significantly across the world and socioeconomic groups [21]. In medical diagnosis, decisions can vary based on a clinician’s training, experience, and normative practices of their professional environment [1]. Radiologists trained in different clinical settings may legitimately disagree on whether a faint shadow in an X-ray signals pathology or artifact [30]. In cardiology, recognized symptoms of a heart attack, such as crushing chest pain radiating down the left arm, are historically based on presentations in White male patients, but women and Black patients often present with different or subtler symptoms [26]. These examples motivate models that preserve annotator-level variation [11].

Modeling Annotator-Content Relationship. Recent works model different dimensions of the annotator–content relationship directly in the modeling pipeline [63]. Jury learning [22] treat annotators as stakeholders and learns to predict annotator-specific labels. Personalized classifiers [70] adapt predictions to an individual annotators’ sensitivity using per-user annotation history, while noise-aware multi-annotator frameworks [32] estimate per-annotator reliability to denoise labels. These approaches capture individual preference, but none of these models the relationship between demographics of annotators and of another stakeholder (e.g. target) as context-dependent variation. Most directly related to our work, Fleisig et al. [19] model annotator disagreement for subjective tasks by conditioning on annotator and target identity, serving as a precursor to our approach and a primary baseline. While prior work demonstrates the value of conditioning on annotator identity, in practice it can be data-hungry or privacy-invasive by requiring per-annotator surveys or identifiers. However, Sachdeva et al. [51] show that annotator behavior in subjective tasks like toxicity prediction can be efficiently modeled by decomposing judgments into independent latent traits (see Appendix § 10 for a psychometric interpretation of SAJ-MTL as a neural generalization of identity-sensitive IRT). Our work contributes to this body of literature by proposing a scalable multi-task learning approach that models the relationship between annotator and target demographics.

Demographic Bias and Fairness in NLP. Models trained on aggregated annotations are well-known to reproduce and even amplify social biases. Sap et al. [53], Davidson et al. [12] show that marginalized perspectives can be systematically underrepresented when datasets collapse annotations. Dixon et al. [15] highlight disparities in toxicity detection across demographic groups. Studies underscore the importance of fairness-aware modeling approaches that explicitly consider group-level variation.

Scalability in MTL. Scaling MTL models to many tasks (annotator ×\times target heads in our work) is challenging. Naive per-task heads typically leads to multiplicative parameter growth in both memory and runtime [5], so recent architectures emphasize parameter-efficiency and sparse conditioning. Adapter modules [27] and adapter fusion [47] allow compact per-task updates, while Low-rank adaptation [28] and prompt/prefix tuning [38] keep the per-task footprint small. Mixture-of-Experts [56, 18] uses learned sparse routing to provide additional model capacity with controlled FLOPs, while multi-gate MoE [41] and PLE [61] propose task-specific gating for MTL. Techniques for gradient conflict mitigation [69, 8] address optimization interference among tasks and remain an active area of research.

Methods

We now formalize the problem of demographic-aware annotation and introduce our Stakeholder-Aware Joint Multi-Task Learning (SAJ-MTL) framework. After highlighting inherent scalability issues with a naive multi-task formulation, we detail our scalable factorized architecture designed to efficiently capture both annotator and target group conditionality, labeling protocols, training objective and viewpoint prioritization scheme.

Notation & Problem Formulation

We formalize the annotation setting as follows. Let XX denote the space of input samples (e.g., text comments), which we map to a structured feature representation via a text encoder (e.g., BERT [14], RoBERTa [40], etc.). Let YY denote the label space (e.g., toxicity score). Each comment is associated with a structured feature vector x∈Xx \in X, an annotation label y∈Yy \in Y, and metadata about the annotator-group and the target-group (mentioned or implicated in the content). We denote A={a1,a2,…,a∣A∣}A = \{a_1, a_2, \ldots, a_{|A|}\} as the set of annotator demographic groups, and T={t1,t2,…,t∣T∣}T = \{t_1, t_2, \ldots, t_{|T|}\} as the set of target demographic groups, with cardinality as ∣A∣|A| and ∣T∣|T| respectively.

Each training instance i∈Ni \in N is a tuple (xi,ai,ti,yi)(x_i, a_i, t_i, y_i), where xi∈Xx_i \in X is the content (feature vector of post), ai∈Aa_i \in A is the annotator’s demographic group, ti∈Tt_i \in T is the target’s demographic group, and yi∈Y{0,1}y_i \in Y^{\{0,1\}} is the binary judgment noting the label given by annotator aia_i to instance xix_i when considered in relation to target tit_i (e.g., is a post hateful towards a particular demographic). We assume the observed label is drawn from a conditional distribution yi∼P(y∣xi,ai,ti)y_i \sim P(y \mid x_i, a_i, t_i), where both annotator- and target-group identities influence the group-conditioned interpretation of the content. This explicitly departs from both traditional single-truth formulations that assume yi∼P(y∣xi)y_i \sim P(y \mid x_i) alone governs labeling, and stakeholder-aware formulations that condition predictions on the content and a single stakeholder at a time.

Naive Extension of Conditional MTL

Gupta et al. [23]’s CondMTL combines a shared encoder with multiple task-specific prediction heads. One simple way to model demographic variation would be to extend CondMTL with conditional gradient routing, as shown in Figure 1, with each prediction head corresponding to a combination of an annotator group a∈Aa \in A and a target group t∈Tt \in T. While this structure allows the model to learn localized annotator–target interactions directly, it suffers from severe scalability and data sparsity issues. In this naive setup, the number of task-specific parameters grows multiplicatively at a rate of O(∣A∣×∣T∣)O(|A| \times|T|). Many demographic pairings in real-world settings, including MHS [52] and KET [37], are quite sparse (e.g., ∼1%\sim1\% of data). This makes it nearly impossible for dedicated all-pair heads to converge effectively without overfitting.

Figure 1. Naive All-Pairs SAJ-MTL. Each annotator-target pair (a,t)(a, t) is modeled as a separate task and dedicated head, allowing direct learning of pairwise interactions but yielding multiplicative parameter growth and data sparsity for underrepresented pairs.

To overcome combinatorial explosion and data sparsity issues, our improved scheme must decouple the annotator-specific variations from the target-specific contexts, while still allowing the model to account for the relationship between them. By factorizing these components, as shown in Figure 2, we can achieve a scalable architecture that learns these effects independently and composes them additively. This ensures the model scales efficiently at O(∣A∣+∣T∣)O(|A| + |T|), sharing general representation knowledge while still effectively modeling the complex, subjective interactions between the annotator and the targeted group.

Figure 2. Scalable SAJ-MTL. Annotator and target effects are factorized into independent modules and combined additively. This enables efficient modeling of annotator–target interactions while reducing complexity to linear scaling, improving both parameter efficiency and generalization in sparse settings.

Modeling Annotator-Target Conditionality

We model the subjective prediction of input xx, annotated by demographic group aa and referring to target group tt, using separate representations for the input and each stakeholder identity. Specifically, the shared encoder (with parameters ϕ\phi) produces a content embedding hϕ(x)∈Rdh_{\phi}(x) \in\mathbb{R}^{d}, while learnable embedding layers produce annotator embeddings ea∈Rdae_a \in\mathbb{R}^{d_a}, and target embeddings et∈Rdte_t \in\mathbb{R}^{d_t}, corresponding to the demographic groups aa and tt, respectively. These three embeddings are then combined through the proposed conditional architecture yielding a stakeholder-conditioned representation:

z(x,a,t)=fcond(hϕ(x),ea,et)(1)z(x, a, t) = f_{\mathrm{cond}}(h_{\phi}(x), e_a, e_t) \tag*{(1)}

where fcondf_{\mathrm{cond}} is a conditional parameterization function that composes the three components into a unified latent representation. The function may take various forms depending on modeling priorities. The fused representation is then passed through branch-specific module heads ga,gtg_a, g_t, yielding conditional predictions:

fθ(y∣x,a,t)=softmax⁡[ga(z(x,a,t))+gt(z(x,a,t))](2)f_{\theta}(y \mid x, a, t) = \operatorname{softmax}\left[g_a(z(x, a, t)) + g_t(z(x, a, t))\right] \tag*{(2)}

This enables annotator-target specific losses to backpropagate through the shared encoder hϕh_{\phi} and the relevant group-specific embeddings eae_a, ete_t. The conditional structure ensures branch-specific updates: for a given example (x,a,t,y)(x, a, t, y), gradients only adjust the corresponding annotator and target embeddings, along with the shared parameters. This selective mechanism allows the model to specialize efficiently for specific annotator-target contexts without interference from unrelated pairs, and is less prone to data sparsity issues.

Architectural Overview

SAJ-MTL implements annotator–target conditionality with each annotator–target pair corresponding to a logical task. Parameters are composed from independent annotator- and target-specific modules, rather than instantiated explicitly (see Figure 3). Whereas conventional MTL tasks are predefined and static, SAJ-MTL tasks are conditionally activated based on annotator and target metadata for each labeled instance. We implement the conditional parameterization function fcondf_{\mathrm{cond}} as a linear additive function to empirically balance expressiveness and computational efficiency.

Figure 3. Overview of the SAJ-MTL architecture. Input vector xx is encoded using a shared encoder hϕ(x)h_{\phi}(x). Annotator-group and target-group embeddings (ea,et)(e_a, e_t) are projected via lightweight linear modules (Wa,Wt)(W_a, W_t) and fused additively with the shared representation to form the conditional representation z(x,a,t)z(x, a, t). Two parallel prediction heads, annotator-specific head gag_a and target-specific head gtg_t, map this representation to logits that are summed to produce the annotator–target–conditioned prediction fθ(y∣x,a,t)f_{\theta}(y \mid x, a, t). During training, gradients update only the shared encoder and the annotator/target modules corresponding to the active pair (a,t)(a, t). At inference, predictions may be aggregated using viewpoint prioritization weights π(a,t)\pi(a, t) enabling socially grounded or policy-driven weighting of perspectives (see § 3.6).

z(x,a,t)=hϕ(x)+Waea+Wtet(3)z(x, a, t) = h_{\phi}(x) + W_a e_a + W_t e_t \tag*{(3)}

Given the conditional representation z(x,a,t)z(x, a, t), SAJ-MTL assigns a distinct prediction head to each annotator group aa and a distinct prediction head to each target group tt. Formally, let ga,gt:Rd→Rkg_a, g_t : \mathbb{R}^{d} \to\mathbb{R}^{k} denote the annotator-specific and target-specific heads, respectively. The final logits for an instance labeled by annotator aa about target group tt are obtained by combining both heads with the conditional representation:

ℓ(x,a,t)=ga(z(x,a,t))+gt(z(x,a,t))(4)\ell(x, a, t) = g_a(z(x, a, t)) + g_t(z(x, a, t)) \tag*{(4)}

The additive structure allows annotator-related semantic shifts and target-related contextual shifts to contribute independently and interpretably, unlike learning a single classifier conditioned on metadata tokens [19], which entangles annotator and target effects for less explicit disagreement modeling.

Training Objective and Optimization

Training on fine-grained annotations rather than aggregated labels preserves and learns from structured disagreement. For each labeled instance (xi,ai,ti,yi)(x_i, a_i, t_i, y_i), annotator-target–conditioned logits are produced where the conditional representation z(xi,ai,ti)z(x_i, a_i, t_i) includes shared content features and group-specific effects.

ℓ(xi,ai,ti)=gai(z(xi,ai,ti))+gti(z(xi,ai,ti))(5)\ell(x_i, a_i, t_i) = g_{a_i}(z(x_i, a_i, t_i)) + g_{t_i}(z(x_i, a_i, t_i)) \tag*{(5)}

The model is trained using the conditional negative log-likelihood:

L(θ)=−1N∑i=1Nlog⁡fθ(yi∣xi,ai,ti)=−1N∑i=1Nlog⁡softmax⁡(ℓ(xi,ai,ti))yi(6)\mathcal{L}(\theta) = -\frac{1}{N}\sum_{i=1}^{N}\log f_{\theta}(y_i \mid x_i, a_i, t_i) = -\frac{1}{N}\sum_{i=1}^{N}\log\operatorname{softmax}\left(\ell(x_i, a_i, t_i)\right)_{y_i} \tag*{(6)}

A key feature of SAJ-MTL is selective gradient routing. For each (xi,ai,ti,yi)(x_i, a_i, t_i, y_i), gradient updates are applied only to: i) the shared content encoder parameters ϕ\phi; ii) the annotator embedding eaie_{a_i} and its head gaig_{a_i}; iii) the target embedding etie_{t_i} and its head gtig_{t_i}; while all other annotator and target modules remain unchanged. This yields branch-specific backpropagation and ensures that each demographic group shapes its own subspace of the representation, allowing the model to learn distinct patterns without interference from unrelated groups.

As a conditional gradient routing model, SAJ-MTL encounters the typical challenge of gradient interference across tasks. However, because SAJ-MTL decomposes annotator and target effects additively, the parameter space naturally reduces conflicting gradients. One may further incorporate MTL optimization techniques such as gradient normalization (GradNorm [8]), gradient projection (PCGrad [69]) or multi-objective balancing (MGDA [55]), to modulate interactions between annotator and target updates. In practice, we find that the additive factorization is stable under standard optimizers such as AdamMax [33] without requiring explicit multi-task gradient control. To prevent overfitting on small groups and ensure meaningful separation between annotator and target effects, SAJ-MTL applies standard L2L_2 regularization on head parameters, dropout within the dense modules, and norm constraints on the embeddings ea,ete_a, e_t to prevent representational collapse.

Viewpoint Prioritization Mechanism

Subjective annotation tasks often require privileging certain perspectives over others, especially in contexts where lived experience or group-specific harm confers epistemic authority. To support such socially grounded decision-making, SAJ-MTL incorporates a viewpoint prioritization mechanism that modulates predictions based on the relationship between annotator and target groups. This allows the model to flexibly reflect platform policies, application goals, or normative choices about whose perspective should guide evaluation.

Given the conditional prediction function fθ(y∣x,a,t)f_\theta(y \mid x, a, t), a prioritization weight π(a,t)\pi(a,t) specifies the influence of annotator group aa when assessing content involving target group tt. The prioritization function: π:A×T→[0,1]\pi: A \times T \to[0,1] is constrained to satisfy ∑a∈Aπ(a,t)=1\sum_{a \in A} \pi(a,t) = 1 for each target group tt. At inference time, the model produces an aggregate prediction by marginalizing over annotator perspectives under the chosen weighting scheme:

y^=P(y∣x,t)=arg⁡max⁡y ∑a∈Aπ(a,t)fθ(y∣x,a,t)(7)\hat{y} = P(y \mid x,t) = \underset{y}{\arg\max}\ \sum_{a \in A} \pi(a,t)f_\theta(y \mid x,a,t) \tag*{(7)}

This general formulation allows SAJ-MTL to implement a variety of prioritization strategies proposed in prior literature on stakeholder-aware and participatory modeling [36, 59, 31, 68]. Below we relate π(a,t)\pi(a,t) to common prioritization strategies. Refer to § 4.3 for the choice and reasoning relevant to our own empirical experiments.

  • Epistemic authority weighting: π(a,t)\pi(a,t) increases for examples where a=ta=t, reflecting that members of a targeted group often have the strongest basis for judging harm.

  • Pluralistic weighting: π(a,t)\pi(a,t) is uniform across groups, where diverse perspectives are modelled equally.

  • Policy-driven weighting: π(a,t)\pi(a,t) reflects community guidelines, safety thresholds, or regulatory constraints.

  • Task-specific weighting: domain expertise or annotator proficiency determines π(a,t)\pi(a,t).

Note that viewpoint prioritization only affects aggregation and not the architecture or training procedure. The conditional model fθf_\theta continues to learn fine-grained differences in how annotators interpret content, while π(a,t)\pi(a,t) determines which perspectives are emphasized once the model is applied. This decoupling allows SAJ-MTL to be used in diverse settings without embedding implicit choices into the learned parameters. This also supports fairness-aware evaluation: by varying π(a,t)\pi(a,t), one can explore how model performance or disparities shift under other prioritization schemes, highlighting whose perspectives current systems advantage or ignore. This is especially valuable in domains like toxicity detection, where harm is not uniformly experienced and where platform governance may require accounting for group-specific viewpoints.

Experimental Setup

Our experiments consider two scenarios: a controlled setting in which target groups, tt, are known (human-annotated) vs. a more realistic setting in which targets must be noisily inferred. We first introduce these scenarios and corresponding datasets, followed by evaluation metrics, baselines, and data pipeline protocols used in our experiments. Additional implementation details can be found in Appendix § 11.

Evaluation Scenarios and Datasets

We evaluate SAJ-MTL on two datasets for toxicity and hate-related classification that include annotator demographics. Both datasets are well-suited for studying annotator disagreement, with multiple labels per instance and systematic variation across demographic groups. We distinguish two evaluation settings. First, the Annotated-Target Scenario (§ 5.1) provides a controlled-setting with definitively known targets via human annotation, i.e. annotations indicating which demographic group is the target of a post. The Measuring Hate Speech (MHS) corpus [52] supports such evaluation. Practical deployment, however, requires target detection. This is the Detected-Target Scenario, a more realistic and noisy evaluation condition (§ 5.2). This particularly matters for SAJ-MTL, since the model conditions on target-group identity during inference. Together, these two settings enable us to assess both the upper-bound performance of the model under known target conditions vs. practical effectiveness in more realistic settings. For the target detection scenario, we detect targets using a custom detection module (see Appendix § 13), then report results on MHS (ignoring human-annotated target labels) and on the KET dataset [37] (which lacks human-annotated targets).

MHS contains posts sourced from YouTube, Twitter and Reddit. Although the dataset has seven demographic race groups, we keep four demographic groups (Asian, Black, Latinx and White), omitting three others (Middle-Eastern, Native-American and Pacific-Islander) due to extremely sparse annotator–target pair instances. KET contains posts sourced from Twitter, Reddit, and 4chan, and we focus on four annotator groups based on race demographics: Asian, Black, Latino, and White. Each annotator has marked the post as being non-toxic (0) or toxic (1-4), which we binarize as y∈{0,1}y \in\{0,1\} for our formulation. Additional statistics on group frequencies and the distribution of targeted demographics are provided in Appendix § 12.

Baselines

To evaluate the effectiveness of SAJ-MTL, we compare it against two representative baseline approaches that incorporate demographic information into toxicity classification models. These baselines reflect commonly used strategies for handling demographic metadata and multi-task structures in subjective annotation tasks.

  1. Text Augmentation (TextAug) [19]. The input text is augmented with tokens describing the annotator’s demographic group (see Figure 4). The intuition is that the encoder can learn context-dependent representations conditioned on the annotator identity through attention mechanisms. The augmented input is processed by a shared encoder hϕh_\phi, and the resulting representation is passed through a classification head to produce toxicity predictions.

TextAug Pipeline

Figure 4. TextAug Pipeline. The input text is internally predicted for toxicity based on the annotator group. An additional encoder is then used to predict conditioned on the target group as well.

While this method allows demographic metadata to influence the learned encoded representation, it does not explicitly model structured relationships between annotator and target groups. The influence of demographic metadata remains entangled with the text representation, making it difficult for the model to learn distinct interaction patterns between stakeholders.

  1. Conditional Multi-Task Learning (CondMTL) Gupta et al. [23] (see § 3.2). Each task corresponds to a target demographic group, with the model learning a distinct classifier for toxicity directed at each group. It thus captures only target-specific variation in toxicity and not the relational structure of who is judging what. This baseline provides a useful reference for assessing whether explicit modeling of annotator–target relations provides useful predictive signal beyond target-aware classification alone.

Evaluation Metrics

Classification: Given the inherent class imbalance and multi-group nature of the task, we primarily focus on macro-level metrics (averaging performance across groups) to ensure that performance on minority demographic groups is weighted equally to that of majority groups. For completeness, we report the micro-level (averaging performance across all predictions) metrics as well. Metrics include Precision, Recall and F1.

Fairness: To assess how well the model respects demographic variation and reduces systemic bias, we utilize fairness-specific metrics that track disparities in error rates. Specifically, we evaluate Equal Opportunity (EO) Gap [25] as the maximum difference in TPR (Recall) between any two demographic groups, as specified in Eq. (8). In toxicity detection, lower TPR indicates that hateful posts targeting a certain group are not flagged as such.

Max. EO Gap=max⁡g∈{A,T}TPRg−min⁡g∈{A,T}TPRg(8)\mathrm{Max. \ EO\ Gap} = \max_{g \in\{A,T\}} \mathrm{TPR}_{g} - \min_{g \in\{A,T\}} \mathrm{TPR}_{g} \tag*{(8)}

Viewpoint Aggregation: We instantiate a group-aligned epistemic authority scheme that assigns larger weights when annotator and target demographics coincide (i.e., where a=ta = t). This is motivated by prior work [22, 19] suggesting that targeted group members may have greater contextual familiarity and lived experience for evaluating potentially harmful content directed toward them.

Data Pipeline Protocols

Subjective annotation datasets often contain multiple labels per instance that capture variation both across groups and within the same group. To formally model this, we consider two labeling strategies distinguishing how labels are indexed and used by SAJ-MTL: AG (Across-Groups) vs. AWG (Across-and-Within-Groups). This distinction is critical for learning differences in subjective data. The AG strategy focuses on disparate belief i.e., modeling systematic differences in labeling behavior between distinct groups. In contrast, the AWG strategy addresses the ecological fallacy of assuming group-level trends apply uniformly to all individuals by explicitly modeling variation both across groups and within each group. This enables labels to capture the fact that not all members within a group necessarily agree. Such an approach enables a granular representation that avoids over-generalization while still capturing inter-group differences.

AG (Across-Groups): Each sample receives a single, group-consolidated label per group representing the group’s average/majority perspective. Different label instances for the same sample therefore correspond strictly to different groups, treating each group’s judgment as a cohesive unit. This approach captures systematic differences between groups but does not account for individual disagreement within a group.

AWG (Across-and-Within-Groups): A single sample can have multiple labels from within the same demographic group, in addition to labels from different groups. This allows the model to see several differing opinions from individuals in the same group. This captures both inter-group and intra-group differences, reflecting the rich spectrum of individual disagreement that exists inside any demographic category.

AG captures systematic differences across demographic groups by learning from a single representative label per group, whereas AWG additionally preserves disagreement among individuals within the same demographic group. Because within-group disagreement provides additional information about the diversity of stakeholder perspectives, we hypothesize that AWG will learn richer stakeholder-conditioned representations and consequently achieve better predictive performance than AG. Our experiments in § 5 evaluate whether preserving this additional disagreement translates into measurable improvements.

Results

We evaluate SAJ-MTL along three complementary axes: 1) predictive performance; 2) fairness across demographic groups; and 3) computational scalability. Per § 4.3, all results are computed using the epistemic authority weighting function π(a,t)\pi(a,t) (§ 3.6); corresponding to the strategy in TextAug baseline [19], i.e., π(a,t)=1\pi(a,t)=1, when group identities match (a=ta=t), else 0. We compare SAJ-MTL against CondMTL and TextAug baselines (§ 4.2). Our main results use DistilBERT as text encoder; see Appendix § 15 additional results using RoBERTa-base.

Predictive Performance on MHS with Annotated-Targets

Table 1 reports classification results on the MHS dataset with human-annotated targets, and reports both micro- and macro-averaged metrics. SAJ-MTL consistently outperforms baselines, showing the benefit of explicitly modeling annotator–target relationships. In regard to SAJ-MTL variants, AWG labeling performs better than AG labeling, reaching a (macro/micro)-F1 of (0.65, 0.64), highlighting the benefit of modeling both across-group disagreements and within-group variation, which are lost when annotations are aggregated or treated as homogeneous within each demographic group. In terms of baselines, CondMTL, which models only the target demographic as a task, performs substantially worse across all metrics. This suggests that modeling the annotator group is critical for some subjective tasks such as toxicity detection, where judgments depend strongly on the relationship between the annotator and the content they are annotating. The inclusion of annotator demographics serve as an additional dimension for the model to learn a better toxicity boundary. Baseline TextAug performs better than CondMTL for most groups, but remains significantly below SAJ-MTL. This suggests that while text augmentation allows annotator metadata to influence the encoder representation, it does not provide explicit structural modeling of annotator–target interactions. Consequently, TextAug appears to struggle to disentangle how different communities interpret the same content.

Table 1: Prediction results on MHS for the Annotated-Target scenario. Mean performance and standard deviation are reported over 5 runs. The top results are bolded. SAJ-MTL with Joint-AWG labeling is best.

MetricLevelJoint-AWGJoint-AGTextAugCondMTL
F1 (↑\uparrow)Micro0.64 ±\pm 0.020.61 ±\pm 0.010.52 ±\pm 0.030.50 ±\pm 0.02
Macro0.65 ±\pm 0.010.64 ±\pm 0.020.55 ±\pm 0.030.49 ±\pm 0.01
Precision (↑\uparrow)Micro0.71 ±\pm 0.010.67 ±\pm 0.020.63 ±\pm 0.030.60 ±\pm 0.02
Macro0.75 ±\pm 0.020.73 ±\pm 0.010.66 ±\pm 0.020.59 ±\pm 0.01
Recall (↑\uparrow)Micro0.59 ±\pm 0.020.57 ±\pm 0.010.45 ±\pm 0.030.43 ±\pm 0.02
Macro0.59 ±\pm 0.010.58 ±\pm 0.020.47 ±\pm 0.020.42 ±\pm 0.02

Table 1.

In sum, results affirm that annotator disagreement contains meaningful signal: modeling this can capture more nuanced interpretations of harmful language. Figure 5 analyzes performance across specific annotator–target demographic pairings under epistemic authority weighting, with AWG labeling showing the largest gains for groups with higher internal disagreement, especially for instances involving White annotators.

F1 scores over Annotator - Target groups

Figure 5. F1 scores over Annotator - Target groups. Both of our Joint-AWG and Joint-AG models outperform the baseline models by better capturing the interaction between annotator and target demographics.

Fairness. We next examine whether modeling demographic disagreement into the framework improves fairness across groups, measured using the Equal Opportunity (EO) gap (§ 4.3) on MHS. Table 2 shows that both AG and AWG SAJ-MTL variants achieve the lowest EO gaps, with Joint-AWG and Joint-AG reducing the gap to 0.17 compared to 0.23 for TextAug and 0.29 for CondMTL. These improvements likely arise because SAJ-MTL explicitly models the relationship between annotator and target demographics. By learning separate annotator- and target-specific representations, the model can capture differences in how communities perceive harmful language without forcing a single universal decision boundary. Notably, the reduction in EO gap occurs without any fairness-specific optimization objective, suggesting that modeling disagreement itself can mitigate disparities that arise when minority perspectives are suppressed during label aggregation.

Table 2: Per-group recall (higher is better) for each demographic group, along with the corresponding Maximum Equal Opportunity (EO) Gap (lower is better). Max. EO Gap is the difference between the maximum and minimum recall across demographic groups. Both Joint-AWG and Joint-AG achieve the lowest EO Gap, indicating more equitable performance across demographic groups compared to other models.

ModelAsianBlackLatinxWhiteMax. EO Gap (↓)
Joint-AWG0.53 ± 0.010.70 ± 0.010.55 ± 0.010.57 ± 0.010.17 ± 0.01
Joint-AG0.53 ± 0.010.70 ± 0.010.55 ± 0.010.54 ± 0.010.17 ± 0.01
TextAug0.42 ± 0.030.64 ± 0.020.41 ± 0.030.41 ± 0.030.23 ± 0.03
CondMTL0.32 ± 0.020.61 ± 0.010.36 ± 0.020.40 ± 0.020.29 ± 0.02

Table 2.

Predictive Performance on MHS & KET with Detected-Targets

We present the predictive performance results of SAJ-MTL under a more realistic deployment setting in which target demographic identities are not assumed to be known a priori. Instead, target groups are predicted automatically using an auxiliary target detection module (see Appendix § 13) and these predictions are used during both training and inference. This reflects practical moderation scenarios where demographic targets (e.g. whether a post is targeted towards Black people or LatinX people) must be inferred rather than provided as human annotations. We present results for MHS (ignoring the human-annotated target labels) and KET (which lacks target annotations) in Table 3.

Table 3: Predictive performance on datasets for the Detected-Target scenario, averaged over five runs.

DatasetMetricLevelMHSKET
Joint-AWGJoint-AGTextAugCondMTLJoint-AWGJoint-AGTextAugCondMTL
F1 (↑)Micro0.58 ± 0.020.56 ± 0.020.48 ± 0.040.45 ± 0.030.75 ± 0.030.72 ± 0.040.65 ± 0.020.62 ± 0.03
Macro0.59 ± 0.010.57 ± 0.020.49 ± 0.030.45 ± 0.020.71 ± 0.020.68 ± 0.030.60 ± 0.040.56 ± 0.03
Precision (↑)Micro0.68 ± 0.020.66 ± 0.010.58 ± 0.020.53 ± 0.010.77 ± 0.030.74 ± 0.030.68 ± 0.010.65 ± 0.04
Macro0.73 ± 0.010.71 ± 0.010.58 ± 0.010.55 ± 0.010.71 ± 0.020.69 ± 0.030.63 ± 0.030.58 ± 0.05
Recall (↑)Micro0.51 ± 0.020.49 ± 0.020.42 ± 0.040.41 ± 0.030.74 ± 0.040.71 ± 0.030.62 ± 0.030.59 ± 0.04
Macro0.50 ± 0.020.49 ± 0.020.43 ± 0.030.43 ± 0.030.70 ± 0.030.67 ± 0.040.58 ± 0.020.54 ± 0.04

Table 3.

Expectedly, introducing prediction noise into the target identities result in a modest decrease in absolute performance across all methods compared to the annotated-target setting. The relative performance across models still remains the same, with SAJ-MTL giving best results. On MHS, Joint-AWG achieves the highest macro-F1 of 0.59, outperforming both TextAug and CondMTL by a substantial margin while also obtaining the strongest micro-F1 performance. Similarly, on KET, the Joint-AWG variant consistently remains the best-performing model across all reported metrics, with Joint-AG also outperforming both baseline approaches. These results demonstrate that the benefits of explicitly modeling annotator-target interactions are preserved even when stakeholder identities are estimated imperfectly, indicating that SAJ-MTL has some degree of robustness to realistic levels of target detection noise.

We further evaluate demographic fairness under the detected-target setting on MHS using the EO Gap measure, with results in Table 4. Although imperfect target predictions do introduce additional uncertainty, both SAJ-MTL variants maintain lowest EO gaps. These results indicate that stakeholder-aware modeling remains effective even when demographic target identities are inferred automatically, and that fairness improvements under human-annotated target setting are not mere artifacts of perfect demographic information, but also translate to scenarios where target detection errors are unavoidable.

ModelRecallMax. EO Gap (↓)
AsianBlackLatinxWhite
Joint-AWG0.49±0.020.49 \pm0.020.64±0.020.64 \pm0.020.44±0.020.44 \pm0.020.53±0.020.53 \pm0.020.20±0.020.20 \pm0.02
Joint-AG0.49±0.020.49 \pm0.020.64±0.020.64 \pm0.020.44±0.020.44 \pm0.020.50±0.010.50 \pm0.010.20±0.020.20 \pm0.02
TextAug0.38±0.030.38 \pm0.030.60±0.020.60 \pm0.020.35±0.030.35 \pm0.030.40±0.020.40 \pm0.020.25±0.030.25 \pm0.03
CondMTL0.30±0.020.30 \pm0.020.60±0.020.60 \pm0.020.30±0.020.30 \pm0.020.53±0.020.53 \pm0.020.30±0.010.30 \pm0.01

Table 4. Maximum EO Gap across demographic groups under noisy target prediction on MHS. Lower values indicate more equitable performance. Both Joint-AWG and Joint-AG maintain lower EO gaps than baseline methods despite using automatically predicted target identities.

Scalability

Table 5 reports the memory (number of trainable parameters) and runtime (model training for 10 epochs) of our SAJ-MTL vs. other models. Target-Only CondMTL, which does not jointly model the annotator-target interactions, serves as baseline. TextAug requires the largest parameter and runtime among all the models. Being a text-augmentation model, the DistilBERT layers were made trainable to learn updated feature representation from the augmented post. These unfrozen DistilBERT layers are responsible for the massive increase in the parameter space. Our scalable SAJ-MTL model only has four annotator and four target branches corresponding to its additive architecture in Figure 3. While maintaining comparable results w.r.t. naive SAJ-MTL model, this proposed framework scales well in terms of both memory and runtime. Even with the four demographic groups (Asian, Black, Latinx and White) and two stakeholder (Annotator and Target) interactions, SAJ-MTL has 18% fewer trainable parameters than naive SAJ-MTL. The 30% runtime increase of SAJ-MTL is expected given its eight branches vs. only four in naive SAJ-MTL. From a computational viewpoint, as the group cardinality increases in any stakeholder, scalable SAJ-MTL can accommodate model training within the same machine, without GPU memory saturation. Additional simulation results of group scaling can be found in Appendix § 15. We note that AG and AWG represent different data protocols rather than different model architectures. Consequently, both variants have identical asymptotic computational and parameter complexity.

Discussion

Role of Disagreement in Annotation. A recurring theme across subjective NLP tasks is that annotator disagreement is not merely noise but a reflection of societal and individual variation in lived experience. Literature increasingly emphasizes that collapsing multiple labels into a single ground truth can erase meaningful minority interpretations [44] and misrepresent the underlying construct being annotated. Davani et al. [11] show that aggregating labels via majority vote systematically suppresses minority demographic variation, effectively erasing the valid perspectives of marginalized groups from the training signal. Gordon et al. [22] argue that heterogeneous judgments should not be discarded, as they encode essential information about how different stakeholder groups distinctively perceive harm, offensiveness, and context. Fleisig et al. [19] highlight that relying on a forced consensus often leads models astray; they show that retaining un-aggregated annotations allows classifiers to learn robustly even when the majority opinion is contextually flawed or socially biased. The proposed methodology contributes to this body of work through a scalable multi-task learning framework for modeling disagreement in annotation tasks, which demonstrates significant improvements for modeling toxicity detection.

Interpreting Viewpoint Prioritization in Practice. While our model can represent a wide array of demographic perspectives, the decision of which viewpoint to prioritize remains a task-specific normative choice. In SAJ-MTL, this prioritization is formalized through a weighting function π(a,t)\pi(a,t), reflecting epistemic, pluralistic, policy-driven, or task-specific decisions about whose perspective should matter most. Following the perspective-aware framework of Basile et al. [3], our work supports a shift beyond a single gold standard toward a system that accounts for disaggregated human perspectives. The flexible design of π(a,t)\pi(a,t) allows practitioners to encode diverse priorities at inference time, integrating annotators’ group membership as a primary component of the model’s prediction logic rather than treating it solely as a feature of data collection. This approach directly addresses the findings of Prabhakaran et al. [48], who demonstrated that standard label aggregation (like majority voting) often introduces systematic biases by over-representing majority perspectives and obscuring the lived experiences of minority groups. In our framework, prioritization is not used to correct the model, but to enable explicit normative choices of how different voices are accounted for. For example, it can be used to center the voices of those with the highest stake in the outcome. In the context of anti-Asian hate speech, for instance, the model can be configured to prioritize the perspective of Asian annotators, ensuring the detection threshold aligns with the community most directly impacted. This flexibility enables a more equitable and context-aware moderation strategy that adapts to different normative choices without requiring retraining. SAJ-MTL thus operationalizes the principle that interpretation depends not only on what is being annotated, but on who is annotating and whom the content targets, without requiring policy priorities to affect the training pipeline.

Model type# Params (↓)Δ (↓)Time(s) (↓)Δ (↓)
Target Only (CondMTL)763,653-2200-
TextAug67,075,795+8684%4800+118%
Naive SAJ-MTL763,653+0%2400+9%
SAJ-MTL (Ours)626,133-18%2850+30%

Table 5. Memory (trainable parameters) and training time (10 epochs) comparison between models. We treat the Target-Only model as baseline for comparison, since it does not account for annotator views. While the model size of Target-Only and Naive SAJ-MTL are the same, SAJ-MTL takes longer time to train due to variation in data. The scalable SAJ variant is the best performing model in terms of trainable parameters.

Generalization to other tasks. Although in this work we explicitly focused on toxicity detection, the SAJ-MTL framework naturally generalizes to many subjective or context-sensitive tasks in NLP and beyond. Tasks such as stance classification, sentiment analysis, misinformation verification, and even medical image analysis have well-documented sources of demographic variation in annotation. For instance, in medical image analysis, diagnoses can vary systematically based on a clinician’s training, experience, and the demographic presentation of the patient. Rather than treating such clinical disagreement as noise, SAJ-MTL could be used to factorize these expertise and patient-context variables to preserve critical diagnostic signals [71]. As noted by Sorensen et al. [58], as AI systems are increasingly deployed in global contexts, they must move away from average human preferences toward a value-pluralistic stance. Because SAJ-MTL’s conditional structure simply combines shared semantic encoders with annotator- and target-specific branches, it does not assume anything special about toxicity labels and can be applied wherever group-conditioned interpretation is relevant. Moreover, because inference-time aggregation is decoupled from model training, domain experts in these diverse fields can tailor viewpoint prioritization to reflect domain-specific norms, regulatory structures, or governance criteria without retraining the underlying model.

Conclusion

Human annotations in subjective tasks often reflect socially grounded disagreement rather than objective ground truth. Single-truth or majority-voted paradigms treat such variation as noise, discarding valuable signal about annotator and target-group perspectives. To address this, we introduced SAJ-MTL, a joint annotator–target–aware multi-task learning framework that explicitly models interactions between annotators’ demographics and content. Using a factorized conditional architecture with selective gradient routing, SAJ-MTL captures structured cross- and within-group variation while scaling linearly with the number of groups and requiring only minimal group-membership information in order to preserve privacy. A flexible viewpoint prioritization mechanism further decouples disagreement modeling from deployment-time choices, allowing practitioners to encode epistemic authority, pluralism, or policy constraints without retraining. Applied to toxicity detection, SAJ-MTL improves macro-level performance on demographic-sensitive content and reduces group-level disparities, all without explicit fairness objectives.

Beyond toxicity, SAJ-MTL offers a general template for learning from subjective annotations where stakeholder perspectives vary systematically. The additive factorization of annotator and target groups suggests a natural extension to a stakeholder triad that incorporates author demographics, as recent works show that author dialect and identity also shape toxicity perception [7, 42]. Integrating such signals additively would capture triadic dynamics while maintaining our privacy-preserving, group-level formulation. More broadly, future work could relax rigid group categories by learning hierarchical or latent group structures, enabling the model to discover intersectional identities obscured by broad demographic labels. Finally, SAJ-MTL’s architecture is well-suited to multi-modal applications: applied to medical imaging with a Vision Transformer encoder, it could reconcile systematic diagnostic disagreements across clinical traditions or specialties, by treating divergent expert judgments as signal rather than noise.

Ethical Considerations

The ethical stakes of modeling annotator disagreement are significant. Without explicit design choices, ML systems risk implicitly weighing majority viewpoints, thereby amplifying harms to marginalized or under-represented groups and reinforcing pre-existing biases [53, 12]. Still, decisions around how groups are defined, aggregated, or weighted require careful ethical consideration. Across application domains, anchoring these decisions on domain-knowledge, and engaging in participatory design practices may be important avenues to ensure their relevance and legitimacy. Additionally, it is important to note that there is a risk of misuse. It could be possible to define viewpoint prioritization functions that perpetuate or exacerbate marginalization by purposefully centering privileged viewpoints. Thus, transparency regarding viewpoint prioritization mechanisms deployed is essential.

The proposed methodology does not explicitly enforce fairness. Instead, fairness benefits stem from improvements in modeling. This is, however, not guaranteed to happen, and it is essential to explicitly evaluate fairness performance, as illustrated in our experiments. In sensitive domains such as toxicity detection, minimizing performance gaps is not merely an algorithmic optimization metric, but a necessary safeguard against asymmetric harms. Errors may disproportionately affect i) the author of content, via unfair over-moderation or mischaracterization, and ii) the communities targeted by that content, via failure to filter out or appropriately react to harmful content. Careful evaluation across stakeholder groups is therefore required and necessary to avoid reinforcing existing social inequities.

Our experiments rely exclusively on publicly available datasets [52, 37], and no additional human participants were recruited or annotations collected. We use the datasets subject to their appropriate access and usage conditions. The use of annotator demographic information introduces privacy and representation concerns. Although our methodology operates on group-level demographic attributes, such information may still be sensitive in some scenarios [60, 2, 66], and its use in deployed ML systems should therefore be subject to appropriate privacy guardrails, informed consent where applicable, and restrictions against discriminatory profiling.

Acknowledgment

We thank the data annotators who made this work possible, as well as our anonymous reviewers for their valuable feedback. This research was supported in part by Amazon, Wipro, the Knight Foundation, the Micron Foundation, and by Good Systems1, a UT Austin Grand Challenge to develop responsible AI technologies. The statements made herein are solely the opinions of the authors and do not reflect the views of the sponsoring agencies.

SAJ-MTL as a Neural Generalization of Identity-Sensitive IRT

This section provides a psychometric interpretation of SAJ-MTL through the lens of identity-sensitive Item Response Theory (IRT) [52]. We show that the proposed architecture preserves the additive inductive structure underlying latent trait models while extending them into a scalable neural conditional representation framework over annotator-target demographic spaces. We also note that while classical IRT formulations such as the 2PL model express probability multiplicatively as P=σ(a(θ−β))P = \sigma(a(\theta- \beta)), the identity-sensitive IRT variant of Sachdeva et al. [52] operates in the log-odds (latent response) space, where item, annotator, and demographic factors enter additively. SAJ-MTL preserves this additive inductive bias, extending it from scalar latent variables to high-dimensional neural representations.

The IRT framework models annotation behavior through additive latent response factors corresponding to item characteristics and annotator-dependent perception effects. Their final formulation defines:

log⁡[pnijkpnij(k−1)]=θn−δi−αj−τk+βjm+μlm(9)\log\left[\frac{p_{nij k}}{p_{nij(k-1)}}\right] = \theta_n - \delta_i - \alpha_j - \tau_k + \beta_{jm} + \mu_{lm} \tag*{(9)}

where θn\theta_n denotes the latent toxicity tendency of item nn, δi\delta_i represents item difficulty, αj\alpha_j captures annotator-specific sensitivity or severity, τk\tau_k denotes ordinal threshold parameters, βjm\beta_{jm} captures whether annotator jj exhibits increased or decreased sensitivity toward content targeting demographic group mm, and μlm\mu_{lm} captures systematic alignment effects between annotator identity ll and targeted demographic identity mm.

SAJ-MTL follows an analogous additive inductive bias, but replaces scalar latent variables with learned high-dimensional neural representations. Specifically, we define a task-conditional prediction function fθ(y∣x,a,t)f_\theta(y \mid x, a, t) jointly on: (i) input text xx, (ii) annotator demographic group aa, and (iii) targeted demographic group tt.

The shared conditional representation z(x,a,t)z(x, a, t) is instantiated through the additive parameterization:

z(x,a,t)=hϕ(x)+Waea+Wtetz(x, a, t) = h_{\phi}(x) + W_{a}e_{a} + W_{t}e_{t}

Under this interpretation, the shared encoder representation hϕ(x)h_{\phi}(x) generalizes latent item-dependent effects (θn,δi)(\theta_n, \delta_i), the annotator component WaeaW_{a}e_{a} generalizes annotator sensitivity effects (αj)(\alpha_j), and the target component WtetW_{t}e_{t} introduces demographic-target conditioning absent from classical IRT formulations.

Thus, SAJ-MTL preserves the additive structure of latent trait models while extending them from scalar psychometric variables to structured high-dimensional neural conditional representations.

The interaction-sensitive term βjm\beta_{jm} aligns closely with the demographic-conditioned prediction structure of SAJ-MTL fθ(y∣x,a,t)f_{\theta}(y \mid x, a, t), and in particular with the additive parameterization, Wa,t=Ua+VtW_{a,t} = U_{a} + V_{t}, where UaU_{a} models systematic annotator-group response tendencies, and VtV_{t} models target-group-specific toxicity sensitivity. While βjm\beta_{jm} represents scalar interaction adjustments in a probabilistic latent trait framework, SAJ-MTL generalizes this interaction sensitivity into high-dimensional conditional neural representations.

The term μlm\mu_{lm} captures alignment effects between annotator identity and targeted demographic identity. Of particular importance are self-interactions (l=m)(l = m), corresponding to annotators evaluating content targeted toward their own demographics (a=t)(a = t). This directly parallels the epistemic authority weighting formulation used in SAJ-MTL as π(a,t)\pi(a, t), which models evaluation according to annotator-target demographic alignment.

A key distinction, however, is that identity-sensitive IRT incorporates these effects directly within probabilistic parameter estimation, whereas SAJ-MTL decouples demographic weighting from representation learning. Consequently, alternative viewpoint prioritization schemes may be applied post hoc during evaluation without retraining the underlying predictive model. Taken together, SAJ-MTL may be interpreted as a structured neural extension of identity-sensitive IRT modeling, where scalar latent demographic effects are generalized into high-dimensional conditional representations and modular viewpoint-aware inference mechanisms. A mapping correspondence of different stages of SAJ-MTL to the IRT formulation is given in Table 6.

IRT ComponentInterpretationSAJ-MTL Correspondence
θn,δi\theta_n, \delta_iItem toxicity and difficultyShared encoder representation hϕ(x)h_{\phi}(x)
αj\alpha_jAnnotator-specific sensitivityAnnotator component UaU_a
βjm\beta_{jm}Annotator–target interaction sensitivityConditional demographic predictor fθ(y∣x,a,t)f_{\theta}(y \mid x, a, t)
μlm\mu_{lm}Group-level demographic alignmentEpistemic authority weighting π(a,t)\pi(a, t)
Eq. (3)Additive latent response decompositionShared conditional representation learning
Eq. (5)Interaction-sensitive demographic modelingDemographic-conditioned prediction parameterization
Eq. (6)Group-level viewpoint weightingViewpoint-prioritized inference aggregation

Table 6. Stage-wise correspondence between identity-sensitive IRT formulations and SAJ-MTL.

Experimental Details

Setup. Experiments use a Nvidia 2060 RTX Super 8GB GPU, Intel Core i7-9700F 3.0GHz 8-core CPU and 16GB DDR4 memory. We use Keras [9] library on a Tensorflow 2.0 backend with Python 3.12 to train the networks in this paper. For optimization, we use AdaMax [33] with parameters (lrlr=0.001) and 1000 steps per epoch. We also setup an EarlyStopping criteria in case the with a min_delta of 1E-4 and patience of 5. The random seed was set to 42 to avoid variability.

Optimizer. We choose AdaMax (infinity-norm variant of Adam) due to its faster convergence from fresh starts, i.e., model training from random weights. It is also more stable than Adam in our case with sparse or unbalanced gradients. This makes tuning the learning rate less fragile.

Regularization. We do not need any explicit regularization term in our SAJ-MTL loss formulation, since all the task branches are treated equally and each branch has a conditional binary cross entropy loss acting on it. For branches having extreme sparsity, we have independent ℓ2\ell_2 regularization for smoothness, which we have explicitly set to λa,t=0.005\lambda_{a,t} = 0.005. However, during our epistemically weighted evaluation over both datasets, this additional term and its variation did not cause any metrics to fall outside the reported variance range.

Training protocol: To ensure a fair comparison with SAJ-MTL, all models are trained using the same shared encoder architecture and optimization procedure. All models are trained on individual annotations rather than majority-vote labels, ensuring that disagreement signals are preserved across methods. For SAJ-MTL, annotator and target embeddings are initialized randomly and projected into the shared representation space using linear transformations Wa,WtW_a, W_t before being combined with the shared representation. Predictions are then produced through the additive combination of annotator-specific and target-specific heads as described in § 3.3. By comparing these approaches, we evaluate whether explicitly modeling the interaction between annotator and target demographics, rather than incorporating metadata implicitly or modeling only target groups, can achieve improved predictive performance and fairness.

Dataset Details

We use the Measuring Hate Speech (MHS) corpus [52], available via the HuggingFace library [16], which includes 135,556 posts sourced from YouTube, Twitter and Reddit. Each post is annotated by one or more human annotators and includes: i) the inferred demographic group(s) targeted by the post; ii) self-reported group-demographic attributes of the annotators; and iii) a binary toxicity label indicating whether the content is perceived as toxic or non-toxic towards the inferred target. Although the dataset has seven demographic race groups as target, we keep only four demographic groups - Asian, Black, Latinx and White, and omit three others (Middle-Eastern, Native-American and Pacific-Islander). This is due to extremely sparse annotator–target (Ai,Tj)(A_i, T_j) pair instances in the three omitted groups. For included groups, both annotators and targets were drawn as shown in Table 7. Because individual posts may be annotated by multiple annotators, we perform the train–validation–test split at the post level to prevent content leakage across splits. Specifically, posts are randomly partitioned into 70%, 10%, and 20% subsets for training, validation, and testing, respectively. All annotations associated with a given post are assigned to the same split. Within each split, a post may still have multiple annotator labels, which allows us to model annotator-specific variation while preserving strict separation of textual content across splits. The dataset exhibits substantial variation in the frequency with which different demographic groups are targeted. For example, the Black community represents the statistical majority among targeted groups.

Target GroupAsianBlackLatinxWhiteAll Annotators
Asian621 (1.2%)721 (1.4%)551 (1.1%)5,501 (10.8%)7,394 (14.6%)
Black1,560 (3.1%)2,292 (4.5%)1,654 (3.3%)18,545 (36.6%)24,051 (47.4%)
Latinx638 (1.3%)961 (1.9%)700 (1.4%)6,657 (13.1%)8,956 (17.6%)
White711 (1.4%)1,086 (2.1%)706 (1.4%)7,825 (15.4%)10,328 (20.4%)
All Targets3,530 (6.7%)5,060 (10.0%)3,611 (7.1%)38,528 (75.9%)50,729 (100%)

Table 7. MHS corpus statistics showing the number of posts that match a specific annotator-target pair (Ai,Tj)(A_i, T_j) over four demographic groups. Cells highlighted in green denote the ones considered under epistemic authority weighting.

The KET dataset [37] consists of posts sourced from Twitter, Reddit, and 4chan. We focus on four annotator groups based on race demographics: Asian, Black, Latino, and White. Each annotator has marked the post as being non-toxic (0) or toxic (1-4), which we binarize as y∈{0,1}y \in\{0,1\} for our formulation. Because target groups are not annotated for this dataset, one must detect targets automatically, as in Fleisig et al. [19]. Again, due to lack of pre-defined splits, we randomly partitioned posts into 70%, 10%, and 20% subsets for training, validation, and testing, respectively. In this dataset, the White group has the majority of posts, as shown in Table 8.

Target GroupAsianBlackLatinxWhiteAll Annotators
Unmarked33,140 (15.3%)56,580 (26.1%)19,500 (9.0%)107,480 (49.6%)216,700 (100%)

Table 8. KET statistics showing posts that match a specific annotator (Ai)(A_i) over four demographic groups.

Automatic Target Detection

In many real-world deployment settings, the demographic group targeted by a post may not be explicitly available and must instead be inferred automatically. While MHS provides human-annotated target-groups, such targets will typically need to be automatically (noisily) detected. Consequently, stakeholder-aware prediction systems often rely on an auxiliary model to estimate target identities before downstream classification. We perform automatic target detection to evaluate SAJ-MTL under this more realistic setting, to assess the robustness of stakeholder-aware modeling when demographic target assignments contain errors.

Specifically, let tannt_{ann} denote the annotated gold target demographic label associated with a sample. Instead of providing tannt_{ann} directly to the toxicity classifier, we replace it with a predicted target label tpredt_{pred} obtained from an auxiliary target-group classifier as tpred=ftarget(x)t_{pred} = f_{target}(x), where ftargetf_{target} predicts one or more of the demographic target categories from the text alone. During both training and evaluation of downstream toxicity models, all methods receive tpredt_{pred} rather than the annotated target label tannt_{ann}.

For MHS, we ignore gold labels and use predicted labels. For KET, target identities are not annotated.

Figure 6. Target-group detection model. The final layer has ∣T∣|T| nodes, each containing a classification layer.

Similar to the protocol by Fleisig et al. [19], we use an custom target detection model. For their work, they used a GPT-2 based module fine-tuned on a different dataset. However, note that different platforms and sources of data come with their own length formats, audience demographics, moderation policies etc. Hence to prevent such artifacts originating from decoupling, our custom target detection model uses the same underlying text encoder architecture as the downstream models and is trained independently using available target annotations where applicable i.e., the MHS dataset in our case. This makes the entire pipeline coupled in terms of data source and text representation. To avoid information leakage, target detection and downstream toxicity classification are trained separately on the same splits. The text encoder is followed by dense neuron layers (512, 128, 64 neurons respectively), with relu activation, added biases and dropout rate of 0.1 to prevent model overfitting. The layers end in ∣T∣|T| classification nodes, each with sigmoid activation, no biases and 0.5 threshold. Given that a single post can target one or more groups simultaneously, we use weighted Binary Cross Entropy (wBCE) as our loss to optimize overall error. The rounded detection accuracy rates for the demographic groups were as follows: Asian: 81%, Black: 87%, Latinx: 82%, White: 77%, with the corresponding confusion matrix shown in Table 9.

Detected Target Group
AsianBlackLatinxWhite
Annotated Target Group — Asian5,9891511071,147
Annotated Target Group — Black24220,9242472,638
Annotated Target Group — Latinx1211737,3441,318
Annotated Target Group — White6879857037,953

Table 9. Confusion matrix (counts) for MHS between using the target-group detection model.

Additional Results

Table 10 presents the statistics on MHS while switching from DistilBERT to RoBERTa-base as the input text encoder with frozen weights. It is expected that RoBERTa’s enhanced capabilities could further improve these scores due to its larger size and advanced pre-training strategies. The results indicate that Joint-AWG and Joint-AG consistently outperforms the other baseline methods in both Micro and Macro evaluations for all metrics. This suggests that SAJ-MTL variants effectively balances precision and recall, demonstrating high accuracy and relevance in its predictions by accounting for demographic disagreements. The Joint-AWG is still the best model by considering within and across group variations. This analysis not only highlights the modeling strength, but also potential improvements with powerful encoders, weighing against computational trade-offs for practical deployment considerations.

MetricLevelJoint-AWGJoint-AGTextAugCondMTL
F1 (↑)Micro0.67 ± 0.010.63 ± 0.020.56 ± 0.020.52 ± 0.02
Macro0.70 ± 0.020.66 ± 0.010.57 ± 0.020.50 ± 0.01
Precision (↑)Micro0.75 ± 0.020.70 ± 0.010.66 ± 0.030.62 ± 0.02
Macro0.78 ± 0.010.74 ± 0.020.66 ± 0.030.60 ± 0.01
Recall (↑)Micro0.61 ± 0.010.58 ± 0.020.48 ± 0.020.45 ± 0.01
Macro0.63 ± 0.020.60 ± 0.020.50 ± 0.020.43 ± 0.02

Table 10. Performance on MHS in the Annotated-Target Scenario using RoBERTa-base as text encoder. Mean model performance and deviations are reported over 5 runs. Best performance is highlighted in bold.

Scalability of SAJ-MTL

A key design goal of SAJ-MTL is to scale nominally with increasing numbers of annotator and target groups. However, in the naive design as shown in Figure 1, this leads to multiplicative parameter growth of O(∣A∣×∣T∣)O(|A| \times|T|). SAJ-MTL addresses this challenge through a factorized parameterization in which annotator and target effects are learned independently and composed additively. With a single shared encoder, a small set of annotator and target embeddings, the total number of group-specific parameters scales additively as (O(∣A∣+∣T∣))(O(|A| + |T|)) (Figure 2). This ensures tractability even in settings with dozens of annotator groups or highly diverse target identities. Concretely, rather than instantiating a separate head or parameter block for each (a,t)(a, t) pair, SAJ-MTL reuses a single shared encoder and a set of small annotator and target modules, allowing the model to represent structured disagreement without incurring combinatorial cost.

In practice, this design leverages parameter-efficient model design ideas from [27, 28]. Beyond memory scalability, this parameterization enhances computational efficiency. Because only the annotator and target modules relevant to an example receive gradient updates, SAJ-MTL amortizes shared backbone updates and computation across tasks. By conditioning predictions on small learned embeddings rather than duplicating large network components, batch-level throughput remains stable even as the number of groups grows. Compared to frameworks such as Mixture-of-Experts [56, 41], which introduce routing and communication overhead, SAJ-MTL achieves scalability with minimal architectural overhead for embedding lookup and head routing. This design ensures fairness-aware modeling remains deployable in resource-constrained environments. Finally, additive factorization promotes generalization across sparse groups. Groups with limited labeled examples benefit from the shared encoder while still receiving individualized modeling capacity. Conversely, well-represented groups can develop more specialized representations through repeated branch-specific updates. This makes SAJ-MTL robust to the long-tailed, heterogeneous distributions common in subjective labeling settings.

Figure 7 shows a simulation of memory scaling across three models vs. some annotator-target pairings. All models have a fixed DistilBERT weight as minimum memory and additional dense layers. When the number of groups is one, we see that CondMTL and SAJ-MTL have effectively the same memory, since they end up with the same architecture. With increasing groups, the quadratic scaling kicks in and we see the memory for naive SAJ-MTL quickly saturating towards GPU limit, a phenomena exhibited by any standard MTL architecture. Furthermore, the Scalable SAJ-MTL variant almost grows asymptotically w.r.t. CondMTL (which just considers target group), all the while capturing interactions across both annotator-target groups.

Figure 7. Synthetic experiment on scalability. Every model has a fixed frozen (non-trainable) DistilBERT representation as their minimum memory. SAJ MTL's quadratic scaling nature with increasing groups is evident, while the Scalable variant is much more memory efficient due to having linear scaling with groups.

References

  1. [1]Erin P Balogh, Bryan T Miller, and John R Ball (Eds.). 2015. Improving Diagnosis in Health Care. National Academies Press, Washington, DC. doi:10.17226/21794DOI
  2. [2]Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. Calif. L. Rev. 104 (2016), 671.
  3. [3]Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silvio Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future. 15–21.DOI
  4. [4]Reuben Binns. 2020. On the apparent conflict between individual and group fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 514–524.arxiv.org/abs/1912.06883
  5. [5]Rich Caruana. 1993. Multitask learning: a knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning (Amherst, MA, USA) (ICML ’93). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 41–48.DOI
  6. [6]Isabel Carvalho, Hugo Goncalo Oliveira, and Catarina Silva. 2023. The importance of context for sentiment analysis in dialogues. IEEE Access 11 (2023), 86088–86103.
  7. [7]Prateek Chaudhry and Matthew Lease. 2022. You are what you tweet: Profiling users by past tweets to improve hate speech detection. In International Conference on Information. Springer, 195–203.arxiv.org/abs/2012.09090
  8. [8]Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normal-ization for adaptive loss balancing in deep multitask networks. In International conference on machine learning. PMLR, 794–803.
  9. [9]François Chollet. 2015. keras. https://github.com/fchollet/keras.
  10. [10]Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023. Hate speech classifiers learn normative social stereotypes. Transactions of the Association for Computational Linguistics 11 (2023), 300–319.DOI
  11. [11]Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics 10 (2022), 92–110.arxiv.org/abs/2110.05719
  12. [12]Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial Bias in Hate Speech and Abusive Language Detection Datasets. In Proceedings of the Third Workshop on Abusive Language Online. 25–35.arxiv.org/abs/1905.12516
  13. [13]Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28, 1 (1979), 20–28.
  14. [14]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). 4171–4186. doi:10.18653/v1/N19-1423
  15. [15]Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Wasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 67–73.
  16. [16]Hugging Face. 2022. ucberkeley-dlab/measuring-hate-speech. https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech
  17. [17]Sina Fazelpour and Maria De-Arteaga. 2022. Diversity in sociotechnical machine learning systems. Big Data & Society 9, 1 (2022), 20539517221082027.arxiv.org/abs/2107.09163
  18. [18]William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39.
  19. [19]Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 6715–6726.
  20. [20]Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2025. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation 59, 2 (2025), 1719–1746. doi:10.1007/s10579-024-09766-4 Published online: August 18, 2024.DOI
  21. [21]William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. Advances in Neural Information Processing Systems 35 (2022), 12979–12990.DOI
  22. [22]Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19.DOI
  23. [23]Soumyajit Gupta, Sooyong Lee, Maria De-Arteaga, and Matthew Lease. 2023. Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection. In Proceedings of the ACM Web Conference 2023. 3689–3700.arxiv.org/abs/2302.07372
  24. [24]Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance detection for mis-and disinformation identification. In Findings of the Association for Computational Linguistics: NAACL 2022. 1259–1277.arxiv.org/abs/2103.00242
  25. [25]Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems (NeurIPS) 29 (2016).arxiv.org/abs/1610.02413
  26. [26]Aaron L Hilliard, David E Winchester, Tanya D Russell, and Rosland D Hilliard. 2020. Myocardial infarction classification and its implications on measures of cardiovascular outcomes, quality, and racial/ethnic disparities. Clinical cardiology 43, 10 (2020), 1076–1083.DOI
  27. [27]Neil Houlsby, Andrei Giurgiu, Stanislau Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790–2799.arxiv.org/abs/1902.00751
  28. [28]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3.arxiv.org/abs/2106.09685
  29. [29]Kristen Intemann. 2010. 25 years of feminist empiricism and standpoint theory: Where are we now? Hypatia 25, 4 (2010), 778–796.DOI
  30. [30]Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 590–597.arxiv.org/abs/1901.07031
  31. [31]Nathan Kallus. 2018. Balanced policy evaluation and learning. Advances in neural information processing systems 31 (2018).arxiv.org/abs/1705.07384
  32. [32]Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Miłkowski, Jan Kocoń, and Przemysław Kazienko. 2022. What if ground truth is subjective? personalized deep neural hate speech detection. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@LREC2022. 37–45.
  33. [33]Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR 2015).arxiv.org/abs/1412.6980
  34. [34]Beata Beigman Klebanov and Eyal Beigman. 2009. Squibs: From annotator agreement to noise models. Computational Linguistics 35, 4 (2009), 495–503.
  35. [35]Miriam Komaromy, Kevin Grumbach, Michael Drake, Karen Vranizan, Nicole Lurie, Dennis Keane, and Andrew B Bindman. 1996. The role of black and Hispanic physicians in providing health care for underserved populations. New England Journal of Medicine 334, 20 (1996), 1305–1310.
  36. [36]Arie W Kruglanski, Amiram Raviv, Daniel Bar-Tal, Alona Raviv, Keren Sharvit, Shmuel Ellis, L Mannetti, et al. 2005. Says who? Epistemic authority effects in social judgment. Advances in experimental social psychology 37, 37 (2005), 345–92.DOI
  37. [37]Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021). 299–318.arxiv.org/abs/2106.04511
  38. [38]Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3045–3059.arxiv.org/abs/2104.08691
  39. [39]Bing Liu. 2012. Sentiment Analysis and Opinion Mining (1 ed.). Springer Cham. XIV, 167 pages. doi:10.1007/978-3-031-02145-9DOI
  40. [40]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).arxiv.org/abs/1907.11692
  41. [41]Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1930–1939.DOI
  42. [42]Zhenxiong Miao, Xingshu Chen, Haizhou Wang, Rui Tang, Zhou Yang, Tiemai Huang, and Wenyi Tang. 2023. Detecting offensive language based on graph attention networks and fusion features. IEEE Transactions on Computational Social Systems 11, 1 (2023), 1493–1505.DOI
  43. [43]An Thanh Nguyen, Matthew Halpern, Byron C. Wallace, and Matthew Lease. 2016. Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 149–158.
  44. [44]Jennifer A Noble. 2012. Minority voices of crowdsourcing: Why we should pay attention to every member of the crowd. In proceedings of the ACM 2012 conference on computer supported cooperative work companion. 179–182.DOI
  45. [45]Stefanie Nowak and Stefan Rüger. 2010. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In Proceedings of the international conference on Multimedia information retrieval. 557–566.
  46. [46]Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 1017–1029.
  47. [47]Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 487–503.arxiv.org/abs/2005.00247
  48. [48]Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the joint 15th linguistic annotation workshop (LAW) and 3rd designing meaning representations (DMR) workshop. 133–138.
  49. [49]Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of machine learning research 11, 4 (2010).DOI
  50. [50]Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies. 175–190.
  51. [51]Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia Von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022. 83–94.
  52. [52]Pratik S Sachdeva, Renata Barreto, Claudia von Vacano, and Chris J Kennedy. 2022. Assessing annotator identity sensitivity via item response theory: A case study in a hate speech corpus. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1585–1603.DOI
  53. [53]Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics. 1668–1678.DOI
  54. [54]Maarten Sap, Swabha Swaydampita, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies. 5884–5906.
  55. [55]Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). 525–536.arxiv.org/abs/1810.04650
  56. [56]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations (ICLR 2017).arxiv.org/abs/1701.06538
  57. [57]Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing. 254–263.
  58. [58]Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024). 46280–46302.arxiv.org/abs/2402.05070
  59. [59]Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Ryting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. Position: A Roadmap to Pluralistic Alignment. In International Conference on Machine Learning. PMLR, 46280–46302.arxiv.org/abs/2402.05070
  60. [60]Latanya Sweeney. 2013. Discrimination in online ad delivery. Commun. ACM 56, 5 (2013), 44–54.
  61. [61]Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In Proceedings of the 14th ACM conference on recommender systems. 269–278.
  62. [62]Yuandong Tian and Jun Zhu. 2012. Learning from crowds in the presence of schools of thought. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. 226–234.DOI
  63. [63]Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research 72 (2021), 1385–1470.DOI
  64. [64]Kristen Vaccaro, Ziang Xiao, Kevin Hamilton, and Karrie Karahalios. 2021. Contestability for content moderation. Proceedings of the ACM on human-computer interaction 5, CSCW2 (2021), 1–28.DOI
  65. [65]Michiel Van Der Meer, Neele Falk, Pradeep K Murukannaiah, and Enrico Liscio. 2024. Annotator-centric active learning for subjective NLP tasks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). 18537–18555. doi:10.18653/v1/2024.emnlp-main.1031DOI
  66. [66]Michael Veale and Reuben Binns. 2017. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society 4, 2 (2017), 2053951717743530.
  67. [67]Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 (2009).
  68. [68]Chung-Hsing Yeh, Robert J Willis, Hepu Deng, and Hongqi Pan. 1999. Task oriented weighting in multi-criteria analysis. European journal of operational research 119, 1 (1999), 130–146.DOI
  69. [69]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836.arxiv.org/abs/2001.06782
  70. [70]Xueying Zhan, Yaowei Wang, Yanghui Rao, and Qing Li. 2019. Learning from multi-annotator data: A noise-aware classification framework. ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 1–28.DOI
  71. [71]Chen Zhong, Yuxuan Yang, Xinyue Zhang, Ruohan Ma, Yong Guo, Gang Li, and Jupeng Li. 2026. Learning from Disagreement: A Group Decision Simulation Framework for Robust Medical Image Segmentation. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7527–7531.arxiv.org/abs/2510.10462

Paper details

vlnc:W_dqnk237x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents