betaFeedback

WorksComputer Science

Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Political Discourse

Abstract

Large language models (LLMs) are increasingly used to interpret, generate, and mediate public-facing language, which makes their reliability especially consequential in politically sensitive settings. Although sentiment analysis is a common task in this area, affective meaning in social media discourse is often conveyed through culturally situated and pragmatically indirect cues that are not well captured by label-centric evaluation alone. In this paper, we evaluate the robustness of LLMs for sentiment analysis of Spanish political discourse from social media, with emphasis on culturally situated affective interpretation. We introduce a diagnostic protocol that combines three independent human annotation views into Gold and Ambiguous subsets, and extends the corpus with a Synthetic subset built from sentiment-flipped counterfactuals selected through model-conditioned perplexity and LLM-as-a-judge criteria. Using this framework, we compare proprietary and open-weight models under zero-shot and few-shot prompting, focusing on predictive robustness across human agreement conditions and counterfactual shift. Our results show that annotation ambiguity and counterfactual perturbation reveal failure modes that conventional evaluation on the original corpus tends to obscure. GPT-4.1-mini remains the most stable among the evaluated models, whereas the two smaller open-weight models degrade more markedly under synthetic perturbations, and few-shot prompting provides only limited gains. Overall, the paper contributes a reproducible protocol for robustness-oriented evaluation in political sentiment analysis, an empirical characterization of model behavior under ambiguity and sentiment-flipped counterfactuals, and a publicly available Spanish social media dataset with three-way human sentiment annotations.

Introduction

Large language models (LLMs) have rapidly become foundational components of contemporary language technologies, supporting tasks such as classification, summarization, explanation generation, content moderation, retrieval support, and synthetic text production [1]. As their deployment expands across education, industry, and public-sector workflows, so does their capacity to shape how information is produced, interpreted, and circulated. This is particularly consequential in democratic contexts, where language is not merely a medium of communication but a core mechanism of participation, deliberation, and accountability [2]. In this sense, LLMs may strengthen the public sphere by improving access to information, facilitating citizen-government interaction, supporting civic engagement, and helping identify citizen-reported issues from large-scale social media data [3, 4]. At the same time, their generative and persuasive capabilities create new risks, including the large-scale production of convincing political content, the simulation of citizen-like discourse, and the amplification of misinformation or disinformation narratives that may affect public trust and democratic representation [5, 6, 7].

Affective signals are central to this problem because political communication often seeks not only to inform, but also to mobilize, persuade, polarize, or delegitimize actors and institutions. Within natural language processing (NLP), sentiment analysis is a long-standing task for identifying opinions, affective orientation, and evaluative stance in text, and it has been widely used to summarize large volumes of user-generated content [8]. In political and participatory settings, these methods can support the analysis of online discussions, public concerns, and deliberative processes [9]. Prior work has also advanced sentiment analysis for Spanish social media in electoral contexts, showing the relevance of annotated corpora and LLM-based evaluation for politically charged discourse [10, 11]. However, political sentiment in social media is rarely reducible to stable labels or surface polarity. Its interpretation often depends on tone, irony, emojis, hashtags, local references, and culturally shared pragmatic cues. Thus, strong performance under standard metrics may conceal deeper interpretive limitations, especially when models face ambiguous or culturally situated language. This aligns with recent arguments that evaluating generative AI should be treated as a measurement challenge, where the target construct and the validity of the evaluation instrument must be explicitly examined [12, 13].

Although recent work has produced increasingly sophisticated protocols for improving and evaluating formal multi-step reasoning in LLMs, including mathematical reasoning through prompting, chain-of-thought variants, and self-reflection mechanisms [14, 15], analogous protocols for culturally situated reasoning remain less mature [16]. Emerging research on cultural awareness, cultural alignment, and cultural benchmarks has shown that multilingual competence does not imply cultural competence, and that LLMs remain brittle when reasoning about culture-specific knowledge, social norms, and situated meanings [17, 18, 19].

Evaluation practice itself is also changing. The LLM-as-a-judge paradigm has emerged as a scalable alternative for scoring, ranking, and selecting open-ended outputs, and has been shown to approximate human preferences reasonably well in some settings [20]. Yet judge models are not neutral evaluators: they can exhibit systematic biases, remain vulnerable to perturbations, and require careful reliability analysis and protocol design [21, 22]. For this reason, LLM-based judgment is best treated not as a replacement for human grounding, but as one component within a broader evaluation framework. This motivates evaluation designs that combine human grounding with model-assisted assessment rather than relying exclusively on either source.

In this paper, we build on these lines of work to study whether LLMs maintain culturally situated affective inferences when Spanish political language is humanly ambiguous or perturbed through counterfactual rewrites. We focus on Spanish social media discourse and move beyond conventional label-centric evaluation by combining three independent human annotation views with perturbation-based stress testing. Concretely, we construct a Gold subset from full annotator agreement, an Ambiguous subset from partial or complete annotator disagreement, and a Synthetic subset composed of sentiment-flipped counterfactuals generated from non-neutral posts. Synthetic instances retain the agreement condition of their source posts, allowing us to compare model behavior on original Gold and Ambiguous instances with counterfactual variants derived from them. We then compare GPT-4.1-mini, Llama3.1:8b, and DeepSeek-r1:8b under zero-shot and few-shot prompting, focusing on predictive robustness across human agreement conditions and counterfactual shift.

Our contributions are threefold:

  • We adapt and operationalize an evaluation protocol for Spanish political sentiment analysis that uses multi-annotator consensus, annotation ambiguity, and counterfactual perturbation as diagnostic conditions for assessing LLM robustness.

  • We empirically characterize robustness differences between proprietary and open-weight LLMs under zero-shot and few-shot prompting, showing how performance changes across Gold, Ambiguous, and Synthetic conditions.

  • We will release the human sentiment annotations, evaluation artifacts, and Spanish political discourse dataset.

The remainder of this paper is organized as follows. Section 2 reviews related work on cultural reasoning, political sentiment analysis, and model-assisted evaluation. Section 3 presents the evaluation protocol, including the dataset, agreement-based subset construction, and counterfactual generation and selection. Section 4 describes the evaluated models, prompting settings, and metrics. Section 5 reports and discusses the main findings regarding annotation ambiguity, counterfactual shift, and prompting strategy. Finally, Section 6 concludes the paper and discusses implications for robust and culturally aware LLM evaluation. Show all of the project history or only labelled versions.

Related Work

This work is grounded in two complementary research lines. The first concerns reasoning-aware evaluation and model-assisted assessment of LLM outputs. Recent work has shown that evaluating LLMs is increasingly difficult because their outputs are open-ended, context-dependent, and not always well captured by single-task accuracy measures [1]. More broadly, natural language reasoning involves heterogeneous forms of inference over unstructured text, which makes evaluation especially challenging when tasks require more than surface-level pattern recognition [16]. In this context, the LLM-as-a-judge paradigm has emerged as a scalable mechanism for scoring, ranking, and selecting generated outputs. Zheng et al. [20] showed that strong LLM judges can approximate human preferences in conversational evaluation, making model-based assessment useful when human evaluation is costly. However, subsequent studies have emphasized that judge models are not neutral instruments. Chen et al. [21] show that both human and LLM judges may exhibit systematic biases, while Li et al. [22] discuss broader opportunities and limitations of using LLMs as evaluators. These findings motivate protocols in which model-assisted judgment may support candidate selection and diagnostic evaluation, but must remain constrained, documented, and grounded in human-annotated data.

The second line concerns cultural reasoning, cultural alignment, and politically sensitive language. Work on cultural alignment has shown that LLM behavior varies with prompting language, training mixtures, and cultural context [17]. Chiu et al. [18] further show that even strong models remain limited on diverse, human-authored cultural knowledge tasks, reinforcing Pawar et al.’s [19] view that cultural awareness requires more than multilingual capability, including sensitivity to social norms, values, and situated context. This gap is especially relevant because recent progress in LLM reasoning has focused heavily on mathematical or formal multi-step reasoning mechanisms [14, 15]. By contrast, culturally situated reasoning remains less mature as an evaluation target, particularly when the task involves affective interpretation rather than factual cultural knowledge.

Sentiment analysis in political discourse provides a concrete setting where these issues converge. As a well-established task for identifying opinions and affective orientation in text [8], it has been widely applied to social media data for analyzing political behavior and electoral dynamics in Latin America [10]. In Spanish electoral social media, prior work has already introduced annotated resources and benchmarked pre-trained and large language models mainly through predictive sentiment performance [11]. The present paper takes this line of research in a different direction by shifting the emphasis from standard label prediction to robustness under annotator ambiguity and counterfactual perturbation. Methodologically, it builds on Ochieng et al. [13], who argue that sentiment in culturally nuanced contexts should be evaluated beyond labels through ambiguity and counterfactual analysis. This use of counterfactual perturbation also connects to broader work on counterfactual reasoning and generation, where minimally altered rewrites are used to examine whether models preserve relevant invariances under controlled changes [23]. By adapting this perspective to Spanish political social media discourse, we position our work at the intersection of model-assisted evaluation and cultural reasoning, examining how ambiguity and counterfactual shifts affect predictive robustness in culturally situated sentiment classification.

Evaluation Protocol

This section describes the protocol used to transform a Spanish political sentiment corpus into a diagnostic evaluation setting for LLM robustness and culturally situated affective reasoning. The protocol treats human consensus, human ambiguity, and counterfactual shift as complementary evaluation conditions for assessing whether models go beyond label prediction and remain consistent under culturally and pragmatically sensitive variation.

Dataset and Human Annotation

We use a Spanish political social media corpus collected between January 1 and July 31, 2024, from Facebook, Instagram, X, and YouTube, originally introduced by Valverde-Rebaza et al. [11]. The corpus contains posts related to the 2024 Mexican electoral process and was annotated independently by three native Spanish-speaking annotators using three sentiment categories: positive, negative, and neutral. In this paper, we use the resulting three-way annotation matrix as the basis for diagnostic evaluation, while the details of data collection, filtering, annotation guidelines, and inter-annotator agreement are provided in the original corpus description.

The resulting dataset comprises 1,354 annotated posts distributed across the four platforms, as summarized in Table 1. In the table, POS, NEG, and NEU denote the consolidated sentiment labels; GOLD and AMBIGUOUS refer to the agreement-based subsets derived from the three human annotations; and SYNTHETIC denotes the non-neutral instances used for counterfactual generation.

PlatformPOSNEGNEUGOLDAMBIGUOUSSYNTHETICTOTAL
Facebook1449995234104243338
Instagram1935297220122245342
X7227631253126348379
YouTube146721414215381295
Total4234944378495059171,354

Table 1. Distribution of sentiment labels and evaluation subsets by platform.

When considering platform-level composition, X provides the largest subset, with 379 posts, followed by Instagram with 342, Facebook with 338, and YouTube with 295. The sentiment composition varies substantially across platforms. Facebook is comparatively balanced, with 144 positive, 99 negative, and 95 neutral posts, whereas Instagram is skewed toward positive sentiment, with 193 positive posts against 52 negative and 97 neutral instances. X exhibits a sharply polarized profile toward negative sentiment, with 276 negative posts and a much smaller presence of 72 positive and 31 neutral instances. YouTube, in turn, is dominated by neutral content, with 214 neutral posts, 67 negative posts, and only 14 positive posts. These differences indicate that the corpus does not represent a homogeneous affective space across platforms. Accordingly, while platform-level variation is reported for transparency, the evaluation in this paper uses agreement structure and counterfactual perturbation as the primary diagnostic axes.

Figure 1 complements Table 1 by visualizing the platform-level distribution of Gold and Ambiguous instances across sentiment categories. The figure shows that ambiguity is not uniformly distributed, with YouTube showing the highest proportion of ambiguous cases, whereas Facebook, Instagram, and X concentrate a larger share of Gold instances. This supports our use of agreement structure as a central diagnostic axis in the evaluation protocol.

Figure 1. Distribution of Gold and Ambiguous instances by platform and sentiment label. The Synthetic subset is constructed from non-neutral Gold and Ambiguous instances through sentiment-flipped counterfactuals.

Gold and Ambiguous Subsets

The three independent annotations associated with each post allow us to derive two agreement-based subsets, reported in Table 1. The Gold subset includes posts for which the three annotators assigned the same sentiment label, representing cases of full human consensus. The Ambiguous subset includes all posts without full agreement, covering both partial agreement cases, in which two annotators selected the same label and one disagreed, and complete disagreement cases, in which the three annotators selected different labels.

Overall, the Gold subset contains 849 posts, corresponding to 62.7% of the annotated corpus, while the Ambiguous subset contains 505 posts, corresponding to 37.3%. At the platform level, Facebook includes 234 Gold and 104 Ambiguous posts, Instagram includes 220 Gold and 122 Ambiguous posts, X includes 253 Gold and 126 Ambiguous posts, and YouTube includes 142 Gold and 153 Ambiguous posts. As shown in Figure 1, YouTube is the only platform where Ambiguous instances slightly exceed Gold instances, whereas Facebook, Instagram, and X show larger proportions of Gold cases.

This partition is central to the evaluation protocol. Gold instances serve as the reference condition for cases in which sentiment interpretation is stable across human annotators. Ambiguous instances, in contrast, are treated as a diagnostic condition rather than as annotation noise. In political social media discourse, disagreement may reflect the presence of implicit stance, irony, platform-specific conventions, or context-dependent affective cues. For this reason, comparing model behavior across Gold and Ambiguous subsets allows us to evaluate whether LLMs remain robust when the sentiment signal is less explicit and less consistently interpreted by humans.

Counterfactual Generation and Selection

To evaluate robustness under controlled affective perturbation, we construct a Synthetic subset composed of sentiment-flipped counterfactuals. In line with reasoning-aware evaluation in culturally nuanced sentiment analysis [13] and broader work on counterfactual rewriting [23], these counterfactuals are used to test whether models respond consistently when the affective orientation of a post is intentionally modified while its topic, informal style, and political context are preserved. Figure 2 summarizes the complete evaluation flow, distinguishing the direct classification path followed by Gold and Ambiguous instances from the counterfactual construction path followed by Synthetic instances.

Experimental flow for evaluating cultural reasoning in political sentiment analysis

Figure 2. Experimental flow for evaluating cultural reasoning in political sentiment analysis. Gold and Ambiguous instances are classified using the original posts and evaluated against the original sentiment label, whereas Synthetic instances are first constructed through counterfactual generation and selection and then evaluated against the expected inverted sentiment.

Counterfactual generation is applied only to non-neutral posts, since polarity inversion is directly defined for positive and negative instances but less straightforward for neutral content. As reported in Table 1, the Synthetic subset is derived from 917 eligible source instances, corresponding to the 423 positive and 494 negative posts in the corpus. At the platform level, X contributes the largest number of eligible source posts, with 348 non-neutral instances, followed by Instagram with 245, Facebook with 243, and YouTube with 81. This distribution reflects the sentiment profile of each platform, since X contributes many counterfactual sources due to its strong concentration of negative posts, whereas YouTube contributes fewer because most of its content is neutral.

For each eligible post, the counterfactual construction process is model-conditioned. That is, for each model-specific run, the same LLM associated with that run is prompted to generate three alternative rewrites that invert the original sentiment. Although Figure 2 abstracts this step as counterfactual candidate generation, the generation itself is performed by the corresponding LLM under an explicit rewriting prompt. The transformation may involve changes in sentiment-bearing keywords, negation, tone, framing, emojis, hashtags, or sentence structure. The objective is not to generate unrestricted paraphrases, but to create plausible variants that alter the affective orientation while preserving the communicative setting of the original post. This model-conditioned design means that the Synthetic condition should be interpreted as an end-to-end diagnostic of each model’s ability to generate, select, and classify sentiment-flipped variants, rather than as an externally validated counterfactual benchmark.

The resulting candidates are then filtered through two complementary selection strategies. The first is based on perplexity and selects the candidate with the lowest model-based expectedness score, using perplexity as a proxy for linguistic plausibility within the corresponding model run. The second uses an LLM-as-a-judge criterion, where the same model-conditioned LLM selects the candidate that best preserves fluency, plausibility, contextual consistency, and clarity of the sentiment shift. Thus, although the figure presents this stage compactly as counterfactual selection, the judge-based selection stage is also model-assisted. These criteria are kept separate because they capture different aspects of counterfactual suitability: perplexity favors linguistically expected formulations, whereas the judge-based criterion prioritizes communicative adequacy and the explicitness of the affective inversion.

After selection, the chosen counterfactual becomes the input for the Synthetic classification condition. In this condition, the model does not classify the original post; instead, it classifies the selected counterfactual text, and the expected label is the inverted sentiment relative to the original non-neutral post. For example, a positive original post is expected to become negative after counterfactual rewriting, and a negative original post is expected to become positive. Synthetic instances also retain the agreement condition of their source posts. Thus, Synthetic-Gold refers to counterfactuals generated from non-neutral posts originally belonging to the Gold subset, whereas Synthetic-Ambiguous refers to counterfactuals generated from non-neutral posts originally belonging to the Ambiguous subset. This distinction allows us to compare model behavior on original posts and counterfactual variants while preserving the role of human agreement as a diagnostic axis.

Experimental Setup

This section describes how the proposed protocol was instantiated across models, prompting strategies, and evaluation metrics. The experiments compare model behavior over the Gold, Ambiguous, and Synthetic conditions, focusing on sentiment classification robustness under human agreement, annotator ambiguity, and counterfactual shift. In all conditions and as shown in Figure 2, the target task remains sentiment prediction; what changes is the input type and the reference label used for evaluation. Gold and Ambiguous instances are classified using the original post and the consolidated sentiment label, whereas Synthetic instances are classified using the selected counterfactual text and the expected inverted sentiment label.

Models

We evaluate three LLMs selected to cover proprietary and open-weight deployment settings. GPT-4.1-mini is used as the proprietary reference model. Llama3.1:8b is included as an open-weight instruction-tuned model, providing a reproducible baseline for local deployment. DeepSeek-r1:8b is included as an open-weight reasoning-oriented model, allowing us to examine whether reasoning-specialized behavior is reflected in sentiment interpretation under ambiguity and counterfactual shift.

The comparison should be interpreted with attention to model scale and availability. Llama3.1:8b and DeepSeek-r1:8b are 8B-parameter models and therefore correspond to relatively small LLMs. By contrast, GPT-4.1-mini is a proprietary model whose exact parameter count is not publicly documented. Consequently, the experiments should not be read as a general comparison between proprietary and open-weight models, nor as representative of larger frontier models or larger open-weight alternatives. Rather, they compare three accessible models under the same robustness-oriented diagnostic protocol.

All models were evaluated under the same diagnostic protocol and label space. Sentiment labels were encoded as three mutually exclusive classes: negative, neutral, and positive. To reduce stochastic variation in classification and rubric-based scoring, temperature was set to 0.0 for sentiment prediction and explanation evaluation. For counterfactual generation, temperature was increased to 0.9 to encourage lexical and pragmatic diversity among candidate rewrites. Counterfactual selection was performed with a low-temperature setting to favor stable candidate choice.

Prompting Settings

We evaluate each model under zero-shot and few-shot prompting. In the zero-shot setting, the model receives the target post and the sentiment label definitions, without labeled examples. In the few-shot setting, the same classification instruction is augmented with labeled examples sampled from the annotated corpus. We use up to k=3k = 3 examples per class with a fixed random seed, yielding a balanced demonstration set whenever sufficient examples are available. The same sampling procedure is applied across models to preserve comparability.

All classification prompts require a structured JSON response containing the predicted sentiment label, a confidence score from 0 to 5, supporting keywords, and a brief explanation. In this paper, explanations are retained as part of the model output for traceability, but the quantitative analysis focuses on predictive robustness. This structured response format constrains the output to the predefined label space and reduces invalid or verbose generations.

For the Synthetic condition, prompting is used in two additional stages already defined in § 3.3. First, the counterfactual generation prompt asks the model to produce three sentiment-flipped rewrites of each eligible non-neutral post while preserving topic, political context, informal tone, and plausibility. Second, the judge-based selection prompt asks the corresponding model-conditioned LLM to choose the candidate that most clearly and plausibly expresses the intended sentiment inversion. The perplexity-based selection condition does not require an additional prompt, since it ranks candidates through model-based expectedness rather than judgment instructions.

Table 2 summarizes the core instructions implemented in the prompt templates used in the experimental pipeline. The table reports functionally equivalent descriptions rather than verbatim prompts, while the complete raw templates will be released with the experimental artifacts.

PromptCore instruction and expected output
Sentiment classificationClassify the input post as negative, neutral, or positive. Return a JSON object containing the predicted label, supporting keywords, a brief explanation, and a confidence score from 0 to 5.
Few-shot classificationApply the same classification schema using labeled examples sampled from the annotated corpus. Include up to k=3k = 3 examples per sentiment class when available.
Counterfactual generationGenerate three alternative rewrites of a non-neutral post that invert its sentiment while preserving language, topic, political context, informal tone, and plausibility. Return each candidate as a JSON object.
Counterfactual selectionSelect the candidate that most effectively inverts sentiment while preserving fluency, realism, and contextual consistency. Return the selected counterfactual, its predicted sentiment, and a brief justification.

Table 2. Functional summary of the prompt templates used in the experimental pipeline.

Metrics

The evaluation metrics are aligned with the three diagnostic conditions defined in the protocol. For Gold and Ambiguous instances, the model prediction is compared against the consolidated sentiment label of the original post. For Synthetic instances, the model prediction is compared against the expected inverted label of the selected counterfactual. Thus, the classification task remains sentiment prediction in all cases, but the Synthetic condition evaluates whether the model recognizes the intended polarity shift introduced by the counterfactual rewrite.

For sentiment prediction, we report F1-scores by class and evaluation condition. Results are computed separately for Gold, Ambiguous, and Synthetic subsets, allowing us to compare performance under full human agreement, annotator disagreement, and controlled sentiment shift. This separation is central to the experimental design because aggregate performance over the original corpus may obscure failures that emerge only when sentiment is ambiguous or deliberately perturbed.

For the Synthetic condition, we also use two auxiliary diagnostics to interpret the counterfactual construction process. The flip rate measures whether the selected counterfactual changes the sentiment label relative to the original post. In this setting, a successful flip indicates that the rewritten text expresses the intended opposite polarity. We also compute token-level Levenshtein distance between the original post and the selected counterfactual, which estimates the magnitude of textual modification introduced by the rewrite. These measures are used as construction-level diagnostics, whereas the main comparative analysis is based on sentiment classification performance across models, prompting strategies, and evaluation conditions.

Results and Discussion

This section reports the empirical results obtained under the proposed diagnostic protocol. We focus on three complementary views. First, we compare F1-scores across Gold, Ambiguous, and Synthetic conditions. Second, we contrast these results with a contextual reference derived from conventional original-corpus sentiment classification. Third, we analyze the drop in F1-score relative to the Original-Gold condition, which represents the clearest setting in terms of human agreement.

F1 Performance Under Diagnostic Conditions

Table 3 reports the F1-scores obtained by each model under zero-shot and few-shot prompting. The table separates the original classification setting from the two Synthetic conditions, in which counterfactuals were selected either by perplexity or by an LLM-as-a-judge criterion. Gold and Ambiguous refer to the agreement condition of the source post. Thus, in the Synthetic conditions, Synthetic-Gold corresponds to counterfactuals derived from non-neutral posts originally belonging to the Gold subset, whereas Synthetic-Ambiguous corresponds to counterfactuals derived from non-neutral posts originally belonging to the Ambiguous subset.

ConditionPromptGPT GoldGPT Amb.DeepSeek GoldDeepSeek Amb.Llama GoldLlama Amb.
OriginalZero-shot0.880.440.760.390.760.39
OriginalFew-shot0.890.470.770.410.720.38
Synthetic-PPLZero-shot0.650.550.270.320.300.34
Synthetic-PPLFew-shot0.660.550.280.300.280.30
Synthetic-JudgeZero-shot0.640.550.290.310.310.30
Synthetic-JudgeFew-shot0.650.540.310.310.290.28

Table 3. F1-scores by model, prompting strategy, and diagnostic condition. Gold and Amb. refer to the agreement condition of the source post. Synthetic-PPL and Synthetic-Judge correspond to model-conditioned counterfactuals selected by perplexity and LLM-as-a-judge, respectively.

The Original-Gold results show that the models can perform competitively when human annotators fully agree on the sentiment label. GPT-4.1-mini obtains the highest scores, with 0.88 in zero-shot and 0.89 in few-shot. DeepSeek-r1:8b reaches 0.76 and 0.77, while Llama3.1:8b obtains 0.76 and 0.72. These results indicate that, under clear human consensus, all models capture a substantial part of the affective signal, although the proprietary model remains stronger.

The Ambiguous condition produces a systematic reduction in performance. In the original dataset, GPT-4.1-mini drops to 0.44 in zero-shot and 0.47 in few-shot. DeepSeek-r1:8b reaches 0.39 and 0.41, while Llama3.1:8b reaches 0.39 and 0.38. This pattern supports the methodological decision to treat annotator disagreement as a diagnostic condition rather than as annotation noise. When human annotators disagree, LLMs also become less reliable, suggesting that political sentiment in social media may depend on pragmatic, contextual, or culturally situated cues that are not captured by surface polarity alone.

Comparison with Conventional Classification

Figures 3 and 4 provide a visual summary of the F1-scores reported in Table 3, complemented by two horizontal reference lines. These lines correspond to a comparable conventional original-corpus sentiment classification reference derived from prior work on the same Spanish political discourse corpus [11]. The reference is contextual rather than a direct baseline, since the prior evaluation was organized by platform and did not include agreement-aware or counterfactual conditions.

The reference values were computed from the mean Weighted-F1 across platforms for the two most comparable models reported in the prior benchmark, namely GPT-4o-mini and Llama-3.1. For each prompting setting, the platform-level Weighted-F1 values were first averaged within each model and then averaged across the two models. Formally, for a prompting setting pp, the reference is computed as:

Ref⁡p=12(14∑s∈SF1GPT4o,p,s+14∑s∈SF1Llama3.1,p,s),(1)\operatorname{Ref}_{p} = \frac{1}{2}\left(\frac{1}{4}\sum_{s \in S} F1_{\mathrm{GPT4o},p,s} + \frac{1}{4}\sum_{s \in S} F1_{\mathrm{Llama3.1},p,s}\right), \tag*{(1)}

where S={Facebook,Instagram,X,YouTube}S = \{\text{Facebook}, \text{Instagram}, \text{X}, \text{YouTube}\}. This gives a zero-shot reference of 0.768 and a few-shot reference of 0.782.

Figure 3 shows that Original-Gold performance for GPT-4.1-mini exceeds the conventional reference, while DeepSeek-r1:8b and Llama3.1:8b are close to or below that reference depending on the prompting setting. However, all Synthetic-Gold scores fall below the conventional reference. This is particularly evident for DeepSeek-r1:8b and Llama3.1:8b, whose Synthetic scores remain near 0.27–0.31. GPT-4.1-mini also decreases under counterfactual shift, but remains substantially above the open-weight models evaluated here, with Synthetic-Gold scores between 0.64 and 0.66.

Figure 3. F1-scores on Gold instances, with contextual reference lines from conventional original-corpus sentiment classification.

Figure 4 shows a stronger contrast. All Ambiguous results remain below the conventional reference, including the original classification condition. This indicates that a conventional original-corpus benchmark can substantially overestimate the robustness of sentiment classification when it does not distinguish between high-consensus and low-consensus instances. Interestingly, GPT-4.1-mini performs better on Synthetic-Ambiguous instances than on Original-Ambiguous instances, reaching approximately 0.54–0.55 under both selection strategies. This suggests that selected counterfactuals may sometimes express the intended polarity more explicitly than original posts that humans themselves judged inconsistently. The open-weight models do not show the same recovery and remain close to 0.28–0.34 in Synthetic-Ambiguous settings.

Figure 4. F1-scores on Ambiguous instances, with contextual reference lines from conventional original-corpus sentiment classification.

Robustness Drop Relative to Original-Gold

To quantify robustness degradation, we compute the F1-score difference between each condition and the corresponding Original-Gold result for the same model and prompting strategy. The drop is defined as:

ΔF1m,p,c=F1m,p,c−F1m,p,Original−Gold,(2)\Delta F1_{m,p,c} = F1_{m,p,c} - F1_{m,p,\mathrm{Original-Gold}}, \tag*{(2)}

where mm denotes the model, pp the prompting strategy, and cc the evaluated condition. Under this definition, the Original-Gold condition has ΔF1=0\Delta F1 = 0 by construction. Negative values indicate performance degradation relative to the clearest human-agreement condition.

Figure 5 confirms that both ambiguity and counterfactual shift produce substantial degradation, but with different profiles across models. For Original-Ambiguous, the drop ranges from −0.34 to −0.44. GPT-4.1-mini loses 0.44 F1 points in zero-shot and 0.42 in few-shot. DeepSeek-r1:8b loses 0.37 and 0.36, while Llama3.1:8b loses 0.37 and 0.34. These values confirm that human disagreement is a strong indicator of model difficulty across all model families.

The Synthetic conditions reveal a sharper separation between GPT-4.1-mini and the open-weight models. For GPT-4.1-mini, Synthetic-Gold produces drops of approximately −0.23 to −0.24, regardless of whether counterfactuals are selected by perplexity or by LLM-as-a-judge. For DeepSeek-r1:8b, the Synthetic-Gold drop reaches −0.49 under perplexity selection and −0.47 or −0.46 under judge-based selection. Llama3.1:8b shows a similar degradation, with drops between −0.43 and −0.46. These results indicate that counterfactual perturbation exposes robustness gaps that are much less visible in Original-Gold evaluation.

The comparison between Synthetic-PPL and Synthetic-Judge also shows that the general pattern is stable across selection strategies. GPT-4.1-mini consistently remains the most robust model, while DeepSeek-r1:8b and Llama3.1:8b remain substantially more vulnerable to synthetic sentiment shifts. This suggests that the main factor driving the observed degradation is not only the specific counterfactual selection criterion, but the model’s ability to recognize and preserve affective inversion under controlled rewriting.

Effect of Few-shot Prompting

We also examine whether few-shot prompting mitigates the degradation observed under ambiguity and counterfactual shift. As shown in Table 3, the gains from few-shot prompting are small and inconsistent across models and conditions. In the Original-Gold condition, GPT-4.1-mini increases only from 0.88 to 0.89, and DeepSeek-r1:8b from 0.76 to 0.77, whereas Llama3.1:8b decreases from 0.76 to 0.72. A similar pattern appears in Original-Ambiguous, where GPT-4.1-mini improves from 0.44 to 0.47 and DeepSeek-r1:8b from 0.39 to 0.41, while Llama3.1:8b decreases slightly from 0.39 to 0.38.

The Synthetic conditions reinforce this interpretation. For GPT-4.1-mini, few-shot prompting produces almost no change under either selection strategy, moving from 0.65 to 0.66 in Synthetic-PPL-Gold and from 0.64 to 0.65 in Synthetic-Judge-Gold, while remaining effectively stable in the corresponding Ambiguous-derived subsets. DeepSeek-r1:8b shows small gains only in some cases, such as Synthetic-Judge-Gold, where it increases from 0.29 to 0.31, but it decreases under Synthetic-PPL-Ambiguous, from 0.32 to 0.30. Llama3.1:8b does not benefit consistently from few-shot examples, decreasing in most Synthetic conditions reported in Table 3.

These results indicate that the main difficulty is not simply the absence of task examples or output-format guidance. If the errors were primarily due to prompt underspecification, few-shot prompting would be expected to produce more systematic improvements. Instead, the limited gains suggest that the performance loss is driven by the intrinsic difficulty of interpreting politically situated affective meaning under annotator ambiguity and controlled sentiment perturbation. This is also consistent with Figure 5, where few-shot rows exhibit degradation patterns comparable to their zero-shot counterparts rather than a clear recovery toward the Original-Gold condition.

Figure 5. F1-score drop relative to Original-Gold. Negative values indicate degradation with respect to the same model and prompting strategy under full annotator agreement.

Limitations

Our results should be interpreted within the scope of the proposed diagnostic design. First, the Synthetic condition was constructed through a model-assisted and model-conditioned pipeline. Counterfactual generation and judge-based selection may therefore introduce self-selection bias, since a model may select rewrites that are more compatible with its own generation style or easier for itself to classify. Consequently, the Synthetic-Judge results in Table 3 should be read as end-to-end diagnostic evidence of model behavior, rather than as externally validated evidence of counterfactual quality.

Second, the comparison between Synthetic-PPL and Synthetic-Judge partially mitigates this concern. The similar degradation patterns in Figure 5 suggest that the main robustness trends are not entirely dependent on a single selection criterion. Nevertheless, neither perplexity-based selection nor LLM-as-a-judge selection replaces human validation. Accordingly, this study does not make claims about counterfactual quality in terms of fluency, naturalness, meaning preservation, or clarity of sentiment inversion, since these dimensions were not systematically assessed.

Third, the scope of the evaluated models and experimental settings is limited. The study compares three relatively small LLMs under zero-shot and few-shot prompting, without larger frontier models, fine-tuned supervised baselines, or sentiment-specific models trained on the target domain. Thus, the observed degradation patterns should not be interpreted as definitive evidence about all LLMs or about open-weight models in general. Differences among GPT-4.1-mini, DeepSeek-r1:8b, and Llama3.1:8b may reflect interacting factors such as model scale, instruction tuning, multilingual coverage, training data composition, and alignment strategy.

Finally, the analysis does not model platform-specific behavior beyond descriptive corpus statistics. Although Table 1 shows substantial variation across Facebook, Instagram, X, and YouTube, the evaluation treats agreement condition and counterfactual perturbation as the primary diagnostic axes. Platform-aware modeling is therefore left for future work. Similarly, model explanations were retained for traceability, but explanation quality was not evaluated as a main empirical axis.

Conclusion

This paper evaluated LLM robustness for Spanish political sentiment analysis through a diagnostic protocol that goes beyond conventional original-corpus classification. Rather than treating sentiment prediction as a single aggregate task, we reorganized evaluation around three conditions: Gold instances with full human agreement, Ambiguous instances with annotator disagreement, and Synthetic instances generated through sentiment-flipped counterfactual perturbation. The results show that conventional sentiment classification can provide an incomplete picture of model reliability. In the Gold subset, the models achieved comparatively strong F1-scores, with GPT-4.1-mini exceeding the contextual conventional reference derived from prior platform-based evaluation. However, performance decreased substantially when the same task was evaluated under human ambiguity and counterfactual shift. This indicates that high performance on standard sentiment benchmarks does not necessarily imply robustness when affective meaning is culturally situated, pragmatically indirect, or unstable even among human annotators.

The main empirical finding is that annotator ambiguity and counterfactual perturbation expose different forms of model fragility. Ambiguous instances produced systematic drops across all models, suggesting that human disagreement is a meaningful signal of interpretive difficulty rather than merely annotation noise. Synthetic instances further revealed robustness gaps that were especially pronounced for the open-weight models. GPT-4.1-mini remained the most stable model across conditions, although it also degraded outside the Gold setting. DeepSeek-r1:8b and Llama3.1:8b performed reasonably on clear original instances but showed stronger degradation under counterfactual shift. These results suggest that politically situated sentiment analysis in Spanish requires evaluation protocols that explicitly account for disagreement, perturbation, and contextual affective cues.

The comparison with conventional sentiment classification is central to the contribution of this work. Standard evaluations organized by platform or aggregate performance remain valuable for benchmarking models under ordinary classification conditions. However, they do not fully address whether models preserve sentiment interpretation when language becomes ambiguous or when sentiment is intentionally inverted while preserving topic and political con text. Our results therefore do not replace conventional sentiment analysis; rather, they complement it by showing where conventional evaluation may overestimate robustness. This distinction is particularly important for political discourse, where sentiment is often expressed through irony, emojis, hashtags, local references, implicit stance, and platform-specific communicative conventions.

This study has several limitations. First, the Synthetic condition was constructed through a model-assisted counterfactual pipeline, and therefore should be interpreted as a diagnostic stress condition rather than as a fully independent human-authored benchmark. Second, although counterfactual generation and selection were supported by perplexity and LLM-as-a-judge criteria, the present paper does not provide a full human validation of counterfactual quality. Third, we retained model explanations as part of the structured output, but we did not include explanation quality as a main empirical axis in this version of the paper. A more rigorous analysis of explanation faithfulness, contextual adequacy, and reasoning consistency remains necessary before making stronger claims about explanatory robustness. Fourth, the analysis did not model platform-specific behavior beyond reporting corpus composition, even though the distribution of sentiment and ambiguity differs across Facebook, Instagram, X, and YouTube. Finally, the study is limited to three LLMs and does not include a broader family of multilingual, sentiment-specific, or instruction-tuned models.

Future work should address these limitations in four directions. First, the counterfactual subset should be strengthened through human validation, including judgments of fluency, naturalness, meaning preservation, and clarity of sentiment inversion. Second, explanation quality should be evaluated systematically, ideally combining human assessment with constrained model-assisted rubrics. Third, the protocol should be extended to platform-aware analysis, examining whether models behave differently across social media environments with distinct linguistic, demographic, and interactional patterns. Fourth, future experiments should include additional affective dimensions beyond polarity, such as emotion, sarcasm, aggressiveness, polarization, uncertainty, and stance. These extensions would allow a more complete characterization of political affect in Spanish social media discourse.

Overall, this work contributes to the study of natural language reasoning in Spanish by showing that sentiment analysis becomes substantially more demanding when evaluated through human ambiguity and controlled counterfactual shift. The proposed protocol provides a practical way to examine whether LLMs merely assign labels or whether they remain stable when affective meaning depends on culturally and pragmatically situated cues. In doing so, it also contributes to a better understanding of Spanish-language social platforms as complex spaces of political expression, where sentiment cannot always be reduced to surface polarity. The dataset annotations, evaluation artifacts, and supporting materials are publicly available in the repository: https://github.com/jvalverr/sentimex-llm-benchmark/tree/main/cultural_reasoning.

References

  1. [1]M. T. R. Laskar, S. Alqahtani, M. S. Bari, M. Rahman, M. A. M. Khan, H. Khan, I. Jahan, A. Bhuiyan, C. W. Tan, M. R. Parvez, E. Hoque, S. Joty, and J. X. Huang, “A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. ACL, 2024, pp. 13785–13816. [Online]. Available: https://doi.org/10.18653/v1/2024.emnlp-main.764
  2. [2]A. Jungher, “Artificial intelligence and democracy: A conceptual framework,” Social Media + Society, vol. 9, no. 3, pp. 1–14, 2023. [Online]. Available: https://doi.org/10.1177/20563051231186353
  3. [3]S. Lazar and L. Manuali, “Using LLMs to Enhance Democracy,” Minds and Machines, vol. 36, no. 1, p. 12, 2026. [Online]. Available: https://doi.org/10.1007/s11023-026-09767-y
  4. [4]V. G. dos Santos, G. L. Santos, T. Lynn, and B. Benatallah, “Identifying Citizen-Related Issues from Social Media Using LLM-Based Data Augmentation,” in Advanced Information Systems Engineering: 36th International Conference, CAiSE 2024, Proceedings. Springer-Verlag, 2024, p. 531–546. [Online]. Available: https://doi.org/10.1007/978-3-031-61057-8_31
  5. [5]K. Hackenburg, B. M. Tappin, L. Hewitt, E. Saunders, S. Black, H. Lin, C. Fist, H. Margetts, D. G. Rand, and C. Summerfield, “The levers of political persuasion with conversational artificial intelligence,” Science, vol. 390, no. 6777, p. eaea3884, 2025. [Online]. Available: https://doi.org/10.1126/science.aea3884
  6. [6]I. A. Qazi, Z. Khan, A. Ghani, A. A. Raza, Z. A. Qazi, W. Sajjad, A. Ali, A. Javaid, M. A. Sohail, and A. H. Azeemi, “Large language models show Dunning-Kruger-like effects in multilingual fact-checking,” Scientific Reports, vol. 16, no. 1, p. 7594, 2026. [Online]. Available: https://doi.org/10.1038/s41598-026-39046-w
  7. [7]I. Vykopal, M. Pikuliak, I. Srba, R. Moro, D. Macko, and M. Bielikova, “Disinformation capabilities of large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds. ACL, 2024, pp. 14830–14847. [Online]. Available: https://doi.org/10.18653/v1/2024.acl-long.793
  8. [8]M. K. Chandan and S. Mandal, “A comprehensive survey on sentiment analysis: Framework, techniques, and applications,” Computer Science Review, vol. 58, p. 100777, 2025. [Online]. Available: https://doi.org/10.1016/j.cosrev.2025.100777
  9. [9]M. Behrendt, S. S. Wagner, C. Weinmann, M. Bormann, M. Warne, and S. Harmeling, “Machine learning for enhancing deliberation in online political discussions and participatory processes: A survey,” 2026. [Online]. Available: https://arxiv.org/abs/2506.02533
  10. [10]K. Brito and P. J. L. Adeodato, “Machine Learning for Predicting Elections in Latin America Based on Social Media Engagement and Polls,” Government Information Quarterly, vol. 40, no. 1, p. 101782, 2023. [Online]. Available: https://doi.org/10.1016/j.giq.2022.101782
  11. [11]J. Valverde-Rebaza, L. Cañete-Sifuentes, F. Castro-Zenteno, E. Jimenez-Garibaldi, J. Trejo, and A. M. del Campo, “Pre-Trained and Large Language Models for Sentiment Analysis of Spanish Social Media in the 2024 Mexican Elections,” in Advances in Computational Intelligence. Proceedings of The 24th Mexican International Conference on Artificial Intelligence (MICAI 2025). Springer Nature Switzerland, 2026, pp. 289–300. [Online]. Available: https://doi.org/10.1007/978-3-032-17930-2_20
  12. [12]H. Wallach, M. Desai, A. F. Cooper, A. Wang, C. Atalla, S. Barocas, S. L. Blodgett, A. Chouldechova, E. Corvi, P. A. Dow, J. Garcia-Gathright, A. Olteanu, N. Pangakis, S. Reed, E. Sheng, D. Vann, J. W. Vaughan, M. Vogel, H. Washington, and A. Z. Jacobs, “Position: evaluating generative AI systems is a social science measurement challenge,” in Proceedings of the 42nd International Conference on Machine Learning, ser. ICML’25. JMLR.org, 2025. [Online]. Available: https://dl.acm.org/doi/10.5555/3780338.3783656
  13. [13]M. Ochieng, A. Thieme, I. Ezeani, R. Ueno, S. C. Maina, K. Ronen, J. Gonzalez, and J. O’Neill, “Reasoning beyond labels: Measuring LLM sentiment in low-resource, culturally nuanced contexts,” in Proceedings of the 7th Workshop on African Natural Language Processing (AfricanNLP 2026). ACL, 2026, pp. 64–81. [Online]. Available: https://doi.org/10.18653/v1/2026.africanlp-main.7
  14. [14]P.-Y. Wang, T.-S. Liu, C. Wang, Z. Li, Y. Wang, S. Yan, C. Jia, X.-H. Liu, X. Chen, J. Xu, and Y. Yu, “A Survey on Large Language Models for Mathematical Reasoning,” ACM Comput. Surv., vol. 58, no. 8, 2026. [Online]. Available: https://doi.org/10.1145/3786333
  15. [15]A. de Souza Loureiro, J. Valverde-Rebaza, J. Noguez, D. Escarcega, and R. Marcacini, “Advancing Multi-step Mathematical Reasoning in Large Language Models Through Multi-layered Self-reflection with Auto-prompting,” in Machine Learning and Knowledge Discovery in Databases. Research Track, ser. ECML PKDD 2025. Springer Nature Switzerland, 2026, pp. 205–223. [Online]. Available: https://doi.org/10.1007/978-3-032-06078-5_12
  16. [16]F. Yu, H. Zhang, P. Tiwari, and B. Wang, “Natural Language Reasoning, A Survey,” ACM Comput. Surv., vol. 56, no. 12, 2024. [Online]. Available: https://doi.org/10.1145/3664194
  17. [17]B. AlKhamissi, M. ElNokrashy, M. Alkhamissi, and M. Diab, “Investigating Cultural Alignment of Large Language Models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2024, pp. 12404–12422. [Online]. Available: https://doi.org/10.18653/v1/2024.acl-long.671
  18. [18]Y. Y. Chiu, L. Jiang, B. Y. Lin, C. Y. Park, S. S. Li, S. Ravi, M. Bhatia, M. Antoniak, Y. Tsvetkov, V. Shwartz, and Y. Choi, “CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2025, pp. 25663–25701. [Online]. Available: https://doi.org/10.18653/v1/2025.acl-long.1247
  19. [19]S. Pawar, J. Park, J. Jin, A. Arora, J. Myung, S. Yadav, F. G. Haznitramа, I. Song, A. Oh, and I. Augenstein, “Survey of Cultural Awareness in Language Models: Text and Beyond,” Computational Linguistics, vol. 51, no. 3, pp. 907–1004, 2025. [Online]. Available: https://doi.org/10.1162/COLI.a.14
  20. [20]L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and Chatbot Arena,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23. Curran Associates Inc., 2023. [Online]. Available: https://dl.acm.org/doi/10.5555/3666122.3668142arxiv.org/abs/2306.05685
  21. [21]G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang, “Humans or LLMs as the judge? a study on judgement bias,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. ACL, 2024, pp. 8301–8327. [Online]. Available: https://doi.org/10.18653/v1/2024.emnlp-main.474
  22. [22]D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, K. Shu, L. Cheng, and H. Liu, “From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. ACL, 2025, pp. 2757–2791. [Online]. Available: https://doi.org/10.18653/v1/2025.emnlp-main.138
  23. [23]L. Qin, A. Bosselut, A. Holtzman, C. Bhagavatula, E. Clark, and Y. Choi, “Counterfactual Story Reasoning and Generation,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). ACL, 2019, pp. 5043–5053. [Online]. Available: https://doi.org/10.18653/v1/D19-1509

Paper details

vlnc:W_kvzd4dqg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents