betaFeedback

WorksComputer Science

Evaluating Human-LLM Alignment on Visualization Preference

Abstract

There are often multiple valid visualizations, spanning different chart types, visual encodings, and aesthetic choices, that can all support a user's goal of analyzing a dataset. Determining which of these candidate visualizations is most appropriate, however, remains challenging. Prior work has leveraged expert-curated taxonomies to guide visualization design and subsequent quality evaluation, a practical evaluation model must also account for human-centric factors. Latest visualization tools integrate Large Language Models (LLMs) to generate and refine visualizations, using LLM-as-a-judge to provide design feedback. Without a deeper understanding of whether, and in what ways, human and LLM perceptions of visualizations align, there is a real risk of deploying low-quality agents that do not reflect human judgment. We study this question through a pilot experiment - we formulate the problem as a preference task over curated chart pairs and compare the judgments of state-of-the-art vision models with those of a panel of human annotators. We establish a cautionary tale: such judges risk filtering out informative, compelling displays. We find that frontier models collapse human subjectivity, a rich signal that reveals contrasting design choices, into near-deterministic verdicts.

Introduction

Visualizations drive the trajectory of business decisions, scientific inquiry, public discourse through journalism, among other fields. For a given dataset and a high-level communicative intent (“a story to tell”), a visualization expert may design multiple valid visualizations–spanning across different chart types, visual encoding and design choices to support them. Determining which candidate visualization is the most appropriate (“tells the most compelling story”) constitutes the visualization preference task: a comparative judgment in which a viewer, presented with valid candidate visualizations, chooses a display that serves their need. A human viewer’s perception of a visualization is inherently subjective, not solely measured by presence or absence of information, but rather by how clearly, reliably and efficiently that information is presented, and how it connects to their prior beliefs and task context. Two displays that encode identical data can produce different levels of understanding or decision confidence when presented to the same viewer. Similarly, a single visualization display might be perceived differently across a panel of viewers. Hence, we deal with preference as a distribution over objective choices, and not as a binary ground truth.

Visualization literature has largely addressed visualization quality assessment through expert-curated taxonomies of isolated design issues—truncated axes, misleading ratios, chart-type mismatch etc. [2, 9]. While valuable as primers to visualization professionals, these taxonomies act as baseline validity filters. In practice, users frequently choose among charts that have already cleared this baseline or are technically “good enough”. At this juncture, the viewers’ latent perception factors drive them to prefer one chart over the other. These factors, rooted in data semantics, aesthetics etc. cannot be accounted for by static taxonomies. Prior benchmarks that evaluate isolated visualizations against predetermined criteria [12] further miss the comparative nature of human perception: pairwise comparison surfaces preferences that a viewer could not articulate when examining a single chart in isolation [15].

Meanwhile, in adjacent AI disciplines, preference-based signals have proven to be critical for building reliable agent-based verifiers. From mathematical reasoning, where step-level human preferences guide problem-solving [10], to molecular design where reinforcement-learning agents are steered by preference-aligned scoring functions [13], the common theme is that once a model’s preferences are aligned with domain-expert judgement, it can be utilized in a scalable feedback loop that evaluates and improves outputs. This paradigm has already been deepened in data analytics tools such as spreadsheet copilots and dashboard generators; systems of this kind [16, 5] adopt multi-step agentic pipelines in which a vision-language model (VLM) acts as the preference critic. The quality of every chart surfaced to an end user therefore pivots on the assumption that the critic’s preferences reflect human judgment.

Yet, we found that no prior work validates this assumption. VLMs have been benchmarked on Data Viz tasks like chart question-answering [11] and summarization. However, these benchmarks are centered around the question “What is presented?” and not “How well is it presented?”. They assess whether a model can read a chart, but not whether it can judge one—let alone whether its judgments align with those of human viewers.

Through this study, we seek to bridge this gap. We pose the inquiry that : When two technically valid charts are constructed to present the same data, do frontier VLMs prefer the same chart that humans prefer? We construct an intent-driven, pair-wise preference benchmark of 24 diverse chart pairs, and evaluate five vision-language models under stochastic multi-run sampling. By treating annotator disagreements as meaningful signals, we find that there exists a disarray between human and model preferences.

Related Works

The foundations of visualization quality assessment span grammatical frameworks, decoding-based tasks (judging position, color etc.) [4], and quantifiable design heuristics such as the data-ink ratio [15]. Collectively, these works establish principled criteria for visualization design but remain anchored to expert judgment and expensive manual application.

Another relevant line of work probes VLM capabilities in chart understanding. Benchmarks such as ChartBench [20] and ChartMuseum [14] frame visualization comprehension as visual question-answering (VQA), requiring models to extract values, interpret encodings, and infer relationships. Notably, the latter demonstrates how current models over-rely on textual reasoning and struggle with visually grounded interpretation of implicit encodings and non-trivial layouts—a finding that motivates our investigation of whether these perceptual limitations in VLMs extend to preference judgments and how they relate to human preferences. On the automation front, a constraint-based recommendation system, DracoGPT [17], encodes empirical visualization design knowledge as soft and hard rules and checks alignment across 3 LLMs’ encoded design practices. VisEval [3] benchmarks LLM-generated visualizations on pre-determined validity, legality, and readability criteria, while C2 [7] introduces a novel, LLM-based automated feedback generator that combines static feedback (over chart-type, color, etc.) with dynamic feedback (faithfulness to user intent, etc.) to iteratively improve chart generation. Agentic visualization pipelines such as LIDA [5] deploy a code-based LLM evaluator inside a (generate-evaluate-refine) loop to score and refine charts without human intervention. However, the evaluation signals in all of these systems are either grounded in static, expert-curated taxonomies (axis truncation, label overlap, chart-junk) or remain unvalidated against fine-grained, latent perceptual factors of their critics. This forms the cornerstone of our contribution.

Dataset

Existing chart evaluation frameworks build primitive visualizations from templates [6] and low-dimensional tables or restrict coverage to narrow domains [18] and chart types [11]. Our data generation pipeline addresses these limitations by modeling visualization construction as a realistic communicative scenario – integrating diverse, complex datasets, accounting for varying target audience in each domain and clear analytical intent – resulting in rich, targeted visualizations that exhibit genuine design trade-offs.

Data Collection

To model data visualization scenarios in the wild, we used the compiled set of real-world datasets from the ChartUIE-8K Dataset from [7]. From 509 source tables, we manually curate 34 datasets spanning 20 thematic categories (education, technology, market research etc.), ensuring broad domain coverage and sufficient structural complexity (3+ data fields each).

Data Generation Pipeline

Stage 1: Structural Annotation - Each source table is parsed into a structured JSON schema. An LLM annotates every field with its semantic role inferred from co-occurring field names and value types/distributions.

Stage 2: Scenario Construction - A Chain-of-Thought (CoT) prompt generates 144 scenario briefs across 34 datasets, each specifying a target audience group, data curator’s intent and shared context between them. Multiple briefs per dataset explore diverse semantics, and hence diverse communicative framings.

Stage 3: Analytical Question Derivation - Schemas paired with scenario briefs are fed to an LLM to derive analytical questions that are tractable for visualization, yet trigger interpretive reasoning (compare/correlate/causal analysis etc.). This yields 888 candidate queries, from which 116 questions are curated2 that satisfy two criteria: (i) Analytically non-trivial. A question like “What is the Attack stat of Pikachu?” is a factual lookup whose answer is independent of chart design. We retain only questions that demand subjective interpretation, such as “Is there a significant difference in ‘Attack’ between Pokémon of different primary types?”—where the viewer’s judgment depends on how the chart encodes the Attack statistic. (ii) Design-space rich. Each query must necessitate utilization of meaningful visualization design choices (using data aggregations, varying chart complexities via layouts, legends, etc.) that affect user perception.

Stage 4: Chart Variant Synthesis - For each of the 116 queries, an LLM generates 10-15 visualizations as executable matplotlib code, applying distinct transformations and visual encodings to the same data.

Question typeExampleCount
Comparative analysisHow do caffeine levels vary among drink types?6
Filtered comparisonAmong top-ranked countries, how do densities compare?3
Ranking / extremaWhich stocks have the highest turnover?3
CorrelationDo properties with more bedrooms have higher prices?3
Anomaly detectionAre there states with far above-average dog volume?3
Part-to-wholeDo any genres dominate global sales in specific years?3
Temporal trendHow has global population changed from 1955 to 2024?2
DistributionHow do undergraduate majors vary among astronauts?1

Table 1. Thematic composition of the study set

same data. Via manual curation, we further yield 396 candidate visualizations.

Study Set Sampling

From the candidate pool, we jointly sample the tuples (query, candidate A, candidate B) to form the study set. Sampling proceeds in 2 passes-

Coverage Pass: Queries are thematically coded into intent types3 (Table 1) to form a coverage scaffold. Queries are manually sampled to spread the study set across intent types, source domains and chart types. The candidate pool is dominated by comparative questions and certain chart types (box plot, bar chart etc.). This ensures we preserve the rarer intents and the most interesting design trade-offs.

Admissibility Pass: The sampled charts are required to satisfy a rubric - (i) Baseline Validity - Every individual chart must be free of standard taxonomy defects (truncated axes, typographical overlaps etc.) (ii) Design Differentiation - The two charts must be separable along at least one of structural (chart type, encoding), semantic (data filtration, transforms) or design (annotations, spatial layouts) axes.

USER STUDY

Prior works in user preference have often relied on thinking-aloud protocols etc. to elicit participants’ thinking process during interaction with informative artefacts. These approaches assume that users have complete and correct introspective access to the factors that shape their choices. Results in cognitive psychology [19] suggest that this methodology degrades task performance quality by shifting their attention to verbalizable but often non-diagnostic features. We posit that this effect shall be especially pronounced for data visualizations—where user perception is rapid and pre-attentive. Our study bypasses the dependence on verbal articulation by eliciting high-level preference signals as an objective choice.

Evaluation Dimensions

To circumvent the problem of burdening annotators with exhaustive visualization taxonomies and to create a scalable annotation platform for future crowd-sourced studies accessible to the common audience, we conducted a structured convergent-design exercise: (i) The authors reviewed literature in visualization research, cognitive science, and human-computer interaction (HCI). (ii) Several candidate evaluation dimensions (grammatical, perceptual and heuristic) were proposed independently and stress-tested through iterative pilot rounds of the annotation task performed on disjoint chart pairs, with the authors serving as trial annotators. (iii) Across multiple iterations, we eliminated ambiguity, extraneous cognitive @page 2/5 starts_mid=1 ends_mid=1 preference judgments and how they relate to human preferences. On the automation front, a constraint-based recommendation system, DracoGPT [17], encodes empirical visualization design knowledge as soft and hard rules and checks alignment across 3 LLMs’ encoded design practices. VisEval [3] benchmarks LLM-generated visualizations on pre-determined validity, legality, and readability criteria, while C2 [7] introduces a novel, LLM-based automated feedback generator that combines static feedback (over chart-type, color, etc.) with dynamic feedback (faithfulness to user intent, etc.) to iteratively improve chart generation. Agentic visualization pipelines such as LIDA [5] deploy a code-based LLM evaluator inside a (generate-evaluate-refine) loop to score and refine charts without human intervention. However, the evaluation signals in all of these systems are either grounded in static, expert-curated taxonomies (axis truncation, label overlap, chart-junk) or remain unvalidated against fine-grained, latent perceptual factors of their critics. This forms the cornerstone of our contribution.

Dataset

Existing chart evaluation frameworks build primitive visualizations from templates [6] and low-dimensional tables or restrict coverage to narrow domains [18] and chart types [11]. Our data generation pipeline addresses these limitations by modeling visualization construction as a realistic communicative scenario – integrating diverse, complex datasets, accounting for varying target audience in each domain and clear analytical intent – resulting in rich, targeted visualizations that exhibit genuine design trade-offs.

Data Collection

To model data visualization scenarios in the wild, we used the compiled set of real-world datasets from the ChartUIE-8K Dataset from [7]. From 509 source tables, we manually curate 34 datasets spanning 20 thematic categories (education, technology, market research etc.), ensuring broad domain coverage and sufficient structural complexity (3+ data fields each).

Data Generation Pipeline

Stage 1: Structural Annotation - Each source table is parsed into a structured JSON schema. An LLM annotates every field with its semantic role inferred from co-occurring field names and value types/distributions.

Stage 2: Scenario Construction - A Chain-of-Thought (CoT) prompt generates 144 scenario briefs across 34 datasets, each specifying a target audience group, data curator’s intent and shared context between them. Multiple briefs per dataset explore diverse semantics, and hence diverse communicative framings.

Stage 3: Analytical Question Derivation - Schemas paired with scenario briefs are fed to an LLM to derive analytical questions that are tractable for visualization, yet trigger interpretive reasoning (compare/correlate/causal analysis etc.). This yields 888 candidate queries, from which 116 questions are curated2 that satisfy two criteria: (i) Analytically non-trivial. A question like “What is the Attack stat of Pikachu?” is a factual lookup whose answer is independent of chart design. We retain only questions that demand subjective interpretation, such as “Is there a significant difference in ‘Attack’ between Pokémon of different primary types?”—where the viewer’s judgment depends on how the chart encodes the Attack statistic. (ii) Design-space rich. Each query must necessitate utilization of meaningful visualization design choices (using data aggregations, varying chart complexities via layouts, legends, etc.) that affect user perception.

Stage 4: Chart Variant Synthesis - For each of the 116 queries, an LLM generates 10-15 visualizations as executable matplotlib code, applying distinct transformations and visual encodings to the same data.

Question typeExampleCount
Comparative analysisHow do caffeine levels vary among drink types?6
Filtered comparisonAmong top-ranked countries, how do densities compare?3
Ranking / extremaWhich stocks have the highest turnover?3
CorrelationDo properties with more bedrooms have higher prices?3
Anomaly detectionAre there states with far above-average dog volume?3
Part-to-wholeDo any genres dominate global sales in specific years?3
Temporal trendHow has global population changed from 1955 to 2024?2
DistributionHow do undergraduate majors vary among astronauts?1

Table 1 (PDF p. 2). Thematic composition of the study set

same data. Via manual curation, we further yield 396 candidate visualizations.

Study Set Sampling

From the candidate pool, we jointly sample the tuples (query, candidate A, candidate B) to form the study set. Sampling proceeds in 2 passes-

Coverage Pass: Queries are thematically coded into intent types3 (Table 1) to form a coverage scaffold. Queries are manually sampled to spread the study set across intent types, source domains and chart types. The candidate pool is dominated by comparative questions and certain chart types (box plot, bar chart etc.). This ensures we preserve the rarer intents and the most interesting design trade-offs.

Admissibility Pass: The sampled charts are required to satisfy a rubric - (i) Baseline Validity - Every individual chart must be free of standard taxonomy defects (truncated axes, typographical overlaps etc.) (ii) Design Differentiation - The two charts must be separable along at least one of structural (chart type, encoding), semantic (data filtration, transforms) or design (annotations, spatial layouts) axes.

USER STUDY

Prior works in user preference have often relied on thinking-aloud protocols etc. to elicit participants’ thinking process during interaction with informative artefacts. These approaches assume that users have complete and correct introspective access to the factors that shape their choices. Results in cognitive psychology [19] suggest that this methodology degrades task performance quality by shifting their attention to verbalizable but often non-diagnostic features. We posit that this effect shall be especially pronounced for data visualizations—where user perception is rapid and pre-attentive. Our study bypasses the dependence on verbal articulation by eliciting high-level preference signals as an objective choice.

Evaluation Dimensions

To circumvent the problem of burdening annotators with exhaustive visualization taxonomies and to create a scalable annotation platform for future crowd-sourced studies accessible to the common audience, we conducted a structured convergent-design exercise: (i) The authors reviewed literature in visualization research, cognitive science, and human-computer interaction (HCI). (ii) Several candidate evaluation dimensions (grammatical, perceptual and heuristic) were proposed independently and stress-tested through iterative pilot rounds of the annotation task performed on disjoint chart pairs, with the authors serving as trial annotators. (iii) Across multiple iterations, we eliminated ambiguity, extraneous cognitive load, and redundancy and refined candidates until stable inter-rater agreement was reached. This process yielded three perceptual dimensions, each operationalized as a first-person statement rated on a 5-point Likert scale (Strongly Disagree to Strongly Agree):

  • Readability — “I think the chart is easy to understand at a glance.”

  • Precision — “I think the chart answers the given question.”

  • Aesthetics — “I think the chart is visually appealing and attractive.”

We do not claim these dimensions to be theoretically orthogonal; rather, they represent the minimal set on which the authors could reach stable, independent agreement. Crucially, within the scope of this work the Likert ratings are not themselves an object of analysis, but serve an instrumental role. They act as lightweight scaffolding that anchors annotators to the salient facets of the task—readability, precision, and aesthetics. While these scores are rich signals in themselves, we leave the analysis of these dimensions to future work.

Procedure

We recruit a panel of 10 volunteers; to avoid injecting authorial bias, none of the authors participate in the study. Each participant is briefed on the task objectives and evaluation dimensions in a synchronous session, after which they complete the task independently at their own pace. Annotations are collected through a custom-built interface, iteratively refined based on pilot feedback to minimize ambiguity and cognitive load during the annotation workflow.4 Each participant first completes an objective, pre-study questionnaire assessing their experience with data handling, their familiarity with visualizations and their self-reported confidence in interpreting data visualizations.

Each of the 24 pairs is then presented in a two-phase protocol:

  1. Independent evaluation: Chart A and Chart B are shown sequentially alongside the analytical question. For each chart in isolation, the participant rates Readability, Precision, and Aesthetics on Likert scales. This phase establishes per-display context rooted in the question and invites analytical engagement before making comparative judgments.

  1. Pairwise preference: After sequential rating, both charts are displayed side-by-side together with the original analytical question. The participant selects one of three options: Chart A, Chart B, or Equal Preference—a three-bin choice that captures objective display preference from the user.

Note: The presentation order of charts was randomized for the participants and the models. Each phase allows participants to optionally add comments about their feelings during the task.5 Following the annotation exercise, participants complete a post-study questionnaire that gauges task engagement and fatigue, which is subsequently used as a control over response quality. The observed mean engagement time was 148.1 seconds per pair.

Participant Description

The study included 10 participants, out of which, 7 hold bachelor’s degrees, 2 hold a masters degree and 1 holds a PhD degree. Most participants reported a professional experience between 1-5 years (including internships etc.), while 1 participant reports less than 1 year of professional experience. The participants were all

Reported familiarity domains by participants

Figure 1. Reported familiarity domains by participants

frequent consumers of data visualizations, reporting high degree of confidence in their ability to perceive data visualizations (90%), interpretation of visualizations with multiple encodings (70%), and handling unconventional visualization types (70%). Our panel is visualization-literate, identifying with routine users of specialized spreadsheet AI tools, and hence form an appropriate population set for this study.

Evaluation

Subjective preferences have no unique gold label. We treat participant disagreement as signal [1]. For each chart pair ii, ten human votes define the empirical distribution hi=(hi,A,hi,B,hi,Equal)\mathbf{h}_i=(h_{i,A},h_{i,B},h_{i,\mathrm{Equal}}) over {A,B,Equal}\{A,B,\mathrm{Equal}\}; analogously, model judgments with stochastic sampling (T=0.75T=0.75) define an empirical distribution mi\mathbf{m}_i [8]. We evaluate Claude Opus 4.5, Claude Sonnet 4.5, GPT-5 Chat, GPT-4V, and Phi-4 Multimodal under identical conditions and prompt scaffold: analytical question, chart images, and evaluation dimensions—eliciting Likert ratings for each chart, a three-way choice and corresponding rationale.6

Metrics

Asking whether a model “agrees with humans” bundles two separable capabilities: whether it picks what the panel picks (agreement), and whether it knows how much the panel disagrees (calibration). We measure each: Jensen–Shannon divergence (JSD) for agreement, and a paired entropy test for calibration. We report the following statistics:

J=1P∑i=1PJSD⁡(hi,mi),T=median⁡i[H(hi)−H(mi)].(1)J = \frac{1}{P}\sum_{i=1}^{P}\operatorname{JSD}(\mathbf{h}_i,\mathbf{m}_i), \qquad T = \operatorname{median}_{i}[H(\mathbf{h}_i)-H(\mathbf{m}_i)]. \tag*{(1)}

JSD is a symmetric, information-theoretic distance between two distributions. The agreement statistic JJ is the mean over P=24P=24 pairs. J∈[0,1]J \in[0,1] measures overall distributional disagreement between the human panel and each model (lower is better). Let H(⋅)H(\cdot) be Shannon entropy in bits. The per-pair sharpness excess is di=H(hi)−H(mi)d_i=H(\mathbf{h}_i)-H(\mathbf{m}_i), and the calibration statistic TT is its median over the study set, in bits. T>0T>0 means the model is typically more certain than the panel, and T<0T<0 means it is typically less certain. JJ identifies whether the distributions differ, while TT helps interpret whether excess confidence contributes to that difference. We also report n+n_+, the number of pairs for which di>0d_i>0, i.e. the number of pairs on which the model exhibited more certainty across runs than the human panel.

Calibrated Reference

Zero is not the right baseline: a perfectly calibrated predictor sampled only K=10K=10 times is itself noisy. To locate that floor we place a symmetric Dirichlet(1) prior on each pair’s vote split and, for each B=2000B=2000 replicates, draw that split from the posterior before sampling KK votes from it. Drawing the split rather than fixing it at the posterior mean lets the reference inherit the panel’s own estimation uncertainty. Built from the observed votes alone, this reference—not zero—is what the model rows of Table 2 should be read against.7

PredictorAgreement J↓J \downarrowCalibration T↓T \downarrown+↓n_+ \downarrow
Calibrated reference0.08 [0.05, 0.11]0.02 [−0.11, 0.20]11 [7, 15]
GPT-4V0.29 [0.21, 0.38]0.9021
GPT-5 Chat0.29 [0.21, 0.38]0.9021
Claude Opus 4.50.29 [0.21, 0.38]0.8824
Claude Sonnet 4.50.31 [0.22, 0.39]0.9124
Phi-4 Multimodal0.16 [0.12, 0.21]0.1915

Table 2. Agreement and calibration relative to the calibrated reference

Observations

Differences between individual models are not resolvable at P=24P=24 and we make no claim about model rankings; we compare frontier models as a class against the calibrated reference. The JJ values show that at the data population level, the model evaluators’ agreement sits roughly 4 times as far from the human panel as a calibrated predictor’s. This confidence effect is universal, as frontier VLMs appear more certain than the panel on 21–24 of the 24 pairs, whereas a calibrated predictor is more certain on 11 pairs on average and on at most 15, representing an evident overconfidence. The models don’t merely disagree with the panel, as the JJ values show, they also collapse genuine human disagreement into deterministic verdicts. Phi-4 is the only model at the calibration floor. However, on closer inspection, it is revealed that in 57% of its total responses, it systematically misidentifies chart types (box plots as scatter plots, violin plots as bar charts etc.) or fails to name one, violating the “visualization literacy” confound. Hence, we attribute its performance to instability. Our results are robust as this trend survives order randomization and prompt framing effects8.

Case Studies

We present 2 example cases from the study – one where humans and models converge, the other where they systematically diverge:

Case I: Agreement on Stock Performance Charts

Only 1 participant chose Chart B as shown in Figure 2, citing its completeness for their choice (“much more informative, not missing any data”), while others cited clutter (“Unreadable,” “Cluttered”) and the irrelevance of Chart B’s trend line for not picking it. The four frontier models unanimously selected Chart A in every run (40/40), strongly aligning with human preferences. The models exhibit a consistent rationale centered around question–chart semantic congruence. The question asks “which stocks” (requires explicit identification) and “how do these compare” (requires comparative encoding). Chart A’s ranked bar layout directly supports both: stocks are named, sorted by 30-day magnitude, and their 365-day changes are shown side-by-side. Opus 4.5 describes it as “directly addressing the question by showing the top 10 positive and negative 30-day performers.”

Clear agreement between models and participants: Chart A, a diverging bar chart, and Chart B, a scatter plot

Figure 2. Clear agreement between models and participants: (i) Chart A (left), a diverging bar chart; and (ii) Chart B (right), a scatter plot.

Case II: Disagreement on Video Game Sales Charts

In Figure 3, 9 of the 10 participants chose Chart A, while the models overwhelmingly selected Chart B: Claude Opus 4.5 and Claude Sonnet 4.5 in all 10 runs, and GPT-4V and GPT-5 Chat in 6 and 8 runs respectively. Claude Opus 4.5 explains how “the heatmap allows immediate identification of which genres dominated and how preferences shifted over time”. Whereas the human annotators are overburdened by the high cognitive load associated with parsing the heatmap, and instead, prefer the intuitive outlook provided by the stacked area chart, despite the fact that it does not answer the question as precisely (via year-by-year breakdown) as the heatmap. The models seem to perform a highly nuanced, analytical breakdown of the task and select the chart that maximizes precision. Instead, humans favor the communicative aspect that is perceived as “unambiguous” and tend to avoid complex analysis, revealing fundamental Cognitive-Load asymmetry.

Disagreement: Chart A, a stacked area chart, and Chart B, a heatmap

Figure 3. Disagreement: (i) Chart A (left), a stacked area chart; and (ii) Chart B (right), a heatmap.

Conclusion

These cases mechanise the aggregate numbers. In Case I, agreement arises when the task’s semantic structure (ranking, comparison) maps unambiguously to one chart type, and both humans and models recognize this. In Case II, disagreement emerges when the question allows for two valid interpretive strategies: narrative appeal vs analytical decomposition. This aspect is inherently invisible in model judgements. Models consistently optimize for “sheer answerability” of the question, hinging upon factors like “chart-type appropriateness” to match the query etc., while humans optimize for the visualization that minimizes cognitive load and tells them a convincing story. We do not position these metrics to evaluate visualization judges, but as diagnostic tools: sufficient to establish that a genuine gap exists in accounting for rich, human-centric preferences that a visualization agent would not surface. To truly leverage the power of agentic visualization tools, a *human-centric framework that integrates subjective design choices and elements of perceptual fluency rather than pure analytical rigor is needed, and subjective human preference should be advocated as a first-class citizen in chart quality evaluation. We call for a broader inquiry via crowdsourced studies and preference-aware post-training objectives.

Supplemental Materials

Supplemental materials are provided under an MIT license. They contain the data pipeline prompts, preference prompts, UI, model and anonymized annotator responses, and the quantitative results to support the claims in this paper.

References

  1. [1]L. Aroyo and C. Welty. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Mag., 36:15–24, 2015. 3DOI
  2. [2]A.-F. Cabouat, T. He, P. Isenberg, and T. Isenberg. Previs: Perceived readability evaluation for visualizations. IEEE Transactions on Visualization and Computer Graphics, 31(1):1083–1093, Jan. 2025. doi: 10.1109/tvcg.2024.3456318 1DOI
  3. [3]N. Chen, Y. Zhang, J. Xu, K. Ren, and Y. Yang. Viseval: A benchmark for data visualization in the era of large language models, 2024. 2arxiv.org/abs/2407.00981
  4. [4]W. S. Cleveland and R. McGill. Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American Statistical Association, 79(387):531–554, 1984. doi: 10.1080/01621459.1984.10478080 1DOI
  5. [5]V. Dibia. Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models, 2023. 1, 2
  6. [6]K. Kafle, S. D. Cohen, B. L. Price, and C. Kanan. Dyga: Understanding data visualizations via question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5648–5656, 2018. 2arxiv.org/abs/1801.08163
  7. [7]W. Koh, J. H. Yoon, M. Lee, Y. Song, J. Cho, J. Kang, T. Kim, S.-Y. Yun, Y. Yu, and B. Lee. c²: Scalable auto-feedback for llm-based chart generation, 2025. 2arxiv.org/abs/2410.18652
  8. [8]L. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation, 2023. 3arxiv.org/abs/2302.09664
  9. [9]X. Lan and Y. Liu. "i came across a junk": Understanding design flaws of data visualization from the public’s perspective. IEEE Trans. Vis. Comput. Graph., 31(1):393–403, 2025. doi: 10.1109/TVCG.2024.3456341 1DOI
  10. [10]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv, abs/2305.20050, 2023. 1arxiv.org/abs/2305.20050
  11. [11]A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In S. Muresan, P. Nakov, and A. Villavicencio, eds., Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279. Association for Computational Linguistics, Dublin, Ireland, May 2022. doi: 10.18653/v1/2022.findings-acl.177 1, 2DOI
  12. [12]T. Munzner. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6):921–928, 2009. doi: 10.1109/TVCG.2009.111 1DOI
  13. [13]M. Olivecrona, T. Blaschke, O. Enkvist, and H. Chen. Molecular de novo design through deep reinforcement learning, 2017. 1
  14. [14]L. Tang, G. Kim, X. Zhao, T. Lake, W. Ding, F. Yin, P. Singhal, M. Wadhwa, Z. L. Liu, Z. Sprague, R. Namuduri, B. Hu, J. D. Rodriguez, P. Peng, and G. Durrett. Chartmuseum: Testing visual reasoning capabilities of large vision-language models, 2026. 1arxiv.org/abs/2505.13444
  15. [15]E. Tufte. The Visual Display of Quantitative Information. Graphics Press, 2001. 1
  16. [16]C. Wang, B. Lee, S. M. Drucker, D. Marshall, and J. Gao. Data formula 2: Iterative creation of data visualizations, with ai transforming data along the way. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25. Association for Computing Machinery, New York, NY, USA, 2025. doi: 10.1145/3706598.3713296 1DOI
  17. [17]H. W. Wang, M. Gordon, L. Battle, and J. Heer. Dracogpt: Extracting visualization design preferences from large language models, 2024. 2arxiv.org/abs/2408.06845
  18. [18]Z. Wang, M. Xia, L. He, H. Chen, Y. Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi, A. Chevalier, S. Arora, and D. Chen. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, eds., Advances in Neural Information Processing Systems, vol. 37, pp. 113569–113697. Curran Associates, Inc., 2024. doi: 10.52202/1079017.3609 2arxiv.org/abs/2406.18521
  19. [19]T. Wilson and J. Schooler. Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of personality and social psychology, 60:181–92, 03 1991. doi: 10.1037//0022-3514.60.2.181 2pubmed.ncbi.nlm.nih.gov/2016668
  20. [20]Z. Xu, S. Du, Y. Qi, C. Xu, C. Yuan, and J. Guo. Chartbench: A benchmark for complex visual reasoning in charts, 2024. 1arxiv.org/abs/2312.15915

Paper details

vlnc:W_tzwugnzj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents