WorksComputer Science
Are Average Estimates in Line Graphs Biased Toward Areas of Higher Variability?
Abstract
When estimating the average value of a data series in line graphs, do viewers systematically overweight regions of higher variability? This is the question that Moritz et al. [14] set out to answer, and across two preregistered experiments they reported evidence for a bias they termed variability overweighting. Using their released data and code, I first reproduce their reported analysis, then ask whether their inference survives an equally valid analytic choice: measuring estimation error against the average of the displayed marks of each stimulus, rather than against a reference series that, for one mark type, is never shown. Under this alternative operationalization, the effect is substantially reduced and, for one mark type, reverses. Splitting the analysis by whether variability appears in the upper or lower region of the graph further reveals an asymmetry that the original direction-corrected analysis collapses away. I argue that the deeper issue is what verbal theories leave unspecified. When a theory is silent about which operationalization it requires, the metric ends up doing the theory's arguing for it. I close with a discussion of what reproduction and reanalysis are for: not verdicts on truth, but instruments for making analytic contingency visible and for forcing theories to say more.
Introduction
Estimating the average value of a data series is one of the most common tasks people perform when reading line graphs [2, 4]. Understanding how properties of a visualization affect this estimation is central to graphical perception research and to the design guidelines that flow from it.
Moritz et al. [14] investigated this question and reported a previously undocumented bias: average estimates in line graphs are systematically pulled toward areas of higher variability in the data, an effect they termed variability overweighting. Across two preregistered experiments with 562 participants, they found that when one region of a line graph displayed more variable data, participants’ average estimates shifted toward that region. In a second experiment comparing three mark types — a continuous line, points equally spaced along the x-axis, and points equally spaced along the arc of the line — the bias was smallest for x-axis-spaced points and largest for arc-spaced points. The authors interpret this pattern through visual salience: marks that capture more attention should exert more pull on the estimate.
In prior work introducing visual decoding operators [13], we used Moritz et al.’s second experiment as a task for our learned operators to generalize to. Using their experiment this way required reconstructing details of the stimulus generation and analysis pipeline from their published paper, supplementary code, and released data. During the course of reproducing the original analysis, we noticed that the original authors defined and computed estimation error differently from the operationalization we had assumed.1 When we recompute estimation error using an equally valid operationalization, the reported effect of variability overweighting is substantially reduced. Additionally, splitting the analysis by whether variability appears in the upper or lower region of the graph reveals an asymmetry that the original analysis did not address. Together, these observations suggest that the evidence for variability overweighting as a salience-driven effect is weaker than originally reported.
In the remainder of the paper, I review prior work on average estimation from visualizations, walk through the experimental design of Moritz et al., present the reproduction and reanalysis, and use the case to discuss the relationship between error definitions, verbal theories, and the conclusions we draw from graphical perception experiments.
Estimating Averages from Visualizations
Mean comparison and estimation has been studied extensively in data visualization. For example, Gleicher et al. [8] examined relative mean judgments in multiclass scatterplots and found that viewers’ performance is largely unaffected by the number of points per class, by redundant or conflicting secondary cues, or by additional distractor classes, but degrades when class membership is encoded through a less salient channel such as shape rather than color. Tseng et al. [16] extended the same multiclass mean-estimation task to between two and ten categories under ten color palettes, and found that both the number of categories and the discriminability of colors within a palette significantly affect viewers’ judgments. Beyond accuracy, Hong et al. [11] report a “weighted average illusion” in scatterplots that encode a third dimension through size or lightness of the dots. They found that larger and darker points are given more weight in x- and y-mean estimates, and the magnitude of the bias depends on the size or lightness range a designer maps onto the data. They account for the effect by modeling each mark’s contribution and measuring attention to individual points using the centroid method from vision science. Correll et al. [4] asked participants to find the month with the highest average value in a year of daily data, and found that a colorfield encoding significantly outperformed the standard line graph on this aggregate judgment. For a broader treatment, a prior survey on ensemble perception [15] identified summary tasks, such as mean, skew, and numerosity, that are common to data visualization and discusses related work.
Overview of Moritz et al. moritz2023average
Moritz et al. [14] investigated what they term variability overweighting in line graphs: the hypothesis that when estimating the average of a time series, viewers’ judgments are systematically pulled toward regions of higher variability. They tested this hypothesis in two pre-registered experiments. In both experiments, stimuli were synthetic time series of data points generated from geometric Brownian motion, with noise linearly interpolated as a function of the base value, so that variability concentrates in the upper region of the graph or, after vertical mirroring, the lower region. Participants placed a draggable horizontal line where they judged the series’ average to be (Figure 1). Moritz et al. [14] release their experiment data and analysis files at https://osf.io/aupbk/.

Figure 1. Screenshot of example stimulus shown in Moritz et al. Figure 3 [14]. Each series is shown as a 500px by 200px graph.
The first experiment () used line graphs only, in a 2 (variability position: upper vs. lower) 2 (variability level: 0.15 vs. 0.4) within-subjects design. 12 seeds 4 conditions gave 48 trials, each series containing 120 data points. Moritz et al. found that average estimates were biased toward the region of the line showing more variable data.
From this finding, Moritz et al. hypothesized that the bias is driven by salience2 and that one “could influence the degree of variability-overweighting by changing the mark type.” Their second experiment () tests this with a mixed design: 3 mark types between subjects — line graph, points equally spaced along the x-axis (point) and points equally spaced along the arc of the line (pointArc) — and within subjects, 2 variability levels (0 vs. 0.4) 2 variability positions (upper vs lower) 12 random seeds, for 48 trials per participant. The results in Experiment 2 replicated Experiment 1. Error increased with variability for every mark type, and the increase was smallest for point, larger for line, and largest for pointArc. Moritz et al. take this ordering as support for the salience account.
Two Definitions of Estimation Error
Moritz et al. [14] observe that in the pointArc condition, “a perfectly accurate viewer cannot estimate the true average of the underlying data series”. This is correct only under one reading, when
“true average” means the average of the time-series data that the dots were sampled from, rather than of the dots themselves (Figure 2). Calling the unshown series the true average is therefore not a description of the encoding but a decision about what an estimate is measured against. Two definitions of error follow from the two readings, and I recalculate the same analysis under both definitions.
Figure 2. Given a line graph (a), a participant could be “sampling” from it using points equally spaced along the x-axis (b), or points equally spaced along the arc of the line (c).
I define displayed error as the difference between a participant’s reported average and the numeric average of the y-values of the marks actually presented in that stimulus. I define reference error as the difference between a participant’s reported average and the reference average, that is, the average of the underlying data series from which all three mark types for a given seed, variability level, and variability position were generated.3
For line and point, the displayed marks are the underlying series, so displayed error is reference error. For pointArc they do not: placing dots equally spaced along the line arc places more points in the high-variability region, so the displayed average sits closer to that region than the reference average does (see, e.g., Figure 3).
Figure 3. Example stimuli for the point and pointArc conditions, holding the variability position fixed to upper.
For either quantity, the direction-corrected version flips the sign so that positive values always indicate displacement toward the high-variability region: it equals the raw value when variability is in the upper region and its negation when variability is in the lower region. Therefore, direction-corrected reference error is what Moritz et al. report as normalized error and plot in their Figure 6. Below I present results under both definitions.

Figure 6. Direction-corrected error, split by whether variability appears in upper or lower y-values. Across mark types, conditions with variability in lower y-values show greater deviation from zero than their upper counterparts.
Recomputing Estimation Error
In figure axes and captions, error without qualification means displayed error; reference error is always labeled as such. I reproduce Moritz et al. [14]’s Figure 6 using the SummarySEWithin function from their supplementary materials (Main_Analysis/Functions.R), plotting direction-corrected reference error by mark type and variability level (Figure 4). The result matches the original.
Figure 4. Reproducing Experiment 2 results presented in Moritz et al. [14] Figure 6. The interval within each density plot shows the 95% CI, with the black dot marking the mean.
I follow the same procedure with direction-corrected displayed error in place of reference error (Figure 5). For line and point, nothing changes, as per definition. For pointArc, error at high variability drops to near zero, and the effect of added variability reverses sign. Participants, in other words, averaged the arc-spaced points about as accurately as they averaged anything. The distance between the point arc distributions in Figure 4 is thus evidence of the distance between the displayed points and a series that was never shown, not a perceptual pull toward variability for line graphs consisted of dots.
Figure 5. Recalculating Moritz et al. Experiment 2 results using the same analysis and plotting code as Figure 4, except with error instead of reference error.
Error is Not Symmetric
Direction correction folds the upper- and lower-variability conditions into one. Figure 6 unfolds them, plotting direction-corrected displayed error for each combination of mark type, variability level, and variability position.
I expected the two conditions to be similar; they are not. Across all three mark types, conditions with more variability in the lower y-values (“Variability Lower”) deviate further from zero than their upper counterparts, and the gap persists at variability 0, where the upper and lower stimuli differ only by vertical reflection. The released data cannot say whether the asymmetry comes from the probe’s starting position, limits on how far it could be dragged, or the stimulus rescaling. Future work could separate these accounts by varying the probe’s starting position, adding un-mirrored controls, and logging the probe’s reachable range.
Discussion
To summarize: what survives the change of metric is the comparison between the point and line conditions. Both display the same underlying data, so the two error definitions coincide there, and the original ordering — line showing greater bias than point — holds under either. What does not survive is the placement of pointArc, which depends entirely on which quantity is treated as ground truth, unless one is prepared to defend the auxiliary assumption that viewers of every mark type are estimating the same latent data series, in which case the reference average is the right truth after all. The salience account, as stated, neither asserts nor rules out that assumption.
When the Theory is Silent, the Metric Speaks
Moritz et al. motivate their error metric with a theoretical commitment: if variability overweighting is driven by visual salience, then the relevant ground truth is the average of the underlying data series, because salience is hypothesized to distort perception of that quantity regardless of mark type. This is a reasonable choice, but not a forced one: the salience account does not distinguish between the underlying data average and the displayed data average, so it cannot say which one error should be measured against. For the line and point conditions the distinction does not matter; for pointArc it does. The theory is silent precisely where the operationalization has consequences.
This is what Akbaba et al. [1] call entanglement: the theoretical commitment motivates the operationalization, and the operationalization produces the pattern that confirms the commitment — an entanglement that stays invisible as long as only one operationalization is examined. In Fried’s [6, 7] terms, salience here operates as a weak theory: a narrative account that specifies neither the functional form of the predicted relationship nor what would count as disconfirming evidence — under a different metric, any result can be read as a mark type simply being “less salient.” A strong theory would specify enough that the choice of error metric is either derivable from the theory or explicitly flagged as a researcher degree of freedom.
This diagnosis suggests its own remedy. Guest and Martin [10] argue that computational modeling forces theory building precisely because a formal model cannot remain silent: to implement a salience account of average estimation, one must commit to what salience is a function of, what quantity it distorts, and therefore which error metric measures the distortion. That is how the present observations arose: composing decoding operators to predict responses on Moritz et al.’s data [13] required committing to which quantity viewers are estimating — a commitment the verbal account never has to make, and whose necessity became visible only when a generative model demanded it. A named effect without a specified mechanism is difficult to falsify, and what cannot be falsified cannot be killed [3]. The field’s growing catalog of named biases is not, in itself, progress; each name is a promissory note for a mechanism, and the note comes due when, as here, two reasonable operationalizations of the same claim disagree.
What is Reproduction For? Where Should Explanation Stop?
It is tempting to read a reanalysis like this one as a verdict: the effect “failed to reproduce,” therefore it is not real. That reading is too strong in both directions — reproducibility and truth are separable. properties: reproducible results need not be true, and irreproducible results need not be false [5]. Nor am I claiming that visual salience is incorrect.
Every theory of the mind (and eye) buys tractability with assumptions; there is no free lunch. The question is where the assumptions live: whether they are explicit or implicit, and whether they were forced by modeling necessity or merely went unexamined. What troubles me about “salience” as an explanatory device is that it is where explanation stops. Once we arrive at “more salient marks capture more attention,” there is nothing beyond it — not what salience is computed from, not how attention weights an estimate, not which quantity gets weighted. Every explanation stops somewhere; it matters where. Vision science stops later — it maintains computational models of salience and formal accounts of ensemble perception — and whether importing that machinery would sustain variability overweighting, dissolve it, or split it into several distinct effects is exactly the kind of question this workshop exists to ask.
Supplemental Materials
All supplemental materials are available on OSF at https://osf.io/g73wc/, released under a CC BY 4.0 license.
Figure Credits
Figure 1 is a screenshot of Figure 3 in Moritz et al and Figure 4 is a recreation of Figure 6 from Moritz. et al. [14], which is in the public domain. The remaining figures (Figure 5 and Figure 6) followed Figure 4 in style. The first author created the diagram in Figure 2.
Acknowledgments
I thank the reviewers for their helpful feedback and Matthew Kay for early discussion of this work and the authors of Moritz et al. [14] for publicly releasing their data and analysis code in an easily accessible form.
References
- [1]D. Akbaba, L. Klein, and M. Meyer. Entanglements for visualization: Changing research outcomes through feminist theory. IEEE Transactions on Visualization and Computer Graphics, 31(1):1279–1289, 2024. doi: 10.1109/TVCG.2024.3456171 3DOI
- [2]D. Albers, M. Correll, and M. Gleicher. Task-driven evaluation of aggregation in time series visualization. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 551–560, 2014. doi: 10.1145/2556288.2557200 1DOI
- [3]M. Correll. Ross-chernoff glyphs or: How do we kill bad ideas in visualization? In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–10, 2018. doi: 10.1145/3170427.3188398 3DOI
- [4]M. Correll, D. Albers, S. Franconeri, and M. Gleicher. Comparing averages in time series data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1095–1104, 2012. doi: 10.1145/2207676.2208556 1DOI
- [5]B. Devezer, L. G. Nardin, B. Baumgaertner, and E. O. Buzbas. Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity. PloS one, 14(5):e0216125, 2019. doi: 10.1371/journal.pone.0216125 4DOI
- [6]E. I. Fried. Lack of theory building and testing impedes progress in the factor and network literature. Psychological Inquiry, 31(4):271–288, 2020. doi: 10.1080/1047840X.2020.1853461 3DOI
- [7]E. I. Fried. Theories and models: What they are, what they are for, and what they are about. Psychological Inquiry, 31(4):336–344, 2020. doi: 10.1080/1047840X.2020.1854011 3DOI
- [8]M. Gleicher, M. Correll, C. Nothelfer, and S. Franconeri. Perception of Average Value in Multiclass Scatterplots. 19(12):2316–2325. doi: 10.1109/TVCG.2013.183 1DOI
- [9]S. N. Goodman, D. Fanelli, and J. P. A. Ioannidis. What does research reproducibility mean? 8(341):34|ps12–34|ps12. doi: 10.1126/scitranslmed.aaf5027 1DOI
- [10]O. Guest and A. E. Martin. How computational modeling can force theory building in psychological science. Perspectives on Psychological Science, 16(4):789–802, 2021. doi: 10.1177/1745691620970585 3DOI
- [11]M.-H. Hong, J. K. Witt, and D. A. Szafir. The Weighted Average Illusion: Biases in Perceived Mean Position in Scatterplots. doi: 10.48550/arXiv.2108.03766 1DOI
- [12]Y. Jansen, J. B. Vornhagen, O. Iarygina, K. S. Niksirat, L. Besançon, P. Dragicevic, J. Gori, and C. Wacharamanotham. The many ways of being transparent in human-computer interaction research. 2024. doi: 10.31219/osf.io/2wze6 1
- [13]S. Long, R. Chang, E. Wu, A. Kale, and M. Kay. Visual decoding operators: Towards a compositional theory of visualization perception. arXiv preprint arXiv:2604.02220, 2026. doi: 10.48550/arXiv.2604.02220 1, 3DOI
- [14]D. Moritz, L. M. Padilla, F. Nguyen, and S. L. Franconeri. Average estimates in line graphs are biased toward areas of higher variability. IEEE Transactions on Visualization and Computer Graphics, 30(1):306–315, 2023. doi: 10.1109/TVCG.2023.3326589 1, 2, 3, 4DOI
- [15]D. A. Szafir, S. Haroz, M. Gleicher, and S. Franconeri. Four types of ensemble coding in data visualizations. 16. doi: 10.1167/16.5.11 1DOI
- [16]C. Tseng, G. J. Quadri, Z. Wang, and D. A. Szafir. Measuring Categorical Perception in Color-Coded Scatterplots. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, pp. 1–14. Association for Computing Machinery. doi: 10.1145/3544548.3581416 1DOI



