WorksComputer Science
One Mean Can't Rule Them All
Abstract
There has been extensive debate on how to summarize performance results, especially within the computer architecture community. This article presents a systematic approach to choosing the correct mean, aiming to answer this question for all computer scientists once and for all.
Motivation
Figuring out how to summarize performance results is challenging, especially given the conflicting literature on the topic. The majority of work presented in MICRO’23 did not state the aggregation metric used, indicating a lack of clarity in the architecture community. Common misconceptions include using the geometric mean exclusively for normalized values [1], and always using the harmonic mean for raw instructions per cycle (IPCs) [2], [3]. Additionally, previous work focuses on a specific metric (mainly IPC) instead of providing a general guideline. The confusion is worsened by the usage of the means—originally intended to reflect central tendency or expected value—to rank systems by their suitability to run a workload.
Some argue that the choice of mean does not matter [4]. This is incorrect as using the correct mean ensures that the result is meaningful. Figure 1 further disproves this, as while the harmonic and arithmetic means of SPECrates are correlated, using the wrong mean can be misleading. For instance, the Supermicro 5019C-WR has a slightly higher arithmetic mean than the NEC Express5800, yet the NEC achieves a 19% higher harmonic mean, which can lead to, for example, buying a less appropriate machine.
Figure 1. The arithmetic vs harmonic means of SPECrates for over 9000 machines, while there is a strong correlation (top), zooming in (bottom) shows that using the wrong mean can lead to the wrong conclusion, based on [3], data from [5].
Table 1 compares select prior work to the proposed aggregation metrics. Some match (✓), some match for incorrect reasons (▲), and many do not match (✗). The problematic assumptions in previous work can be categorized into: incorrectly assuming the workload is sequential (Sequential assumption), focusing on expected values instead of scores (Expected value), using an aggregation metric designed to compare different classes of machines to compare the same class (Apples to oranges), assuming the means are all approximately the same (Triviality assumption), and finally, assuming that human intuition, rather than solid mathematical foundations are the basis for choosing a mean (Intuition). No previous work provides a generic framework for selecting a mean for different applications.
This paper starts by comparing the use of the mean to indicate an expected value vs a score, followed by what means mean and how they work and ends by providing examples of how to use this methodology.
| Work | Year | Workload | Metric | Advocates | This Work | Match | Problem |
| SPEC [6] | - | Benchmarks | Latency | Geomean | GMS | ✓ | - |
| How not to lie... [1] | 1986 | Benchmarks | Any | Geomean | GMS | ✓ | - |
| Characterizing... [2] | 1988 | Benchmarks | FLOPS | Hmean | GMS | ✗ | Sequential assumption |
| War of the... [7] | 2003 | Benchmarks | Latency | Geomean | GMS | ✓▲ | Expected value |
| The harmonic or... [4] | 2006 | Benchmarks | Any | Any | GMS | ✗ | Triviality assumption |
| R.I.P geomean... [3] | 2024 | Benchmarks | IPC | Hmean | GMS | ✗ | Sequential assumption |
| Symbiotic... [8] | 2000 | Multiprogram | IPC | WS | GMS | ✗ | Apples to oranges |
| System-level... [9] | 2008 | Multiprogram | IPC | WS | GMS | ✗ | Apples to oranges |
| Demystifying... [10] | 2012 | Multiprogram | IPC | A/Hmeans | GMS | ✗ | Assumes intuition |
| CRC1... [11] | 2016 | Multiprogram | IPC | WS | GMS | ✗ | Reasoning not stated |
| Amdahl’s law [12] | 1967 | End-to-end | Speedup | Hmean* | HMS | ✓▲ | Not generally applicable |
| Simulation Point... [13] | 2003 | End-to-end | IPC | Hmean | HMS | ✓ | - |
Table 1. Means throughout the years in Computer Architecture. WS: Weighted speedup, FLOPS: FP operations per second.
Expecting a value or a score?
“The mean, also called the average, is a measure of central tendency of a group of values." [14] It is traditionally used to obtain the expected, or most likely value of a set of data. This is useful when running an experiment multiple times to cancel out error sources. Means here are selected based on the distribution of data; the arithmetic mean should be used if the data are normally distributed, the geometric mean if it is log-normally distributed, and the harmonic mean if the reciprocals of the data are normally distributed. An example of such use case is Ricau’s experiments [15].
Obtaining a representative value may not always be the goal. Sometimes, the aim is to rank two sets of numbers based on a certain property. In this case, these basic mathematical operations may still be useful. For instance, when evaluating the impact of a disturbance on a baseline: comparing the “mean" of the baseline numbers to the “mean" of the numbers after the disturbance, can reveal some properties of this disturbance. For example, when summarizing the runtime lengths of different sections of a program, we are not interested in the expected value of each section’s runtimes, but rather in which configuration minimizes the total program runtime. Thus, the smallest mean must reflect the configuration with the shortest total runtime. Furthermore, an absolute decrease in runtime for any section must be considered equal as they all contribute equally to decreasing the total runtime.
In this case, the mean should not follow the distribution of the data because we are not interested in an expected value but rather in whether the workload takes longer to run or not. In fact, in the previous example, if the runtimes are log-normally distributed and the geometric mean is used, the smallest mean may
not correspond to the shortest total runtime, rendering the results useless. The selection of the mean should thus follow the properties we are interested in. To preserve the distinction between both cases, different terms should be used for each one. For the latter case, one suggestion is adding “As a Score" to the end (i.e. Geometric Mean as a Score (GMS)), or simply calling them scores, (i.e. Geometric Score).
In summary, means are used for two purposes:
To obtain a number representing a group of values. Here, we are interested in an expected value that represents a central, most likely value. An example would be an experiment that is repeated multiple times to remove errors affecting each value. The appropriate mean depends on the distribution of data as exemplified by Mashey [7] and Iqbal et al [16]. Here, calling them means is appropriate (e.g. Geometric Mean)
To obtain a score to compare multiple different sets of values. Here, we are interested in which set maximizes (or minimizes) a certain property. Normally, each value in each set would have a corresponding value in the other sets. An example would be different benchmark results across multiple machines. The appropriate mean depends on the statistical property of interest. This use case is the focus of this work. Adding “as a Score" to the end (e.g. Geometric Mean as a Score (GMS)) or simply calling them scores (e.g. Geometric Score) helps distinguish both cases.
What Do the Means Mean?
If we care about comparing multiple sets of data instead of obtaining an expected value, the selection of a mean shifts from the data distribution to the relevant metrics. While we may measure a certain metric (such as IPC), we may be more interested in another metric (such as maximizing throughput). Defining those quantities and their relationship gives the mean a meaning. For example, does the machine with the highest mean IPC always have the highest throughput?
| BL | ❶ | ❷ | ❸ | ❹ | ❺ | ❻ | |
| IPC GMS | 0.45 | 0.63 | 0.63 | 0.63 | 0.47 | 0.49 | 0.63 |
| IPC HMS | 0.33 | 0.36 | 0.57 | 0.36 | 0.36 | 0.34 | 0.57 |
| IPC AMS | 0.6 | 1.1 | 0.7 | 1.1 | 0.61 | 0.7 | 0.7 |
| Rel. Lat. Dec. | 0% | 50% | 50% | 50% | 10% | 17% | 50% |
| Rel. TP Inc. | 0% | 100% | 100% | 100% | 22% | 20% | 100% |
| Abs. Lat. Dec. | 0 | 1 | 5 | 1 | 1 | 0.33 | 5 |
| Abs. TP Inc. | 0 | 10 | 2 | 10 | 0.22 | 2 | 2 |
Table 2. Numerical Results for Figure 2. TP=Throughput
Mathematical Background
The AMS is defined as . Since addition is associative and commutative, any absolute additive change to any value will affect the mean equally, i.e. equals .
The GMS is defined as . Multiplication is also associative and commutative, so any multiplicative (percentage/relative) change to any value will affect the mean equally, i.e. equals .
The HMS is the reciprocal of the AMS of the reciprocals of the constituent values, and is defined as , any absolute change in the reciprocals of the values will affect the mean equally. Note that since the GMS is multiplicative, a relative change in any of the reciprocals of the constituent values also affects the mean equally.
IPC as an Illustrative Example
When performing experiments, two important metrics arise: a “measured metric" and a “relevant metric", the former refers to the metric being measured, such as the time to complete a workload, IPC, or misses per thousand instructions (MPKI), while the latter refers to a metric we would like to maximize or minimize, such as latency, throughput, or energy consumption. Finding the correct mean relies on finding the relationship between them.
In computer architecture, we are typically interested in minimizing the latency of each application, or maximizing its throughput; thus, both serve as relevant metrics. IPC is usually easier to measure, especially for microarchitecture research. Thus, it is usually used as a measured metric. IPC1 is related to both metrics by , .
Figure 2 shows how different IPC means are affected by changes in throughput and latency for a workload composed of two benchmarks: Benchmark A (high IPC) and Benchmark B (low IPC). Both benchmarks are run for 20 time units either independently or sequentially. We measure the mean IPC when each benchmark is sped up separately (i.e. speeding up Benchmark A but not Benchmark B, and vice versa). Three speedup cases are shown: (i) a 2× relative IPC increase (0.5× CPI decrease), (ii) a −0.5 absolute IPC decrease, and (iii) a +0.2 absolute IPC increase.

Figure 2. The relevant metric (latency/throughput) is changed to achieve the same measured metric(IPC) mean. Benchmark A and Benchmark B are run (sequentially or independently) for 20 time units and each is sped up separately.
In ❶ and ❷, the same relative change in throughput or latency (both directly and inversely proportional to the measured metric, by 2× and 0.5×, respectively) yields the same GMS of IPCs, regardless of which benchmark is affected. This shows that the GMS is appropriate when relative changes to any value are equally important.
In ❸ and ❹, an equal absolute change in latency (inversely proportional to the measured metric, by −1 time unit per instance) yields the same HMS of IPCs, again, independent of which benchmark is affected. This means that applications with a higher baseline IPC (or lower CPI/latency) need to be sped up more in relative terms to achieve the same HMS. Although this may appear unfair, it can be desirable in some cases as will be discussed later.
In ❺ and ❻, the same absolute change in throughput (directly proportional to the measured metric, by +2 instances) yields the same AMS, independent of the affected benchmark. This means that applications with a lower baseline IPC/throughput need to be sped up more in relative terms to achieve the same AMS, which may also be appropriate in some cases.
Both the HMS and AMS IPC insights only hold if the number of instructions is the same for both benchmarks, otherwise, weighting is needed as will be discussed later.
This leads to the following, for any relevant and measured metric.
If relative changes to the relevant metric matter most use the GMS on the measured metric.
If absolute changes to the relevant metric matter most:
Use the AMS on the measured metric if both are directly proportional.
– Use the HMS on the measured metric if both are inversely proportional.
Since the GMS deals with relative instead of absolute values, it has the special property of providing a qualitative analysis, which is useful when the constituent values do not have any inherent weighting. For example, a benchmark suite, where a relative change to the performance of any constituent benchmark matters equally, regardless of the baseline performance. i.e. applications with high latency are not more or less important than ones with low latency.
Weights
We discussed how different means preserve different properties, but the insights for the AMS and the HMS do not hold if the instruction count for each benchmark is not the same. This indicates that weighting must be used to preserve the properties for the relevant metric.
For the AMS, weighting is done by multiplying each input value by a weight indicating how much it contributes to the final score, then summing up the results: , weighting for the harmonic mean is similar:
. Note that in both cases, the total sum of all weights must be one.
The weights are taken from the relationship between the measured metric, and the relevant metric. If the proportionality constant is not the same for all values, weighting will be needed, and this proportionality constant will form the weight. Since the sum of all proportionality constants will not always be one, all weights must be divided by their sum. Weighting can be thought of as partially converting the measured metric to the relevant metric then back again.
An example is when the relevant metric is latency, the measured metric is IPC, and absolute changes to latency are prioritized. The equation relating IPC and latency is , here, the frequency is constant throughout all benchmarks, but the number of instructions may not be, so , since the latency is inversely proportional to the IPC, so the HMS should be used, the weights are the instruction count. Since the sum of all instruction counts may not be equal to one, the instruction count for each benchmark should be normalized to the sum of all instruction counts, i.e. .
Visualizing Means
To visualize the difference between the means, the novel "iso-mean line plots" are shown in Figure 3 and Figure 4. This plot shows a line where the performance results of two benchmarks (x-axis and y-axis) vary, but combine to produce the same mean. The benchmarks from Figure 2 are reused, with the circled numbers indicating the same configurations as before. The performance of Benchmark A and Benchmark B is shown on the X-, and Y-Axes, respectively with the relative change (speedup) shown on the main (bottom and left) axes, and the absolute IPC shown on the secondary (top and right) axes. “A baseline” and “B baseline” indicate the mean that produces a 100% speedup in A with no speedup in B and vice versa, respectively.
Figure 3. Iso-mean plots (for Figure 2 benchmarks, Benchmark A and Benchmark B) depict relative (bottom/left axes) and absolute (top/right axes) speedups. Left: The AMS (green, dashed) holds for an absolute IPC increase of +0.2, the GMS (red, solid) for a 2.0× relative IPC increase, and the HMS (purple, dotted) for a −0.5 CPI decrease. Middle: Iso-mean lines for applications with equal baseline IPCs or when using weighted speedups. Right: Similar to left, includes additional arithmetic (+1.0 IPC) and harmonic (−2.5 CPI) lines showing how different the results are for reversed relative speedups.
Figure 4. Iso-mean plots following Figure 3 with the maximum speedup increased to 10× to demonstrate how the three means handle outliers differently. Left: linear axes (arithmetic iso-mean is linear), Middle: log axes (geometric iso-mean is linear), Right: 1/x axes (harmonic iso-mean is linear). The effects of stacking on the AMS are shown in purple.
In Figure 3 (Left), the geometric iso-mean line (red) is mirrored across the X=Y line, indicating that a speedup of (a,b) is equivalent to a speedup of (b,a), which means switching the speedup values for two applications produces the same result. The arithmetic iso-mean line preserves absolute changes, as the sum of the IPC shift is the same (+0.2 from the baseline).
In Figure 3 (middle), both benchmarks have the same baseline IPC (or are normalized to a baseline as will be discussed later.) removing the asymmetry. The arithmetic iso-mean line (green) crosses the axis at 1.5, indicating that speeding up one benchmark by 100% produces the same mean as speeding up both by 50%.
by 50%. The geometric iso-mean line (red) equates a 100% speedup in one to a 40% speedup in both, while the harmonic iso-mean line (blue) reduces this further to 30%. This illustrates that the AMS does not attenuate outliers, whereas the HMS does so the most. Figure 4 (left) amplifies the relative increase in IPC from to . Here, a 10× speedup in one application is equal to a 5.5×, 3.3×, and 1.8× speedup in both applications for the AMS, GMS, and HMS, respectively. Showing more clearly how the means handle outliers.
The additive nature of the AMS and the HMS limits the effect of the input values when they approach zero or infinity, respectively. Assuming all other values remain constant and that there are 10 values, when one value approaches infinity, the HMS cannot increase by more than , which is 11%. On the other hand, when a value approaches zero, the AMS cannot decrease by more than , or 10%. The multiplicative nature of the GMS prevents this issue.
Figure 4 plots the iso-mean lines on linear, log and reciprocal scales showing how each iso-mean line is a straight line in the corresponding scale.
Stacked Changes
When changes are stacked, i.e. when a change to a benchmark is repeated, the mean will still follow the above properties. For example, doubling the IPC of a workload twice results in the same GMS as doubling the IPC of two different workloads. Similarly, for the
AMS, adding +1 twice to a workload’s IPC is equivalent to adding +1 to the IPC of two different workloads. But adding +1 to the same IPC represents a smaller percentage increase the second time, this means the AMS amplifies repeating relative changes, while the HMS, similarly, diminishes them.
Figure 4 (left) illustrates this for the AMS: after a 5.5× speedup for one application, only a further 1.8× speedup is equal to a 5.5× speedup for the other application. The HMS reverses this; after a 1.8× speedup of one application, a further 5.5× speedup is needed to match a 1.8× speedup for the other application. For the GMS, equal percentage speedups always yield the same change in mean.
Normalizing to a Baseline
Some studies [8], [9], [11] propose normalizing values to a baseline before taking the mean —known as weighted speedup— to counter the bias in the arithmetic and harmonic means. However, if the baselines differ for each set of data [17], the results can be misleading or even counter-intuitive. For example, when normalizing the multicore results for each cache replacement policy to the single-core results of the respective policy, the faster policy will score less if it speeds up the system so much that when used in a multicore setup, the system reaches a bottleneck causing the faster system to scale less well than the slower system, even if it was faster in both cases.
When all baselines are the same, this becomes a speedup aggregation metric. For the geometric mean, normalization does not affect rankings or relative values because division is commutative, i.e.:
On the other hand, using the arithmetic or harmonic means is problematic because:
The choice of baseline can distort both the relative differences and even rankings [1].
While percentage speedups for any application are equal, the non-logarithmic nature of these means causes stacked changes to the same application to have diminishing or growing effects.
The final aggregation metrics are meaningless and are harder to interpret and reason about.
This becomes even more convoluted for multiprogrammed workloads, where the ratio is not simply an enhanced system compared to a baseline system, but rather an enhanced system (e.g. with 4MB L3 cache) running in a multicore configuration to a baseline system (e.g. with 1MB L3 cache) running in a single-core configuration [11], conflating the speedup from the enhancement with the slowdown of sharing resources, along with the complex interactions between them, producing a meaningless aggregation metric.
In conclusion, normalizing to a baseline is unnecessary at best and may even be harmful. It can be useful to measure scalability (e.g. of more cores), but should not be used to measure performance.
Steps to Choose the Correct Mean
To select which mean to use in a certain scenario, the following steps should be followed:
Find out what metric should be maximized or minimized, and whether absolute, or relative changes to it matter more. (relevant metric)
Find out a measurable metric that is related to the relevant metric. (measured metric)
If relative changes to the relevant metric matter most, use the GMS on the measured metric.
If absolute changes matter most, figure out the relationship between both metrics.
If the relationship is directly proportional, use the AMS on the measured metric.
If the relationship is inversely proportional, use the HMS on the measured metric.
If using the AMS or the HMS, figure out the weights using the proportionality constant. The variable components in this proportionality constant should be used as weights, normalized to their sum to make the total equal to one.
The HMS
The HMS preserves absolute changes when the relevant and measured metrics are inversely proportional. Examples include Amdahl’s law [12] and SimPoint [18].
Amdahl’s Law on Steroids
Speeding up a portion of an application has diminishing returns, as the more one portion is sped up, the less significant it becomes to overall performance.
Amdahl’s law quantifies this by estimating speedup in a program with one parallelizable and one non-parallelizable section executed sequentially [12].
Amdahl’s law not only falls in line with this work, but this work also allows us to derive a generalized version allowing arbitrary sections to be sped up independently. Following the above approach, the relevant metric is the overall latency of a program, prioritizing absolute changes, the measured metric is the speedup. The runtime of a program section is inversely proportional to speedup, i.e. . The original time serves as the proportionality constant, so weighting should be each section’s fraction of the total original runtime.
Amdahl’s law’s workload has a parallel portion and a serial portion , where only sped up by .
Thus, the HMS is a generalization of Amdahl’s law.
Sampled Simulation
Sampled simulation is used to speed up simulation time by only simulating smaller, but representative, portions (samples) of a benchmark [18] from which the overall application performance is estimated. Here, the relevant metric is the total execution time with absolute changes being valued equally. When the measured metric is the IPC, the HMS should be used. Since the proportionality constant is the number of instructions, the weights should be the normalized number of instructions. Indeed, the authors of SimPoint [18], a popular sampled simulation technique, suggest using the weighted arithmetic mean of CPI [13], which is equivalent to the weighted HMS of IPC.
The GMS
Some workloads consist of benchmarks that reflect expected tasks qualitatively but not quantitatively. Benchmark length does not indicate importance [7], so none should be prioritized. Examples include benchmark suites and multiprogrammed workloads.
Benchmark Suites
SPEC is a benchmark suite comprising independent benchmarks whose length does not reflect their importance [6]. For example, on the Supermicro 5019C-WR, 600.perlbench_s takes 503 seconds to execute vs. 113 seconds for 625.x264_s, but neither is 4.5× more important than the other [5, 7]. This means that relative or percentage changes to each benchmark’s latency or throughput matter equally and should produce the same mean; therefore, the GMS must be used.
Although SPEC uses the geometric mean,4 it divides each result by that of a reference machine due to the assumption that this specific ratio is log-normally distributed. This assumption is irrelevant, and the GMS should be used regardless of the distribution since we are not interested in an expected value, but rather in comparing two machines. (This shows the importance of distinguishing between both use cases.) Using a reference machine is also superfluous as the multiplicative nature of the geometric mean means that the denominators cancel out when comparing two machines. On the other hand, this contradicts recent work which suggests using the harmonic mean [3], which is only appropriate if the relevant metric is absolute. reduction in runtime, indicating an implicit assumption that all benchmarks are run back-to-back.
Multiprogrammed Workloads
Multiprogrammed workloads run benchmark mixes in parallel, restarting any that finish early until all are done in order to keep the pressure on shared resources. While the benchmarks share resources, they do not communicate or wait for each other. Since they are independent, their baseline performance is not an indicator of their importance, and relative changes to latency and throughput are more relevant, therefore, the GMS should be used. These workloads have primarily used the weighted speedup which yields incorrect results as discussed previously.
The AMS
The AMS preserves absolute changes when the relevant and measured metrics are directly proportional.
MPKI
MPKI is a common metric used to measure the efficiency of a cache. A higher MPKI means more energy and power are consumed. As a common measured metric, this is roughly directly proportional to memory power consumption, which is directly proportional to energy used, with the proportionality constant being . When measuring the MPKI for simpoints, the weighted AMS should be used since the relevant metric is an absolute decrease in the energy used, which is directly proportional to the measured metric (MPKI).
On the other hand, for benchmark suites, the MPKI of each benchmark is independent, meaning the GMS is more appropriate. Most computer architecture work uses the AMS for benchmark suite MPKI, this can be reasonable assuming all benchmarks are run for an equal amount of time, and have the same IPC as then applications with high MPKI would contribute more to energy consumption than those with low MPKI indicating the importance of clearly stating the underlying assumptions, though misses per thousand cycles (MPKC, ) may be more suitable since it eliminates the assumption that all IPCs are equal5.
Hit Ratios
Hit ratios are used to characterize cache performance, if a program is split into multiple sections, and the overall hit ratio is desired, the relevant metric is the number of hits, which should be maximized6, and the measured metric is the hit ratio for each section. As they are directly proportional, the AMS should be used. The proportionality constant is the number of accesses in each section, which —divided by the total number of accesses— is the weight.
Conclusion
This paper provides a general framework to help summarize performance results by rigorously analyzing the means and their properties. We make the important distinction between the two use cases for means; as an expected value, and to compare two populations of data. We call on the research community to distinguish between both use cases by using different names for each use case (e.g. geometric mean vs GMS) This is followed by an analysis of when each mean is most appropriate, along with different aspects of the means such as weights, or normalizing to a baseline. Finally, we provide examples of applying these properties to each different means. What sets this work apart from previous work is mathematically deriving the general properties of each mean and how to use them, instead of using ad-hoc examples and common myths to connect a mean to a certain workload. This work should help guide all computer scientists when summarizing performance.
ACKNOWLEDGMENTS
This work was supported by the Knut and Alice Wallenberg Foundation through the Wallenberg Academy Fellows Program under Grant 2015.0153, the European Research Council (ERC) under the European Union’s Horizon 2020 Research and Innovation Program under Grant 715283, and the Swedish Research Council under Grant 2019-02429.
REFERENCES
P. J. Fleming and J. J. Wallace, “How not to lie with statistics: the correct way to summarize benchmark results,” Commun. ACM, vol. 29, no. 3, p. 218–221, mar 1986. [Online]. Available: https://doi.org/10.1145/5666.5673
Biographies
Ahmed Nematallah
Ahmed Nematallah is a PhD student at Uppsala University, Sweden. His current research interests include Computer Architecture, Microelectronics, and Security. Contact him at [email protected].
References
- [1]P. J. Fleming and J. J. Wallace, “How not to lie with statistics: the correct way to summarize benchmark results,” Commun. ACM, vol. 29, no. 3, p. 218–221, mar 1986. [Online]. Available: https://doi.org/10.1145/5666.5673
- [2]J. E. Smith, “Characterizing computer performance with a single number,” Commun. ACM, vol. 31, no. 10, p. 1202–1206, oct 1988. [Online]. Available: https://doi.org/10.1145/63039.63043
- [3]L. Eeckhout, “R.I.P. geomean speedup use equal-work (or equal-time) harmonic mean speedup instead,” IEEE Computer Architecture Letters, vol. 23, no. 1, pp. 78–82, 2024. [Online]. Available: https://doi.org/10.1109/LCA.2024.3361925
- [4]D. Citron, A. Hurani, and A. Gnadrey, “The harmonic or geometric mean: does it really matter?” SIGARCH Comput. Archit. News, vol. 34, no. 4, p. 18–25, sep 2006. [Online]. Available: https://doi.org/10.1145/1186736.1186738
- [5]SPEC, “All Published SPEC CPU 2017 Integer Speed Results.” [Online]. Available: https://www.spec.org/cpu2017/results/cint2017/
- [6]“Overview - CPU 2017,” Nov 2022. [Online]. Available: https://www.spec.org/cpu2017/Docs/overview.html#metrics
- [7]J. R. Mashey, “War of the benchmark means: time for a truce,” SIGARCH Comput. Archit. News, vol. 32, no. 4, p. 1–14, sep 2004. [Online]. Available: https://doi.org/10.1145/1040136.1040137
- [8]A. Snavely and D. D. Tullsen, “Symbiotic job scheduling for a simultaneous multithreaded processor,” in Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS IX. New York, NY, USA: Association for Computing Machinery, 2000, p. 234–244. [Online]. Available: https://doi.org/10.1145/378993.379244
- [9]S. Eyerman and L. Eeckhout, “System-level performance metrics for multiprogram workloads,” IEEE Micro, vol. 28, no. 3, pp. 42–53, 2008.DOI
- [10]P. Michaud, “Demystifying multicore throughput metrics,” IEEE Computer Architecture Letters, vol. 12, no. 2, pp. 63–66, 2013. [Online]. Available: https://doi.org/10.1109/L-CA.2012.25
- [11]“Cache Replacement Championship.” [Online]. Available: https://jilp.org/jwac-1/
- [12]G. M. Amdahl, “Validity of the single processor approach to achieving large scale computing capabilities,” in Proceedings of the April 18-20, 1967, spring joint computer conference, 1967, pp. 483–485, reprinted in IEEE Solid-State Circuits Society Newsletter, vol. 12, no. 3, pp. 19–20, 2007. [Online]. Available: https://doi.org/10.1109/N-SSC.2007.4785615
- [13]B. Calder, “Simulation Point Tutorial and FAQ,” Jun 2005. [Online]. Available: https://cseweb.ucsd.edu/~calder/simpoint/sim-points-FAQ.htm
- [14]Brilliant.org, “Mean (Average) | Brilliant Math & Science Wiki.” [Online]. Available: https://brilliant.org/wiki/data-mean/
- [15]P.-Y. Ricau, “Statistically Rigorous Android Macrobenchmarks.” [Online]. Available: https://blog.p-y-wt/statistically-rigorous-android-macrobenchmarks
- [16]M. F. Iqbal and L. K. John, “Confusion by all means,” in Proceedings of the 6th International Workshop on Unique chips and Systems (UCAS-6), 2010. [Online]. Available: https://lca.ece.utexas.edu/people/iqbal/ucas6-paper.pdf
- [17]I. Shah, A. Jain, and C. Lin, “Effective mimicry of belady’s min policy,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022, pp. 558–572. [Online]. Available: https://doi.org/10.1109/HPCA53966.2022.00048
- [18]T. Sherwood, S. Sair, and B. Calder, “Phase tracking and prediction,” in Proceedings of the 30th Annual International Symposium on Computer Architecture, ser. ISCA ’03. New York, NY, USA: Association for Computing Machinery, 2003, p. 336–349. [Online]. Available: https://doi.org/10.1145/859618.859657


