WorksComputer Science
A predictive, per-engine energy roofline of the Apple Neural Engine
Abstract
The Apple Neural Engine (ANE) is in over a billion devices, but is characterized indirectly via CoreML. Using a CoreML-free fused-graph dispatch path, we measure the ANE, along with the Apple GPU and CPU on the same device, with per-engine energy accounting. The consumer-available roofline does not predict the ANE in practice. A critical correction is that the ANE sees two effective memory bandwidths: a slow activation path and a fast weight-stream path. Two roofline ridges follow, and the weight-stream ridge is below that of the GPU. The ANE single-operand compute peak decreases instead of plateauing. Its weight stream is byte-bound in fp16 but limited by the dequantizer's element rate in int8. As a result, int8 compression saves energy at unchanged throughput and an int4 lookup-table stream raises throughput and efficiency at a cost of 2.55 points of ImageNet top-1. The corrected roofline identifies the lower-energy ANE paths and bounds it from below. A constructed device map reproduces across four unique ANE generations. Results show that operators should be placed by arithmetic intensity and fused for optimal performance.
Introduction
The Apple Neural Engine (ANE) is among the most widely deployed machine-learning accelerators, present in every recent iPhone, iPad, and Apple-silicon Mac. Despite this ubiquity, the engine’s performance and energy profile has had little open measurement. The publicly quoted figure is a peak 38 TOPS (tera-operations per second) for the M4 Neural Engine. Community guidance, discussed in a community knowledge base [14] and implicit in Apple’s transformers-on-the-engine material [1], states that the engine is worthwhile mainly for inference on large transformer models. Both the principal figure and the large-model guidance come from CoreML, which is the only sanctioned interface to the ANE. However, these figures instrument the hardware indirectly. CoreML decides, by an opaque placement policy, which operators run on the ANE versus the GPU or CPU, and it exposes neither a fair per-operator dispatch nor energy accounting [6]. The community’s working assumptions about the ANE thus reflect a scheduler’s defaults and optimization schemes, not the hardware envelope.
This work measures which workloads favor the ANE over the Apple GPU and CPU for the same device on speed and watts. No CoreML-free method exists to dispatch arbitrary operators with controlled batch, dtype, and warm state. We provide one with anaforge [5], a fused-graph compiler and runtime that lowers a user’s operator graph to a single fused ANE program and submits it through Apple’s e5rt execution runtime. Every timed operator runs on the ANE with a controlled batch, data type, and warm state. This gives a fair per-operator dispatch and a controlled device comparison. The access mechanism is documented elsewhere [5, 6], though the instrument is described in full in the online supplement so the measurements can be independently judged. This includes two checks that every benchmarked operator runs on the ANE and not a fallback device. Against the ANE we run MLX, Apple’s Metal-based array framework (fp16), for the GPU and numpy/Accelerate (fp32, via Apple’s AMX matrix coprocessor) for the CPU, on an Apple M5 Pro (ANE architecture h17s; a 48 GB MacBook Pro, 18-core CPU and 20-core GPU). The setup repeats the three-way roofline comparison, neural processing unit (NPU, or ANE here) versus GPU versus CPU, that was established for a datacenter TPU [19], for an on-device NPU with energy measured throughout. The fp32 Accelerate/AMX CPU path is a contextual reference, not a like-for-like fp16 comparand. The main comparison is therefore the ANE against its companion GPU.
Prior characterizations reach the engine through CoreML or target other accelerators: smartphone-AI benchmarking [16, 26], the per-engine Edge TPU evaluation [27], edge and mobile NPU energy studies [18, 21, 24, 29, 32], and a small body of direct-access ANE work [20, 28, 34]. These works do not isolate the ANE in a per-engine energy roofline across operating regimes (see § 11 for further discussion).
Here, we present a corrected roofline. The textbook single-roofline model mispredicts the ANE. The primary correction is that the ANE has two effective bandwidths: a slow activation path and a fast weight-stream path, so two ridges, not one. The weight-stream ridge that governs all compute-bound work lies below the GPU’s, so an operator becomes compute-bound on the ANE at a lower arithmetic intensity than on the GPU. Its weight stream is byte-bound in fp16 but limited by the dequantizer’s element rate in int8, so what weight compression buys depends on the encoding and on the part. Its single-fused-operand compute roof for general matrix–matrix multiplication (GEMM) falls instead of plateauing, so fusion, not a larger operand, keeps work on the fast tier. We combine these three corrections into a per-engine energy roofline. From a workload’s arithmetic intensity it assigns each operator to the lower-energy engine and bounds its energy per operation, validated on held-out archetypes and a decode loop on the measured host (§ 5). The regime-dependent device map that prior NPU-versus-GPU work reports follows from this two-ridge structure. We test whether that map is a property of one part or of the engine family, and separate what reproduces across generations from what is specific to the measured host (§ 7).
The measurement method ties idle-subtracted total-package energy to the roofline per engine (§ 2). From it follow the three corrections (§ 3.1, § 3.3 and § 9), a closed-form energy predictor and placement rule fit from two per-device constants (§ 5), and the device map’s reproduction across four ANE generations (§ 6 and § 7).
Methodology
Measurement tool and dtype. The ANE is driven by aneforge’s fused-graph dispatch (fp16): each operator graph is lowered to one fused MIL program and submitted directly to the engine, with no CoreML model and therefore no CoreML placement policy. Every timed operator is verified to run on the ANE rail. The per-rail power attribution below confirms this, and a route-and-capability gate test compiles both lowerings of every selectable operator on-device and validates each against a reference. The full instrument description is in the supplemental material. The GPU is driven by MLX v0.31.2 (Metal, fp16; v0.32.2 for the compiled MiniLM and ViT-B/16 baselines). The real-model rows also use torch-MPS (torch’s Metal Performance Shaders backend) in fp16, with a single-session fp32 energy column. The CPU is driven by numpy/Accelerate (fp32, AMX), with torch as the reference where it is the canonical implementation. The GPU and ANE therefore run the same fp16 math; the CPU runs fp32 because Accelerate/AMX has no fast fp16 GEMM (it upcasts), which doubles its bytes-per-element and halves its arithmetic intensity, carried through the roofline arithmetic wherever it matters.
Speed. The measured latency is the minimum end-to-end latency over repetitions after warmup, a dedicated probe separate from the sustained power window; at the smallest shapes it measures each device’s dispatch floor. Each repetition forces device completion (mx.eval for MLX, the compiled-net call for the ANE, torch.mps.synchronize for MPS) so that no work is left in flight. All measurements are at warm caches and steady state, with matched batch sizes. Throughput is GFLOP/s where a closed-form FLOP count exists, otherwise elements/s. The latency includes each device’s host and dispatch overhead; at sub-millisecond shapes the overhead is part of the cost a deployment pays and is not subtracted out.
Power. Power is read from powermetrics per ANE, GPU, or CPU rail. The state of programmatic Apple-silicon power measurement (IOReport versus powermetrics, and the absence of a validated estimator) is the subject of [23]; prior Apple-silicon efficiency studies have read powermetrics per-rail across the M1 to M4 generations [15]. We adopt total-package active power herein. Following [12], a no-workload baseline is sampled once per run and subtracted (package 1196 mW in the device-map run). We take powermetrics’ own Combined Power (CPU + GPU + ANE) line, not a hand-summed rail total, because an ANE workload still draws from host CPU power on dispatch (as does an MLX workload).
The measurement boundary. By per-engine energy accounting we mean that every operating point reports total-package power, latency, and accuracy for the engine that ran it. The claim is one of attribution, not of an exhaustive power domain. The domain is the CPU, GPU, and ANE rails as powermetrics reports them. It excludes the DRAM rail, quantified next, and rests on a modeled SoC estimator, not a wall-socket meter. For each measurement a per-rail breakdown is reported. Median loaded package power with the coefficient of variation (CV%) over per-sample readings are included, with CV above 35 % flagged low-confidence measurement. Each loop drives the workload for the entire powermetrics window and samples at 500 ms to integrate the ANE’s sub-millisecond duty cycle; a naive fixed-time stop integrating at 100 ms produces a 60–140 % CV quantity, whereas this strategy gives 1–25 % CV (30 of 31 measurements below 17 %).
Cross-validation against an independent estimator. The total-package active-power metric is cross-validated against Apple’s IOReport energy-model counters at five operating points spanning the attribution axis (see supplement, Table S1), where the estimators agree to a median of 97–99 %. Both remain modeled SoC estimators, not wall-socket meters; the relative device comparisons are unaffected by a shared offset that cancels via ratios.
| Operating point | powermetrics (W) | IOReport (W) | Agreement (median; 10-run range) |
| GPU GEMM (, fp16) | 52.9 | 53.1 | 99.3 % (93.0–99.9) |
| CPU GEMM (, fp32, 100 % CPU rail) | 16.8 | 16.5 | 97.6 % (71.9–99.8) |
| ANE stream (standalone relu, CPU-dispatch-heavy) | 8.3 | 8.3 | 98.6 % (87.9–99.9) |
| ANE attention (ViT block, , ) | 7.9 | 8.2 | 97.6 % (74.7–99.3) |
| ANE conv (4-layer , 16 64ch @32²) | 5.5 | 5.7 | 96.9 % (73.2–99.5) |
Table S1. Comparison of idle-subtracted total-package active power from powermetrics and IOReport at five operating points over ten runs. Entries report medians and ten-run ranges. Both totals are ANE+GPU+CPU with the DRAM rail excluded (quantified separately below). Agreement measures cross-estimator consistency, not wall-meter accuracy.
The DRAM rail. The combined power excludes the DRAM rail, which the IOReport DRAM channel puts at 2.4–12.9 % of package across the loaded operating points. That share is device-dependent, not uniform: 4.1–5.3 % on an ANE convolution against 12.5–12.9 % on a saturating GPU GEMM. Including DRAM takes the GPU-over-ANE package-power ratio on that pair from 8.4 to 9.0. Since the GPU’s DRAM share is the largest measured, excluding the rail is the conservative choice for the ANE-versus-GPU comparisons this paper draws (online supplement, Table S28).
| Operating point | DRAM W (4 runs) | Package W (4 runs) | DRAM % of package |
| idle | 0.038 / 0.076 / 0.771 / 0.076 | 0.34 / 0.80 / 5.22 / 0.98 | 7.7–14.8 |
| ANE conv stack | 0.238 / 0.360 / 0.307 / 0.256 | 5.39 / 6.77 / 7.03 / 6.33 | 4.1–5.3 |
| ANE attention | 0.811 / 1.127 / 0.868 / 0.790 | 8.16 / 12.27 / 8.69 / 8.45 | 9.2–10.0 |
| ANE weight stream | 0.934 / 1.061 / 1.024 / 0.893 | 8.34 / 9.33 / 9.63 / 8.48 | 10.5–11.4 |
| GPU GEMM (saturating) | 6.719 / 7.112 / 6.490 / 6.646 | 53.18 / 54.99 / 51.82 / 52.87 | 12.5–12.9 |
| CPU GEMM | 0.406 / 0.657 / 1.170 / 0.393 | 16.68 / 18.29 / 19.26 / 16.46 | 2.4–6.1 |
Table S28. The DRAM rail, measured on the IOReport Energy Model DRAM channel across four independent runs of the estimator cross-validation (M5 Pro; idle-subtracted active power). Package here is the ANE+GPU+CPU total that powermetrics reports as Combined Power, which excludes DRAM, so the last column is the fraction the main metric omits.
Derived metrics: single-window. Perf/watt and mJ/inference (active W × sustained ms/iter) are computed within a single sustained measurement, never spliced across executions. The sustained per-iteration time is the wall time of the measurement loop divided by the inferences completed inside the same window over which power is summarized. Each 6 s window holds 354 to 61 446 inferences, so the reported energy integrates completed work within one execution, and does not combine statistics from separate runs. At sub-millisecond timings that time exceeds the separate minimum-latency probe on every ANE path, and perf/watt therefore cannot be reconstructed from the latency column, by design. Across Table 5 the two statistics differ by a median factor of 1.05, and by more than 25 % only at the smallest shapes, where the gap measures per-call dispatch overhead. The online supplement (Table S21) reports both per class, with the sensitivity of each energy figure to the choice of power and latency statistic; both choices are conservative for the ANE.
Table 5. Single-stream device map (M5 Pro, 6 s power window, idle package = 1196 mW); metrics defined in § 2, with CV% in parentheses in the W columns; fp16 relative error per class is in the online supplement (Table S20). Best speed and perf/W per row in bold; GF/W is GFLOP/s per watt, the norm rows elements/s per watt. Shapes: GEMM ; attention sequence length (, ); conv rows are a single 16 64 layer and a stack of one 64 256 then fifteen 256 256 layers (18.4 GFLOP); stencil counts the dense 3 3 conv (a strict 5-point count scales every device by 5/9, leaving ratios unchanged). †The stencil GPU column is an optimized GPU stencil (shifted-slice arithmetic under mx.compile), not the dense 3 3 convolution the ANE runs, and both columns are means; the DFT row’s single-session ordering is not a verdict (online supplement, formulation-fairness appendix). ∗Replicated over independent cold starts: the stencil (), group_norm and the DFT (, intervals in Figure 5), and the conv stack, ViT attention, and GEMM-compute rows (online supplement, Table S17). The seven unmarked rows are single-session.
| Class | Workload | ANE lat | GPU lat | ANE W (CV) | GPU W (CV) | ANE perf/W | GPU perf/W |
| GEMM floor | 0.092 | 0.124 | 2.93 (11 %) | 6.54 (5 %) | 15.7 GF/W | 7.8 GF/W | |
| GEMM bandwidth | 0.086 | 0.187 | 9.22 (7 %) | 9.40 (6 %) | 274 GF/W | 160 GF/W | |
| GEMM compute∗ | 1.293 | 0.461 | 8.71 (4 %) | 33.05 (8 %) | 751 GF/W | 499 GF/W | |
| conv | single 3×3 (16 → 64) | 0.103 | 0.173 | 5.99 (12 %) | 7.61 (3 %) | 24.5 GF/W | 14.7 GF/W |
| conv∗ | ResNet 3×3×16 (64 → 256) | 0.757 | 3.159 | 10.12 (6 %) | 32.70 (2 %) | 2289 GF/W | 175 GF/W |
| attention∗ | ViT block | 0.230 | 0.269 | 8.50 (17 %) | 16.01 (11 %) | 522 GF/W | 197 GF/W |
| attention | long-seq | 0.817 | 0.390 | 6.89 (7 %) | 23.22 (11 %) | 561 GF/W | 305 GF/W |
| norm | layer_norm (197,768) | 0.195 | 0.176 | 5.76 (12 %) | 9.03 (6 %) | 1.25e8/W | 9.24e7/W |
| norm | rms_norm (197,768) | 0.373 | 0.151 | 3.44 (6 %) | 7.75 (6 %) | 1.13e8/W | 1.12e8/W |
| norm∗ | group_norm (1,256,64²) | 0.350 | 0.196 | 7.89 (7 %) | 10.82 (9 %) | 3.63e8/W | 3.85e8/W |
| sci∗ | 5-pt stencil 256, 32 steps† | 0.137 | 0.382 | 7.54 (14 %) | 11.58 (1 %) | 37.1 GF/W | 7.86 GF/W |
| sci | Jacobi , 25 iters | 0.170 | 0.285 | 5.24 (13 %) | 9.98 (6 %) | 14.5 GF/W | 3.98 GF/W |
| sci∗ | DFT-as-matmul | 0.176 | 0.185 | 4.06 (25 %) | 8.24 (5 %) | 2.45 GF/W | 2.99 GF/W |
Table 5.
Table S21. The two latency statistics per device-map class (M5 Pro). min is the minimum-latency probe reported in the main paper’s device map. sust. is the sustained per-iteration time measured inside the power window and used for every perf/watt and mJ/inference figure. The ratio compares the two. It exceeds 1.25 only at the two smallest ANE shapes and at the DFT, and stays at 1.01 to 1.02 on the three real models. Substituting the mean package power for the median across these points changes the reported energies by a median of 0.5 % (range −5.9–12.8 %, under 5 % at 76 % of points) and reverses no device comparison.
| Class (workload) | ANE min (ms) | ANE sust. (ms) | ANE ratio | GPU min (ms) | GPU sust. (ms) | GPU ratio |
| GEMM floor () | 0.092 | 0.182 | 1.99 | 0.124 | 0.165 | 1.33 |
| GEMM bandwidth () | 0.086 | 0.106 | 1.24 | 0.187 | 0.179 | 0.95 |
| GEMM compute () | 1.293 | 1.314 | 1.02 | 0.461 | 0.521 | 1.13 |
| conv single | 0.103 | 0.129 | 1.25 | 0.173 | 0.168 | 0.97 |
| conv ResNet-like | 0.757 | 0.795 | 1.05 | 3.159 | 3.215 | 1.02 |
| attention ViT () | 0.230 | 0.236 | 1.03 | 0.269 | 0.333 | 1.24 |
| attention long-seq () | 0.817 | 0.833 | 1.02 | 0.390 | 0.455 | 1.17 |
| layer_norm (197,768) | 0.195 | 0.210 | 1.08 | 0.176 | 0.181 | 1.03 |
| rms_norm (197,768) | 0.373 | 0.391 | 1.05 | 0.151 | 0.175 | 1.15 |
| group_norm (1,256,) | 0.350 | 0.366 | 1.05 | 0.196 | 0.251 | 1.28 |
| DFT-as-matmul () | 0.176 | 0.422 | 2.39 | 0.185 | 0.170 | 0.92 |
| 5-pt stencil (, 32 steps) | 0.128 | 0.134 | 1.05 | 1.256 | 1.327 | 1.06 |
| Jacobi (, 25 iters) | 0.170 | 0.173 | 1.02 | 0.285 | 0.331 | 1.16 |
| ResNet-18 forward | 0.330 | 0.337 | 1.02 | – | – | – |
| MiniLM encoder | 0.525 | 0.536 | 1.02 | – | – | – |
| ViT-B/16 forward | 18.336 | 18.450 | 1.01 | – | – | – |
Table S21.
Repeated measurements. Some results are replicated over independent cold starts, each a fresh process with its own idle baseline and randomized device order, and carry a 95 % confidence interval. These are the ANE and fp16 torch-MPS energies and ratios in Table 6, the six device-map classes marked in Table 5, and the ceiling and serving comparisons and crossovers of Table S17 in the online supplement. The compiled-MLX columns of Table 6 are repetitions within one session, also with 95 % intervals. The other seven device-map classes, the fp32 GPU energy column, the serving sweeps apart from their crossovers, the decode sweeps, and the cross-generation maps are each a single sustained measurement with a within-window CV and no across-run interval. Under repetition the DFT ratio moved from its single-session 0.82× to 1.35×, with an interval spanning parity (§ 6).
| Model | Latency (ms) | Energy (mJ/inf) | ANE energy advantage | |||||||
| ANE | MPS | MLX-c | ANE relerr | ANE | MPS fp32† | MPS | MLX-c | adv. MPS | adv. MLX-c | |
| ResNet-18 | 0.330 | 2.033 | 1.645 | 2.3e-3 | 2.30 | 39.1 | 21.3 | 32.6 | 8.9× [7.5, 10.7] | 14.6× [12.1, 17.2] |
| MiniLM (1 sent) | 0.525 | 1.920 | 0.650 | 5.5e-3 | 2.74 | 28.8 | 17.6 | 11.5 | 6.7× [5.9, 7.6] | 3.4× [3.0, 3.9] |
| ViT-B/16 | 18.34 | 15.92 | 2.959 | 7.8e-3 | 65.7 | 1382.7 | 577.9 | 100.4 | 9.4× [7.9, 11.1] | 1.5× [1.3, 1.8] |
Table 6. Real models (latency ms; energy mJ/inference; all GPU arms fp16 unless marked). MPS is torch-MPS (.half()) and MLX-c is graph-compiled MLX (mx.compile); MiniLM and ViT-B/16 run as MLX ports loading the same weights (online supplement, formulation-fairness appendix). The ANE and MPS energies are means over cold starts, and adv. MPS is their geometric-mean ratio with a 95% interval, whose width is set mostly by the GPU arm (online supplement, Table S6). The fp32 MPS column† is single-session and carries no interval. The MLX-c energies and adv. MLX-c are paired repetitions interleaved with the ANE within one session ( for ResNet-18, for the others), with a 95% -interval on the mean ratio. Each adv. MLX-c ratio pairs a compiled-MLX run with an ANE run from the same session, so it is not the quotient of the printed cold-start ANE energy (the in-session ANE energies are in the online supplement, formulation-fairness appendix). Latency is the device-map probe, except MLX-c, which is the sustained time of its energy window.
Accuracy. Every workload reports relative L2 error against an fp32/fp64 numpy reference, printed beside the speed and watt numbers so that a speed gain at worse accuracy is visible. The GPU is consistently more accurate in fp16 than the ANE, most so on large GEMMs. This error is computed on a synthetic input and is not a uniform proxy for real data: on their own domains ResNet-18 falls to , MiniLM holds at , and ViT-B/16 rises to . Compiling the ViT encoder to each depth localizes that gap. Error grows by 7–9 % per block for both inputs, and their difference appears only at the classification head, where a peaked logit vector carries larger absolute rounding over a smaller norm. Relative L2 also does not say whether a divergence changes an answer, so all three real models are scored on their task. ResNet-18 reaches 69.76 % top-1 on the ANE, the GPU, and the fp32 CPU reference alike over all 50 000 ILSVRC2012 validation images. ViT-B/16 reaches 81.00 % against the reference’s 81.06 %. Both reproduce the published torchvision accuracies. Neither ANE path shifts accuracy: its label changes run in both directions (ResNet 14 against 14, ViT 33 against 21, McNemar and 0.13), though on ViT it changes ten times as many individual labels as the GPU does. MiniLM has no label set, so the decision measured there is which neighbor an embedding retrieves: the ANE returns the fp32 nearest neighbor for 99.2 % of queries and reproduces the pairwise similarity ranking exactly to four decimals. Weight compression produces the measurable task-level losses: per-channel int8 costs 0.11 percentage points of ResNet-18 top-1 and two points of MiniLM retrieval agreement, and int4 costs 2.55 points (online supplement, Tables S25 and S26).
Exclusions. Sixteen subprocess-bridge operators (sort, cost-volume, farthest-point sampling, radius search, and others) run at 25 ms to 2.5 s because each call spawns a one-shot probe process and pays the full process-spawn, graph-author, and ANE-load cost every call, a dispatch artifact, not silicon speed. Timing them against MLX would misrepresent the hardware, so they are excluded from the speed comparison and treated as a capability statement instead. The sixteen are listed in the online supplement, and the three bridge operators that carry a persistent worker (sdpa, argmax, topk) are timed on silicon and remain in the comparison.
Compute and bandwidth ceilings
The corrected roofline uses measured ceilings for each device. A representative shape need not saturate the device: a 512 × 512 GEMM reaches only about 4 % of the GPU’s fp16 peak and therefore understates its capability.
Compute roof
The compute-saturation sweep scales two compute-bound primitives (square GEMM, FLOPs, and a 3 × 3 same-pad convolution, the ANE’s highest-throughput primitive) until each device plateaus, measuring throughput, idle-subtracted active package power, and fp16 relative error at every size. Table 1 gives the peaks.
| Primitive | Device | Dtype | Peak GFLOP/s | @ size | relerr there | Peak GFLOP/s/W |
| GEMM | GPU | fp16 | 30 862 | 3.6e-4 | 603 () | |
| GEMM | ANE | fp16 | 10 191 | 1.2e-2 | 866 () | |
| conv | GPU | fp16 | 31 790 | 7.4e-3 | 1 392 () | |
| conv | ANE | fp16 | 18 771 | 5.5e-4 | 1 372 () |
Table 1. Compute and efficiency peaks (M5 Pro), GPU and ANE in fp16.
The GPU has the higher raw throughput on both primitives compared to the other devices; GEMM 3.0× the ANE (30.9 versus 10.2 TFLOP/s) and conv 1.7× the ANE (31.8 versus 18.8 TFLOP/s), confirming that the large square GEMM of large-language-model (LLM) inference is appropriate for the same SoC’s GPU. Perf/watt divides between the primitives. At the ANE is slower but draws 9.5 W against the GPU’s 53.3 W, so its efficiency at that size is the GPU’s (866 against 558 GFLOP/s/W). Peak against peak, the ANE’s efficiency is higher (866 at against the GPU’s 603 at ). Cold-start replication of the device map’s GEMM-compute shape (, ) gives (811 against 558 GFLOP/s/W, online supplement, Table S17); this is the most variable contested claim, but it remains Holm-significant. For conv the single-sweep gap is small (ANE 1372 against GPU 1392 GFLOP/s/W) and replication resolves it to a 5 % GPU edge. This bounds the efficiency claim: it holds for GEMM at peak and reverses to a small GPU edge at conv saturation. The utilization shows in the power curves: the GPU’s rises from 9 W at and plateaus at 49–53 W for , while the ANE’s peaks at 12.7 W at and then falls with its throughput.
| Claim (perf/W) | ANE mean ± SD | GPU mean ± SD | Diff [95 % CI] | (Holm) | Verdict |
| conv @ (GF/W) | 1352 ± 49 | 1427 ± 38 | −74 [−94, −55] | < 0.001 | GPU diff. (~5 %) |
| GEMM @ serving (items/s/W) | 2075 ± 34 | 1945 ± 5 | +130 [+87, +174] | 0.003 | ANE difference |
| conv ResNet-like (GF/W) | 2253 ± 77 | 172 ± 2 | +2081 [+1988, +2173] | < 0.001 | ANE difference |
| attention (GF/W) | 517 ± 12 | 190 ± 4 | +326 [+316, +337] | < 0.001 | ANE difference |
| GEMM-compute (GF/W) | 811 ± 88 | 558 ± 15 | +253 [+194, +312] | < 0.001 | ANE difference |
| Attention crossover | 2.23 ± 0.02 | – | CI [2.20, 2.25] | – | Stable (est. only) |
| Encoder crossover | 31.1 ± 1.3 | – | CI [29.5, 32.7] | – | Stable (est. only) |
Table S17. Per-claim replication of the contested verdicts of the main paper’s device-map, ceiling, and serving sections. Each claim is measured over independent cold-start runs (idle re-sampled and outlier-gated per run, workload and device order randomized, thermal pressure Nominal throughout); the two compute-bound rows are measured in isolation under inter-run cooldown so no workload pre-heats the die. perf/W is idle-subtracted; values are mean ± across-run SD; Diff is the paired ANE–GPU mean with a paired- 95 % CI; is two-sided paired , Holm-adjusted across the difference tests. A difference requires the CI to exclude 0 at adjusted ; equivalent requires two one-sided tests within the ±5 % margin at ; crossover rows are estimation-only.
The second roofline correction is determined via the throughput curve: the ANE’s compute roof falls instead of plateauing. ANE GEMM peaks at (10.2 TFLOP/s), then degrades to 8.3 TFLOP/s at , 4.8 for , and 4.9 for , a 52 % drop that reproduces across runs (Figure 1), remaining at roughly half its peak for . The active power falls with the throughput (12.7 W to 6 W), so the engine is stalled, not busy at a reduced rate. Once a square operand exceeds the ANE’s working-set window (about to 4096), on-chip tiling and weight streaming no longer feed the multiply–accumulate (MAC) array fully. The computation then serializes on off-chip streaming. The same peak-then-decline shape appears on an M2 Pro (h14), peaking at 7.5 TFLOP/s at and falling to 2.2 at N = 8192, so it is a property of the engine, not of one part. Accuracy degrades over the same range: relative error grows from $2.6 \times 10^{-3} at to at . Large- ANE GEMM therefore loses both throughput and accuracy. That growth is consistent with per-product fp16 rounding over a long contraction, not a narrow accumulator: the MAC array carries a wide, fp32-class accumulator, and the error tracks as independent rounding errors accumulate.
Figure 1. Square-matmul throughput versus dimension on the M5 Pro (ANE fp16, GPU MLX fp16). The ANE roof peaks then falls; the GPU’s rises and plateaus.
The falloff is specific to the large square single-matmul shape: ANE conv throughput rises monotonically to its 18.8 TFLOP/s peak with relative error constant at , because conv’s natural tiling keeps the array fed. This is why conv, not GEMM, is the ANE’s highest-throughput primitive. A check robust to the falloff isolates the raw ceiling: deepening a fused chain of small matmuls grows compute at fixed I/O, and its slope reaches 19.6 TFLOP/s against the 10.2 single-matmul peak, so the gap is utilization, not a silicon limit.
Bandwidth roof
The compute-bound primitives (matmul, conv, and their variants) are the minority of aneforge’s 50 fused operators. The rest (activations, elementwise binaries, reductions, softmax, the norm family, pooling, and data-movement ops) are memory-bandwidth-bound at size. We scale each class from 256K to 64M elements and read its saturation plateau; the rise toward that plateau identifies the dispatch-bound small sizes. Table 2 shows the ceilings.
| Archetype (bytes) | ANE GB/s | GPU GB/s | ANE GB/s/W | GPU GB/s/W |
| streaming relu / (2N) | 24.1 | 229.7 | 1.99 | 27.13 |
| reduction sum (1N) | 22.4 | 193.0 | 2.24 | 27.28 |
| softmax (2N) | 15.1 | 238.4 | 1.78 | 18.54 |
| layer_norm (2N) | 4.4 | 41.9 | 0.90 | 2.43 |
| gelu light-compute (2N) | 23.2 | 48.6 | 2.03 | 3.70 |
Table 2. Bandwidth ceilings (M5 Pro), GPU and ANE in fp16.
The streaming row sets the measured GPU ceiling at 230 GB/s, with softmax reaching 238. This single-program rate is about 75 % of the part’s 307 GB/s unified-memory specification and is the appropriate ceiling for a measured roofline. The ANE’s achieved bandwidth through aneforge’s single-fused-program path is flat at 22–24 GB/s across streaming and reduction, reached already at the smallest sizes measured, about 10× below the memory ceiling the GPU attains. That flat-from-small signature is the finding: on pure elementwise and reduction graphs the ANE is bound by its program-streaming and dispatch path, not by the memory system, so the 24 GB/s is a path ceiling and not a silicon-memory one. The GPU therefore has both the higher raw bandwidth (230 against 24) and the higher bandwidth-per-watt on every archetype (27.1 against 1.99 GB/(s W) on streaming). The ANE’s usual perf/watt advantage does not survive here, because it came from doing useful compute at low power and a pure memory stream is the one regime with no compute to do. The consequence is that standalone large softmax, elementwise, and reduction ops belong on the GPU; they are cheap only when fused into a larger compiled program.
Weight-stream bandwidth (GEMV-K)
This measurement establishes the first roofline correction: the ANE has two effective bandwidths, not one. The streaming roofline of § 3.2 measures the standalone-activation path. A decode GEMV (matrix–vector multiply, ) exercises a different path: it streams a weight matrix once and does negligible compute (arithmetic intensity ), so its effective bandwidth is that of the path the ANE uses to stream weights, not of its standalone-activation path. A single GEMV at gives an effective 112 GB/s, well above the 24 GB/s activation ceiling of § 3.2. To resolve whether this is anomalous or a second bandwidth tier, we sweep and read effective GB/s = (weight + I/O bytes, fp16) / minimum latency (the per-archetype byte models are in the online supplement, Table S16). Figure 2 reports sweep (the per- values are tabulated in the online supplement, Table S3); the same rise-to-plateau standard we applied to the 24 GB/s figure applies here.
Figure 2. Weight-stream effective bandwidth versus weight dimension for a decode GEMV () on the M5 Pro. The ANE’s bandwidth exceeds the GPU’s up to (76 GB/s versus 32 GB/s) as it amortizes its dispatch floor, then plateaus near 130–143 GB/s; the GPU’s unified memory is higher from and reaches 221 GB/s.
| ANE GB/s | GPU GB/s | CPU GB/s* | ANE GB/s/W | GPU GB/s/W | |
| 512 | 6.1 | 3.7 | 277.6 | 0.8 | 0.4 |
| 1024 | 28.9 | 11.0 | 401.8 | 9.2 | 1.1 |
| 2048 | 75.6 | 32.4 | 63.6 | 13.0 | 4.9 |
| 4096 | 113.5 | 129.3 | 52.9 | 35.9 | 18.0 |
| 6144 | 129.8 | 169.1 | 62.7 | 53.1 | 21.1 |
| 8192 | 142.6 | 221.4 | 61.0 | 60.8 | 41.9 |
Table S3. Weight-stream effective bandwidth and energy, GEMV (), , fp16 (CPU *fp32). GB/s is effective weight and I/O bytes divided by minimum latency. GB/s/W is GB/s divided by idle-subtracted active package power. The ANE reaches a 130–143 GB/s plateau, and the GPU’s unified-memory bandwidth is higher from on, but the ANE moves bytes at higher energy efficiency at every .
The ANE’s bandwidth increases 6 29 76 GB/s at small , and so is dispatch-bound and the flat-from-small signature of § 3.2 is inverted: here the per-program overhead is amortized as the weight matrix grows (Figure 2). The ANE reaches 113.5 GB/s for , reproducing the 112 GB/s figure to within 2 %, then plateaus to 130–143 GB/s at larger . The 112 figure is thus a conservative lower bound, and the weight-stream path is 5 to 6 the 24 GB/s standalone path.
The standalone-activation path plateaus at 23 GB/s with a stable large-size ceiling (a plateau coefficient of variation of 3.3 %) and does not rise with size, so it is a true ceiling and not an under-amortized operating point of the same DMA; the weight-stream tier is 5.4 higher. Fragmenting a fixed weight volume into many small programs also lowers the achieved weight-stream bandwidth, consistent with a separate weight-stream tier. An M2 Pro (h14, a desktop Mac mini) shows the same flat low activation ceiling at 14 GB/s and a weight-stream tier 3.8 higher; an M4 (h16, a second Mac mini) shows the same flat ceiling near 10 GB/s with a weight-stream path that runs 2.4 to 6.6 above it.
Scope of the two-tier measurement. The two effective bandwidths are a property of the dispatch path as reached by aneforge, measured end to end from the host, and the measurement does not establish a dedicated weight DMA as the mechanism. In these programs the weight matrix is a compile-time constant of the compiled graph and the activation is the only runtime input, so a warm repetition reissues the same program against the same constant. The faster tier is also consistent with constant handling, on-chip tensor layout, compiler placement, or residency across repetitions. An external measurement cannot separate these from a distinct transfer engine, so we report a dedicated weight path as a candidate mechanism, not an established one. Every weight point exceeds the activation ceiling by at least 2.4, so the two-tier separation holds across sizes; the online supplement notes one h16-only weight-path stride-aliasing dip at exact powers of two, which does not affect the separation. The activation path’s saturation at 14 GB/s on a part with roughly 200 GB/s of unified-memory bandwidth indicates the ceiling is a dispatch-path limit, not a DRAM limit. The absolute ceilings track the part, not the generation: the two Pro parts (23 and 14 GB/s) sit above the smaller base M4 (10 GB/s), even though the M4 is the newer silicon.
The ANE’s single-GEMV bandwidth exceeds the GPU’s through (29 versus 11 at , 76 versus 32 at $K=2048); from the GPU’s unified memory is higher, rising 129 221 GB/s while the ANE saturates near 143. (The CPU’s fp32 figures are highest only at the smallest sizes, where the tiny weight fits in cache and the comparison is call-overhead-bound, not bandwidth-bound; it falls to 50–60 GB/s once the matrix spills.)
Figure 3. Per-device roofline on the M5 Pro: five operator archetypes on the ANE (triangles) and the GPU (circles), placed by arithmetic intensity. The ANE shows two bandwidth slopes, a 24 GB/s standalone-activation path and a faster weight-stream path used by compiled programs; per-archetype, per-device placement is in Table 3 and Table 4.
The energy column shows the ANE streams weights at higher GB/s/W than the GPU at every , 2.0× at (35.9 versus 18.0) and still 1.45× at (60.8 versus 41.9), past the bandwidth crossover. Even where the GPU moves more bytes per second, the ANE moves them at lower energy on the exact path decode uses, which is the roofline-level reason the ANE’s decode advantage is energy (§ 9).
The weight-stream path also accepts compressed weights, decompressed during the DMA (the on-the-fly-decode pattern of EIE [13]); aneforge emits int4 lookup-table palettization and unstructured sparse forms behind one opt-in flag. On a single matmul this raises arithmetic intensity and speeds the stream (1.65 to 1.82× on large layers, online supplement Table S4), moving the int4 operating point to 104 % of the matmul roof at (Table 4). End to end, int8’s benefit is footprint and energy, not latency: the int8 weight stream is bound by the dequantizer’s element rate, not by bytes, the third roofline correction, developed in § 9 (halving the weight bytes yields energy savings on a part at its element-rate ceiling, and throughput only on a bandwidth-starved one, though a lookup-table encoding raises the ceiling itself).
| Operator archetype | AI | GPU GFLOP/s (%roof) | ANE GFLOP/s (%roof) |
| matmul | 171 | 1235 (4 %) | 1819 (18 %) |
| matmul | 683 | 22345 (72 %) | 10191 (100 %) |
| matmul | 2731 | 30527 (99 %) | 4882 (48 %) |
| conv () | 287 | 5343 (17 %) | 5885 (31 %) |
| conv () | 2020 | 31790 (100 %) | 18771 (100 %) |
| attention () | 52 | 4685 (39 %) | 598 (48 %) |
| GEMV / decode | 1.0 | 75 (33 %) | 112 (78 %)‡ |
| softmax | 1.25 | 298 (104 %) | 19 (63 %) |
| gelu | 2.5 | 122 (21 %) | 58 (96 %) |
| layer_norm | 2.5 | 105 (18 %) | 11 (18 %) |
| relu / copy | 0.25 | 57 (100 %) | 6 (100 %) |
| matmul int4 (, ) | 4 | 171 (19 %) | 165 (29 %) |
| matmul int4 (, ) | 410 | 5218 (17 %) | 10579 (104 %) |
Table 4. Operator archetypes placed on each device’s roofline: arithmetic intensity (AI, the fp16 value), achieved performance, and the fraction of the applicable roof () reached, M5 Pro. Only the high-AI matmul and conv rows exceed any ridge (compute-bound); the rest fall on the bandwidth slope. ‡The ANE GEMV is scored against the weight-path roof (78 %), not the 24 GB/s activation roof (a spurious 466 %), since a weight-streaming GEMV uses the weight path.
| Matmul (), batch | fp16 (ms) | int4 (bytes) | sparse (bytes) |
| 1024 1024, | 0.095 | 0.94 (0.25) | 0.94 (0.38) |
| 2048 2048, | 0.142 | 1.32 | 1.38 |
| 4096 4096, | 0.312 | 1.65 | 1.63 |
| 8192 4096, | 0.522 | 1.67 | 1.82 |
| 256 256, | 0.089 | 0.71 | 0.77 |
| 4096 4096, | 0.356 | 1.37 | 1.52 |
Table S4. Compressed weight-stream speedup (single matmul, ANE via aneforge; median of five 200-eval blocks, batch 1 unless noted). Speedup is fp16 median latency divided by the compressed median. The bytes column is the compressed weight-blob fraction of fp16. int4 is per-tensor LUT. Sparse is the unstructured weight at about 68 % zeros (about 0.38 bytes).
Per-device roofline synthesis
The corrected roofline turns on one inequality: the ANE’s weight-stream ridge (about 71 FLOP/B) lies below the GPU’s (134), so the path that carries every compute-bound ANE operator becomes compute-bound at a lower arithmetic intensity than the GPU does.
A roofline positions each operator by its arithmetic intensity (AI = FLOPs / bytes-moved) and predicts an achievable ceiling of min(compute_roof, bandwidth_roof × AI); the ridge point (compute_roof / bandwidth_roof) divides memory-bound (left) from compute-bound (right). Table 3 gives each device’s roofs and ridges (recomputed from the two ceiling JSONs, not re-measured); Table 4 then tabulates every measured archetype by its arithmetic intensity, reporting its achieved rate and the fraction of the applicable roof it reaches on each device. Only the high-AI matmul and conv archetypes exceed any device’s ridge (compute-bound); everything below falls on the bandwidth slope.
| Device | Dtype | Compute roof (GFLOP/s) | Bandwidth roof (GB/s) | Ridge (FLOP/byte) |
| GPU | fp16 | gemm 30862 / conv 31790 | 229.7 | 134.4 |
| ANE (activation path) | fp16 | gemm 10191 / conv 18771 | 24.1 | 423.6 |
| ANE (weight-stream path) | fp16 | gemm 10191 / conv 18771 | ∼143 | ∼71 |
Table 3. Per-device roof and ridge (recomputed from the saturation and bandwidth JSONs). The ANE has two bandwidths (§ 3.2 and § 3.3) and therefore two ridges: a standalone-activation path near 24 GB/s and a weight-stream path near 143 GB/s.
On its standalone-activation path the ridge is about 424 FLOP/B (3.2× the GPU’s 134), a consequence of that path’s bandwidth being only 24 GB/s against 10 TFLOP/s of compute. This is why dispatching an elementwise, norm, or softmax op one at a time underutilizes the engine: such ops lie far to the left of 424 and never approach compute-bound.
Compute-bound work (GEMM, conv, GEMV, linear, anything that streams weights) instead uses the weight-stream path (§ 3.3), whose ridge is only about 71 FLOP/B, below the GPU’s 134. On the path that carries the ANE’s compute-bound work, an operator becomes compute-bound at a lower arithmetic intensity than on the GPU, not higher. This marks where the ANE’s energy advantage begins, not which device is faster: throughput follows the lower, falling compute roof (§ 3.1), so on large square GEMM the GPU is the faster device while the ANE still draws far less power, whereas on conv and the stencil the ANE is faster and lower-power. A single ridge at 424 conflates the activation and weight paths and places the weight-streaming points above the roof. With both ridges, the GEMV reaches 78 % of its applicable weight-path roof instead of the 466 % the activation ridge implies. The CPU (fp32 via Accelerate/AMX) is a contextual reference whose full figures are in the online supplement (Table S18); the like-for-like comparison is GPU versus ANE, which run identical fp16 math.
Table S18. CPU (fp32, Accelerate/AMX) reference figures removed from the main paper’s body tables, M5 Pro. Compute peaks (conv im2col), bandwidth ceilings, and end-to-end decode throughput, as a contextual fp32 reference. The CPU is not directly comparable to the fp16 GPU/ANE columns (its AI is the fp32 value, half the fp16 AI).
| Quantity | CPU value | Note |
| GEMM peak GFLOP/s | 1 899 (), relerr 8.1e-7 | 112 GFLOP/s/W () |
| conv peak GFLOP/s | 11 (im2col) | 1.05 GFLOP/s/W |
| streaming GB/s | 130.4 | 9.04 GB/s/W |
| reduction sum GB/s | 41.7 | 2.95 GB/s/W |
| softmax GB/s | 6.1 | 0.51 GB/s/W |
| layer_norm GB/s | 10.9 | 0.76 GB/s/W |
| gelu light-compute GB/s | 175.8 | 10.41 GB/s/W |
| decode tok/s () | 11 / 16 / 32 / 60 / 112 / 194 | fp32 reference; relerr 6e-6 |
Table S18.
The fused 3-conv block at about 110 % of the single-conv roof exceeds a standalone-dispatch ceiling by amortizing per-dispatch overhead and I/O (the fusion discussion below). The GPU softmax at 104 % is FLOP-convention sensitivity (the flat per-element transcendental allowance of the online supplement).
The ANE roofline carries two caveats. It is not a single clean line: its compute roof differs by operator (conv about 18.8 versus square-matmul about 10.2 TFLOP/s, and matmul falls off past to about 4.9 TFLOP/s at ), so we draw both roofs. The 24 GB/s bandwidth and the ridge near 424 FLOP/B are also a path operating point of aneforge’s single-program streaming, not a silicon DMA limit (§ 3.2); the weight-stream path reaches 5 to 6× higher at its plateau (§ 3.3). On the figure we draw the weight-stream line at the conservative 112 GB/s operating point corroborated at (ridge about 91 FLOP/B); the measured plateau reaches about 143 GB/s, which gives the ridge of about 71 used in Table 3, and the regime conclusions are identical under either convention.
The archetype arithmetic intensities (fp16) span four orders of magnitude, from 0.25 for elementwise ops to the thousands for conv, and scoring achieved throughput against roof@AI sorts them into three regimes. Large matmul and conv are the only compute-bound archetypes, so they are the only classes that approach the compute roof. Convolution gives the ANE’s largest measured advantage, about 18.8 TFLOP/s and 1370 GFLOP/s/W. GEMV and decode (AI ≈ 1.0) lie below every ridge and are memory-bound on both devices, so decode throughput follows whichever device streams weights fastest. The operator-level GEMV result corroborates the second bandwidth tier but does not bound a fused token loop; § 9 measures that loop end to end. Standalone elementwise, norm, and softmax ops reach only a fraction of even the ANE’s low bandwidth roof (layer_norm 18 %, softmax 63 %), since its ridge near 424 demands about 3× the AI of the GPU to become compute-bound.
Fusion acts through two separable mechanisms. One is arithmetic intensity: fusing operators into one compiled program keeps intermediates on-chip and streams weights once, raising effective AI (a standalone conv at about 1076 rises to about 2854 in a fused 3-stack) and moving low-AI points rightward past the ridge, so elementwise, norm and softmax ops run at negligible marginal cost inside a fused program instead of standalone at low utilization. The other is overhead amortization: a fused graph of any depth lowers to a single AneInference dispatch, so depth amortizes the per-program floor and fixed I/O setup that a standalone dispatch pays in full, while each bridge-induced graph cut becomes a separate dispatch.
The three- and sixteen-conv stacks reach 110 % and 124 % of the single-conv saturation rate even though both lie beyond every ridge, where raising AI predicts no further gain. These values reflect amortized dispatch and I/O cost, not a violated hardware bound. We therefore call the measured single-dispatch curve an empirical operating envelope, not a hardware upper bound: it is the highest rate one dispatch reaches at that shape, carrying the per-dispatch cost and DRAM I/O that a fused program amortizes, and the fused-chain slope of § 3.1 says the same for GEMM. An entry above 100 % in Table 4 therefore measures amortization, not a violated bound.
Fusion also produces the gain the roofline does predict from arithmetic intensity. Fusing a memory-bound neck block (a 1×1 convolution, group norm, and two gelus) moves its arithmetic intensity from 32.7–52.6 below the weight ridge to 119.9–131.2 above it. Throughput rises by 3.7× at and 2.3× at .
A fused conv or attention graph is efficient on the ANE; the identical ops dispatched standalone are not.
An energy roofline and a predictor
The time roofline above bounds throughput; the energy roofline of Choi et al. [10] bounds energy per operation, , with an energy balance point that, like the time ridge, divides a memory-dominated from a compute-dominated regime but in joules, seconds. Because every operating point in this paper already carries active package power, the two constants are fitted directly from the ceiling sweeps with no new measurement: is the inverse of each device’s peak GFLOP/s/W and the inverse of its GB/s/W streaming plateau (Table 1 and Table 2). Figure 4 draws the result for all three engines, with the ANE split across its two byte paths.
Figure 4. Energy roofline (energy per FLOP versus arithmetic intensity) on the M5 Pro, fitted from the same active-package-power measurements as the perf/watt tables. Solid green is the ANE weight-stream path, dashed green the standalone-activation path; open circles mark each path’s energy balance point and gray droplines the corresponding time ridge .
The fitted constants explain the ANE’s efficiency: a compute floor of 1.16 pJ/FLOP (GPU 1.66) and the cheapest byte in the system, 16.5 pJ/B on the weight path (GPU 36.3), while its standalone-activation byte is the most expensive at 503 pJ/B, 96 % of it host dispatch. The per-watt advantage thus comes from cheaper FLOPs, much cheaper weight bytes, and fusion keeping activation bytes off DRAM, not from the activation path. The energy balance point lies left of the time ridge on both fast paths ( versus on the GPU, 14 versus 71 on the ANE weight path), Choi’s regime where raising arithmetic intensity yields speed but no further energy savings; only the ANE activation path has the two coincident (435, 424).
The two per-device constants also make the energy roofline predictive. Given a workload’s arithmetic intensity and dispatch path, estimates its energy per FLOP on device , and comparing against identifies the lower-energy engine. The dispatch path is a second model input and cannot be inferred from arithmetic intensity. An operator compiled into the fused graph reaches the weight-stream path at 16.5 pJ/B; the same operator issued standalone reaches the activation path at 503 pJ/B. The predictor takes that dispatch state as given, and does not derive it. The device-pair energy crossover is . On the ANE weight-stream path the ANE has both the lower compute floor and the cheaper byte, so is negative: the ANE is the lower-energy device at every arithmetic intensity, with no crossover. The GPU has no separate weight path, so it carries a single streaming byte (36.3 pJ/B); that byte reproduces its measured weight-streaming energy (online supplement, Table S19), so the crossover describes each device’s energy byte, not mismatched paths. On the standalone-activation path the expensive 503 pJ/B moves the crossover up to , above which only fusing the op back onto the weight path keeps the ANE ahead. The predictor therefore has one lever: keep FLOP-denominated work on the weight-stream path (compiled and fused), where the ANE is the lower-energy device unconditionally, while a standalone activation-path op below AI 933 belongs on the GPU.
| Workload | regime | AI | ANE pJ/FLOP pred / meas (err) | GPU pJ/FLOP pred / meas (err) |
| near the roof (bound tight) | ||||
| GEMM compute | compute-bound | 227.6 | 1.23 / 1.33 (−7%) | 1.82 / 2.00 (−9%) |
| Attention long-seq | near-roof | 52.2 | 1.48 / 1.78 (−17%) | 2.36 / 3.28 (−28%) |
| Decode loop | weight-roof | 1.0 | 17.7 / 22.9 (−23%) | 38.0 / 42.5 (−11%) |
| away from the roof (one-sided bound, loose) | ||||
| GEMM bandwidth | bandwidth | 102.4 | 1.32 / 3.65 (−64%) | 2.01 / 6.25 (−68%) |
| conv single (16 → 64) | dispatch-floor | 224.8 | 1.23 / 40.8 (−97%) | 1.82 / 67.8 (−97%) |
| GEMM floor | dispatch-floor | 42.7 | 1.55 / 63.6 (−98%) | 2.51 / 128.7 (−98%) |
Table S19. Held-out predictor validation across the device-map classes (M5 Pro). Predicted energy per FLOP from the ceiling-fitted constants alone (ANE weight path , ; GPU 1.66, 36.3), against the measured value 1000/(perf/W) from the main paper’s device map (decode from the main paper’s decode table). The predictor is a one-sided lower bound: tight near the roof, loose in the dispatch-bound regimes where energy is dominated by the fixed per-call cost the roofline omits. The GPU constant (36.3 pJ/B) is its streaming-roof byte; the GPU has no separate weight path, and the device-pair crossover stays negative, so the ANE remains the lower-energy device, even under the GPU’s own cheaper weight-stream byte (about 24 pJ/B). Constants are fitted on the M5 Pro.
We test the predictor on workloads it was not fitted to, since the constants come only from the ceiling sweeps of § 3.1 and § 3.2. The predictor makes two separable claims, on placement and on magnitude. On the weight path the crossover is negative, so the model makes one prediction, the ANE, for every weight-streaming workload; the device-map archetypes and the decode loop bear it out (Table 5 and Table 8), but this is a single sign, not a test per workload. The discriminating predictions are on the activation path, where the AI-933 crossover sends a standalone low-intensity op to the GPU. The standalone group norm (AI 2.5) is GPU-favored in Table 5, as predicted, and the below-ridge fusion block switches from the GPU when standalone (AI 32.7 to 52.6) to the ANE when fused (AI 119.9 to 131.2), the measured 3.7× and 2.3× gains confirming the ANE side.
| ANE tok/s | int8 tok/s | GPU tok/s | ANE-fp16 tok/s/W | ANE-int8 tok/s/W | GPU tok/s/W | |
| 1 | 115 | 123 | 200 | 35.9 | 44.7 | 19.3 |
| 2 | 204 | 204 | 231 | 50.6 | 68.7 | 9.9 |
| 4 | 341 | 342 | 431 | 68.6 | 64.8 | 18.5 |
| 8 | 517 | 515 | 834 | 88.2 | 100.3 | 36.9 |
| 16 | 704 | 698 | 1444 | 106.2 | 113.7 | 65.1 |
| 32 | 847 | 850 | 2322 | 88.0 | 105.4 | 116.4 |
Table 8. End-to-end LLM decode, batch sweep (M5 Pro; 8-layer forward per token at seq pos 1; KV-len 256; ANE attention = bmm+softmax; idle pkg ≈ 1480 mW). Throughput (tok/s) and idle-subtracted total-package efficiency (tok/s/W) for the ANE (fp16 and per-channel int8 weight-stream) and GPU (MLX fp16). The fp32 relative-error reference is numpy/Accelerate. Best throughput and best efficiency per batch in bold. fp16-versus-fp32 relative error ≈ (ANE fp16), (ANE int8), (GPU). The GPU’s efficiency dip is a power-state step (the GPU rail pins at 19.9 W for all before batching amortizes it).
The magnitude claim follows from the roofline being a lower bound on energy per FLOP for single-dispatch work: the predicted joules are accurate near the roof and a one-sided bound elsewhere. Tested against the held-out device-map classes (online supplement, Table S19), the predicted energy per FLOP is below measurement on every single-dispatch class, as a lower bound requires. Near the roof it lands within 7–28 % (median 14 % across both devices, on compute-bound GEMM, long-sequence attention, and the decode loop). Away from the roof it under-predicts the dispatch-bound rows by large factors, whose package energy is dominated by the fixed per-call cost the roofline omits. The bound’s domain excludes dispatch-amortized graphs: the fused 16-deep conv stack runs at 0.44 pJ/FLOP, beneath the fitted floor, because one dispatch covers sixteen layers (§ 4).
A dispatch term. The gap away from the roof is a fixed per-call cost, so we model it. Writing the roofline per call as with , and adding one constant,
where , , and keep the values already fitted from the ceiling sweeps, so each equation gains exactly one parameter. Across the held-out classes the work spans a factor of 1024 while the roofline’s residual spans only 2.8, from 354–987 µJ on the ANE. The omitted cost therefore does not scale appreciably with work. The fitted ANE constants are µJ and µs per call; the GPU values are 1771 µJ and 67 µs. Adding them cuts the median absolute error on energy per FLOP from 56.6 % to 13.7 %. Scoring each class with the constant refitted without it gives 16.5 %. The dispatch-floor classes carry the improvement: the smallest GEMM moves from −97.6 % to 30.1 % and the single conv from −97.0 % to −8.4 %, while the near-roof classes stay within 5 %.
An independent measurement confirms that the fitted is a dispatch floor, not a free parameter absorbing error. The per-program floor of about 70 µs (§ 9) comes from the minimum-latency probe, one dispatch under the most favorable conditions. Equation (1) is instead fitted to sustained execution, whose per-call cost is larger by the measured sustained-to-probe ratio of 1.99 at that shape. The two agree to 2.5 %: 70 × 1.99 = 139 µs expected against 136 µs fitted. We report the GPU’s term as fitted only. Its independently measured 184 µs intercept comes from the stencil step-count sweep, a compiled 32-step program launch, not a single-operator dispatch, and therefore measures a different quantity.
The corrected roofline is thus a placement rule: from a workload’s arithmetic intensity and its dispatch path, assign it to the lower-energy engine and fuse to keep it on the weight-stream path. Two of the three inputs reproduce across generations: the compute floor stays near 1 pJ/FLOP (1.16 on the M5 Pro, 0.96 on the M4, 1.3 on the M2 Pro) and the weight-path placement holds on every measured part (§ 7). The byte constant does not: the bandwidth-starved base M4 carries a higher effective weight byte (about 34 against the M5 Pro’s 16.5 pJ/B), the byte-bound-versus-element-bound contrast of § 9, so the predictor’s energy magnitude is host-specific and reported on the fitted M5 Pro alone.
Single-stream device map
From each operator’s arithmetic intensity, the corrected roofline predicts the single-stream device map measured here. The single-stream () device map addresses the first practical choice: for one inference on one input, which engine a given operator class should run on. Table 5 reports latency, fp16 relative error, idle-subtracted active package power with CV%, and perf/watt for the ANE and GPU across thirteen synthetic workload classes; Table 6 reports three real models with energy per inference.
The single-stream map shows that the ANE draws less power on every in-graph-expressible workload tested (Figure 5), a 1.02 to 3.8× reduction in active package power, even on the workloads where the GPU is faster. On large square fp16 GEMM () the GPU is 2.8× faster and more accurate ( versus the ANE’s ), yet the ANE still does the work on 8.71 W against the GPU’s 33.05 W. The ANE has the lower latency on convolution, convolution stacks (4.2× on the ResNet-like block), the stencil (2.8× against an optimized GPU stencil), short attention, mixed GEMM, and the smaller real models (ResNet-18 5.0× and MiniLM 1.2× against compiled fp16 MLX). It has the higher perf/watt on 11 of 13 classes, by the largest margin on the ResNet-like conv stack (13×), then the stencil (4.7×) and the Jacobi solve (3.6×). The remaining two classes sit at or near parity, and repetition separates them. Over ten independent cold starts the GPU advantage on group_norm is small and reproducible (0.949, 95% CI [0.900, 0.997]). The DFT ratio is 1.35 CI of [0.96, 1.73] that spans parity, so that class carries no directional verdict (online supplement, formulation-fairness appendix). The exception among the real models is ViT-B/16, on which the GPU is faster (Table 6). Against torch-MPS fp16 the margin is small (15.9 versus the ANE’s 18.3 ms) and the ANE’s energy advantage is 9.4×. Graph-compiled MLX runs the model in 2.96 ms, 6.2× faster than the ANE, and the ANE’s energy advantage drops to 1.5× (95% CI [1.3, 1.8]). MLX also halves MiniLM’s advantage, from 6.7× to 3.4×, though the ANE remains the faster device there. The GPU has the lower latency on large square GEMM, long-sequence attention, and the small norm reductions, and is more accurate in fp16 everywhere.
Figure 5. Single-stream perf/watt advantage, ANE divided by GPU, per workload class on the M5 Pro (Table 5), sorted and drawn against a parity line. The ANE is more energy-efficient on 11 of 13 classes, by up to 13× on the ResNet-like convolution stack; the stencil is scored against an optimized GPU stencil, which places it at 4.7×. Error bars are 95% CIs over independent cold starts for the stencil, the DFT, and group_norm; the DFT interval spans parity, so that class supports no directional verdict (n.s.). The conv stack, ViT attention, and GEMM-compute classes are also replicated (perf/watt differences with CIs in online supplement Table S17), and Table 5 marks every replicated class.
A precision-versus-speed rule follows from the measured fp16 accuracy. Use the ANE’s fp16 speed and efficiency where the math is well-conditioned and not a large square GEMM (convolution, stencils, short attention, mid-size GEMM, encoders); there its relative error of to is the same order as GPU fp16 and it is usually both faster and lower-power. Hand large compute-bound square matmuls () and long sequences to the GPU. Use fp32 wherever cancellation can dominate (an adversarial, ill-conditioned reduction), where fp16 error is unbounded. The headline perf/watt is the conservative total-package number that charges the ANE for its own host CPU dispatch, and the ANE is the more efficient device on it for 11 of the 13 device-map classes, so the efficiency is silicon; the ANE-rail-only silicon bound is 1.2 to 4× larger (online supplement, dual-bound table).
Cross-generation reproduction
The single-stream watt harness also runs on three further hosts: an M2 Pro (h14 ANE, a 16 GB Mac mini), an M1 Max (h13, a MacBook Pro), and an M4 (h16, a second 16 GB Mac mini, base part), so the reproduction spans four generations and both chassis classes. The online supplement gives the per-class maps (Table S11–Table S13); h16’s three real-model rows were not obtained, as ResNet-18 hung at compile on that part. Those maps cover the eleven synthetic classes common to all four hosts, two fewer than Table 5, since Jacobi and the DFT-as-matmul were added to the reference map afterwards and are M5 Pro only.
| Class | Workload | ANE lat | GPU lat | ANE W (CV) | GPU W (CV) | ANE perf/W | GPU perf/W |
| GEMM floor | M64·K256·N256 | 0.164 | 0.184 | 2.50 (16 %) | 1.23 (8 %) | 17.0 GF/W | 27.1 GF/W |
| GEMM bandwidth | M128·K1024·N1024 | 0.405 | 0.242 | 1.92 (5 %) | 3.51 (32 %) | 470 GF/W | 207 GF/W |
| GEMM compute | M256·K4096·N4096 | 2.394 | 1.179 | 4.42 (11 %) | 32.47 (1 %) | 775 GF/W | 192 GF/W |
| conv | single 3×3 (16 → 64) | 0.191 | 0.211 | 3.94 (4 %) | 3.43 (11 %) | 23.2 GF/W | 17.7 GF/W |
| conv | ResNet-like 3×3×16 (64 → 256) | 1.918 | 3.944 | 4.35 (3 %) | 30.09 (6 %) | 2063 GF/W | 142 GF/W |
| attention | ViT block | 0.613 | 0.573 | 4.48 (8 %) | 4.04 (8 %) | 370 GF/W | 187 GF/W |
| attention | long-seq | 1.815 | 0.855 | 3.54 (15 %) | 9.82 (23 %) | 544 GF/W | 217 GF/W |
| norm | layer_norm (197,768) | 0.321 | 0.308 | 4.39 (14 %) | 3.84 (13 %) | 9.09e7/W | 1.01e8/W |
| norm | rms_norm (197,768) | 1.311 | 0.255 | 3.51 (10 %) | 4.06 (15 %) | 6.29e7/W | 1.07e8/W |
| norm | group_norm (1,256,642) | 1.025 | 0.358 | 4.66 (9 %) | 2.54 (27 %) | 3.49e8/W | 6.14e8/W |
| sci | 5-pt stencil , 32 steps | 0.223 | 2.066 | 4.46 (5 %) | 26.06 (2 %) | 33.8 GF/W | 0.60 GF/W |
Table S11. Single-stream device map on the M1 Max host (h13 ANE; 6 s power window, idle package = 2014 mW). The three real-model classes were skipped on this host (torchvision and transformers unavailable), so the table reports the eleven synthetic classes.
| Class | Workload | ANE lat | GPU lat | ANE W (CV) | GPU W (CV) | ANE perf/W | GPU perf/W |
| GEMM floor | M64·K256·N256 | 0.076 | 0.202 | 1.31 (9 %) | 2.46 (7 %) | 27.9 GF/W | 18.1 GF/W |
| GEMM bandwidth | M128·K1024·N1024 | 0.184 | 0.284 | 4.09 (6 %) | 6.31 (7 %) | 407 GF/W | 133 GF/W |
| GEMM compute | M256·K4096·N4096 | 2.117 | 2.692 | 3.94 (5 %) | 14.90 (12 %) | 1020 GF/W | 209 GF/W |
| conv | single 3×3 (16 → 64) | 0.092 | 0.198 | 3.51 (5 %) | 2.63 (5 %) | 57.9 GF/W | 32.1 GF/W |
| conv | ResNet-like 3×3×16 (64 → 256) | 1.158 | 6.045 | 5.65 (5 %) | 15.37 (6 %) | 2746 GF/W | 197 GF/W |
| attention | ViT block | 0.334 | 0.762 | 4.26 (5 %) | 9.89 (6 %) | 780 GF/W | 122 GF/W |
| attention | long-seq | 1.100 | 1.478 | 3.36 (3 %) | 11.10 (9 %) | 868 GF/W | 186 GF/W |
| norm | layer_norm (197,768) | 0.285 | 0.275 | 2.62 (5 %) | 3.56 (6 %) | 2.04e8/W | 1.66e8/W |
| norm | rms_norm (197,768) | 0.231 | 0.221 | 4.05 (4 %) | 3.06 (6 %) | 2.78e8/W | 2.10e8/W |
| norm | group_norm (1,256,642) | 0.661 | 0.336 | 4.31 (5 %) | 5.45 (6 %) | 4.02e8/W | 5.07e8/W |
| sci | 5-pt stencil 256², 32 steps | 0.139 | 3.638 | 3.00 (5 %) | 9.77 (6 %) | 77.6 GF/W | 1.02 GF/W |
Table S13. Single-stream device map on the M4 host (h16 ANE, the base-M4 engine; 6 s power window, idle package = 135 mW). The three real-model rows are absent because ResNet-18 hung at compile on h16. The synthetic map provides the h16 comparison, and reproduces to within a few percent across two independent runs.
Every substantial-workload result reproduces, where substantial means the classes above the dispatch floor: conv stacks, attention, compute-scale GEMM, the stencil, and the real models where they ran. Every substantial class favors the same device on all four generations, and the ANE is the more efficient device in each case. The conv-stack perf/watt ratio stays between 12.9 and across the four parts. Non-reproducing rows are confined to the dispatch-floor and tiny-norm classes, where host dispatch dominates package power. These shift toward the GPU on the older parts and back on the newest. Across the eleven classes, the faster device matches the M5 map on 7 and the more efficient device on 7, with 3 matching on both axes. On h16 the disagreements shrink instead of growing, the GPU keeping a perf/watt edge only on group_norm.
The bandwidth, serving, and decode sweeps reproduce in structure on the desktop parts. Both h14 and h16 show the same low-activation-ceiling asymmetry. On h14 the batched-serving throughput crossover falls near , with no throughput-per-watt crossover on any of the four workloads. On h16 the ANE leads in tokens-per-watt at every batch, by a wider margin than on the M5 Pro. It does not run on h14, whose 16 384 maximum tensor dimension is the 32 000-token vocabulary head exceeds. A fused-GPU baseline (mx.compile) shows the large gaps are not MLX launch overhead: fusion helps only the small elementwise rows.
Formulation is a separate confound, and one class is bound by it: expressing the 5-point stencil as a dense convolution costs the GPU 3.3× in time and 2.9× in power, moving that row from to (online supplement, fused-GPU and formulation-fairness appendices).
Batched serving
The device map of § 6 is the single-stream () choice. Serving is different: a deployment batches requests. We sweep batch size with true batching, one compiled program with a real batch dimension, across four serving workloads, and report the two deployment-relevant crossovers: the batch at which GPU throughput first exceeds the ANE’s, and the batch at which GPU throughput-per-watt does. Both devices run the same fp16 math at the same , warm, steady-state; accuracy is flat in .
The throughput crossover appears on three of the four workloads, and the throughput-per-watt crossover on none of them within the swept range. The online supplement (Table S7–Table S10) gives the sweeps; Table 7 summarizes the crossovers.
| Workload | Throughput crossover | Throughput/W crossover |
| Vision (conv classifier) | never (3.1–5.7×) | never (6–10×) |
| Encoder (transformer block) | ∼23–31 (replicate 31.1 ± 1.3) | never (1.9–5.4×) |
| Attention (self-attention block) | small batch (replicate 2.23 ± 0.02) | never (1.7–2.4×) |
| Batched GEMM | ∼5 () | never (1.06× at ; § 10) |
Table 7. Crossover summary: the batch at which GPU throughput, and GPU throughput-per-watt, first exceed the ANE’s, per serving workload. A never entry marks a workload where the ANE leads at every swept , with the ANE/GPU multiplier range in parentheses.
Table S7. Vision, conv (ch, ), then a global average pool and a 100-way fully connected layer; items = images.
| B | ANE thr | GPU thr | ANE perf/W | GPU perf/W | ANE mJ/it | GPU mJ/it |
| 1 | 11759 | 2067 | 1536 | 232 | 0.65 | 4.32 |
| 4 | 28768 | 9183 | 2904 | 367 | 0.34 | 2.72 |
| 16 | 50962 | 13984 | 4484 | 457 | 0.22 | 2.19 |
| 64 | 62204 | 16312 | 4607 | 466 | 0.22 | 2.14 |
| 256 | 66969 | 17338 | 4840 | 475 | 0.21 | 2.10 |
Table S7.
Table S10. Batched GEMM, ; items = batch elements.
| B | ANE thr | GPU thr | ANE perf/W | GPU perf/W | ANE mJ/it | GPU mJ/it |
| 1 | 8276 | 3734 | 736 | 382 | 1.36 | 2.62 |
| 4 | 18540 | 11228 | 1508 | 716 | 0.66 | 1.40 |
| 16 | 23495 | 54748 | 1903 | 1841 | 0.53 | 0.54 |
| 64 | 27198 | 94407 | 2135 | 2148 | 0.47 | 0.47 |
| 256 | 27577 | 109589 | 2223 | 2063 | 0.45 | 0.48 |
Table S10.
For vision both crossovers are absent: the ANE leads at every batch on throughput and perf/watt. For the transformer workloads the GPU’s raw throughput crosses above the ANE’s at small-to-moderate batch (attention near , encoder near ; § 10), while the ANE keeps the higher throughput-per-watt across the full swept range, and on bare large-batch GEMM it retains about 6 % (resolved under the replicate protocol of § 10). The GPU is the throughput device for batched transformers; the ANE’s serving role there is the low-batch and energy-capped regime. The throughput crossover depends on the workload, falling near for a self-attention block and near for a transformer encoder. Above its own crossover the GPU maximizes throughput. The ANE remains preferable whenever sequences-per-joule is the target, and for vision under both metrics.
End-to-end LLM decode
The serving sweeps above measure encoder, attention, and conv blocks. The remaining serving case, and the one where a single operating point most easily misleads, is autoregressive LLM decode, the per-token forward pass of a generative transformer. The synthetic GEMV operating point (Table 4: ANE 112 > CPU 81 > GPU 75 GFLOP/s at AI ≈ 1.0) suggests an ANE single-stream decode advantage. To test whether that op-level point transfers, we measured a full decode loop: one token is a complete 8-layer transformer forward at sequence position 1, with the key–value (KV) cache (length 256) fed as graph inputs. The configuration is a TinyLlama/GPT-small class model (d_model=2048, 8 layers, 16 heads (d_head=128), a 4× feed-forward network (FFN, d_ff=8192), vocab 32000, KV-len 256) at 1.22 GFLOP/token () and 39.1 GFLOP/token (). The ANE attention is built from bmm and softmax primitives, not af.sdpa (which needs equal-shape query/key/value tensors and so does not apply to the decode step). The GPU runs MLX fp16, the CPU numpy/Accelerate fp32 (the reference); ANE and GPU run the same fp16 math. Table 8 reports the result with the same per-engine energy accounting as the rest of the paper.
The op-level GEMV ordering does not carry to end-to-end decode throughput. Measured on a real loop, the GPU has the higher throughput at every batch in fp16 and int8, 1.74× the ANE at (200 versus 115 tok/s), widening to 2.7× at (2322 versus 847), the reverse of the single-GEMV microbenchmark.
The per-token time budget explains both the deficit and the reversal. The decode forward compiles to a single fused program per token step, so the per-program dispatch floor [6] (about 70 μs on this part) is paid once per token, under one percent of the 8.7 ms fp16 token. The budget is instead dominated by the weight stream: the model as measured carries 602.4 M weight parameters across the eight decoder blocks and the vocabulary projection, 1.20 GB in fp16. The batch sweep separates the costs cleanly, since per-step latency is linear in batch with intercept 7.9 ms and slope 0.93 ms per stream (residuals under 1.5 % at all six batch points). The intercept is the batch-invariant weight stream, 1.20 GB at an effective 154 GB/s. This agrees with the standalone GEMV sweep after removing 56 amortized dispatch floors from the 7.6 ms total for its 57 constituent GEMVs. The slope is the per-stream traffic, chiefly the 16.8 MB KV cache fed through the slower activation path. Single-stream decode on the ANE is therefore weight-stream-bound; dispatch would dominate only if the stack were issued one program per operation, costing 57 floors and capping the loop near 80 tok/s.
The GPU streams the same way faster: at 200 tok/s it moves the same 1.20 GB in 5.0 ms, about 240 GB/s, near its achieved streaming ceiling. Decode is weight-stream-bound on both devices, as the roofline predicts, and the GPU’s higher decode throughput follows from its faster weight stream. The GEMV microbenchmark corroborates the weight-stream tier (§ 3.3 and § 4), though standalone small- GEMVs are dispatch-floor-dominated and do not bound the fused loop.
int8 weight streaming identifies which resource binds the stream. Decode is the AI ≈ 1.0 memory-bound regime, so per-channel int8 (half the weight bytes) is the one change that could move throughput, not energy alone. It barely does: int8 lifts throughput by 7 % (115 123 tok/s) and then matches fp16 for every (within 1 %). A purely byte-bandwidth-bound stream would roughly double under this compression, and the measured change is near null. Combined with the weight-stream-bound budget above, that places the limit at the rate weight elements pass the dequantize-and-multiply stage, about 77 Gelem/s, and not at DRAM bytes.
That ceiling belongs to the dequantize stage, not to the weight stream, and a third encoding separates the two. Streaming the same weights as an int4 lookup table quarters the weight blob (1149–287 MB) and raises throughput to 203 tok/s, the int8 rate, where an element-rate limit predicts no change. That is level with the GPU’s fp16 rate of 200 tok/s; we did not measure an int4 GPU arm, so this is not a like-for-like throughput comparison. The encodings compile to different constant operators, constexpr_affine_dequantize for int8 and constexpr_lut_to_dense for int4, and the measured rates separate with them: fp16 streams at 140 GB/s and 70 Gelem/s, int8 at 74 GB/s and 74 Gelem/s (the 77 Gelem/s above, re-measured), and int4 at 61 GB/s and 122 Gelem/s. The fp16 stream is thus byte-bound at the 130–143 GB/s GEMV plateau, the int8 stream is element-rate-bound below it, and the lookup-table stream clears both. The ceiling is therefore a property of the weight encoding, not of precision: which constant operator the encoding lowers to sets the rate, and the three operators measured give three rates. The int4 program is inaccurate on this synthetic model, disagreeing with the fp64 reference on the greedy token at a rank correlation of 0.82 against 0.998 for int8, so its output alone does not establish that the encoding streams every weight. ResNet-18 does. Requested at the default budget the encoding falls back to int8 and reproduces the int8 row exactly (69.65 % top-1, 2.33 % label disagreement); engaged, it is a distinct and working classifier at 67.21 % top-1 over all 50 000 validation images (§ 2). A path that skipped weights would not classify at that rate. The synthetic model’s error reflects randomly initialized weights, which give a 4-bit palette no cluster structure to exploit. The online supplement (Table S23) reports the four encodings with weight blobs read from the compiled artifact. This measurement is one configuration at on the M5 Pro and is not checked across generations, which matters because int8 decode is already at a different bound on the base M4.
| Encoding | weight blob | emitted operator | lat (ms) | tok/s | GB/s | Gelem/s |
| fp16 | 1149.1 MB | (dense const) | 8.598 | 116.3 | 140.1 | 70.1 |
| int8 | 575.0 MB | affine_dequantize | 8.100 | 123.5 | 74.4 | 74.4 |
| int41 | 575.0 MB | affine_dequantize | 8.114 | 123.2 | 74.3 | 74.2 |
| int4 | 287.3 MB | lut_to_dense | 4.923 | 203.1 | 61.2 | 122.4 |
| blockwise | 610.5 MB | blockwise_shift_scale | 8.892 | 112.5 | 72.0 | 67.7 |
Table S23. Decode weight encodings at on the configuration of the main paper’s decode table (M5 Pro). A requested encoding is not always the applied one, so the weight blob is read from the compiled artifact rather than inferred from the compile flag, and the emitted operator is read from the generated MIL. int4 at the default reconstruction-error budget of 0.05 is rejected for every weight of this model and falls back silently to per-channel int8, which the int41 row records. Latencies are means over five interleaved rounds of 30 repetitions, round-robin across encodings. Across-round standard deviations are 0.03–0.3 % of the mean.
What compression yields instead is energy savings: halving the weight bytes cuts energy per token by 20 % at (27.8 22.4 mJ, about 9 pJ per avoided byte, an LPDDR-scale figure) and improves the ANE’s tok/s/W at most batches (by up to 36 %, though it dips about 6 % at ), at roughly 2 the fp16 relative error (about ). int8 does not change the GPU’s throughput advantage at any batch. This element-rate ceiling is host-specific: on the more bandwidth-starved base M4 (h16) int8 does raise ANE decode throughput (1.3 to 2), which shows that part’s decode is at the DRAM-byte bound, not at this element-rate limit (§ 7). The accuracy cost is bounded: int8 holds the greedy token 92 % of the time versus the fp64 reference.
The energy axis is a measured crossover, not an immediate switch. The ANE has the higher tok/s/W at every batch up to in both precisions, int8 by 2.3 at (44.7 versus 19.3), 2.7 at (100.3 versus 36.9), and still 1.75 at (113.7 versus 65.1), with fp16 close behind (35.9 at , 106.2 at ). The GPU’s efficiency crosses above only at (116.4 versus 105.4 int8, 88.0 fp16), where the ANE’s package draw grows with the host CPU feeding MB of KV cache per step. (The GPU’s dip at is a power-state step: its rail jumps from 6.3 to a pinned 19.9 W before batching amortizes the full-power clocks.) Sweeping the KV cache from 128 to 2048 positions at , per-step latency is linear in context on both devices, and the fit depends on where the cache lives. Holding it resident in the engine’s input buffers, the ANE runs at 7.75 ms plus 1.34 s per position against the GPU’s 4.74 ms plus 0.73 s. Both intercepts recover the batch-invariant weight stream on an axis it was not fitted against, and the KV term reaches the weight term only at on the ANE and on the GPU, so single-stream decode is weight-stream-bound across any context either part will hold. The ANE’s KV path is 1.8 slower per position, the activation path of § 3.2 and not the weight path, and its efficiency advantage is 2.6–2.0 across the sweep, with no crossover.
Re-feeding the cache to the engine on every token instead, as a graph input instead of a resident buffer, is what the roofline predicts it to be and is worth separating from the silicon. The cost is 0.51 ms and 0.92 W at , rising to 4.38 ms and 2.18 W at 2048, where the transfer is 128 MB per token. It also triples the fitted per-position slope, to 3.49 s, and it erases the ANE’s efficiency advantage by . The signature is a falling engine rail, 1878–1428 mW, beside a rising package, 3.75–5.86 W: the engine idles while the host marshals. Held resident, the rail is flat at about 1.93 W and the package falls to 3.68 W at the same point. This host-feeding term also produces the batch crossover above; it is an implementation cost, not a property of the part.
Batched decode gives the GPU the throughput axis immediately but the energy axis only past . At small batch, on-device decode uses 2 to 3 less energy on the ANE. That advantage ends near on the batch axis, and across context length it persists only while the caller keeps the cache resident.
These magnitudes are for the one model measured. Decode here is weight-stream-bound because the 1.20 GB weight stream is about 70 the per-token KV traffic at , and the throughput ordering follows the two devices’ weight-stream rates. A model whose weight bytes dominate its KV bytes should order the same way. The energy crossover depends on the ratio of per-stream KV and host-feeding cost to the weight stream, which changes with width, depth, attention layout, and vocabulary (the 32 000-token vocabulary projection alone is 65.5 M of the 602.4 M streamed parameters), and we make no claim for where it falls on other models.
Discussion
The corrected roofline has a corollary the four sweeps confirm: the ANE’s performance-per-watt advantage is regime-dependent, set by where a workload’s arithmetic intensity falls relative to the two ridges. It is largest in the compute-bound, low-to-moderate-utilization regime, where the ANE is both faster and more efficient (Table 5, and ResNet-18 and MiniLM in Table 6). Four regimes favor the GPU. It has the higher raw throughput once a device is saturated, where it also takes a small conv perf/watt edge (§ 3.1), and it leads the standalone memory-bound regime, where the ANE’s activation path runs about below the GPU’s unified-memory rate (§ 3.2). It also leads batched-transformer serving (§ 8) and, in the measured decoder, decode throughput at every batch (§ 9). Vision serving is the exception. ViT-B/16 and single-stream decode are energy-only exceptions: the ANE uses less energy but has the higher latency. The decode result holds across the measured context range while the KV cache stays resident. The two-ridge structure accounts for this split, and HeteroLLM [8] reports the same split independently on a different NPU (§ 11).
This corrects the community guidance [1, 14]. The 38-TOPS shorthand (an M4 figure) reduces a regime-dependent engine to one peak number. The measurements also contradict the guidance that the ANE is worthwhile mainly for big models: its largest advantages are in low-batch, latency-and energy-capped on-device inference. The claim that the ANE is more efficient is true in the compute-bound regime, narrows to a small GPU edge on conv at peak, and is false in the standalone memory-bound regime.
Fusion is the program-level control on arithmetic intensity, and so on regime: fusing operators into one compiled program raises effective AI and moves the operating point past the ridge (§ 4). An arithmetic-intensity-aware scheduler should place work by arithmetic intensity and target utilization. Operators that would otherwise be memory-bound on the ANE should be fused into a larger graph where possible. The choice is also constraint-dependent. Under a tight power or energy cap the ANE is preferable for every serving workload measured here, though on bare large-batch GEMM by only about 6 %. Without such a cap the GPU is preferable past each workload’s crossover. CoreML exposes model-level placement through computeUnits, which can assign vision, convolution, and small-encoder models to the ANE. It does not expose the per-operator fusion control that moves a memory-bound operator across the ridge, nor does it report the per-engine energy measured here.
Replication. Re-measured over independent cold-start runs, conv at peak resolves to a 5 % GPU edge and batched GEMM at to a 6 % ANE edge (both Holm-significant), the large ANE margins (conv stack, attention, GEMM-compute) confirm, and both transformer crossovers are stable ( and ). The full protocol and per-claim table are in the online supplement (Table S17).
Related work
The analysis descends from the roofline model [31], extended to energy by Choi et al. [10] and to cache-aware, hierarchical, and instruction-level variants [11, 17, 33], with an efficiency-oriented framing for ML accelerators [30]. The ANE exposes two effective bandwidths, so the single-line form does not describe it; the multi-tier roofline [33] is what the data-re-derive for this NPU (§ 4). We build the roofs empirically from microbenchmarks, not datasheet peaks [22], and the weight-path-versus-activation-path split is the energy-per-data-movement dataflow taxonomy [9] seen from outside the black box. Pagoda [25] builds dual time-and-energy rooflines on Jetson, and RooflineBench [3] scores on-device LLMs by operational intensity but is throughput-only and does not isolate the ANE. Smartphone-AI benchmarking was popularized by AI-Benchmark [16] and standardized, with a power methodology, by MLPerf Inference [26], though that line reaches the ANE only through CoreML and cannot isolate it. Seshadri et al. [27] report throughput, utilization, and energy for Google’s Edge TPU. Edge-AI rooflines have targeted Jetson and Coral [18], mobile-NPU LLM inference has been studied at the systems level and in GPU-NPU hybrids [24, 32], and sustained-load mobile energy has been measured across NPU and GPU including Apple devices [21, 29], none isolating the ANE rail.
HeteroLLM [8] characterizes a Qualcomm Hexagon NPU with an Adreno GPU and finds the same regime split with absolute power, compute-bound prefill to the NPU and memory-bound decode to the GPU; that an independent study reaches this on a different NPU is consistent with the device map reflecting a general property of NPU-versus-GPU economics, not of one vendor. For the energy methodology, idle-subtracted net energy has precedent in NPU-MCU work [12], and programmatic energy measurement on Apple silicon has been discussed but not resolved into a validated estimator [23].
A small body of concurrent work establishes direct or reverse-engineered ANE access: Orion [20], the maderix M4-ANE writeups [28], and the Asahi-Linux ANE driver [34]; neither direct access nor the observation that the engine can be energy-efficient is new here. Orion reports ANE throughput for LLM training and inference, with an ANE-versus-CPU speed comparison only and no power or roofline measurements. It independently finds CPU decode outpacing ANE decode on an M4 Max (283 versus 170 tok/s). Our decode loop compares the ANE against the GPU, not the CPU, but finds the same direction, a faster companion engine, on a different chip and access path (§ 9), and attributes the deficit to the ANE’s weight-stream rate, not to aneforge. NPUMoE [2] runs mixture-of-experts on the ANE through CoreML, as do Apple’s transformer guidance [1] and the community knowledge base [14]. Pagoda [25] and HeteroLLM [8] are the closest prior characterizations. Neither isolates the ANE rail in a per-engine roofline with idle-subtracted, package-level energy across the single-stream, batched, compute-saturation, and bandwidth regimes.
Conclusion
We presented a per-engine energy roofline characterization of the Apple Neural Engine against the Apple GPU and CPU on an M5 Pro, with the single-stream watt map reproduced on an M1 Max, an M2 Pro, and an M4.
The ANE’s efficiency advantage is regime-dependent. It is faster and lower-power for compute-bound work at low-to-moderate utilization, including conv, attention, mid-size GEMM, ResNet-18, and the MiniLM encoder; on ViT-B/16 it is only the lower-energy device, by against compiled MLX. The GPU has the higher throughput at saturation and at transformer-serving scale, a small perf/watt edge on conv at peak, and both higher speed and higher efficiency for standalone memory-bound operators. The ANE, as reachable today, has two effective bandwidths (about 24 GB/s standalone-activation, 112–143 GB/s weight-stream) and two corresponding ridges, the fast one below the GPU’s. Given arithmetic intensity and dispatch path, the energy roofline predicts the lower-energy engine and bounds its energy from below. Its median error is 14% near the roof, and it stays conservative elsewhere within its stated domain. ML-systems schedulers should place work by arithmetic intensity and utilization, not by operator name.
Limitations of this work
The saturation, bandwidth, serving, and decode magnitudes are from the one Apple M5 Pro (h17s); their structure reproduces across the four generations (§ 7), but the absolute multipliers are host- and thermal-dependent and should not be quoted for other parts, and the cross-generation reproductions are single- or dual-run point estimates, and are not replicated under the cold-start protocol. The end-to-end decode results are for one model configuration (one size, one 32 000-token vocabulary, synthetic weights), and their magnitudes and energy crossover are not claimed for other models (§ 9). The activation-path energy crossover () is computed from the two fitted constants and lies well beyond the measured range, so it is an extrapolation, not a measurement. Both power estimators are modeled SoC telemetry, not an external meter, though they cross-validate (§ 2, online supplement Table S1), and the ANE bandwidth ceiling is a path limit of aneforge’s single-program streaming, not a claim about the silicon DMA. This is a plugged-in, actively cooled host exposing only an OS thermal-pressure signal, so a fanless or battery-constrained device may behave differently. The access path uses undocumented but unprotected user-space runtime interfaces [6]. This work characterizes performance only and crosses no content-protection or security boundary. Despite these limitations, the contested small-margin verdicts are settled under the cold-start replicate protocol with formal inference attached (§ 10).
Data and code availability
The code, fail-soft driver, scripts, and committed result files behind every reported value are public at https://github.com/sbryngelson/ANEForge under a permissive open-source license, archived as release artifact-v1 at https://doi.org/10.5281/zenodo.24210581 [4].
Acknowledgments
During preparation of this manuscript the author used a large language model (Anthropic Claude) for prose and grammar polishing. The author reviewed and edited all such text and takes responsibility for the content.
References
Supplementary material
These appendices are the paper’s supplementary material. Tables use an S prefix. Unprefixed section and table references point to the main paper.
A Supplementary measurement tables
This appendix reports the measurements summarized in the main text.
IOReport lands within about 2 % of powermetrics at the median on every point, in no systematic direction. Two of the five points are CPU-dispatch-dominated (CPU GEMM, ANE stream), so the cross-check covers the attribution the total-package figure depends on. The lowest-power ANE points carry the widest run-to-run spread, dipping to about 73 % on occasional runs while their medians hold near 97 %.
Run-to-run variation is asymmetric. The ANE arm’s coefficient of variation is 11–52 % against 1.7–30 % for torch-MPS. ResNet-18’s torch-MPS energy alone spans 20–49 mJ across cold starts, so one GPU measurement does not determine the ratio. Moving the GPU from fp32 to fp16 cuts its energy by 39–58 %.
The three device maps that follow share the columns and conventions of the main paper’s device map. Best speed and best perf/W per row are in bold. Each caption states only the conditions and omissions specific to its host.
Table S8. Encoder, pre-LN transformer block, , , , ; items = sequences.
| B | ANE thr | GPU thr | ANE perf/W | GPU perf/W | ANE mJ/it | GPU mJ/it |
| 1 | 2756 | 1262 | 374 | 69 | 2.67 | 14.51 |
| 4 | 2984 | 2546 | 391 | 103 | 2.56 | 9.73 |
| 16 | 2786 | 2584 | 402 | 129 | 2.48 | 7.73 |
| 64 | 2364 | 2947 | 414 | 145 | 2.41 | 6.88 |
| 256 | 1031 | 2948 | 285 | 150 | 3.51 | 6.65 |
Table S8.
Table S9. Attention, self-attention block, , , ; items = sequences; decomposed scaled-dot-product attention (SDPA), true rank-4 batch. †The GPU point did not reproduce. An independent run measured about 11 000 it/s. The replicated throughput crossover is .
| B | ANE thr | GPU thr | ANE perf/W | GPU perf/W | ANE mJ/it | GPU mJ/it |
| 1 | 6842 | 1948 | 715 | 297 | 1.40 | 3.37 |
| 4 | 8972 | 4238† | 1114 | 495 | 0.90 | 2.02 |
| 16 | 9778 | 21888 | 1122 | 580 | 0.89 | 1.72 |
| 64 | 7782 | 25463 | 1044 | 631 | 0.96 | 1.58 |
| 256 | 8088 | 24880 | 1027 | 618 | 0.97 | 1.62 |
Table S9.
Stride aliasing on h16. A dense weight-stream sweep on h16 shows a dip to about 25 GB/s at exactly the powers of two , , and . Every neighboring size, including , streams at full rate. Neither the M2 Pro nor the M5 Pro shows the dip, and every point still exceeds the activation ceiling by at least 2.4×.
Per-class cross-generation agreement. Counting against the M5 reference map (the main paper’s device map) across the eleven synthetic classes in Table S11–Table S13, the faster device is unchanged on 7 and the more efficient device on 7 (3 classes on both axes). The perf/watt disagreements all fall in sub-millisecond dispatch-bound and tiny-norm rows, that is, outside the substantial classes above the dispatch floor that the main paper defines. They are GEMM-floor perf/watt (GPU on both h13 and h14), single-conv perf/watt (GPU on h14), and layer_norm and rms_norm perf/watt (GPU on both). On those rows package power is dominated by host dispatch. The speed disagreements are the GEMM and attention rows, which switch to the GPU only on the older parts and back to the ANE on the newest (h16).
| Class | Workload | ANE lat | GPU lat | ANE W (CV) | GPU W (CV) | ANE perf/W | GPU perf/W |
| GEMM floor | M64·K256·N256 | 0.110 | 0.206 | 2.66 (7 %) | 3.04 (5 %) | 11.4 GF/W | 16.6 GF/W |
| GEMM bandwidth | M128·K1024·N1024 | 0.135 | 0.222 | 5.57 (2 %) | 7.78 (10 %) | 298 GF/W | 130 GF/W |
| GEMM compute | M256·K4096·N4096 | 2.043 | 1.858 | 5.61 (7 %) | 21.25 (1 %) | 773 GF/W | 214 GF/W |
| conv | single 3×3 (16 → 64) | 0.107 | 0.219 | 2.93 (3 %) | 3.37 (7 %) | 25.1 GF/W | 31.4 GF/W |
| conv | ResNet-like 3×3×16 (64 → 256) | 1.265 | 5.126 | 6.53 (1 %) | 20.52 (1 %) | 2235 GF/W | 173 GF/W |
| attention | ViT block | 0.589 | 0.643 | 5.52 (11 %) | 12.88 (11 %) | 481 GF/W | 120 GF/W |
| attention | long-seq | 1.253 | 1.144 | 4.63 (4 %) | 17.74 (8 %) | 567 GF/W | 150 GF/W |
| norm | layer_norm (197,768) | 0.283 | 0.278 | 4.30 (7 %) | 3.71 (8 %) | 1.08e8/W | 1.75e8/W |
| norm | rms_norm (197,768) | 1.080 | 0.182 | 3.40 (10 %) | 3.47 (7 %) | 7.47e7/W | 2.19e8/W |
| norm | group_norm (1,256,642) | 0.679 | 0.288 | 5.36 (3 %) | 5.72 (9 %) | 4.12e8/W | 5.80e8/W |
| sci | 5-pt stencil , 32 steps | 0.371 | 3.349 | 4.21 (8 %) | 14.13 (2 %) | 37.6 GF/W | 0.79 GF/W |
| model | ResNet-18 (1 × 3 × ) | 1.064 | 3.379* | 4.19 (8 %) | 3.75 (26 %) | 2.6 mJ | 22.3 mJ |
| model | MiniLM encoder (1 sent) | 1.753 | — | 3.73 (9 %) | — | 3.0 mJ | — |
| model | ViT-B/16 (12-layer, 197 tok) | 21.34 | 37.28* | 3.14 (1 %) | 18.45 (4 %) | 67.9 mJ | 714.3 mJ |
Table S12. Single-stream device map on the M2 Pro host (h14 ANE, the A14-generation engine; 6 s power window, idle package = 57 mW). The bottom block reports the three real models. There the GPU baseline is torch-MPS fp32 (marked *, not a like-for-like fp16 comparison) and the last two columns report energy in mJ/inference instead of perf/W. MiniLM has no torch-MPS reference in the harness.
FLOP and byte accounting per archetype
Every throughput, perf/watt, and arithmetic-intensity figure in this paper derives from the FLOP and byte formulas below, which are taken directly from the measurement scripts. Three conventions apply throughout. (i) Multiply–accumulates are counted as 2 FLOPs. (ii) Element traffic is priced at 2 B (fp16) for the GPU and ANE and 4 B (fp32) for the CPU, so every CPU AI is half the fp16 value listed. (iii) Elementwise transcendentals are charged a flat per-element allowance: 1 FLOP/element for relu/copy-class ops, 5 FLOPs/element for softmax (max, subtract, exp, sum, divide), and 10 FLOPs/element for gelu/silu (the erf/tanh polynomial approximation) and for layer_norm (mean, variance, normalize, scale, shift). These are order-of-magnitude conventions, not instruction counts. Scaling any allowance changes the derived FLOP rate but not the measured bandwidth or the regime classification. The memory-bound archetypes are measured in bytes per second and reported in FLOP-denominated units only through these factors, and no tested archetype crosses a ridge.
A silicon-versus-package perf/watt bound
The headline perf/watt divides throughput by total-package active power, which charges the ANE for its own CPU dispatch. An ANE program still draws several watts of host CPU power submitting
Table S15. Device-map workload classes (the main paper’s device map): the exact work count used to convert the measured latency and active power into throughput and perf/watt. The norm family is counted in elements normalized (Gelem/s), not FLOPs; the real models report energy per inference (mJ/inf) with no FLOP count. AI is shown for the fp16 fused-program traffic model of Table S14 (input + weights + output; on-chip intermediates uncounted) and is derived here, not computed by the script. The ViT attention count is the four projections () plus and (, ), with in-graph softmax FLOPs uncounted (conservative); the stencil is counted as the dense 3×3 convolution executed, per the main paper’s device map caption.
| Archetype | FLOPs | Bytes (fp16) | AI (FLOP/B) |
| matmul square | (AI = ) | 170.7 | |
| matmul square | 682.7 | ||
| matmul square | 2730.7 | ||
| GEMV / decode (, ) | MF | 1.0 | |
| conv 3×3 (, , ) | GF | 286.7 | |
| conv 3×3 (, , ) | GF | 2019.9 | |
| attention (, , ) | GF | 52.2 | |
| softmax | /elem | elem/elem × | 1.25 |
| gelu / silu | /elem | elem/elem × | 2.5 |
| layer_norm | /elem | elem/elem × | 2.5 |
| relu / copy | /elem | elem/elem × | 0.25 |
| matmul int4 (, ) | MF | 4.0 | |
| matmul int4 (, ) | GF | 409.6 | |
| fused 3-conv (, , ) | GF | fused in/out + ; standalone | 2854 / 1076 |
Table S14. Roofline-placement archetypes (the main paper’s roofline-placement table): FLOP formula, fp16 byte formula, and resulting arithmetic intensity. denotes bytes per element (2 for fp16; the CPU’s halves each AI). For the memory-bound rows the achieved GFLOP/s in the main paper’s roofline-placement table is derived as AI × measured GB/s; for the compute-bound rows it is the size-specific measurement. Attention counts and at FLOPs each with softmax at 5 FLOPs per score; its bytes are , , , and the output plus the score matrix written and re-read for softmax , the standalone/spilled convention).
| Device-map class | Work formula (per call) | Value | AI (fp16) |
| GEMM floor (, ) | 8.39 MF | 42.7 | |
| GEMM bandwidth (, ) | 268.4 MF | 102.4 | |
| GEMM compute (, ) | 8.59 GF | 227.6 | |
| conv single 3×3 (16 → 64, ) | 18.9 MF | 103.5 | |
| conv ResNet-like ×16 (64 → 256, then ×15, ) | 18.42 GF | 988 | |
| attention ViT (, , ) | 1.049 GF | – | |
| attention long-seq () | 3.221 GF | – | |
| layer_norm / rms_norm (197, 768) | elements | 0.151 Melem | 2.5 |
| group_norm (1, 256, ) | elements | 1.049 Melem | 2.5 |
| 5-pt stencil (, 32 steps) | steps | 37.7 MF | – |
| ResNet-18 / MiniLM / ViT-B/16 | energy per inference | mJ/inf |
Table S15. Device-map workload classes (the main paper’s device map): the exact work count used to convert the measured latency and active power into throughput and perf/watt. The norm family is counted in elements normalized (Gelem/s), not FLOPs; the real models report energy per inference (mJ/inf) with no FLOP count. AI is shown for the fp16 fused-program traffic model of Table S14 (input + weights + output; on-chip intermediates uncounted) and is derived here, not computed by the script. The ViT attention count is the four projections () plus and (, ), with in-graph softmax FLOPs uncounted (conservative); the stencil is counted as the dense 3×3 convolution executed, per the main paper’s device map caption.
Table S16. Bytes-moved models for the bandwidth roofline: achieved bandwidth = bytes/min-latency with the stated traffic factor; is the element count and the dtype size (2 B fp16 for GPU/ANE, 4 B fp32 for the CPU, whose proxy op is a memory-bound scalar multiply or copy, never a transcendental). The saturation-sweep FLOP counts are for square GEMM and for 3×3 conv at 64×64 spatial.
| Bandwidth type | Bytes moved | Reason |
| streaming (relu / ) | read + write ; the primary bandwidth-roof probe | |
| gelu (light compute) | same traffic; arithmetic does not add bytes | |
| reduction (sum, last axis) | read , write | |
| layer_norm (last axis) | read-once/write-once, statistics on-chip | |
| softmax (last axis) | logical two passes (max+exp, sum+div) are fused; read-once/write-once is paid | |
| DRAM traffic |
Table S16. Bytes-moved models for the bandwidth roofline: achieved bandwidth = bytes/min-latency with the stated traffic factor; is the element count and the dtype size (2 B fp16 for GPU/ANE, 4 B fp32 for the CPU, whose proxy op is a memory-bound scalar multiply or copy, never a transcendental). The saturation-sweep FLOP counts are for square GEMM and for 3×3 conv at 64×64 spatial.
work. The per-rail breakdown puts that CPU share between 15 % on a compute-heavy fused conv and 86 % on a single tiny op. We therefore report two figures from the same per-rail data: total-package perf/watt, which charges the ANE the full dispatch power, and ANE-rail-only perf/watt, which excludes it.
On the device-map classes of the main paper the ANE is more efficient on the total-package figure (Table S2), and the rail-only figure is 1.2 to 4× larger. The regime where the CPU dominates the package draw is the single tiny op at 80–86 % CPU, where standalone dispatch leaves the engine mostly idle and the rail-only column reads 176 against 25 GFLOP/s/W. We keep total-package as the headline because it is the marginal cost a deployer pays today, and report the rail-only figure alongside it.
| Workload | total-package perf/W | ANE-rail-only perf/W | CPU-dispatch share |
| conv ResNet-like (, fused) | 2289 | 2677 | 15 % |
| GEMM compute () | 751 | 1181 | 37 % |
| attention ViT () | 522 | 1074 | 52 % |
| GEMM bandwidth () | 274 | 1007 | 73 % |
| conv single (tiny, dispatch-bound) | 25 | 176 | 86 % |
Table S2. Silicon-versus-total-package perf/W (representative single-stream rows; idle-subtracted). Total-package perf/W includes CPU dispatch power. ANE-rail-only perf/W excludes it. Their difference is the measured host-dispatch share.
Silicon versus toolchain: a fused-GPU baseline
The single-stream device map compares anforge’s fused ANE programs against MLX’s default (un-fused) GPU dispatch. Part of the measured gap could therefore be MLX’s kernel-launch overhead and not silicon. Fusing the GPU with mx.compile tests this. Five representative workloads (a ViT-sized layer_norm and gelu, a self-attention block, a 16-deep conv stack, and the 32-step stencil that carries the device map’s largest perf/watt gap) run on the GPU in both default and mx.compile-fused form, against the ANE, under the same per-engine energy harness. Table S5 reports them.
| Workload | GPU def ms | fused ms | ANE ms | def W | fused W | ANE W | def perf/W | fused perf/W | ANE perf/W |
| layer_norm | 0.298 | 0.145 | 0.195 | 9.11 | 8.78 | 5.20 | 0.091 | 0.099 | 0.135 |
| gelu | 0.222 | 0.120 | 0.088 | 8.53 | 7.32 | 7.37 | 0.100 | 0.129 | 0.204 |
| attention | 0.270 | 0.288 | 0.233 | 15.07 | 15.10 | 7.64 | 212 | 212 | 566 |
| conv stack | 3.166 | 3.164 | 0.763 | 31.89 | 31.93 | 9.16 | 180 | 180 | 2477 |
| 5-pt stencil | 1.323 | 1.281 | 0.132 | 33.02 | 33.16 | 7.28 | 0.86 | 0.86 | 38.3 |
Table S5. Fused-GPU baseline (M5 Pro, 6 s window, idle package = 1017 mW). GPU default versus mx.compile-fused versus ANE: latency (ms), idle-subtracted active package W, and perf/W (Gelem/s/W for the elementwise rows, GFLOP/s/W for attention, conv, and the stencil). Best latency and best perf/W per workload in bold. The stencil was measured in a separate session. Its active W is idle-subtracted and therefore comparable. Shapes: layer_norm and gelu at 197 768; attention at , , ; conv stack of 16 layers, 64 256 channels, spatial, ; 5-point stencil on over 32 steps.
Fusion moves some rows and not others. It helps the multi-pass elementwise ops: layer_norm speeds up and gelu with mx.compile, and on one workload it reverses the within-session ordering: fused-MLX layer_norm latency (0.145 ms) is below the ANE’s (0.195 ms), where this session’s default-MLX baseline (0.298 ms) had been slower than the ANE. The device-map session measured default MLX at 0.176 ms, so the small elementwise-latency rows are both session-and toolchain-sensitive.
The ANE has the higher perf/watt on all five workloads, because it draws far less power (conv at about 9 W versus about 32 W, attention at about 7.6 W versus about 15 W), so even where the GPU matches or improves on its latency, the ANE does the work at a fraction of the energy. Fusion does not help the compute-bound or the bandwidth-bound multi-pass workloads: attention is (fusion slightly slows it), the conv stack is , and the stencil is only , because those are either already single fused MLX kernels or, in the stencil’s case, a chain whose per-step cost is the bandwidth of streaming a field, not kernel-launch overhead that fusion could remove. The large gaps are unchanged under fusion: the ANE is faster than even fused-MLX on the conv stack, faster on attention, and faster on the stencil. Fusing the GPU barely moves the stencil (), so kernel-launch overhead does not explain that row. § 24 shows the formulation does: the dense convolution is a poor GPU expression of a 5-point stencil, and replacing it moves that row from to . These results exclude kernel-launch overhead as the main cause of the tested gaps. They do not establish formulation fairness, and a fused baseline is not by itself an optimized one.
Replicate and variance protocol
The within-window coefficient of variation (CV%) reported per measurement in the main paper does not measure variation across independent executions. We therefore repeated the contested verdicts in fresh processes: the conv and GEMM perf/watt comparisons at peak, the large-margin ANE advantages, and the two serving crossovers. Every run re-samples its own idle baseline and randomizes the workload and device order (ANE-first versus GPU-first) to break warm-up and ordering bias. The two compute-bound rows were run separately, one workload per process under an inter-run cooldown, so no prior workload pre-heats the die. Both devices are measured within the same run and share its idle baseline and ambient state, so the run is the pairing unit and inference is on the per-run differences.
We test each difference with a two-sided paired -test and report the mean difference with a -based 95% confidence interval (CI). Equivalence is not inferred from failure to reject a difference. Near-equivalent rows are tested with two one-sided tests (TOST) against a declared margin of of the pooled device mean, and equivalence is declared only when both reject at . Simultaneous difference verdicts are corrected with Holm–Bonferroni, so a difference requires an adjusted and a CI excluding zero. Rows establishing neither a difference nor equivalence are reported as indeterminate, and the two serving-crossover rows are estimation problems that carry confidence intervals only. At these sample sizes an equivalence verdict is bounded by its CI, and margins much below a few percent cannot be certified.
Run-to-run idle-baseline drift is the dominant variance source. The per-run idle package draw varies by several hundred milliwatts, and a run whose idle baseline is corrupted by transient background activity is gated out. A run is dropped when its idle package power exceeds max(2 W, 2× median idle). The criterion is evaluated on the idle sample alone, before the workload is measured, so it is blind to the outcome it could affect. It is protective only while most runs are clean: the median rises with widespread contamination, and the max then widens the threshold instead of tightening it. The real-model replication of Table S6 therefore uses an absolute 2 W threshold. Background activity during the idle sample raises the baseline, over-subtracts, and inflates perf/watt, so the corrupted run is treated as a measurement fault, not a sample of the quantity of interest. One run of the ten on the saturation pair exceeded the gate. On the compute-bound GEMM row its exclusion tightens the statistics (ANE SD from 456 to 88, paired- from 0.019 to , minimum detectable effect from 56.5 % to 11.9 % of pooled) and reduces the estimated ANE advantage from 387.6 to 253.2. On conv_C512 it moves an indeterminate verdict to one favoring the GPU (mean difference −70.1, CI [−230.1, 89.9] ungated, −123.7, CI [−242.6, −4.8] gated). The gate therefore improves precision without systematically favoring the ANE. Table S17 reports the gated statistics, and the artifact retains both variants. The absolute idle floor differs between sessions, for example 1196 mW in the device-map run against 1017 mW in the fused-GPU run, so every run re-samples its own baseline instead of reusing a stored one. Every perf/watt number is idle-subtracted, so this drift propagates into the across-run standard deviation (SD), which is the variance figure we quote. Thermal pressure remained Nominal throughout, but this M5 Pro exposes only the OS thermal-pressure level through powermetrics and not a die temperature, so that rules out a pressure-level throttle and not a sub-threshold drift in clocks.
| Model | GPU fp32 mJ | GPU fp16 mJ | GPU fp16 lat (ms) | ANE mJ (committed) | ANE energy adv. (fp16) |
| ResNet-18 | 39.1 | 21.3 | 2.033 | 2.30 | 8.9 [7.5, 10.7] |
| MiniLM | 28.8 | 17.6 | 1.920 | 2.74 | 6.7 [5.9, 7.6] |
| ViT-B/16 | 1382.7 | 577.9 | 15.92 | 65.7 | 9.4 [7.9, 11.1] |
Table S6. Like-for-like fp16 GPU baseline for the three real models (torch-MPS .half()), paired per run against the ANE. Energy columns are means over independent cold starts. Each run re-samples its own idle baseline, shuffles the workload order, and is discarded if that baseline exceeds 2 W. The advantage is a geometric mean of the per-run ratios with a 95 % interval, aggregated in log space because a ratio is bounded below and right-skewed. The fp32 column is a single-session result without an interval; every other energy figure in this table is replicated. The compiled-MLX comparison for all three models is in the formulation-fairness appendix (§ 24).
F CPU (fp32/AMX) reference figures
The CPU enters the main paper as the fp32 accuracy reference and as a contextual roofline command, not as a like-for-like fp16 device. This appendix collects the CPU figures that the body tables omit so the comparison stays focused on the ANE versus the GPU. The CPU runs numpy/Accelerate (fp32, via Apple’s AMX matrix coprocessor): it moves twice the bytes per element of the fp16 devices, halving its arithmetic intensities, so it is the least-comparable of the rooflines and is fastest only on tiny dispatch-bound shapes where call overhead, not FLOPs, dominates. Its conv roof has two values, the 11 GFLOP/s im2col path the saturation harness defaults to and a tuned native conv (torch, fp32, the same shape) at 2077 GFLOP/s; the im2col figure is an implementation floor, and power was not separately metered for the tuned path. The CPU GEMM power is flat at about 16 W from to while throughput holds at about 1.8 TFLOP/s, the signature of an AMX-bound ceiling [7]. On the roofline placement the CPU GEMV (124 % of roof) and gelu (135 %) entries are cache residency: at those working sets the CPU streams from cache at 277–402 GB/s, far above its DRAM-derived roof, the phenomenon the cache-aware roofline [17] models.
Held-out predictor validation
The energy predictor of the main paper, , is fitted from the ceiling sweeps alone (ANE weight path , ; GPU 1.66, 36.3, all in pJ). The device-map classes are held out from that fit, so Table S19 tests the predicted energy magnitude against all of them, grouped by regime. For single-dispatch workloads the prediction lies below measurement, and near the roof it is within 7–28 % (median 14 % across both devices). Away from the roof it under-predicts the dispatch-bound rows by large factors, because their package energy is dominated by the fixed per-call cost the roofline omits. The fused 16-deep conv stack lies outside this domain because it amortizes one dispatch across sixteen layers, and it runs below the single-dispatch floor (0.44 measured against 1.18 pJ/FLOP predicted).
Both loose regimes are the ones the corrections already identify.
fp16 numerical accuracy per class
Table S20 reports the relative error of the fp16 device path against an fp32/fp64 numpy reference for each device-map class. Accuracy is set by the fp16 arithmetic and the reference rather than by the host, and the per-class error matches across the M5 Pro, M1 Max, M2 Pro, and M4 to within last-digit measurement noise. The one large-error class is the compute-bound square GEMM (), where fp16 accumulation over a long reduction reaches . Every other class stays at to , the same order as the GPU’s fp16.
| Class (workload) | ANE relerr | GPU relerr |
| GEMM floor (M64·K256·N256) | 1.3e-3 | 3.6e-4 |
| GEMM bandwidth (M128·K1024·N1024) | 5.5e-3 | 3.6e-4 |
| GEMM compute (M256·K4096·N4096) | 2.7e-2 | 3.6e-4 |
| conv single (16 → 64) | 5.1e-4 | 3.6e-4 |
| conv ResNet-like (64 → 256, 16-deep) | 3.3e-3 | 1.4e-3 |
| attention ViT block () | 1.6e-3 | 1.2e-3 |
| attention long-seq () | 2.4e-3 | 1.3e-3 |
| layer_norm (197,768) | 6.7e-4 | 6.1e-4 |
| rms_norm (197,768) | 5.7e-4 | 5.4e-4 |
| group_norm (1,256,642) | 7.4e-4 | 6.0e-4 |
| 5-pt stencil (256, 32 steps) | 1.4e-2 | 6.6e-4 |
Table S20. fp16 relative error per device-map class (M5 Pro; host-invariant to measurement noise). GPU values are MLX fp16 against the same reference.
The dispatch floor at the smallest GEMM reproduces on all three generations. No other class does. The DFT-as-matmul is strongly dispatch-bound on the M5 Pro alone, and ratios below unity, where the closed loop outruns the single-call probe, appear on both older hosts. The dispatch term of the main paper’s cost model is therefore fitted on the M5 Pro only.
Engagement is a binary switch on the reconstruction-error budget, not a continuum. At the default 0.05 every weight of this model is rejected and the encoding falls back to int8. From 0.15 upward the encoder emits the same palette at every budget we tested: sweeping 0.15, 0.25, 0.40 and 0.60 gives one weight blob (301 283 360 B) and one relative (0.5766) at all four points, with latency varying by 0.3 %. That sweep therefore fixes where the int4 engages, and it does not vary the palette’s accuracy, so it cannot show whether speed tracks accuracy.
The relative is measured against an fp64 reference on randomly initialized weights, which are unfavorable for 4-bit palettization because a Gaussian gives the palette no cluster structure to exploit. The evidence that the LUT path streams every weight is Table S25 rather than this error figure. Requested at the default budget on ResNet-18 the encoding falls back and reproduces the int8 row exactly (69.65% top-1 and 2.33% disagreement in both rows). Engaged, it is a distinct and working classifier at 67.21% top-1 over all 50 000 validation images, which a path that skipped weights could not reach.
| Device | top-1 | 95 % CI | top-5 | disagree | McNemar | |
| ResNet-18, all 50 000 validation images | ||||||
| CPU fp32 (reference) | 69.76 % | [69.36, 70.16] | 89.08 % | — | — | — |
| ANE fp16 | 69.76 % | [69.36, 70.16] | 89.09 % | 0.11 % | 14/14 | 1.00 |
| GPU MPS fp16 | 69.76 % | [69.36, 70.16] | 89.10 % | 0.14 % | 16/17 | 1.00 |
| ANE int8 | 69.65 % | [69.24, 70.05] | 88.97 % | 2.33 % | 319/263 | 0.022 |
| ANE int4 | 69.65 % | [69.24, 70.05] | 88.97 % | 2.33 % | 319/263 | 0.022 |
| ANE int4 | 67.21 % | [66.79, 67.62] | 87.89 % | 13.89 % | 2514/1237 | |
| ViT-B/16, 20 000 validation images | ||||||
| CPU fp32 (reference) | 81.06 % | [80.51, 81.60] | 95.38 % | — | — | — |
| ANE fp16 | 81.00 % | [80.45, 81.54] | 95.35 % | 0.40 % | 33/21 | 0.134 |
| GPU MPS fp16 | 81.07 % | [80.52, 81.61] | 95.38 % | 0.04 % | 3/5 | 0.727 |
Table S25. Task-level accuracy on the ILSVRC2012 validation set for the two classifiers (M5 Pro). Every device is given the identical preprocessed tensor, so the only difference between rows is arithmetic and weight encoding, and the ANE loader carries the same torchvision weights as the reference. Disagree is the fraction of images whose predicted label differs from the fp32 reference. counts images the reference gets right and the device gets wrong, the reverse, and is McNemar’s exact test on those discordant pairs, which tests whether the reduced precision changes decisions in a direction. The int4 row is the default accuracy budget, which falls back to int8 for every weight of this model and so reproduces the int8 row exactly. ViT uses 20 000 images and not all 50 000 because its fp32 CPU reference costs 35.7 ms per image.
Per-step latency is linear in context on both devices. The ANE fit is 7.750 ms plus 3.496 μs per KV position and the GPU fit 4.769 ms plus 0.702 μs, with maximum residuals of 0.71% and 0.24%. The intercepts recover the batch-invariant weight stream fitted from the batch sweep, and the KV term reaches the weight term only at on the ANE and on the GPU. Re-feeding costs the ANE a host-side copy that reaches 128 MB per token, which is why the efficiency ratio in the last column falls to 0.81 at 2048. Held resident on both devices, the ANE’s advantage is 2.6–2.0 across
Table S22. The dispatch ratio = sustained per-iteration time over minimum-latency probe, ANE path, across three generations. The M5 Pro column is measured directly from the retained iteration counts (Table S21). The M4 and M2 Pro records retain the probe throughput, the active power and the derived perf/watt, but not the sustained time, so is recovered from the identity = GFLOP/s/(GFLOP/s/W W), which is exact up to the stored precision. Re-rounding the M5 Pro record to that precision and applying the identity reproduces the measured to a median of 0.04 % and a worst case of 0.3 % over 20 operating points, so the recovered columns are limited by the stored precision and not by the method. The three normalization classes are omitted because their perf/watt is reported in items/s/W, which does not close the identity.
| Class (workload) | measured M5 Pro | recovered M4 | recovered M2 Pro |
| GEMM floor () | 1.99 | 3.01 | 2.46 |
| GEMM bandwidth () | 1.24 | 0.88 | 1.23 |
| GEMM compute () | 1.02 | 1.01 | 0.98 |
| conv single | 1.25 | 1.01 | 1.35 |
| conv ResNet-like | 1.05 | 1.03 | 1.00 |
| attention ViT () | 1.03 | 0.94 | 1.02 |
| attention long-seq () | 1.02 | 1.00 | 0.98 |
| DFT-as-matmul () | 2.39 | 0.90 | 0.66 |
| 5-pt stencil (, 32 steps) | 1.05 | 1.17 | 1.11 |
| Jacobi (, 25 iters) | 1.02 | 1.06 | 1.04 |
Table S22.
the same sweep with no crossover, and its fitted per-position slope drops from 3.49 to 1.34 μs. The weight-stream intercept is unaffected by the transport, so the linearity and the intercept recovery stand either way.
Both fp32 references reproduce the published torchvision top-1 (69.758 and 81.072 %), which validates the preprocessing and label mapping. McNemar finds no directional change for either fp16 path. ViT-B/16 has the largest ANE relative of the paper’s three real models, on real images against the 2 random-input measurement gives. Even there the ANE’s top-1 is statistically indistinguishable from fp32, and although it changes ten times as many individual labels as the GPU does (0.40 against 0.04 %), those changes are not directional.
The ordering matches the classifiers. fp16 on either device preserves the retrieved neighbor for more than 99 % of queries and the similarity structure to four decimals, and per-channel int8 costs about two points of retrieval agreement. For the two classifiers, where both agreement and accuracy are measurable, they give the same verdict.
Operators excluded from the speed comparison. Sixteen operators reach the engine through a one-shot subprocess bridge, paying the full process-spawn, graph-author and ANE-load cost on every call, so timing them against MLX would measure the bridge overhead (main paper, § 2).
They are sort, cross_product, cross_correlation, cost_volume, fps, radius_search, minmax_norm, lrn, space_to_channel, channel_to_space, space_to_batch, batch_to_space, flatten, input_view, dynamic_slice and scaled_elementwise.
| Device | rel. | top-1 NN | top-10 overlap | similarity Spearman |
| CPU fp32 (reference) | — | 100.00 % | 100.00 % | 1.0000 |
| ANE fp16 | 99.18 % | 99.39 % | 1.0000 | |
| GPU MPS fp16 | 99.78 % | 99.85 % | 1.0000 | |
| ANE int8 | 97.31 % | 97.84 % | 0.9994 |
Table S26. Task-level check for the sentence encoder (MiniLM, 4977 wikitext-2-raw sentences, M5 Pro). For this embedding model, task-level agreement is measured by nearest-neighbor retrieval. Each device embeds the same corpus. top-1 NN is the fraction of sentences whose nearest neighbor under cosine similarity is unchanged from the fp32 reference, top-10 is the mean overlap of the ten nearest, and the last column is the Spearman correlation of the pairwise similarities themselves. These values measure agreement with fp32, not accuracy against ground truth, because no labeled sentence-pair set was available offline.
The three bridge operators that carry a persistent worker (sdpa, argmax, topk) are timed on silicon and stay in the comparison.
Scope of the relative-error column. The per-class relative reported throughout uses seeded Gaussian input. It does not consistently bound the error on real data or preserve the ranking across models. Re-measuring the three real models on their own domains, ResNet-18 falls from to , MiniLM holds at against a reported , and ViT-B/16 rises from to .
ViT-B/16 permits a layerwise decomposition of this discrepancy. We compile encoder prefixes of depth 1 to 12 and compare each against the corresponding fp32 reference. Error grows by a median factor of 1.07 to 1.09 per block for both input types, and through depth 11 the real-over-synthetic ratio stays between 0.85 and 1.30. The entire discrepancy appears at the last block and the classification head. There the absolute error on real images jumps to 2.24× the synthetic figure while the reference logit vector’s RMS falls to of it, and their ratio, 3.34, reproduces the observed 3.33× ratio in relative . Both terms move the same way because a trained classifier on a real image produces a peaked logit vector. A few large entries carry larger absolute fp16 rounding, while the vector’s RMS over all 1000 classes is lower than for the diffuse response to noise. Relative over a logit vector is therefore least informative for confident predictions, the regime that decides the class. The encoder shows ordinary accumulation of fp16 rounding rather than a single-operator failure.
The rail columns separate the two transports. Re-fed, the ANE rail decreases with context (1878–1428 mW) while package power rises, because the engine waits on the host. Resident, the rail is flat between 1915 and 1950 mW and package power is lower. The efficiency ratio turns on this choice alone, from an apparent GPU advantage at long context to a steady ANE advantage of about 2×.
The latency distribution. The probe used throughout the paper reports the minimum wall time over 30 repetitions after warmup; here we also give the median and the mean. We re-ran the probe over the device-map workloads with the full distribution retained, using the same workload builders so that shapes, dtypes and seeds are identical by construction. Across 46 device-and-workload probes the median exceeds the minimum by a median factor of 1.086 (mean 1.117, p90 1.178), and 85 % of probes lie within 25 % of their minimum. Every probe whose median exceeds its minimum by more than 30 % is a GPU or CPU path. group_norm on the GPU is the extreme at 2.51 (per-repetition CV 41 %), followed by GPU attention at 1.58 and GPU convolution at 1.57. The ANE paths sit between 1.02 and 1.09 on every class. Minimum latency benefits the more variable GPU and CPU paths more strongly, so using medians would increase the reported ANE/GPU ratios.
Formulation fairness of the GPU baselines
Kernel fusion and workload formulation are separate confounds. The fused-GPU appendix (Section D) tests the first and finds that MLX kernel-launch overhead does not explain the device map’s large gaps. This appendix audits formulation and remeasures the stencil and the DFT.
Formulation-exposed classes. A class is formulation-exposed when the GPU’s idiomatic expression discards a better algorithm for the same problem. The GEMM, convolution, attention and norm classes are not. Matrix multiply, conv2d and elementwise reductions are the standard formulations tested here, and the fused-GPU appendix additionally checked convolution, attention and two norms under mx.compile. The Jacobi solve is a dense Rx recurrence, which is likewise standard. The stencil and the DFT both have known lower-work formulations. Two classes are exposed. The 5-point stencil is expressed as a dense convolution whose kernel carries four structural zeros, so the GPU reads nine neighbors to use five. The DFT is expressed as two dense matrix–vector products, where a fast transform is .
The formulation-bound stencil. Rewriting the GPU stencil as direct shifted-slice arithmetic on a zero-padded field, , and compiling it with mx.compile, gives 0.382 ms against the dense convolution’s 1.256 ms and 11.6 W against 33.4 W. The rewrite is 3.3× faster and draws 2.9× less power, and it is also slightly more accurate (relative against , both against the same fp64 reference). The optimized baseline preserves the direction but reduces the perf/watt ratio by 9.3×, from 43.8× (95% CI [35.6, 52.0], ) to 4.71× ([3.94, 5.49]), and the latency advantage from 9.2× to 2.8×. The main paper’s device map reports the optimized baseline.
Both devices are close to their dispatch floors at this size. A linear fit to the GPU’s step-count sweep gives 5.87 μs per step plus 0.184 ms fixed, so 49 % of the GPU time at 32 steps is fixed cost, and aneforge’s own cost model flags the ANE program as dispatch-floor-bound. The comparison is asymmetric in the GPU’s favor: it runs hand-written code while the ANE runs the framework’s natural expression. Convolution is the ANE’s native primitive, it has no 5-tap operator, and a shift formulation would place the work on the slow activation path, so we take the comparison to be between each device’s best available expression.
The DFT’s unresolved verdict. Replacing the dense matrix–vector products with mx.fft is only 1.17× faster (0.172 ms against 0.201 ms) despite an 82× smaller operation count, because at every variant is dispatch-bound, not compute-bound. The FFT is, however, times more accurate (relative against ), which is worth stating because this class was chosen as an fp16 cancellation stress and much of that stress is an artifact of the formulation.
Over ten independent cold starts the ANE-over-GPU perf/watt ratio is 1.35 with a 95% CI of [0.96, 1.73], a paired- of 0.073, and a TOST equivalence of 0.93, so neither a difference nor an equivalence is established. The minimum detectable effect at this is 35 % of the pooled mean, and the device map’s single-session ratio (0.82×, the GPU ahead) lies outside the interval. The cause is the power scale. This class draws 3–9 W on the ANE while the idle baseline being subtracted varied over 156–632 mW across cold starts, so the subtraction is a large and drifting fraction of the signal. The DFT result is therefore indeterminate.
The remaining GPU-favorable class. Applying the same protocol to group_norm, the only remaining class where the GPU leads, gives a ratio of 0.949 with a 95% CI of [0.900, 0.997] over ten cold starts (paired- = 0.035, minimum detectable effect 7 % of pooled), reproducing the single-session 0.943. That class draws about 10 W, where the idle drift is a small fraction of the signal. The contrast between the two rows locates the instrument’s working range, not a property of either workload.
Real models. The real-model rows were first measured on torch-MPS, the one framework that differs from the rest of the paper. Re-measuring ResNet-18 against a graph-compiled MLX baseline gives 1.645 ms and 32.6 mJ per inference, against torch-MPS at 2.351 ms and 21.1 mJ, so compiled MLX is 30 % faster while drawing the power and using more energy per inference. The ANE’s energy advantage on this model is 14.6× (95% CI [12.1, 17.2], =5) against compiled MLX and 8.9× ([7.5, 10.7], =16) against torch-MPS. The torch-MPS ResNet-18 energy spans 20–49 mJ over independent cold starts, a coefficient of variation of 30 % against 11 % for the ANE, so most of the uncertainty in the ratio comes from the baseline and not the device under test.
MiniLM and ViT-B/16. The same check on the two transformers needs an MLX implementation of each, and the MLX image library used for ResNet-18 has neither. We ported both to plain MLX, loading the identical sentence-transformers and torchvision weights and using MLX’s fused attention and layer-norm kernels (mx.fast.scaled_dot_product_attention and mx.fast.layer_norm). Run in fp32 on the MLX CPU stream, the ports agree with the torch fp32 references to a relative error of (MiniLM) and (ViT-B/16). In fp16 on the GPU their errors are and , and compilation leaves the output bit-identical. The four arms (ANE, torch-MPS fp16, eager MLX fp16, compiled MLX fp16) were interleaved within one session for ten repetitions (MLX v0.32.2).
On MiniLM, compiled MLX takes 0.650 ms and 11.5 mJ per inference against the ANE’s 0.529 ms and 3.49 mJ, an ANE energy advantage of 3.42× (95% CI [2.99, 3.86]). On ViT-B/16, compiled MLX takes 2.96 ms and 100.4 mJ, against torch-MPS fp16 at 15.8 ms and 598 mJ. The ANE took 18.8 ms in this session, 6.4× the compiled-MLX time (6.2× against the probe latency of the main paper’s Table 6), and its energy advantage falls to 1.53× ([1.26, 1.81]). One ANE repetition read 199 mJ at 10.4 W, against 56–77 mJ for the other nine. We kept it in; dropping the three ANE windows with harness flags as low-confidence gives 1.71× ([1.66, 1.76]). The torch-MPS ratios from the same session, 6.58× on MiniLM and 9.12× on ViT-B/16, match the cold-start values of Table S6. Compiled MLX is the faster GPU baseline on all three models, but it uses less energy than torch-MPS only on the two transformers. Result files record the framework versions alongside each measurement.
Per-operator energy per FLOP. The predictor uses one compute-energy constant per device, fit from the saturating GEMM and convolution sweeps. That quantity is operator-dependent, so we report the spread here rather than leave the constant unqualified. At the device-map operating points the ANE spends 1.33 pJ/FLOP on saturating GEMM, 1.78 and 1.91 on the two attention shapes, and 0.44 on the fused 16-deep convolution stack. The GPU spends 2.00, 3.28, 5.08 and 5.71 on the same four. The roughly fourfold spread on each device does not call for per-class constants, because the model is not a per-class estimator. The fitted constant characterizes single-dispatch work. Deep fusion can fall below it by amortizing the per-call cost across layers, as the 16-deep stack does at 0.44 against the fitted 1.16 pJ/FLOP, for the same reason that puts those points above the compute envelope.
The int8 decode agreement protocol. The greedy-token agreement quoted for int8 decoding is computed on a synthetic 8-layer decoder (, 16 heads, , vocabulary 32 000, KV length 256) with randomly initialized weights and a random batch of 64 decode positions. For each position we take the argmax over the 32 000 logits and compare against an fp64 reference of the same decode, so 92 % is 59 of 64 positions, alongside a top-5 overlap of 93 % and a softmax KL of . The measurement is one of numerical fidelity on random weights, not of task quality on a trained model. The fp16 baseline agrees with fp64 on 95 % of the same positions, so most of the 8 % int8 disagreement is already present in fp16.
The measurement instrument
Every measurement in the main paper is taken through aneforge, the CoreML-free fused-graph dispatch path documented in the ANEForge software paper [5], over the runtime stack described in the companion guide [6]. The software paper establishes how the path reaches the engine. This section states the guarantees and limitations of that path as a measurement instrument.
aneforge requests direct ANE execution without CoreML placement. Rail power, compiler and runtime structure, MLComputePlan, and correctness tests support that placement, but the platform exposes no per-operator rail-exclusive trace, so these are necessary-condition checks and not a direct placement readout.
aneforge accepts a user operator graph and lowers it to a single fused program in Apple’s Model Intermediate Language (MIL), with one MIL operation emitted per graph node and all weights packed into one constant blob. It compiles that program with Apple’s ANE compiler (ANEСCompile) and submits it through Apple’s e5rt execution runtime directly to the engine. There is no CoreML model and therefore no CoreML placement heuristic deciding which operations run where. Measurements are taken under a chosen batch size, fp16 weights and activations throughout, and warm program and cache state.
The first check is the per-rail power attribution of the main paper’s methodology. powermetrics reports the ANE rail separately, and an ANE workload draws measurable power on that rail while it runs, which a CPU or GPU fallback would not produce.
The second check is structural and ships with the artifact. The route and capability gate compiles both lowerings of every selectable operator on the device and asserts they agree within fp16 operation noise, which verifies the set of operators the instrument claims to reach. The correctness corpus runs each shipped operator against a reference, so a node that failed to lower or executed incorrectly would surface as a test failure rather than as a fast but wrong measurement.
The third is a first-party triangulation. Querying Apple’s own planner through MLComputePlan for single-operator fp16 programs, it prefers the Neural Engine for matmul and convolution and the CPU for standalone elementwise operators, the same compute-bound-favors-ANE split the device map reports. It confirms placement, not the per-engine energy CoreML does not expose.
| ANE lat (ms) | ANE tok/s | ANE rail (mW) | GPU lat (ms) | GPU tok/s | GPU/ANE tok/s | ANE/GPU tok/s/W | |
| 128 | 8.238 | 121.4 | 1906 | 4.871 | 205.3 | 1.69 | 1.66 |
| 256 | 8.605 | 116.2 | 1876 | 4.940 | 202.4 | 1.74 | 1.69 |
| 512 | 9.482 | 105.5 | 1822 | 5.116 | 195.4 | 1.85 | 1.56 |
| 1024 | 11.411 | 87.6 | 1644 | 5.499 | 181.8 | 2.07 | 1.10 |
| 2048 | 14.885 | 67.2 | 1431 | 6.205 | 161.2 | 2.40 | 0.81 |
Table S24. Decode against KV-cache length at (M5 Pro), the axis the main paper’s decode table holds fixed at 256. In this sweep the GPU cache is resident while the ANE cache is re-fed as a graph input on every step, so the last column is not a like-for-like device comparison. It is retained for comparison with Table S27.
| arm | lat (ms) | pkg (W) | ANE rail (mW) | CPU rail (mW) | ANE/GPU tok/s/W | |
| 256 | ANE re-fed | 8.600 | 3.75 | 1878 | 1730 | 1.79 |
| ANE resident | 8.089 | 2.83 | 1950 | 884 | 2.56 | |
| GPU | 4.918 | 11.42 | — | 4692 | — | |
| 1024 | ANE re-fed | 11.384 | 4.96 | 1640 | 3316 | 1.09 |
| ANE resident | 9.124 | 2.60 | 1950 | 652 | 2.63 | |
| GPU | 5.499 | 11.15 | — | 4036 | — | |
| 2048 | ANE re-fed | 14.872 | 5.86 | 1428 | 4428 | 0.87 |
| ANE resident | 10.492 | 3.68 | 1915 | 1740 | 1.97 | |
| GPU | 6.223 | 12.09 | — | 4563 | — |
Table S27. KV-cache residency and its cost (M5 Pro, , TinyLlama-class decoder of Table S24). The GPU path holds the cache resident in device arrays across steps. The ANE path can either re-feed it as a graph input every step or hold it in the engine’s own input buffers and re-run without rebinding. Both produce identical logits (relative exactly zero), so the difference is transport and not arithmetic. Re-feeding charges the ANE a host-side copy the GPU never pays, and at that copy is 128 MB per token.
References
- [1]Apple Machine Learning Research. 2022. Deploying Transformers on the Apple Neural Engine. https://machinelearning.apple.com/research/neural-engine-transformers. Apple Machine Learning Research article.
- [2]Afsara Benazir and Felix Xiaozhu Lin. 2026. Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs. arXiv:2604.18788 [cs.LG]arxiv.org/abs/2604.18788
- [4]Spencer H. Bryngelson. 2026. ANEForge: CoreML-free programming of the Apple Neural Engine. https://github.com/sbryngelson/ANEForge. doi:10.5281/zenodo.24210581 Software artifact, release artifact-v1.
- [5]Spencer H. Bryngelson. 2026. ANEForge: Python for Direct Computation on the Apple Neural Engine. Software: Practice and Experience (2026). doi:10.1002/spe.70124DOI
- [6]Spencer H. Bryngelson. 2026. The Apple Neural Engine: A Complete Guide. arXiv preprint arXiv:2606.22283 (2026). doi:10.48550/arXiv.2606.22283
- [7]Peter Clower. 2024. Apple AMX Instruction Set. https://github.com/corsix/amx. Reverse-engineered documentation of Apple’s undocumented AMX matrix coprocessor.
- [8]Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yupin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25). Seoul, Republic of Korea. doi:10.1145/3731569.3764808DOI
- [9]Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016. Eyeris: A Spatial Architecture for Energy-Efficient Dataflow Convolutional Neural Networks. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). 367–379. doi:10.1109/ISCA.2016.40DOI
- [10]Jee Hwan Choi, Daniel Bedard, Robert Fowler, and Richard Vuduc. 2013. A Roofline Model of Energy. In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing (IPDPS). 661–672. doi:10.1109/IPDPS.2013.77DOI
- [11]Nan Ding and Samuel Williams. 2019. An Instruction Roofline Model for GPUs. In 2019 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). 7–18. doi:10.1109/PMBS49563.2019.00007DOI
- [12]Anastasios Fanariotis, Theofanis Orphanoudakis, and Vasilis Fotopoulos. 2025. Evaluating the Energy Efficiency of NPU-Accelerated Machine Learning Inference on Embedded Microcontrollers. arXiv:2509.17533arxiv.org/abs/2509.17533
- [13]Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. EIE: Efficient Inference Engine on Compressed Deep Neural Network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). 243–254. doi:10.1109/ISCA.2016.30DOI
- [14]Matthijs Hollance. 2020. The Neural Engine – What Do We Know About It? https://github.com/hollance/neural-engine. Community-maintained repository.
- [15]Paul Hübner, Andong Hu, Ivy Peng, and Stefano Markidis. 2025. Apple vs. Oranges: Evaluating the Apple Silicon M-Series SoCs for HPC Performance and Efficiency. arXiv:2502.05317 [cs.AR]arxiv.org/abs/2502.05317
- [16]Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Luc Van Gool. 2019. AI Benchmark: All About Deep Learning on Smartphones in 2019. arXiv:1910.06663arxiv.org/abs/1910.06663
- [17]Aleksandar Ilic, Frederico Pratas, and Leonel Sousa. 2014. Cache-Aware Roofline Model: Upgrading the Loft. IEEE Computer Architecture Letters 13, 1 (2014), 21–24. doi:10.1109/L-CA.2013.6
- [18]Rakshith Jayanth, Neelesh Gupta, and Viktor Prasanna. 2024. Benchmarking Edge AI Platforms for High-Performance ML Inference. arXiv:2409.14803arxiv.org/abs/2409.14803
- [19]Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantín, Clifford Cho, Chris Clark, Jeremy Chvatal, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William G. lland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA ’17). 1–12. doi:10.1145/3079856.3080246DOI
- [20]Ramchand Kumaresan. 2026. Orion: Characterizing and Programming Apple’s Neural Engine for LLM Training and Inference. arXiv:2603.06728 [cs.LG] Concurrent preprint.arxiv.org/abs/2603.06728
- [21]Stefanos Laskaridis, Kleomenis Kateveas, Lorenzo Minto, and Hamed Haddadi. 2024. MELting Point: Mobile Evaluation of Language Transformers. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking (MobiCom). doi:10.1145/3636534.3690668DOI
- [22]Yu Jung Lo, Samuel Williams, Brian Van Straalen, Terry J. Ligocki, Matthew J. Cordery, Nicholas J. Wright, Mary W. Hall, and Leonid Oliker. 2015. Roofline Model Toolkit: A Practical Tool for Architectural and Program Analysis. In High Performance Computing Systems. Performance Modeling, Benchmarking and Simulation (PMBS 2014) (Lecture Notes in Computer Science, Vol. 8966). 129–148. doi:10.1007/978-3-319-17248-4_7DOI
- [23]ML.ENERGY / Zeus. 2025. ML.ENERGY Energy Consumption Measurement on Apple Silicon (macOS). https://github.com/ml-energy/zeus/issues/159. GitHub issue #159.
- [24]Seungjae Moon, Junseo Cha, Hyunjun Park, and Joo-Young Kim. 2025. GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context Window. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA). 808–820. doi:10.1145/3695053.3713051DOI
- [25]S. K. Prashanthi, Kunal Kumar Sahoo, Samotya Ranjan Saikia, Pranay Gupta, Atharva Vinay Joshi, Priyanshu Pansari, and Yogesh Simmhan. 2025. Pagoda: An Energy and Time Roofline Study for DNN Workloads on Edge Accelerators. arXiv:2509.20189 [cs.DC]arxiv.org/abs/2509.20189
- [26]Vijay Jayaraddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, et al. 2020. MLPerf Inference Benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). 446–459. doi:10.1109/ISCA45697.2020.00045
- [27]Kiran Seshadri, Berkin Akın, James Laudon, Ravi Narayanaswami, and Amir Yazdanbakhsh. 2022. An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks. In 2022 IEEE International Symposium on Workload Characterization (IISWC). 79–91. doi:10.1109/IISWC55918.2022.00017
- [28]Manjeet Singh. 2026. Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering. https://maderix.substack.com/p/inside-the-m4-apple-neural-engine; repo: https://github.com/maderix/ANE. Blog series and repository.
- [29]Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal, and Kautuk Kundan. 2026. LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load. arXiv:2603.23460 [cs.DC]
- [30]Marian Verhelst, Luca Benini, and Naveen Verma. 2025. How to Keep Pushing ML Accelerator Performance? Know Your Rooflines! IEEE Journal of Solid-State Circuits (2025). doi:10.1109/JSSC.2025.3553765DOI
- [31]Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM 52, 4 (2009), 65–76. doi:10.1145/1498765.1498785
- [32]Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2025. Fast On-device LLM Inference with NPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Volume 1. doi:10.1145/3669940.3707239DOI
- [33]Charlenne Yang, Thorsten Kurth, and Samuel Williams. 2020. Hierarchical Roofline analysis for GPUs: Accelerating performance optimization for the NERSC-9 Perlmutter system. Concurrency and Computation: Practice and Experience 32, 20 (2020), e5547. doi:10.1002/cpe.5547DOI
- [34]Eileen Yoon. 2022. Are Reverse-Engineered Linux Driver for the Apple Neural Engine (+anecc). https://github.com/eiln/ane; https://github.com/eiln/anecc. Asahi Linux driver and compiler repositories.




