betaFeedback

WorksComputer Science

Accelerating LLM Decoding with In-DRAM Parallelism

Abstract

Autoregressive decoding of large language models (LLMs) on edge devices is dominated by memory-bound general matrix–vector multiplication (GeMV) operations. Processing-using-DRAM (PUD) is a promising computing paradigm that can alleviate this bottleneck by reducing processor–memory data transfers and enabling massively parallel computation within DRAM (e.g., across 65,536 columns). However, weight matrices in individual LLM layers (e.g., a QKV matrix in Llama3 8B with 6,144 rows) are typically too small to fully utilize parallelism in PUD. To address this challenge, we propose an algorithm–system co-design for PUD-based LLM inference. At the algorithm level, we enable a subsequent projection to be packed with the current projection by reformulating it as a computation on the current residual state plus a correction for the residual update. Layer redundancy allows this correction to be approximated using fewer bit-serial steps. At the system level, we tailor data mapping and execution to improve utilization of either columns within DRAM subarrays (SIMD) or independently controlled banks (MIMD). Overall, we achieves up to 1.41×1.41\times end-to-end decoding speedup over the PUD baseline, corresponding to a 2.45×2.45\times speedup over processor-based execution.

Introduction

Efficient inference for large language models (LLMs) is important for edge devices with limited power and hardware resources. Autoregressive decoding on these devices runs at batch size one, where each linear projection becomes a matrix–vector multiplication (GeMV) between a weight matrix and an activation vector. These operations have low arithmetic intensity because they perform little computation per weight fetched from memory. Consequently, decoding on CPUs and GPUs is typically bottlenecked by weight transfers, leaving much of their computational capacity unused [2, 4, 3]. In this regime, increasing computational throughput alone offers limited improvements in decoding latency.

Processing-in-memory (PIM) is a promising computing paradigm to address this bottleneck by performing computation near or within memory, reducing costly data movement [5]. Processing-near-memory places dedicated compute units close to memory arrays [6]. However, integrating compute units into DRAM incurs significant area overhead and increases design complexity under the constraints of DRAM fabrication. In contrast, processing-using-DRAM (PUD) is a promising computing paradigm that leverages the operational principles of existing DRAM cells and sense amplifiers to perform massively parallel computation directly within DRAM arrays.

Figure 1. PUD is a promising computing paradigm for memory-bound LLM decoding. PUD-LLM unlocks its underutilized parallelism through algorithm–system co-design.

PUD exploits DRAM’s column-level parallelism to achieve high computational throughput [8, 11]. DRAM command sequences implement bitwise primitives that are combined into bit-serial arithmetic operations [11, 7]. Each primitive operates across many DRAM columns simultaneously, enabling single-instruction, multiple-data (SIMD) execution [9]. Independently controlled regions, such as channels and banks, further provide multiple-instruction, multiple-data (MIMD) parallelism [12, 10]. Although bit-serial arithmetic requires long sequences of DRAM commands, prior work [13, 67] demonstrates that PUD can outperform CPU and GPU execution for sufficiently large low-bit matrix–vector multiplications by leveraging its massive parallelism and exploiting bit sparsity to reduce the number of required DRAM commands.

However, LLM matrix sizes are typically insufficient to fully utilize this parallelism. Moreover, the sequential dependencies between layers prevent overlapping their execution during batch-size-one decoding. As a result, PUD parallelism remains underutilized, significantly limiting end-to-end LLM inference performance. For example, the QKV projection in Llama 3.1 8B [61], with a combined output dimension of 6,144, uses only 24,576 of the 65,536 available DRAM columns with 4-bit weights, leaving 62.5% of the column-level parallelism unused.

System-level optimizations alone cannot fully resolve this challenge while preserving the original computational dependencies. Prior work improves PIM utilization through data placement [27], fine-grained resource control [10], and scheduling across independent sub-batches [29]. However, these approaches preserve the sequential dependencies between consecutive LLM sublayers. Consequently, parallel computations remain limited to operations within the current layer, leaving substantial PUD resources underutilized.

Figure 2. Our system design improves column (SIMD) utilization through packing and bank (MIMD) utilization through pipelining.

To address this challenge, we propose PUD-LLM, a hardware-aware algorithm and system co-design for PUD-based LLM inference. At the algorithm level, we use linearity to decompose a subsequent projection into a lookahead computation on the current residual state and a correction based on the residual update. Although this decomposition increases the total computational work, it reduces decoding latency in two ways. First, the lookahead computation overlaps with the preceding sublayer by exploiting otherwise idle PUD resources. Second, the similarity between consecutive representations in LLMs, known as layer redundancy [34], allows the correction to be approximated using fewer bit-serial computation steps.

At the system level, we develop two execution schemes to improve utilization of PUD’s column-level (SIMD) and bank-level (MIMD) parallelism as illustrated in Figure 2. The SIMD scheme reformulates Gate/Up lookahead to share QKV’s input and maps its weights to otherwise unused DRAM columns, allowing both projections to execute under a common instruction stream. The MIMD scheme applies the decomposition to both QKV and Gate/Up, allowing each projection’s correction and its successor’s lookahead to execute in parallel on independently controlled banks or channels. These schemes improve utilization within subarrays and across independent resources, respectively.

In summary, our contributions are as follows:

  • We identify how individual projection dimensions and sequential sublayer dependencies leave PUD parallelism underutilized in PUD-based LLM decoding.

  • We propose a training-free algorithm that exposes parallelism across sublayers and exploits inter-layer redundancy to reduce bit-serial steps on the critical path.

  • We develop system-level optimizations for two execution schemes that exploit unused SIMD or MIMD parallelism available in a given model–memory configuration.

  • PUD-LLM achieves up to 1.41×1.41\times end-to-end decoding speedup over the PUD baseline, with 1.85×1.85\times higher mean PUD resource utilization.

Background and Motivation

This section reviews Transformer-based LLMs, DRAM, and PUD, highlighting how PUD executes GeMV and why its parallelism remains underutilized during LLM decoding.

Transformer-based LLM

A Transformer model [1] consists of a stack of blocks, each containing an attention sublayer followed by a feed-forward network (FFN) sublayer. Both sublayers share a common structure: input linear projections, intermediate nonlinear processing, and an output linear projection. In attention, the input is projected into queries, keys, and values, followed by self-attention and an output projection back to the model’s hidden dimension. A common FFN design uses gate and up projections, followed by nonlinear gating, element-wise multiplication, and a down projection back to the hidden dimension [14]. Each sublayer operates on a normalized input and adds its output back to the original input via a residual connection:

x′=x+WoutΦ ⁣(WinNorm(x)),\begin{equation} \mathbf{x}' = \mathbf{x} + \mathbf{W}_{\mathrm{out}} \Phi\!\left( \mathbf{W}_{\mathrm{in}} \mathrm{Norm}(\mathbf{x}) \right), \tag*{(1)} \end{equation}

where x\mathbf{x} and x′\mathbf{x}' denote the residual states before and after the update, respectively. The matrix Win\mathbf{W}_{\mathrm{in}} vertically concatenates the weight matrices of the input projections, Φ\Phi denotes the intermediate processing such as attention or nonlinear gating, and Wout\mathbf{W}_{\mathrm{out}} denotes the output projection. This expression summarizes a standard sublayer; architectural variants may incorporate expert routing [15] or alternative attention [64].

Processing-using-DRAM

Figure 3. (a) DRAM organization and (b) PUD primitives.

DRAM Organization and Operation.

A DRAM memory system comprises channels, ranks, chips, banks, and subarrays, as shown in Figure 3(a) [16]. Each rank groups chips that operate in lockstep under shared command and address signals. Within a subarray, one-bit cells are arranged in rows and columns. Cells in each row share a wordline, while those in each column connect to a bitline and its sense amplifier (SA). The SAs collectively form the local row buffer. An ACT command opens a row, allowing the SAs to sense, amplify, and restore its data. READ and WRITE commands transfer selected portions of the open row over the data bus, while PRE closes the row and precharges the bitlines for the next activation.

In-DRAM Computation.

PUD exploits the analog behavior of DRAM cells and sense amplifiers to perform row copying and bitwise logic without transferring data outside DRAM (Figure 3(b)). Row copying transfers data between rows within the same subarray through shared sense amplifiers and bitlines [8], and simultaneous activation of multiple rows enables majority operations [11]. These operations can be implemented using timing-violating command sequences in unmodified DRAM (u-PUD) [7] or additional hardware support in modified DRAM (m-PUD) [11]. Multi-bit arithmetic is implemented through sequences of these bitwise operations. Applying a common operation across many columns provides single-instruction, multiple-data (SIMD) parallelism [9]. Interleaving commands to different banks over a channel’s shared command/address bus allows independent operation sequences to execute concurrently, providing multiple-instruction, multiple-data (MIMD) parallelism [12, 10].

GeMV Execution in DRAM

Figure 4. PUD-based GeMV consists of two stages: (1) in-DRAM computation with cost proportional to the number of retained nonzero input terms; (2) partial-sum readout for reduction.

Execution Principle.

Prior work [13, 67] demonstrates that PUD can outperform processor-based execution for sufficiently large low-bit matrix–vector multiplications by exploiting massive parallelism and input bit-level sparsity. We follow the data mapping and execution scheme of MVDRAM [13], as illustrated in Figure 4. Consider y=Wx\mathbf{y}=\mathbf{W}\mathbf{x}, where W∈RM×K\mathbf{W}\in\mathbb{R}^{M\times K} has input dimension KK and output dimension MM. Each weight vector W:,k\mathbf{W}_{:,k} is mapped to a DRAM row, with output dimensions and weight-bit positions mapped to columns. For bwb_w-bit weights, each row requires MbwMb_w columns. Thus, output dimension and weight precision determine SIMD utilization for a given column capacity. Execution consists of PUD computation and host-side reduction. During computation, the host translates each nonzero bit of xkx_k into DRAM commands that accumulate the appropriately scaled contribution of W:,k\mathbf{W}_{:,k} from its stored row, exploiting input bit-level sparsity. During reduction, the partial sums within DRAM are transferred to the host and combined to obtain the output.

Quality–Efficiency Trade-off.

In this PUD GeMV execution scheme, computation time scales approximately with the number of nonzero terms. Using more terms improves numerical accuracy but increases latency, creating a quality–efficiency trade-off, as illustrated in Figure 5 (left). This trade-off can be controlled at runtime by truncating less significant terms under a specified budget [45]. Our goal is to improve this Pareto frontier by optimizing end-to-end LLM inference beyond individual GeMV kernels.

Figure 5. Left: Schematic quality–efficiency trade-off in bit-serial computation. Right: Partition-dependent latency of PUD.

Tensor Parallelism.

The input dimensions of LLM weight matrices exceed the row capacity of a DRAM subarray (typically 512–1024 rows), requiring input partitioning. Input partitioning is a form of tensor parallelism [17], decomposing the projection into independent partial products: y=∑n=1NWnxn\mathbf{y}=\sum_{n=1}^{N}\mathbf{W}_n\mathbf{x}_n. These partial products can execute concurrently on independently controlled banks or channels, exploiting MIMD parallelism. Each partition processes K/NK/N input elements, where NN is the input partition count. Increasing NN therefore reduces computation per partition but increases the number of partial sums to read and aggregate, creating a trade-off between computation and reduction.

Partitioning Optimization.

We model this trade-off to select the input partition count that minimizes GeMV latency. Let τ\tau denote the average number of retained nonzero terms per input element. Assuming NN is large enough to saturate the readout bandwidth but small enough for all partitions to execute concurrently, latency is approximated by

T(N)=αKτN+βMN,\begin{equation} T(N)=\frac{\alpha K\tau}{N}+\beta MN, \tag*{(2)} \end{equation}

where α\alpha is the PUD computation cost per term and β\beta is the reduction cost per output element per partition. Figure 5 (right) illustrates this trade-off. The unconstrained continuous optimum is N⋆=αKτ/(βM)N^\star=\sqrt{\alpha K\tau/(\beta M)}, at which computation and reduction contribute equally. The cost coefficients α\alpha and β\beta are detailed in Appendix § 8. The selected partition count determines how many independent resources can be used, and hence MIMD utilization.

Challenges in PUD-Based LLM Inference

Figure 6. PUD mapping and execution of a Transformer block.

Figure 6 shows the mapping and execution of a Transformer block on PUD. We offload QKV and Gate/Up to PUD, exploiting their wide output dimensions to accelerate these projections through column parallelism, while the remaining operations execute on the CPU. However, QKV and Gate/Up execute sequentially due to dependencies between sublayers. Each projection’s output dimension and weight precision determine its column demand, while its input partition count determines the number of execution resources used concurrently. When these demands fall below the available capacity, SIMD or MIMD parallelism remains underutilized. We illustrate these two cases using Llama 3.1 8B on two memory configurations.

Case 1: Underutilized SIMD Parallelism.

On a u-PUD system with one 8 GiB single-rank DDR4 DIMM and 16 banks on a single channel, 4-bit Llama 3.1 8B uses 16 latency-optimal input partitions for QKV, occupying all 16 banks. However, its 6,144 output dimensions require only 24,576 of the 65,536 available columns per partition, leaving 62.5% of the SIMD capacity unused.

Case 2: Underutilized MIMD Parallelism.

On an m-PUD system with one 16 GiB dual-rank DDR4 DIMM per channel across two channels, the same model’s QKV and Gate/Up projections occupy 32 and 30 of the 42 effective MIMD resources, respectively, under latency-optimal input partitioning (Appendix § 8). This leaves 23.8% and 28.6% of the MIMD capacity unused.

Algorithm Design

This section presents an algorithm that exploits inter-layer redundancy in LLMs to expose parallel work for PUD.

Figure 7. Our algorithm decomposes a subsequent projection into lookahead that uses PUD parallelism to execute alongside the current projection and a correction approximated with fewer terms by exploiting layer redundancy.

Algebraic Transformation

Let x\mathbf{x} denote the current residual state and u\mathbf{u} the update produced by the current sublayer. The successor’s input is

x′=x+u.\begin{equation} \mathbf{x}' = \mathbf{x} + \mathbf{u}. \tag*{(3)} \end{equation}

Ordinarily, the successor’s input projection of x′\mathbf{x}' must wait until the current sublayer produces u\mathbf{u}. We restructure this dependency by separating the contributions of x\mathbf{x} and u\mathbf{u}.

Let Win′\mathbf{W}'_{\mathrm{in}} and γ′\boldsymbol{\gamma}' denote the successor’s projection weights and learned normalization gains, respectively. Writing Norm(x′)=diag⁡(γ′)x′/ρ(x′)\mathrm{Norm}(\mathbf{x}') = \operatorname{diag}(\boldsymbol{\gamma}')\mathbf{x}'/\rho(\mathbf{x}'), where ρ\rho is the input-dependent RMS normalization factor, we absorb the gains into W~′=Win′diag⁡(γ′)\widetilde{\mathbf{W}}' = \mathbf{W}'_{\mathrm{in}}\operatorname{diag}(\boldsymbol{\gamma}'). By linearity, we obtain

Win′Norm(x′)=1ρ(x′)(W~′x⏟lookahead+W~′u⏟delta).\begin{equation} \mathbf{W}'_{\mathrm{in}}\mathrm{Norm}(\mathbf{x}') = \frac{1}{\rho(\mathbf{x}')} \left( \underbrace{\widetilde{\mathbf{W}}'\mathbf{x}}_{\text{lookahead}} + \underbrace{\widetilde{\mathbf{W}}'\mathbf{u}}_{\text{delta}} \right). \tag*{(4)} \end{equation}

As illustrated in Figure 7, the decomposition enables lookahead to run concurrently with the current sublayer using the already available state x\mathbf{x}. Although it splits one projection into two computations, it requires only one readout and reduction. To achieve this, we retain the lookahead partial results in DRAM without readout or reduction. Once u\mathbf{u} becomes available, we compute delta and accumulate it into these partial results. We then read out and reduce the combined results and apply the normalization factor.

Why Is This Algorithm Particularly Effective on PUD?

The benefit of this decomposition depends on the ability to overlap the current projection with the successor’s lookahead, which uses a different weight matrix. In memory-bound processor-based execution, these computations compete for weight-transfer bandwidth. In contrast, PUD performs computation within DRAM, allowing lookahead to exploit otherwise idle computational resources without additional weight transfers to the host. This distinction reflects two different forms of parallelism. Token-parallel methods [19, 20] exploit processor-side parallelism by processing multiple input vectors with the same weight matrix, increasing weight reuse, whereas our design exploits in-memory parallelism by concurrently executing computations on different weight matrices.

Efficient Approximation via Layer Redundancy

The decomposition replaces one matrix–vector product with two, increasing arithmetic work. Even if lookahead is fully overlapped, the decomposition alone does not reduce latency. We therefore exploit redundancy across LLM layers to approximate the delta projection with fewer terms.

It has been shown that residual states in LLMs change relatively little between consecutive sublayers [41, 34]. For a residual update u\mathbf{u} and the updated state x′=x+u\mathbf{x}'=\mathbf{x}+\mathbf{u}, we define the relative update norm as η=∥u∥2/∥x′∥2\eta=\|\mathbf{u}\|_2/\|\mathbf{x}'\|_2. Figure 8 shows how η\eta varies across model depth on WikiText-2 [46].

Figure 8. Layer redundancy in LLMs.

A small residual update allows a coarser approximation at the same relative state-error tolerance. Let v^τ\widehat{\mathbf{v}}_{\tau} approximate v\mathbf{v} using an average of τ\tau significant nonzero terms per input element, with relative error ev(τ)=∥v^τ−v∥2 / ∥v∥2.e_{\mathbf{v}}(\tau) = \|\widehat{\mathbf{v}}_{\tau}-\mathbf{v}\|_2 \,/\, \|\mathbf{v}\|_2. Assuming exact lookahead computation, approximating the update as u^τdelta\widehat{\mathbf{u}}_{\tau_{\mathrm{delta}}} yields the reconstructed state x^′=x+u^τdelta\widehat{\mathbf{x}}'=\mathbf{x}+\widehat{\mathbf{u}}_{\tau_{\mathrm{delta}}} with relative error

∥x^′−x′∥2∥x′∥2=∥u∥2∥x′∥2∥u^τdelta−u∥2∥u∥2=η eu(τdelta).\begin{equation} \frac{\|\widehat{\mathbf{x}}'-\mathbf{x}'\|_2} {\|\mathbf{x}'\|_2} = \frac{\|\mathbf{u}\|_2}{\|\mathbf{x}'\|_2} \frac{\|\widehat{\mathbf{u}}_{\tau_{\mathrm{delta}}} -\mathbf{u}\|_2} {\|\mathbf{u}\|_2} = \eta\,e_{\mathbf{u}}(\tau_{\mathrm{delta}}). \tag*{(5)} \end{equation}

Thus, when η<1\eta<1, achieving a target relative error ε\varepsilon in the full state requires an update approximation error of at most ε/η\varepsilon/\eta, allowing a coarser approximation of the update. To estimate how many fewer terms this permits, we model the relative approximation error as a function of the term count:

ev(τ)=avexp⁡(−cvτ),av,cv>0.\begin{equation} e_{\mathbf{v}}(\tau) = a_{\mathbf{v}}\exp(-c_{\mathbf{v}}\tau), \qquad a_{\mathbf{v}},c_{\mathbf{v}}>0. \tag*{(6)} \end{equation}

For simplicity, we assume common coefficients, i.e., av=aa_{v} = a and cv=cc_{v} = c. Under this model, matching the relative state error of direct approximation requires η eu(τdelta)=ex′(τ)\eta\,e_{\mathbf{u}}(\tau_{\mathrm{delta}}) =e_{\mathbf{x}'}(\tau), yielding

τdelta=τ+log⁡ηc.\begin{equation} \tau_{\mathrm{delta}} = \tau+\frac{\log\eta}{c}. \tag*{(7)} \end{equation}

For 0<η<10<\eta<1, delta approximation requires fewer terms than direct approximation at the same error. When lookahead latency is fully hidden, the reduced cost of delta computation translates into a shorter critical path.

Figure 9. Execution flows and data mappings of the PUD baseline and the two proposed schemes. SIMD executes QKV and Gate/Up lookahead under a shared instruction stream. MIMD overlaps the current projection’s delta with the successor’s lookahead.

System Design

This section describes how to efficiently execute the parallel work exposed by lookahead–delta decomposition. We present two execution schemes tailored to underutilized SIMD and MIMD parallelism, respectively, then describe their data mappings and optimize input partitioning.

Execution Schemes

Different model–memory configurations leave different forms of parallelism underutilized. We develop two execution schemes: one uses SIMD parallelism to execute lookahead alongside the current projection, while the other uses MIMD parallelism to pipeline lookahead and delta computations across successive sublayers. Figure 9 shows their execution flows, while Figure 10 illustrates their timelines for representative configurations, using input partitioning optimized as described in the next subsection.

SIMD Execution through Packing

We apply lookahead–delta decomposition only to Gate/Up, allowing its lookahead to execute alongside direct QKV computation. SIMD execution requires the two projections to share an input and instruction stream. However, the Gate/Up lookahead in Equation (4) uses x\mathbf{x}, whereas QKV uses the normalized input x~=x/ρ(x)\tilde{\mathbf{x}}=\mathbf{x}/\rho(\mathbf{x}). To share QKV’s input, we rewrite the Gate/Up projection as

W~′x′ρ(x′)=ρ(x)ρ(x′)[W~′x~⏟lookahead+W~′uρ(x)].\begin{equation} \widetilde{\mathbf{W}}' \frac{\mathbf{x}'}{\rho(\mathbf{x}')} = \frac{\rho(\mathbf{x})}{\rho(\mathbf{x}')} \left[ \underbrace{ \widetilde{\mathbf{W}}'\tilde{\mathbf{x}} }_{\text{lookahead}} + \widetilde{\mathbf{W}}' \frac{\mathbf{u}}{\rho(\mathbf{x})} \right]. \tag*{(8)} \end{equation}

As shown in Figure 9, execution begins with QKV and Gate/Up lookahead. After QKV completes, the CPU reads and reduces its partial sums, then executes attention and the Out projection to produce u\mathbf{u}. Once both u\mathbf{u} and the lookahead result are available, Gate/Up delta accumulates into the lookahead result retained in DRAM. The combined result is then read out, reduced, and scaled by ρ(x)/ρ(x′)\rho(\mathbf{x})/\rho(\mathbf{x}').

Figure 10. Execution timelines comparing the PUD baseline with (a) SIMD packing and (b) MIMD pipelining.

MIMD Execution through Pipelining

MIMD execution allows independently controlled resources to process different inputs with separate instruction streams. To exploit this parallelism, we apply lookahead–delta decomposition to both QKV and Gate/Up, overlapping each projection’s delta with the successor’s lookahead. Figure 9 shows the dependencies enabling this execution. Each residual update makes inputs available for two independent computations. Once the attention update u\mathbf{u} is available, the GeMV components of Gate/Up delta and the next block’s QKV lookahead are

W~G(ℓ)u⏟Gate/Up delta,W~Q(ℓ+1)(x+u)⏟next QKV lookahead.\begin{equation} \underbrace{ \widetilde{\mathbf{W}}_{G}^{(\ell)}\mathbf{u} }_{\text{Gate/Up delta}}, \qquad \underbrace{ \widetilde{\mathbf{W}}_{Q}^{(\ell+1)} (\mathbf{x}+\mathbf{u}) }_{\text{next QKV lookahead}}. \tag*{(9)} \end{equation}

Figure 10 illustrates the resulting pipeline. QKV delta overlaps Gate/Up lookahead, while Gate/Up delta overlaps the next block’s QKV lookahead. Each delta accumulates into its retained lookahead result and can begin once both its input update and that result are available.

Lookahead can overlap with result readout because full DQ utilization does not necessarily imply full C/A utilization, as illustrated in Figure 11. Each READ command initiates a data burst spanning multiple cycles, leaving C/A slots available between READ commands. The controller can use these slots to issue PUD commands to separate banks while DQ remains fully occupied with data transfers, subject to shared timing and resource constraints.

Figure 11. Overlapping data transfer and PUD computation.

Data Mapping and Tensor Partitioning

We model latency under each scheme’s data mapping. Let KK denote the common input dimension of QKV and Gate/Up, and let MQM_Q and MGM_G denote their respective output dimensions. Let HQH_Q and HGH_G denote the CPU processing times following the QKV and Gate/Up, respectively. These include attention for HQH_Q, and Down for HGH_G.

PUD-LLM (SIMD).

Executing QKV and Gate/Up lookahead under a shared instruction stream requires their weights to be mapped within the same subarrays. The paired projections use a common partition count NN. When packing does not lengthen the QKV instruction stream, Gate/Up lookahead is fully hidden by QKV computation. The decoder-block latency is then

Tsimd(N)=αK(τQ+τΔ,G)N+β(MQ+MG)N+HQ+HG,\begin{equation} \begin{aligned} T_{\mathrm{simd}}(N) ={}& \frac{\alpha K(\tau_Q+\tau_{\Delta,G})}{N} \\ &+\beta(M_Q+M_G)N+H_Q+H_G, \end{aligned} \tag*{(10)} \end{equation}

where τQ\tau_Q and τΔ,G\tau_{\Delta,G} are the average retained term counts per input element for direct QKV and Gate/Up delta, respectively. The optimum is Nsimd⋆=αK(τQ+τΔ,G)β(MQ+MG).N_{\mathrm{simd}}^\star = \sqrt{ \frac{\alpha K(\tau_Q+\tau_{\Delta,G})} {\beta(M_Q+M_G)} }.

PUD-LLM (MIMD).

Executing QKV and Gate/Up with independent instruction streams requires their weights to be mapped to independently controlled resource groups, such as different banks. The two projections can use separate partition counts NQN_Q and NGN_G. Lookahead overlaps the concurrent delta computation, reduction, and subsequent CPU processing. Let LGL_G and LQL_Q denote the execution times of Gate/Up lookahead and the next block’s QKV lookahead, respectively. Assuming both computations in each pair can start concurrently, lookahead is fully hidden when

Lj≤αKτΔ,iNi+βMiNi+Hi,\begin{equation} L_j \le \frac{\alpha K\tau_{\Delta,i}}{N_i} +\beta M_iN_i+H_i, \tag*{(11)} \end{equation}

where (i,j)∈{(Q,G),(G,Q)}(i,j)\in\{(Q,G),(G,Q)\} and τΔ,i\tau_{\Delta,i} is the average term count per input element for projection ii’s delta. Under these conditions, the decoder-block latency is

Tmimd(NQ,NG)=∑i∈{Q,G}(αKτΔ,iNi+βMiNi+Hi),\begin{equation} T_{\mathrm{mimd}}(N_Q,N_G) = \sum_{i\in\{Q,G\}} \left( \frac{\alpha K\tau_{\Delta,i}}{N_i} +\beta M_iN_i+H_i \right), \tag*{(12)} \end{equation}

and the unconstrained continuous optimum for each projection is Nmimd,i⋆=αKτΔ,i/(βMi)N_{\mathrm{mimd},i}^\star =\sqrt{\alpha K\tau_{\Delta,i}/(\beta M_i)}. Reducing the delta term count therefore favors fewer input partitions, also reducing partial-sum traffic.

Scheme Selection.

We first optimize input partitioning within each scheme, then select the scheme with the lowest predicted latency. The SIMD scheme is favorable when the paired projections fit within otherwise unused columns without requiring additional output tiles. The MIMD scheme can be favorable when sufficient independent resources and overlap time are available to hide lookahead. Figure 12 shows how the latency-minimizing partition count and preferred scheme depend on the model–memory configuration.

Figure 12. GeMV latency versus partition count.

Evaluation

We evaluate the decoding performance of PUD-LLM and analyze how acceleration of the targeted projections contributes to the overall speedup. We also examine the utilization, quality–efficiency trade-off, and sensitivity to host-side optimizations and hardware configurations.

Figure 13. Latency breakdown, normalized to the PUD baseline. Stacked segments show operator contributions to the critical path, with host operators in gray and QKV and Gate/Up in blue. Thick outlines identify the fastest method in each group; annotations show its speedup over the corresponding PUD baseline. Lower bars indicate better performance.

Experimental Setup

Models and Hardware.

We evaluate ten models from the Llama [61], Mistral [62], Qwen [64], and Gemma [63] families on both unmodified and modified DRAM. We adopt DDR4-2400 with 8 GiB and 16 banks per rank, using two ranks per channel for all models. For each model, we select the smallest channel count from {1,2,4,8,16}\{1,2,4,8,16\} that permits all schemes to be placed under the u-PUD limit of 256 input elements per subarray. For host processing, we assume memory-bandwidth-bound execution with an effective bandwidth of 85% of the peak memory bandwidth.

Quality Evaluation.

The term budget controls both approximation quality and PUD computation latency, linking our quality and performance evaluations. We evaluate its impact on model quality by applying the approximation to pretrained models and measuring perplexity on WikiText-2 [46]. For each model and execution scheme, we select term budgets that satisfy the specified quality constraint for performance simulation.

Performance Evaluation.

We combine an analytical performance model with a PUD simulator built on Ramulator 2.0 [47]. We evaluate batch-one autoregressive decoding with a context length of 128 or 1024 tokens. The default weight precision is 4 bits; selected SIMD configurations use mixed precision with an average of 3–4 bits. Following prior architectural evaluations [68, 29], we report decoding latencies per generated token across all Transformer blocks, excluding prefill, embedding lookup, final normalization, the language-model head, and sampling. We assume input sparsification for the host-side Out and Down projections, allowing weight transfers and computations associated with pruned input elements to be skipped [60]. Guided by the sparsity levels reported in that work, we assume input sparsities (i.e., fractions of input elements pruned) of 70% for Out and 65% for Down across all models, with Down-input sparsity of 90% for Mistral 7B (TS) [59].

Decoding Performance

For each model and execution scheme, we select term budgets that limit the perplexity increase to at most 5% relative to the quantized model without term truncation. Figure 13 shows decoding latency and its breakdown, normalized to the PUD baseline. Taking the faster of SIMD and MIMD for each model, PUD-LLM achieves geometric-mean speedups of 1.17×1.17\times on u-PUD and 1.18×1.18\times on m-PUD. The maximum speedups are 1.23×1.23\times for Gemma 31B on u-PUD and 1.41×1.41\times for Gemma 12B on m-PUD.

Figure 14. Decoding latency normalized to CPU.

Figure 15. Quality–efficiency trade-offs: perplexity on WikiText-2 versus average terms per input element on the critical path. PUD-LLM achieves comparable perplexity with fewer terms. Faint points show evaluated settings; highlighted points trace the Pareto frontiers.

Figure 16. Lookahead overlaps projection computation, increasing mean utilization while reducing block latency.

The latency breakdown illustrates both the benefit of projection acceleration and the limit imposed by host processing. For Gemma 12B on m-PUD, the QKV and Gate/Up projections account for 54.6% of the PUD baseline latency, while the remaining operations account for 45.4%. MIMD accelerates these projections by 2.13×2.13\times. With the remaining latency unchanged, this yields an overall speedup of 1/(0.454+0.546/2.13)≈1.41×1/(0.454+0.546/2.13)\approx1.41\times over the PUD baseline, corresponding to a 2.45×2.45\times speedup over modeled CPU execution (Figure 14).

Figure 16 illustrates PUD parallel resource utilization during one Gemma 12B decoder block on m-PUD. Utilization is defined as the fraction of available column resources occupied by PUD computation, including lookahead, estimated from Ramulator command traces and mapped column counts and averaged over 1 μ1\,\mus intervals. For this block, mean utilization rises from 8.03%8.03\% to 14.81%14.81\% (1.85×1.85\times), while latency decreases from 0.8070.807 to 0.5680.568 ms (1.42×1.42\times speedup).

Quality–Efficiency Trade-off

The preceding performance comparison uses a fixed quality constraint. The analysis in § 3.2 suggests that PUD-LLM requires fewer terms than direct computation at the same input-approximation error. Assuming comparable input-approximation errors have similar effects on model quality, we expect this term reduction to also hold at comparable perplexity.

Figure 15 shows the results. The horizontal axis measures the average number of nonzero terms per input element on the critical path, assuming fully hidden lookahead. Across all evaluated models, the proposed schemes achieve comparable perplexity with fewer terms than the PUD baseline, supporting our analysis. Since projection computation latency scales with the number of nonzero terms on the critical path, these results indicate a better quality–efficiency trade-off. The extent of the reduction varies by model: MIMD requires substantially fewer terms than SIMD at comparable perplexity on Gemma 12B, whereas SIMD offers a comparable trade-off to MIMD on Qwen 27B and Mistral 128B.

Sensitivity Analysis

We vary memory configuration, context length, and Down-input sparsity to examine how PUD resources and CPU processing costs affect the benefits of PUD-LLM.

Figure 17. Increasing memory parallelism can shift the preferred execution scheme from SIMD to MIMD for the same model.

Memory Configuration.

The preferred execution scheme depends on model dimensions and available memory parallelism. Figure 17 examines this dependence by varying channel and rank counts for Llama 1B on u-PUD. SIMD is preferable in the 8 and 16 GiB configurations, exploiting unused columns without requiring separate banks for lookahead. Additional independent resources favor MIMD in the 32 and 64 GiB configurations by reducing competition between lookahead and delta computation. For example, SIMD achieves a 1.26×1.26\times speedup over the PUD baseline at 8 GiB, while MIMD achieves 1.28×1.28\times at 32 GiB.

Figure 18. Longer CPU processing limits overall speedup, while shorter CPU processing can leave lookahead exposed.

CPU Processing.

We examine how CPU processing time affects decoding speedup by varying context length on u-PUD and Down-input sparsity on m-PUD for Gemma 12B (Figure 18). At short contexts, the small attention cost allows SIMD to achieve its highest observed speedup of 1.10×1.10\times at 128 tokens. MIMD, however, has insufficient time to hide Gate/Up lookahead. Longer contexts provide more overlap time, allowing MIMD to reach its highest observed speedup of 1.21×1.21\times at 1k tokens. Beyond this point, attention computation and KV-cache accesses account for an increasing share of decoding latency, reducing overall speedup despite continued projection acceleration. Our approach is therefore complementary to techniques that accelerate attention [65, 66]. The Down-input sparsity sweep models potential CPU-side acceleration through activation sparsification [59]. At the highest evaluated sparsity of 95%, both schemes achieve their largest speedups: 1.12×1.12\times for SIMD and 1.66×1.66\times for MIMD. Although Down execution becomes shorter, QKV lookahead remains hidden.

In-Memory Computing Systems for LLM Inference.

PIM systems improve execution efficiency through data placement and address mapping [27, 28], as well as operator assignment, partitioning, and execution overlap [32, 29, 30, 31]. Hardware/software support for arithmetic synthesis and fine-grained resource control enables flexible in-DRAM execution [9, 10]. Further optimizations adapt precision, data representations, and arithmetic implementations to reduce computation latency [33], eliminate input preparation and output bit-transposition overheads [13], and remove runtime data transposition while enabling bit-level pipelining [50]. In contrast, PUD-LLM restructures the computation itself by exploiting LLM-specific properties.

Parallelizing Dependent Computations.

Prior work exposes parallelism in dependent computations through finer task granularity, pipelining, and computational reformulation. Task-based execution advances computations as their inputs become available, while software pipelining overlaps stages without violating data dependences [52, 53, 51]. Other approaches use algebraic transformations [54], predict unavailable values and subsequently validate them [55], or relax selected dependences under application-specific accuracy constraints [56, 57]. For neural networks, prior work either evaluates provisional intermediate states in parallel and refines them iteratively [18, 58] or exploits similarity between layer inputs [44]. At the token level, candidate generation and parallel verification reduce sequential decoding steps [19, 20]. Our work also algebraically reformulates dependent computations using redundancy in LLMs.

Redundancy-Aware Approximate Inference.

Redundancy and error tolerance in neural networks allow selected computations to be skipped or performed at lower precision with limited impact on model quality. Mixed-precision quantization allocates precision according to its impact on model quality [21, 23, 22], while precision-adaptive execution translates reduced precision into lower execution cost [24, 25, 26]. Redundancy across LLM layers [36, 35] motivates layer omission or consolidation [37, 38, 39, 34], approximation or reuse of intermediate representations [40, 41, 42, 43], and input substitution for concurrent layers [44]. Spatial and temporal redundancy also enables differential computation [48, 49]. Our work exploits inter-layer redundancy to approximate the corrections, rather than skipping layers, without model retraining or architectural modifications.

Conclusion

We presented PUD-LLM, an algorithm–system co-design whose algorithm exploits LLM layer redundancy to expose inter-layer parallel work for efficient execution on PUD. Its system design improves SIMD and MIMD utilization through packing and pipelining, respectively. PUD-LLM improves the quality–efficiency Pareto frontier. Performance simulations show up to 1.41×1.41\times end-to-end decoding speedup over the PUD baseline. These results demonstrate the importance of restructuring computation around model dependencies and numerical properties to use PUD resources effectively, complementing optimizations of individual operations. Overall speedup remains constrained by operations executed on the CPU. Coordinating their acceleration with PUD execution offers opportunities for further improvement. Future work will explore other PIM architectures and investigate opportunities and conditions for applying the approach to model structures such as MoE.

PUD Execution Model

This appendix describes the m-PUD operation costs and their relationship to α\alpha, β\beta, and PP in our latency model. We adopt the addition sequence from SIMDRAM [9] and the primitive-operation timings from the MIMDRAM implementation [10].

Primitive Operations and Timing.

Modified DRAM supports ACT–PRE (AP) and ACT–ACT–PRE (AAP) sequences. AP activates the addressed row or row group and then precharges the bank. AAP additionally copies the activation result to a destination row or row group before precharging. Selecting designated compute rows enables these sequences to perform majority operations and preserve their results [11].

The public DDR4-2400 configuration specifies tCK=0.833t_{\mathrm{CK}}=0.833 ns, tRAS=32t_{\mathrm{RAS}}=32 ns, and tRP=14.16t_{\mathrm{RP}}=14.16 ns. Overlapping the two activations in AAP adds tov=4t_{\mathrm{ov}}=4 ns. The resulting primitive latencies are

TAP=tRAS+tRP=46.16 ns,TAAP=tRAS+tov+tRP=50.16 ns.\begin{align} T_{\mathrm{AP}} &= t_{\mathrm{RAS}}+t_{\mathrm{RP}} = 46.16\,\mathrm{ns}, \tag*{(13)} \\ T_{\mathrm{AAP}} &= t_{\mathrm{RAS}}+t_{\mathrm{ov}} +t_{\mathrm{RP}} = 50.16\,\mathrm{ns}. \tag*{(14)} \end{align}

These isolated latencies include final precharge recovery and exclude queuing and resource contention.

Addition and Per-Term Computation Cost.

The addition loop executes six AAPs and two APs per bit, including operand preparation and result preservation [9]. Thus, one full-adder step takes

TFA=6TAAP+2TAP=393.28 ns.\begin{equation} T_{\mathrm{FA}} =6T_{\mathrm{AAP}}+2T_{\mathrm{AP}} =393.28\,\mathrm{ns}. \tag*{(15)} \end{equation}

Including one initialization AAP, the DRAM-operation time for a bb-bit addition is

Tadd(b)=TAAP+bTFA=(50.16+393.28b) ns.\begin{equation} T_{\mathrm{add}}(b) =T_{\mathrm{AAP}}+bT_{\mathrm{FA}} =(50.16+393.28b)\,\mathrm{ns}. \tag*{(16)} \end{equation}
Reduction Cost.

The coefficient β\beta captures partial-sum readout and CPU aggregation per output element per input partition. With weight precision bwb_w and bit-plane partial-sum precision bpsb_{\mathrm{ps}}, readout transfers bwbps/8b_wb_{\mathrm{ps}}/8 bytes per output element per partition. At effective readout bandwidth BW\mathrm{BW}, this contributes bwbps/(8BW)b_wb_{\mathrm{ps}}/(8\mathrm{BW}) to β\beta; the remaining contribution accounts for CPU aggregation.

Effective MIMD Parallelism.

Computations in different banks can overlap, but banks within a channel share the C/A bus. In our analytical model, each ACT or PRE occupies one C/A cycle. For a repeated operation with isolated duration LL and CC commands, the command-bandwidth bound on concurrency at the isolated execution rate is

P=LC tCK.\begin{equation} P =\frac{L}{C\,t_{\mathrm{CK}}}. \tag*{(17)} \end{equation}

Expanding each AAP into three commands and each AP into two, the full-adder loop requires C=6×3+2×2=22C=6\times3+2\times2=22 commands. Its per-channel bound is therefore

P=393.2822×0.833≈21.46.\begin{equation} P =\frac{393.28}{22\times0.833} \approx21.46. \tag*{(18)} \end{equation}

References

  1. [1]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
  2. [2]Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need. https://arxiv.org/abs/1911.02150
  3. [3]Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024). AI and memory wall. IEEE Micro, 44(3), 33–39.
  4. [4]Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 606–624.
  5. [5]Mutlu, O., Olgun, A., & Yüksel, İ. E. (2025). Memory-centric computing: solving computing’s memory problem. 2025 IEEE International Memory Workshop (IMW), 1–4.
  6. [6]Devaux, F. (2019). The true processing in memory accelerator. 2019 IEEE Hot Chips 31 Symposium (HCS), 1–24.
  7. [7]Gao, F., Tziantzioulis, G., & Wentzlaff, D. (2019). ComputeDRAM: in-memory compute using off-the-shelf DRAMs. Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 100–113.
  8. [8]Seshadri, V., Kim, Y., Fallin, C., Lee, D., Ausavarungnirun, R., Pekhimenko, G., Luo, Y., Mutlu, O., Gibbons, P. B., Kozuch, M. A., & Mowry, T. C. (2013). RowClone: fast and energy-efficient in-DRAM bulk data copy and initialization. Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture, 185–197.
  9. [9]Hajinazar, N., Oliveira, G. F., Gregorio, S., Ferreira, J. D., Ghiasi, N. M., Patel, M., Alser, M., Ghose, S., Gómez-Luna, J., & Mutlu, O. (2021). SIMDRAM: a framework for bit-serial SIMD processing using DRAM. Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 329–345.
  10. [10]Oliveira, G. F., Olgun, A., Yağlıkçı, A. G., Bostancı, F. N., Gómez-Luna, J., Ghose, S., & Mutlu, O. (2024). MIMDRAM: An end-to-end processing-using-DRAM system for high-throughput, energy-efficient and programmer-transparent multiple-instruction multiple-data computing. 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 186–203.
  11. [11]Seshadri, V., Lee, D., Mullins, T., Hassan, H., Boroumand, A., Kim, J., Kozuch, M. A., Mutlu, O., Gibbons, P. B., & Mowry, T. C. (2017). Ambit: In-memory accelerator for bulk bitwise operations using commodity DRAM technology. Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, 273–287.
  12. [12]Kim, Y., Seshadri, V., Lee, D., Liu, J., & Mutlu, O. (2012). A case for exploiting subarray-level parallelism (SALP) in DRAM. ACM SIGARCH Computer Architecture News, 40(3), 368–379.
  13. [13]Kubo, T., Tokuda, D., Nagatani, T., Usui, M., Qu, L., Cao, T., & Takamaeda-Yamazaki, S. (2025). MVDRAM: Enabling GeMV execution in unmodified DRAM for low-bit LLM acceleration. https://arxiv.org/abs/2503.23817
  14. [14]Shazeer, N. (2020). GLU variants improve transformer. https://arxiv.org/abs/2002.05202
  15. [15]Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Renard Lavaud, L., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., … El Sayed, W. (2024). Mixtral of experts. https://arxiv.org/abs/2401.04088
  16. [16]Jacob, B., Wang, D., & Ng, S. (2010). Memory systems: cache, DRAM, disk. Morgan Kaufmann.
  17. [17]Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: training multi-billion parameter language models using model parallelism. https://arxiv.org/abs/1909.08053
  18. [18]Song, Y., Meng, C., Liao, R., & Ermon, S. (2021). Accelerating feedforward computation via parallel nonlinear equation solving. International Conference on Machine Learning, 9791–9800.
  19. [19]Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from transformers via speculative decoding. International Conference on Machine Learning, 19274–19286.
  20. [20]Fu, Y., Bailis, P., Stoica, I., & Zhang, H. (2024). Break the sequential dependency of LLM inference using lookahead decoding. International Conference on Machine Learning, 14060–14079.
  21. [21]Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., Mahoney, M., & Keutzer, K. (2021). HAWQ-V3: dyadic neural network quantization. International Conference on Machine Learning, 11875–11886.
  22. [22]Zheng, Z., Song, X., & Liu, C. (2026). Mixllm: Llm quantization with global mixed-precision between output-features and highly-efficient system design. Proceedings of Machine Learning and Systems, 8, 1771–1785.
  23. [23]He, Y., Liu, L., Liu, J., Wu, W., Zhou, H., & Zhuang, B. (2023). Ptqd: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 13237–13249.
  24. [24]Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., & Moshovos, A. (2016). Stripes: bit-serial deep neural network computing. Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture.
  25. [25]Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Kim, J. K., Chandra, V., & Esmaeilzadeh, H. (2018). Bit fusion: bit-level dynamically composable architecture for accelerating deep neural networks. Proceedings of the 45th Annual International Symposium on Computer Architecture.
  26. [26]Sabri, M., Riera, M., & González, A. (2024). ReDy: a novel ReRAM-centric dynamic quantization approach for energy-efficient CNNs. Proceedings of the 53rd International Conference on Parallel Processing, 1042–1051.
  27. [27]Ibrahim, M. A., Islam, M., & Aga, S. (2024). PIMnast: balanced data placement for GEMV acceleration with processing-in-memory. Proceedings of the SC24 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, 970–981.
  28. [28]Seo, S. H., Kim, J., Lee, D., Yoo, S., Moon, S., Park, Y., & Lee, J. W. (2025). FACIL: flexible DRAM address mapping for SoC-PIM cooperative on-device LLM inference. 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA).
  29. [29]Heo, G., Lee, S., Cho, J., Choi, H., Lee, S., Ham, H., Kim, G., Mahajan, D., & Park, J. (2024). NeuPIMs: NPU-PIM heterogeneous acceleration for batched LLM inferencing. Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS).
  30. [30]Gu, Y., Khadem, A., Umesh, S., Liang, N., Servot, X., Mutlu, O., Iyer, R., & Das, R. (2025). PIM is all you need: a CXL-enabled GPU-free system for large language model inference. Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS).
  31. [31]He, Y., Mao, H., Giannoula, C., Sadrosadati, M., Gómez-Luna, J., Li, H., Li, X., Wang, Y., & Mutlu, O. (2025). PAPI: exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system. Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS).
  32. [32]Zhou, M., Wang, X., & Rosing, T. (2023). OverlaPIM: overlap optimization for processing in-memory neural network acceleration. 2023 Design, Automation and Test in Europe Conference and Exhibition (DATE).
  33. [33]Oliveira, G. F., Kabra, M., Guo, Y., Chen, K., Yağlıkçı, A. G., Soysal, M., Sadrosadati, M., Olivares Bueno, J., Ghose, S., Gómez-Luna, J., & Mutlu, O. (2025). Proteus: enabling high-performance processing-using-DRAM with dynamic bit-precision, adaptive data representation, and flexible arithmetic. Proceedings of the ACM International Conference on Supercomputing (ICS).
  34. [34]Men, X., Xu, M., Zhang, Q., Yuan, Q., Wang, B., Lin, H., Lu, Y., Han, X., & Chen, W. (2025). ShortGPT: layers in large language models are more redundant than you expect. Findings of the Association for Computational Linguistics: ACL 2025, 20192–20204.
  35. [35]Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., & Roberts, D. A. (2025). The unreasonable ineffectiveness of the deeper layers. International Conference on Learning Representations.
  36. [36]Lad, V., Lee, J. H., Gurnee, W., & Tegmark, M. (2024). The remarkable robustness of LLMs: stages of inference?. https://arxiv.org/abs/2406.19384
  37. [37]Corro, L. D., Giorno, A. D., Agarwal, S., Yu, B., Awadallah, A., & Mukherjee, S. (2023). SkipDecode: autoregressive skip decoding with batching and caching for efficient LLM inference. https://arxiv.org/abs/2307.02628
  38. [38]Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., & Kim, J.-J. (2024). SLEB: streamlining LLMs through redundancy verification and elimination of transformer blocks. International Conference on Machine Learning, 46136–46155.
  39. [39]Yang, Y., Cao, Z., & Zhao, H. (2024). LaCo: large language model pruning via layer collapse. Findings of the Association for Computational Linguistics: EMNLP 2024, 6401–6417.
  40. [40]Yom Din, A., Karidi, T., Choshen, L., & Geva, M. (2024). Jump to conclusions: short-cutting transformers with linear transformations. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 9615–9625.
  41. [41]Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Gerasimenko, N., Oseledets, I., Dimitrov, D., & Kuznetsov, A. (2024). Your transformer is secretly linear. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5376–5384.
  42. [42]Chen, X., Hu, Y., Zhang, J., Wang, Y., Li, C., & Chen, H. (2025). Streamlining redundant layers to compress large language models. International Conference on Learning Representations.
  43. [43]Mu, Y., Wu, Y., Fan, Y., Wang, C., Li, H., Zeng, J., He, Q., Yang, M., Meng, F., Zhou, J., Xiao, T., & Zhu, J. (2026). Cross-layer attention sharing for pre-trained large language models. Transactions of the Association for Computational Linguistics, 14, 656–688.
  44. [44]Zou, L., Wang, Q., Zhao, H., Kong, J., Yang, Y., & Deng, Y. (2024). CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers. https://arxiv.org/abs/2404.06709
  45. [45]Kung, H. T., McDanel, B., & Zhang, S. Q. (2020). Term revealing: furthering quantization at run time on quantized DNNs. https://arxiv.org/abs/2007.06389
  46. [46]Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2017). Pointer sentinel mixture models. International Conference on Learning Representations.
  47. [47]Luo, H., Tuğrul, Y. C., Bostancı, F. N., Olgun, A., Yağlıkçı, A. G., & Mutlu, O. (2023). Ramulator 2.0: A modern, modular, and extensible dram simulator. IEEE Computer Architecture Letters, 23(1), 112–116.
  48. [48]Mahmoud, M., Siu, K., & Moshovos, A. (2018). Diffy: A déjà vu-free differential deep neural network accelerator. 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 134–147.
  49. [49]Kim, S., Lee, H., Cho, W., Park, M., & Ro, W. W. (2025). Ditto: Accelerating diffusion model via temporal value similarity. 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 338–352.
  50. [50]Oliveira, G. F., Olgun, A., Yüksel, I. E., Bostancı, F. N., Becker, P. H. E., Sadrosadati, M., Ghose, S., Gómez-Luna, J., & Mutlu, O. (2026). PipeDRAM: A Data-Transposition-Free In-DRAM Architecture with Hardware/Software Pipelining. https://arxiv.org/abs/2609.30998
  51. [51]Buttari, A., Langou, J., Kurzak, J., & Dongarra, J. (2009). A Class of Parallel Tiled Linear Algebra Algorithms for Multicore Architectures. Parallel Computing, 35(1), 38–53. https://doi.org/10.1016/j.parco.2008.10.002
  52. [52]Lam, M. S. (1988). Software Pipelining: An Effective Scheduling Technique for VLIW Machines. Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, 318–328. https://doi.org/10.1145/53990.54022
  53. [53]Ottoni, G., Rangan, R., Stoler, A., & August, D. I. (2005). Automatic Thread Extraction with Decoupled Software Pipelining. Proceedings of the 38th Annual IEEE/ACM International Symposium on Microarchitecture, 105–116. https://doi.org/10.1109/MICRO.2005.13
  54. [54]Blelloch, G. E. (1990). Prefix sums and their applications.
  55. [55]Lipasti, M. H., & Shen, J. P. (1996). Exceeding the Dataflow Limit via Value Prediction. Proceedings of the 29th Annual ACM/IEEE International Symposium on Microarchitecture, 226–237. https://doi.org/10.1109/MICRO.1996.566464
  56. [56]Udupa, A., Rajan, K., & Thies, W. (2011). ALTER: Exploiting Breakable Dependences for Parallelization. Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, 480–491. https://doi.org/10.1145/1993498.1993555
  57. [57]Misailovic, S., Kim, D., & Rinard, M. C. (2013). Parallelizing Sequential Programs with Statistical Accuracy Tests. ACM Transactions on Embedded Computing Systems, 12(2s), 88:1–88:26. https://doi.org/10.1145/2465787.2465790
  58. [58]Danieli, F., Sarabia, M., Suau Cuadros, X., Rodriguez, P., & Zappella, L. (2023). Deeppcr: Parallelizing sequential operations in neural networks. Advances in Neural Information Processing Systems, 36, 47598–47625.
  59. [59]Song, Y., Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., & Chen, H. (2024). Turbo sparse: Achieving llm sota performance with minimal activated parameters. https://arxiv.org/abs/2406.05955
  60. [60]Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., & Athiwaratkun, B. (2025). Training-free activation sparsity in large language models. International Conference on Learning Representations, 2025, 98302–98322.
  61. [61]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., … Ma, Z. (2024). The llama 3 herd of models. https://arxiv.org/abs/2407.21783
  62. [62]Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Renard Lavaud, L., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., & El Sayed, W. (2023). Mistral 7b. https://arxiv.org/abs/2310.06825
  63. [63]Gemma Team, El Abd, S., Aggarwal, V., Algayres, R., Andreev, A., Bachem, O., Ballantyne, I., Brick, C., Cărbune, V., Casbon, M., Chaturvedi, M., Chawla, A., Cotruta, V., Coucke, A., Culliton, P., Dadashi, R., Dixon, L., Elhawaty, M., Evci, U., … Zou, C. (2026). Gemma 4 Technical Report. https://arxiv.org/abs/2607.02770
  64. [64]Qwen Team. (2026). Qwen3.5-omni technical report. https://arxiv.org/abs/2604.15804
  65. [65]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., & Han, S. (2024). Quest: Query-aware sparsity for efficient long-context llm inference. https://arxiv.org/abs/2406.10774
  66. [66]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X. (2024). Kivi: A tuning-free asymmetric 2bit quantization for kv cache. https://arxiv.org/abs/2402.02750
  67. [67]de Lima, J. P. C., Morris, B., Khan, A. A., Castrillon, J., & Jones, A. K. (2026). Count2multiply: Reliable in-memory high-radix counting. 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA), 1–15.
  68. [68]Park, J., Choi, J., Kyung, K., Kim, M. J., Kwon, Y., Kim, N. S., & Ahn, J. H. (2024). Attacc! unleashing the power of pim for batched transformer-based generative model inference. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 103–119.

Paper details

vlnc:W_jzfgdspp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 18

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents