betaFeedback

WorksComputer Science

LazyWeights: Decode-on-Demand Edge LLMs

Abstract

Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, llama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best llama.cpp format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU llama.cpp, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

Introduction

Generating a token reads every weight of a large language model (LLM) once, so on an edge device the weight footprint decides whether a model fits and how fast it runs: a 7B model needs 14 GB in 16-bit floats, more than most edge modules have, and even at 4 bits each token moves gigabytes through a phone-class memory system. llama.cpp [1] has made post-training quantization the default, with GGUF files offering fixed-width integer formats with per-block scales from 8 down to about 3 bits per weight.

These formats and the quantizers that target them (GPTQ [2], AWQ [3], the llama.cpp K- and I-quants) share an unstated commitment: every code of a matrix costs the same number of bits. This wastes bits twice. Weights are bell-shaped, so a 4-bit alphabet’s inner codes dominate and its entropy sits well below 4 bits [4], [5]; and many code assignments reach the same quality, so a quantizer that knows its codes will be entropy-coded can pick ones that compress.

Lossless weight coding, from Deep Compression [6] to recent work on 16-bit LLM weights [7, 8, 9, 10] or a quantizer’s integers [11], [12], takes the quantizer as given and mostly decodes in software. A coded 4-bit model is worth having only if it beats the next-smaller fixed-width format at equal quality, a comparison rarely made, and only if its codes can be expanded at DRAM rate every token: software decoders fall 15–20× short of that on an Orin Nano Super, making a token 2.5–3.2× slower than the same codes at fixed width. LazyWeights removes the commitment end to end (Figure 1). It shapes the codes inside the deployed format, codes them with a Huffman layout built for parallel decoding, and decodes them in a small unit between DRAM and L2, where bytes are counted. The GPU still reads fixed-width codes, so its kernels are untouched, the decoded weights are bit-identical to a valid GGUF file, and quality is the quantizer’s alone. On the bandwidth-bound Orin, fewer bytes then mean faster, cheaper tokens; on the instruction- and capacity-bound Nano, the unit

Figure 1. LazyWeights overview: entropy-aware quantization and a per-matrix Huffman bundle in the cloud; on the device, a decode unit between the memory controller and L2 restores fixed-width codes. Without the unit, the GPU kernels decode in software inside GEMV (32 lanes) as the reference. Bottom: bits per weight.

removes software decoding and the coded size decides whether Mistral-7B runs at all (Table I).

Jetson Orin Nano SuperJetson Nano
Memory / bandwidth8 GB, 102 GB/s4 GB, 22.5 GB/s (meas.)
GPUAmpere, 1024 coresMaxwell, 128 cores
int8 dp4a / toolchainyes / CUDA 12.6no / CUDA 10.2
DRAM use, packed40–64%34–35%
Mistral-7B fitsyesno (streamed, §IV-E)
Bound bymemory bandwidthinstructions + capacity
Compression buyslatency, energyability to run; the unit buys latency, energy

Table I. Edge platforms. DRAM use: weight bytes per token over time per token, as a fraction of bandwidth, for our fixed-width kernels.

Our contributions:

  • Entropy-aware quantization in standard formats: GPTQ with shared sub-block steps and a wider step, emitting valid GGUF 5.7–31.4% smaller than the best llama.cpp format at ≤1% and ≤3.5% ΔPPL.

  • A parallel-decodable Huffman bundle: a length-limited canonical code per matrix, 32 interleaved lane streams per tile, pair symbols for 4-bit codes; within 1.2–2.1% of the zeroth-order entropy, all overheads included.

  • A Huffman decode unit between DRAM and L2: 0.05 mm² per 32-lane engine at 1 GHz (ASAP7), sized from the bandwidth, bit-exact, prototyped on an AMD ZCU102; never larger than rANS in rate, smaller in area at equal throughput.

  • End-to-end evaluation in two edge regimes: latency and energy per token on a Jetson Orin Nano Super and a Jetson Nano, greedy and with prompt-lookup speculation, against llama.cpp, from measured time, energy and DRAM energy per byte and synthesized decoder power.

Weight quantization. GPTQ [2] quantizes column by column, feeding each column’s Hessian-weighted error into the rest; AWQ [3] rescales salient channels. Both target fixed-width formats, like llama.cpp’s [1] K- and I-quants (optionally with an importance matrix), which edge modules ship in and which are our baseline. AQLM [13], QuIP# [14] and QTIP [15] reach lower rates at equal quality with vector and trellis codebooks but need their own formats and kernels; we keep the deployed scalar format and code its integers.

Entropy-constrained quantization. ECSQ adds a rate term to the distortion; Choi et al. [16] applied Hessian-weighted ECSQ to network weights. Inside GPTQ, structural changes to the grid matter far more than this term (§VII).

Lossless coding of weights. Deep Compression [6] Huffman-coded the cluster indices of pruned CNNs; DeepCABAC [17] arithmetic-codes weights for storage and transmission. For LLMs, ZipNN [8] compresses model files for storage and transfer, DFloat11 [7] entropy-codes BF16 exponents and decodes them in a GPU kernel, ZipServ [9] fuses a fixed-length bitmap encoding of BF16 into the Tensor-Core GEMM, and Huff-LLM [10] Huffman-codes 16-bit weights end to end; all keep roughly 70% of the BF16 size, around 11 bits per weight, well above what edge devices deploy. EntroLLM [11] codes tensor-level 8- and 4-bit integers and decodes them in parallel on a Jetson’s CPU.

Entropy decoders in hardware. Huffman decoding runs in FPGA Deflate cores [18] and in parallel on GPUs [19]; ANS/rANS [20] reaches arithmetic-coding rates but has a multiply in its state recurrence. APack [21] places an arithmetic decoder next to a DNN accelerator’s memory controller; we place a Huffman unit there in a GPU memory hierarchy, size it from the bandwidth, and compare it with rANS at equal throughput.

Low-bit runtimes. BitLLM [22] speeds up fixed-width low-bit decoding on mobile GPUs (KV-cache layouts, packing, deferred type conversion); it cuts compute per code, we cut bytes per code, so the two compose.

Where we sit. LazyWeights is, to our knowledge, the first design to code the deployed integer format rather than merely store it: it shapes the codes inside llama.cpp’s formats, keeps files valid GGUF, decodes in hardware where DRAM traffic is counted, and compares against the best fixed-width format at equal perplexity, the comparison that decides whether coding is worth its decoder. Any quantizer that emits integer codes in a fixed format can feed the bundle and the unit.

Background

Decoding is memory-limited

Each token runs one GEMV per weight matrix, two FLOPs per weight, far below what an edge GPU sustains per byte of bandwidth. With SS the weight bytes read per pass, β\beta the DRAM bandwidth, τarith\tau_{\mathrm{arith}} the arithmetic time, and SstS_{\mathrm{st}} the bytes streamed from storage at βst\beta_{\mathrm{st}} when memory cannot hold SS, a weight pass takes

t≈max⁡(Sβ, τarith)+Sstβst.(1)t \approx\max\left(\frac{S}{\beta},\ \tau_{\mathrm{arith}}\right) + \frac{S_{\mathrm{st}}}{\beta_{\mathrm{st}}}. \tag*{(1)}

Fewer bytes are faster only while S/βS/\beta dominates; when the model does not fit, the storage term does.

On the Orin Nano Super, our fixed-width kernels match or beat llama.cpp on the same GGUF and use 40–64% of the bandwidth (Table I), so time tracks bytes, and a unit that makes coded bytes the only bytes read turns compression into latency and energy.

On the Nano, time tracks fp32 arithmetic and fewer bytes save little time; but software Huffman decoding is 54–69% of a token, which the unit removes, and Mistral-7B does not fit, which puts the storage term of (1) in charge.

The entropy of quantized codes

A fixed-width format stores each code at its alphabet’s width; an entropy coder stores it near the code entropy H(q)H(q). For weights of density p(w)p(w) quantized with step Δ\Delta, at high resolution

H(q)≈h(w)−log⁡2Δ,h(w)=−∫p(w)log⁡2p(w) dw,(2)H(q) \approx h(w) - \log_2 \Delta,\qquad h(w) = -\int p(w)\log_2 p(w)\,dw, \tag*{(2)}

so scaling the step by cc saves about log⁡2c\log_2 c bits per weight: 0.32 at ×1.25\times1.25, 0.58 at ×1.5\times1.5. At 4 bits, far from that regime, the measured saving is smaller (0.19–0.20 b/w at ×1.25\times1.25 on Phi-3, 0.46 at ×1.5\times1.5 on Mistral), but the lever is the same. Conversely, a quantizer that spreads codes evenly, as min-max fitting per sub-block does, pushes H(q)H(q) toward the full width: the coder’s saving is decided by the quantizer.

The LazyWeights Design

Problem statement

Let F\mathcal{F} be llama.cpp’s formats (blocks, scales, code alphabet; e.g. IQ4_XS, Q5_K, Q6_K) and θ\theta the quantizer settings. With b(f,θ)b(f,\theta) the coded bits per weight, counting scales, code tables, lane offsets and embeddings, and ΔPPL(f,θ)\Delta\mathrm{PPL}(f,\theta) the WikiText-2 perplexity increase over fp16, we seek for a budget ε\varepsilon

(f⋆,θ⋆)=arg⁡min⁡f∈F, θb(f,θ)s.t.ΔPPL(f,θ)≤ε,(3)(f^\star,\theta^\star)=\arg\min_{f\in\mathcal{F},\ \theta} b(f,\theta)\quad\text{s.t.}\quad\Delta\mathrm{PPL}(f,\theta)\le\varepsilon, \tag*{(3)}

against the budget baseline, the smallest llama.cpp file (with or without importance matrix) within the same budget. ε=1%\varepsilon=1\% is near-lossless, the traditional role of 8 bits; 3.5% is a small, usable loss within which every model has a llama.cpp point. Perplexity selects; downstream tasks confirm. The output must decode losslessly to valid GGUF of format ff, so quality is the quantizer’s alone, measured once before coding.

Entropy-aware quantization inside standard formats

LazyWeights keeps each format’s blocks, scales and alphabet, so files load in unmodified llama.cpp, and changes how codes are chosen (Figure 2).

Code selection diagrams comparing llama.cpp and LazyWeights

Figure 2. Code selection (histograms illustrative). LazyWeights keeps the format but shares one step per super-block (γ\gamma) and widens it (cc), so the codes concentrate; GPTQ feeds each column’s error to the remaining columns, so the coarser grid keeps its quality.

Shared sub-block steps (γ\gamma). llama.cpp fits each sub-block of a 256-weight super-block to its own range, spreading its codes over the whole alphabet: right for fixed width, wrong for a coder. We bound each sub-block’s range from below by γ\gamma times the largest in its super-block: γ=0\gamma= 0 is llama.cpp’s fit; at γ=1\gamma= 1 the super-block shares one step and its codes concentrate where its weights do.

Step scale (cc). By (2), a wider step lowers the entropy: we scale the fitted step by c∈{1,1.25,1.5}c \in\{1,1.25,1.5\}, chosen per point, and let GPTQ absorb the error.

GPTQ error feedback [2] makes the coarser grid affordable; round-to-nearest (RTN) does not (§VII). Weight ww in column jj takes

qj∗=arg⁡min⁡q hjj−1(w−w^q)2+λℓ(q),(4)q_j^* = \underset{q}{\arg\min}\ h_{jj}^{-1}(w-\hat{w}_q)^2+\lambda\ell(q), \tag*{(4)}

with Hessian term hjj−1h_{jj}^{-1}, reconstruction w^q\hat{w}_q and code length ℓ(q)\ell(q); all points use λ=0\lambda= 0, as the ECSQ rate term [16] does not pay (§VII). Calibration: 128 sequences of 512 tokens from C4. Settings differ per point (e.g. γ=1\gamma= 1, c=1.5c = 1.5 for Mistral-7B ≤3.5%\leq3.5\%; γ=0\gamma= 0, c=1.25c = 1.25 for Phi-3 ≤1%\leq1\%), hence the search (3) rather than a fixed recipe.

A lossless Huffman bundle built for parallel decoding

Huffman decoding is sequential: a code’s start is known only once the previous code is decoded. The bundle (Figure 3a) breaks this. Each matrix has one canonical code, length-limited to 11 bits so an 11-bit window always resolves a code. Each 64×256 tile is 32 lane streams of two rows, decoded in parallel without waiting for each other. The lanes’ 32-bit words are interleaved, so the words needed next are adjacent and a GPU warp or the unit’s read port fetches them in wide, aligned reads; per-tile lane offsets let any tile start independently. In 4-bit formats two codes form one 8-bit pair symbol, two weights per decode step.

Figure 3. (a) Bundle layout: a 64 × 256 tile of 4-bit codes is 32 lanes of two rows; each lane is one Huffman stream of 32-bit words (code boundaries illustrative); the lanes’ words are interleaved, so one aligned read fetches the next word of every lane, and per-tile lane offsets let any tile start independently; two 4-bit codes form one 8-bit pair symbol. (b) Decode unit between the memory controller and L2 (as in Figure 1): TT engines of 32 lanes sharing an 8-word read port through a masked round-robin arbiter; a lane compares an 11-bit peek against per-length limits (no 2048-entry table), adds the length’s offset to index the symbol table, and shifts its buffer by the length.

A Huffman decode unit between DRAM and L2

Decoding in the GPU kernels spends instructions on bit manipulation, costing more time than the bytes save; expanding weights back into DRAM saves storage, not bandwidth. LazyWeights decodes between the memory controller and L2 (Figure 3b): DRAM holds and moves only coded weights, L2 receives fixed-width codes, and the kernel is unchanged, so the first term of (1) follows the coded size.

Lane. A request side feeds the lane’s stream through a 4-entry prefetch FIFO into a 64-bit bit buffer. The next 11 bits are compared against all 11 per-length limits at once; the thermometer result gives the length, the symbol is the code plus a per-length offset, and the buffer shifts. A lane emits one pair symbol (two weights) per cycle; the per-matrix-limit and offset tables are broadcast by the engine.

Engine. 32 lanes share an 8-word read port through a masked round-robin arbiter, decoding 31.9 symbols per cycle (0.2% stall), i.e. 60–64 weights per cycle for pair-coded formats.

Sizing and clock. To keep pace with DRAM, the unit must decode

θ∗=8βbc(5)\theta^* = \frac{8\beta}{b_c} \tag*{(5)}
ChangeMeasured onEffect
Speculative attention, GGUFW-fidth twinsSmolLM 8-token pass, depth 40044.4 → 31.2 ms
Phi-3 8-token pass, depth 90077.5 → 52.5 ms
Nano kernelsSmolLM ≤1% twin, greedy token474 → 152 ms
SmolLM ≤3.5%282 → 134 ms
Phi-3585 → 261 ms
Ordered reduction CUDA graphsNano, 2-token coded pass418 → 301 ms
Orin pass (485–645 kernels)−1.9–3.5 ms (7–12%)
per greedy token−6–11%

weights per second (bcb_c: coded bits per weight), 151–229 G weights/s at 102 GB/s. At 1 GHz, close to the Orin’s 1.02 GHz GPU clock (MAXN_SUPER), four engines (0.206 mm2^2) cover θ∗\theta^* for five of six points; SmolLM ≤1%\leq1\%, whose 5- and 6-bit codes decode singly (32 weights per cycle per engine), needs five (0.258 mm2^2). Lower clocks need proportionally more engines; the Nano needs 1–2. The lane’s compare-and-shift recurrence closes at 1 GHz without the repair arbiter needs, while a 2048-entry lookup table per lane is 9× larger and misses it (Table V).

Verification. A C model reproduces every coded matrix of 17 bundles with zero mismatches; Verilator co-simulation of the SystemVerilog RTL is bit- and cycle-exact against it on 29/29 matrices in 5 configurations; and NN engines against a bandwidth-capped DRAM model with read latencies up to 1024 cycles are bit-exact on all 6 points at 95–96% of θ∗\theta^*. Synthesis: Yosys/ABC, OpenROAD, OpenSTA, ASAP7 RVT TT, logic synthesis (pre-layout).

FPGA prototype. We map four engines onto an AMD ZCU102: a Zynq UltraScale+ XCZU9EG MPSoC (speed grade −2; 274k LUTs, 548k flip-flops, 32.1 Mb of block RAM, 2,520 DSP slices) with a quad-core 1.2 GHz Arm Cortex-A53 running Linux, 4 GB of 64-bit DDR4 on the processor side and 512 MB of 16-bit DDR4 on the fabric side. The Arm cores write each matrix’s bundle in chunks into block RAM over AXI and start the engines together; every lane’s output (CRC-32) and every chunk’s cycle, stall and word counts are checked against the C model.

Runtime

Our runtime runs both the twin — the same codes without entropy coding, as the GPU sees them behind the unit — and the coded bundle, decoded inside the GEMV kernels as the software reference. A fixed-order reduction across tile columns, in the int8 and fp32 kernels, makes the two bit-identical, so losslessness is tested end to end: identical logits on both devices, and speculation (up to 8 tokens per pass) token-identical to greedy.

Speculation. Attention reads each key and value once for all draft tokens of a head, with per-query arithmetic independent of the draft length, and Q5_K/Q6_K twins keep the GGUF width (nibbles plus a high-bit plane), not bytes (Table II).

Nano kernels. The instruction-bound Nano needs few instructions per weight. A 32-bit fetch yields four (5/6-bit) or eight (4-bit) fixed-width codes; Q5_K/Q6_K codes are rebuilt from their nibble and high bits by byte permutes, one sub-block per fetch. Byte permutes also replace integer-to-float conversion (a code placed under the exponent of 2232^{23}, recovered by one subtraction) and table lookup (the 16 IQ4 XS grid values, biased to unsigned bytes); the multiply-add order is unchanged. The ordered reduction replaces Maxwell’s slow global atomics.

CUDA graphs. Each kernel launch idles the GPU for about 4 μ\mus. We replay each pass shape (1–8 tokens) as a CUDA graph, with the position read from device memory, with bit-identical logits on all 14 bundles; llama.cpp already uses graphs on the Orin.

Streaming on a 4 GB device. The Nano has 3.6 GB free at idle; CUDA takes about 200 MB, and we keep 384 MiB free because an out-of-memory kill takes down system services. Tegra device memory is pinned host memory and counts against that floor, so at most 2.56 GB of weights stay resident, while the Mistral-7B bundles are 3.2–3.6 GB. The rest is streamed every pass in forward order: stored pre-interleaved in the device layout (4096-byte aligned) and read with O_DIRECT by a loader thread into two pinned staging buffers, overlapping I/O and compute. Streamed tokens equal resident ones.

Experimental Setup

Models and quality. Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, SmolLM-1.7B-Instruct. WikiText-2 perplexity (all tokens, 512-token chunks); HellaSwag, ARC-easy, PIQA (0-shot); GSM8K (8-shot chain of thought, flexible extraction: the last number in the answer).

Baselines. On the Orin the budget baseline uses llama.cpp’s CUDA kernels. The Nano is frozen at CUDA 10.2 and gcc 7.5 (its last JetPack): current llama.cpp no longer builds its CUDA backend there, and releases that do predate the I-quants (IQ4 XS, IQ3 M) several baselines use, so the Nano baseline is current llama.cpp cross-compiled for the 4-core CPU, the best llama.cpp available. Our runtime builds for sm 53 with fp32 activations. The fixed-width twin isolates the unit’s share with identical kernels; other runtimes (MLC-LLM, TensorRT-LLM) alter only the kernel share.

Latency and energy. 128 new tokens for each of 6 prompts (32 for streamed Mistral-7B on the Nano); energy integrated from the module input rail, idle included; Orin clocks pinned (MAXN_SUPER). Every runtime runs greedy (one weight pass per token) and with prompt-lookup speculation: drafts of up to 7 tokens from an nn-gram table of the prompt and accepted text, with no drafter model, verified in one pass, yielding 1.7–2.4 tokens per pass (0.41–0.58 passes per token). Each runtime uses its own drafter (llama.cpp’s; ours, nn=3); our runtime also ran llama.cpp’s (ported), isolating the drafter’s share (Figure 5a).

Figure 5. (a) Factors from the llama.cpp baseline (greedy) to LazyWeights + unit with our drafter, one change per step, on a log scale (steps below 1× extend left; unit time conservative; product at bar end); top: Orin; bottom: Nano, resident points, where our drafter is a loss (0.73–0.76×). (b) Decode rate. Left: software decoders on the coded operating points; Mistral-7B is streamed on the Nano. Right: area vs rate for 1, 2, 4, 8 and 16 engines of 32 lanes (ASAP7 RVT TT, pre-layout) at the clock each closes; rANS engines share the Huffman arbiter; stars: the sized units. Bands: the rate DRAM delivers, θ∗=8β/bc\theta^* = 8\beta/b_c over the operating points; AGX Orin (not tested) at its 204.8 GB/s specification.

End-to-end model with the unit. Time with the unit is the twin’s, measured in the same run and mode (thw=ttwint_{\mathrm{hw}} = t_{\mathrm{twin}}): conservative, since fewer DRAM bytes could also shorten the token; the bandwidth-bound end scales ttwint_{\mathrm{twin}} by the coded-to-fixed byte ratio. Energy is

Ehw=Etwin−k(ϵDRAMΔS−Edec,pass),(6)E_{\mathrm{hw}} = E_{\mathrm{twin}} - k\left(\epsilon_{\mathrm{DRAM}} \Delta S - E_{\mathrm{dec,pass}}\right), \tag*{(6)}

with kk weight passes per token (1 greedily, measured under speculation) and ΔS\Delta S the bytes saved per pass. ϵDRAM\epsilon_{\mathrm{DRAM}}, the input-power slope of a streaming DRAM read minus the same read from L2, is 90.5 pJ/byte on the Orin and 34.9 pJ/byte on the Nano; EdecE_{\mathrm{dec}} uses synthesized energy per view (SVI-D), bounded by clock-gated vs. always-on engines with a 2× margin.

Results

Memory at equal quality

Every coded model is 5.7–31.4% smaller than the best llama.cpp format within its budget, most for Mistral-7B ≤\leq1% (Table III, Figure 4). Huffman coding alone gives 6.2–20.7% (coded vs twin); the quantizer gives the rest, letting a coded file in a wider format undercut a narrower llama.cpp format at equal quality. HellaSwag, ARC-easy and PIQA are at most 1.5 points below the baseline, and GSM8K at most 3.2 (Mistral ≤\leq3.5%).

Figure 4. Rate–quality frontiers. Dots: coded models, including llama.cpp’s imatrix codes behind our Huffman stage; lines: lowest ΔPPL reachable at or below a rate; shading: where coding wins; rings: released operating points; dashed: the 1% and 3.5% budgets and fp16. Axis linear below 1%.

Latency and energy on the Orin Nano Super

With the unit, every greedy point beats the budget baseline by 10–36% in time (up to 44% bandwidth-bound) and 9–38% in energy (Table III); the decoder costs 3–13 mJ per token against 5–61 mJ of DRAM energy saved, and with always-on engines at 2× power energy still drops 6–36% greedily and 17–42% with speculation. Without the unit, coding would slow the Orin down (8I, Figure 5b).

Speculation composes with the unit: it cuts passes per token, the unit cuts bytes per pass. Against llama.cpp with speculation, every point is 29–54% faster (up to 57% bandwidth-bound) at 21–44% less energy. One change at a time (Figure 5a), speculation is the largest step on the Orin (1.3–2.0×), the format gives 1.3–1.4× where it changes the file, and the drafter is within 5%: ours accepts more tokens per pass (1.96 vs 1.63 on Mistral), but larger passes cost more.

ModelBudgetFormatb/wΔPPLHellaSwagARC-ePIQAGSM8KSavingOrin, greedy ms / mJΔtΔEOrin, speculation ms / mJΔtΔE
Phi-3-mini≤1%IQ4 XS+im4.311+0.65%.783.807.798.78535.0 / 69127.4 / 495
LazyWeights4.046+0.11%.778.803.789.7736.2%31.4 / 627−15/−10%−9%17.9 / 366−39/−35%−26%
≤3.5%Q3 K M+im4.108+2.91%.776.786.799.77944.0 / 84027.6 / 512
LazyWeights3.649+2.97%.774.811.796.77011.2%31.4 / 616−37/−29%−27%18.3 / 370−42/−34%−28%
Mistral-7B≤1%Q5 K M5.668+0.44%.842.716.768.42583.3 / 178461.6 / 1276
LazyWeights3.886+0.94%.840.720.770.40231.4%53.0 / 1107−41/−36%−38%41.5 / 871−37/−33%−32%
≤3.5%IQ3 M+im3.628+1.01%.838.722.779.40077.7 / 161171.6 / 1397
LazyWeights3.422+1.46%.838.719.782.3685.7%52.9 / 1068−44/−32%−34%38.5 / 786−56/−46%−44%
SmolLM-1.7B≤1%Q5 K M5.721+0.65%.607.633.737.06426.4 / 48114.3 / 246
LazyWeights5.162+0.88%.607.630.739.0679.8%18.6 / 356−33/−29%−26%10.1 / 194−33/−29%−21%
≤3.5%IQ4 XS+im4.388+3.09%.605.625.732.05523.7 / 37516.5 / 254
LazyWeights3.934+3.27%.601.610.736.05510.3%15.3 / 287−40/−36%−23%7.6 / 143−57/−54%−44%

Table 3. Operating points (the released GGUF files) and per-token cost on the Orin Nano Super. Upper row: the llama.cpp budget baseline (its format). b/w: all overheads; ΔPPL vs fp16; GSM8K: flexible extraction; Saving = 1 − bours/bllama.cppb_{\mathrm{ours}}/b_{\mathrm{llama.cpp}}. Orin: ms / mJ per token, each runtime with its own drafter; LazyWeights with the unit at the measured twin time; Δt: bandwidth-bound / conservative; ΔE: (6), clock-gated decoder.

ModelBudgetBaselineLazyWeights GGUFSoftware+ unit
Greedy Phi-3≤1%657 / 4.48654 / 4.40591 / 4.24261 / 2.05
≤3.5%736 / 4.80663 / 4.45590 / 4.46263 / 2.08
Greedy SmolLM≤1%366 / 2.47364 / 2.21484 / 3.22152 / 1.17
≤3.5%317 / 2.12317 / 2.03289 / 2.10134 / 0.97
Greedy Mistral≤1%226.6 / 71.218.0 / 58.33.40 / 15.8n/a
≤3.5%193.1 / 12.715.1 / 49.0208 / 12.0n/a
Prompt-lookup speculation Phi-3≤1%840 / 5.60865 / 5.74423 / 2.94244 / 1.71
≤3.5%956 / 6.23864 / 5.74415 / 3.05239 / 1.69
Prompt-lookup speculation SmolLM≤1%475 / 3.10437 / 2.66286 / 1.84141 / 0.91
≤3.5%419 / 2.73469 / 2.94170 / 1.17101 / 0.65
Prompt-lookup speculation Mistral≤1%16.7 / 56.613.3 / 46.12.30 / 11.3n/a
≤3.5%2.90 / 19.111.5 / 40.21.49 / 8.85n/a

Table 4. Jetson Nano, per token (ms / J; Mistral: s / J). CPU: llama.cpp with the budget baseline or our GGUF. GPU: our runtime, decoding in software or with the unit (twin time). Mistral streams from USB 3 at a 2560 MB resident budget; its twin is no proxy for the unit (§VIII).

The Jetson Nano

For resident models (Table IV), LazyWeights with the unit takes 58–64% less time and 53–57% less energy per token than llama.cpp on the CPU, greedy, and 70–76% less time and 69–76% less energy with speculation in both. Most of this comes from the GPU kernels (Fig. 5a, bottom), as Table I predicts, but the unit matters: software decoding is 2.2–3.2× slower than the twin and beats the CPU only on Phi-3 and SmolLM ≤3.5% (by 9–20%), not on SmolLM ≤1%, whose 5/6-bit codes decode singly. With the Nano’s low DRAM energy per byte, energy with the unit is within ±0.03 J of the measured twin energy. Speculation helps our GPU runtime (SmolLM ≤3.5% twin −25%; Phi-3 only −7 to −9%, as an 8-token pass costs 3.6× a 1-token pass) and hurts llama.cpp on the instruction-bound CPU by 28–32%.

Mistral-7B, streamed. Here the coded side decides: at ≤1%, LazyWeights is 6.6× faster than llama.cpp at 4.5× less energy, even decoding in software. At ≤3.5%, llama.cpp wins greedy time narrowly but more energy and LazyWeights wins with speculation, because llama.cpp’s memory-mapped IQ3_M file (3.28 GB) stays in the reclaimable page cache, which does not count against our floor. A slightly larger file gets almost no hits, since every token scans the weights in the same order: our IQ4_XS GGUF (3.91 GB) takes 15.1 s. Our bundle (3.18 GB) is smaller still, but the cache cannot hold its streamed matrices beside our pinned resident weights and the kernel’s 464 MB CMA reservation, hence O_DIRECT. A pass costs about 1.0 s of compute plus the streamed bytes, and about the same for 1 or 8 tokens, so accepted tokens per pass set speculation’s gain; a USB 3 stick (317 MB/s) instead of the SD card (70 MB/s) cuts our software-decoding token time 2.1–2.7×.

Decoder cost, and Huffman versus rANS

A small unit clears the software wall (Fig. 5b; Table V).

Huffman outperforms rANS on both axes. Rate: rANS is never smaller (0.0 to +1.0% more bits): Huffman is already within 1.2–2.1% of entropy, and rANS pays 32 bits of state plus padding per lane stream. Hardware: at equal throughput (2 G weights/s), rANS needs 4.0× as wide a pair symbols (270 MHz) or 1.15× with single codes (645 MHz), as its one-cycle recurrence holds a symbol search, a table select and a 13×20-bit multiply against Huffman’s compare and shift.

BlockASAP7 μm2\mu\mathrm{m}^2ASAP7 mWASAP7 slackNangate45 μm2\mu\mathrm{m}^2Nangate45 mWNangate45 slack
Comparator lane1,5663.59+117 ps23,46225.2+0.5 ns
Table lane (2048 entries)14,07694.4−31 ns–––
Arbiter, 32 lanes1,4430.83met†18,6394.25+1.5 ns
Arbiter, 64 lanes9,174–−0.29 ns–––
Engine51,550115.7––––

Table 5. Decoder blocks, logic synthesis (pre-layout): ASAP7 RVT TT at 1 GHz, Nangate45 at 500 MHz. †after OpenROAD repair. Power: co-simulation activity for the comparator lane, default otherwise. Shaded: Engine = 32 comparator lanes + arbiter.

ModelBudgetFullγ=0\gamma=0RTN
Phi-3≤3.5%\le3.5\%3.649, +2.974.046, +0.113.646, +12.07
Mistral≤1%\le1\%3.886, +0.944.241, +0.323.886, −0.04
Mistral≤3.5%\le3.5\%3.422, +1.463.837, +2.613.422, +2.78
SmolLM≤1%\le1\%5.162, +0.885.621, +0.315.162, +0.57
SmolLM≤3.5%\le3.5\%3.934, +3.274.321, +2.243.934, +6.14

Table 6. Ablations at the operating-point settings (b/w, Δ\DeltaPPL).

Interleaving several rANS states per lane would relax its recurrence; we did not evaluate it.

FPGA prototype

On the ZCU102, four engines close timing at 112.5 MHz (WNS +0.06 ns; the decode-to-arbiter loop caps the design near 115 MHz) using 183.0k LUTs (67%), 37.2k flip-flops, 464 BRAM36 (51%, mostly payload copies) and no DSPs: 46k LUTs and 116 BRAM36 per engine. All 29 matrices (2,723 chunks, 484M weights) decode bit-exact, with cycle, stall and word counts equal to the C model on every chunk. Each engine sustains 31.8 weights per cycle, 14.3G weights/s for four; the fabric (VCCINT) draws 0.88 W idle plus 3.7 W at full rate, 0.26 nJ per weight.

Ablations

Quantizer (Table VI). Shared sub-block steps carry most of the saving: with γ=0\gamma=0 (llama.cpp’s fit), Δ\DeltaPPL is lower but every point costs 9–12% more bits. GPTQ feedback rescues the coarse points, which reach up to +12.1% with RTN at the same bits, but not the fine ones, where RTN is better. The step scale is non-monotone in quality (Mistral ×1.25 +2.57%, ×1.5 +1.46%, ×1.75 +15.5%), hence the per-model search. λ>0\lambda> 0 saves at most 1.2% of the bits and usually raises perplexity; int8 vs fp32 activations move perplexity by −0.07 to +0.18%.

Decoder. Comparator vs table lookup: Figure 5b. An 8-word read port with cross-stream prefetch reaches 31.9 symbols per cycle; pair symbols cost −0.2 to +0.8% in size but double an engine’s rate; 99% of the rate needs latency/4 FIFO words per lane (or: +0.014 mm2^2 per engine); wider engines (64 lanes: arbiter 6.4×, misses 1 GHz; Table V) only rescale the engine count.

Run-to-run variance. Five more repeats of every measured Orin row of Table III, clocks pinned, vary by at most 0.6% in time and energy per token (median 0.1%), and every Δt\Delta t and ΔE\Delta E stays within 1.7 points of the table.

Limitations and Conclusion

The unit’s latency and energy are projections from measured device numbers and pre-layout synthesis, not a fabricated SoC; the FPGA shows bit- and cycle-exact decoding, but at FPGA clocks, not 1 GHz. For streamed Mistral-7B the twin is no proxy for the unit: it streams the fixed-width bytes (1.05 GB per token at a 2560 MB budget vs 0.68/0.26 GB coded at ≤1%/≤3.5%\le1\%/\le3.5\%), leaves less than our memory model free, and at a 2048 MB budget from the SD card was 1.2–1.8× slower than software decoding. We therefore do not report the unit’s streaming time; reading its coded bytes alone takes 2.2/0.8 s per token from USB. We did not evaluate Llama-3 or Qwen3, which quantize worse [23], or long contexts, where the uncoded fp16 KV cache dilutes the saving (Mistral-7B at 4K tokens: 0.54 GB per pass vs 3.5 GB of weights).

The fixed width of edge LLM formats is a commitment, not a necessity: the codes are far from uniform, the quantizer can make them less so at no quality cost, and an entropy coder removes the difference if the decoder sits where bytes are counted. Shaped and Huffman-coded, models in llama.cpp’s formats are 5.7–31.4% smaller at equal perplexity; a 0.2 mm2^2 unit between DRAM and L2 turns this into faster, cheaper tokens on a bandwidth-bound Orin Nano Super, with or without speculation, and lets a capacity-bound Nano run a 7B model that otherwise does not fit. Coding the KV cache, newer model families, and a layout next to a real memory controller are independent next steps.

Acknowledgment

We thank Radu Marculescu for lending us the AMD ZCU102 board, and Geffen Cooper for his help in setting it up.

References

References

  1. [1]G. Gerganov et al., “llama.cpp,” https://github.com/ggml-org/llama.cpp.
  2. [2]E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” CoRR, vol. abs/2210.17323, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2210.17323
  3. [3]J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,” in Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., vol. 6, 2024, pp. 87–100. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95ccefef21-Paper-Conference.pdfDOI
  4. [4]Z. Allen-Zhu and Y. Li, “Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=FxNNiUGftha
  5. [5]J. X. Morris, C. Sitarwin, C. Guo, N. Kokhlikyan, G. E. Suh, A. M. Rush, K. Chaudhuri, and S. Mahloujifar, “How much can language models memorize?” in Forty-third International Conference on Machine Learning, 2026. [Online]. Available: https://openreview.net/forum?id=bA6BgsSbaUiarxiv.org/abs/2505.24832
  6. [6]S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2016. [Online]. Available: https://doi.org/10.48550/arXiv.1510.00149
  7. [7]T. Zhang, M. Hariri, S. Zhong, V. Chaudhary, Y. Sui, X. Hu, and A. Shrivastava, “70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11),” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=xdNAVPT7TGy
  8. [8]M. Hershcovitch, A. Wood, L. Choshen, G. Girmonsky, R. Leibovitz, O. Ozeri, I. Ennmouri, M. Malka, P. Chin, S. Sundararaman, and D. Harnik, “ZipNN: Lossless Compression for AI Models,” in 2025 IEEE 18th International Conference on Cloud Computing (CLOUD), 2025, pp. 186–198. [Online]. Available: https://doi.org/10.1109/CLOUD67622.2025.00028
  9. [9]R. Fan, X. Yu, X. Pan, Z. Li, W. Luo, Q. Wang, W. Wang, and X. Chu, “ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 2264–2280. [Online]. Available: https://doi.org/10.1145/3779212.3790250
  10. [10]P. Yubeatov, T. Mahmoud, S. Naga, P. Taheri, T. Xia, A. George, Y. Khalil, S. Q. Zhang, S. Joshi, C. Hegde, and S. Garg, “Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference,” 2025. [Online]. Available: https://arxiv.org/abs/2502.00922
  11. [11]A. Sanyal, G. Datta, P. Mukherjee, S. P. Chinchali, and M. Orshansky, “EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices,” in ICASSP 2026 – 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 19742–19746. [Online]. Available: https://doi.org/10.1109/ICASSP55912.2026.11464777
  12. [12]A. Sanyal, S. P. Chinchali, and G. Datta, “EntroMCU: A Hardware-Algorithm Vision for LLM Inference on Microcontrollers,” in 2026 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), 2026. [Online]. Available: https://doi.org/10.26153/tsw/64855
  13. [13]V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh, “Extreme Compression of Large Language Models via Additive Quantization,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. PMLR, 21–27 Jul 2024, pp. 12 284–12 303. [Online]. Available: https://proceedings.mlr.press/v235/egiazarian24a.htmlarxiv.org/abs/2401.06118
  14. [14]A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. D. Sa, “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,” in Forty-first International Conference on Machine Learning, 2024. [Online]. Available: https://openreview.net/forum?id=9BrydUVcq0earxiv.org/abs/2402.04396
  15. [15]A. Tseng, Q. Sun, D. Hou, and C. D. Sa, “QTIP: Quantization with Trellises and Incoherence Processing,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=7sdkLVuYCUarxiv.org/abs/2406.11235
  16. [16]Y. Choi, M. El-Khamy, and J. Lee, “Towards the Limit of Network Quantization,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [Online]. Available: https://openreview.net/forum?id=rJ8uNptglarxiv.org/abs/1612.01543
  17. [17]S. Wiedemann, H. Kirchhoffer, S. Matlage, P. Haase, A. Marban, T. Marinč, D. Neumann, T. Nguyen, H. Schwarz, T. Wiegand, D. Marpe, and W. Samek, “DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 4, pp. 700–714, 2020. [Online]. Available: https://doi.org/10.1109/JSTSP.2020.2969554
  18. [18]M. Lewon, B. F. Cockburn, and J. Han, “High-Throughput FPGA-Based Hardware Accelerators for Deflate Compression and Decompression Using High-Level Synthesis,” IEEE Access, vol. 8, pp. 62 207–62 217, 2020. [Online]. Available: https://doi.org/10.1109/ACCESS.2020.2984191
  19. [19]A. Weißenberger and B. Schmidt, “Massively Parallel Huffman Decoding on GPUs,” in Proceedings of the 47th International Conference on Parallel Processing, ser. ICPP ’18. New York, NY, USA: Association for Computing Machinery, 2018. [Online]. Available: https://doi.org/10.1145/3225058.3225076
  20. [20]J. Duda, “Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding,” 2014. [Online]. Available: https://arxiv.org/abs/1311.2540
  21. [21]A. D. Laszorz, M. Mahmoud, and A. Moschovos, “APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference,” CoRR, vol. abs/2201.08830, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2201.08830
  22. [22]H. Zheng, Z. Shu, K. Zhu, M. Qin, X. Ma, G. Agrawal, and W. Niu, “BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs,” in Proceedings of the 32nd ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ser. ASPLOS ’27. New York, NY, USA: Association for Computing Machinery, 2027, pp. 120–135. [Online]. Available: https://doi.org/10.1145/3548514.3850028
  23. [23]W. Huang, X. Zheng, X. Ma, H. Qin, C. Lv, H. Chen, J. Luo, X. Qi, X. Liu, and M. Magno, “An empirical study of LLaMA3 quantization: from LLMs to MLLMs,” Visual Intelligence, vol. 2, no. 1, p. 36, Dec 2024. [Online]. Available: https://doi.org/10.1007/s44267-024-00070-x

Paper details

vlnc:W_7wph7msh
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents