betaFeedback

WorksComputer Science

Scute: Exact Batched Serving for PEFT Adapters Beyond LoRA

Abstract

Existing serving systems for large language models batch requests for many different LoRA adapters on one GPU. However, these systems reject adapters from other parameter-efficient fine-tuning (PEFT) methods such as DoRA, (IA), VeRA and OFT. One solution to this is to convert these adapters to LoRA, but this induces a significant memory and accuracy cost. Instead, we show that, under some modifications, a batched LoRA kernel can serve all of these methods exactly and efficiently. We achieve this by writing each method in one normal form, a LoRA computation with six slots for an adapter's own vectors and matrices. Some slots can merge into an adapter's weights at load time; the others remain live and induce a cost. We give ways to efficiently pay this cost and predict it from a zero-parameter byte model. We package this system, Scute, and build it on top of vLLM. Scute serves all five methods, coming as checkpoints from the Hugging Face PEFT library, in one batch, and it matches that library with a mean error of at most 1.5% of each adapter's effect. Under a burst of requests on an RTX A6000, a mix of these methods takes 5-17% more decode time than the same number of LoRA adapters. Furthermore, using Scute, the same GPU can serve 64 to 1024 (IA) and VeRA adapters at 1.5x to at least 2.5x the request rate of their LoRA conversions.

Introduction

Parameter-efficient fine-tuning (PEFT) adapts a pretrained model to a task by training a small adapter while the model’s own weights stay frozen. A platform that serves many customers, or tenants, can then keep one copy of the base model on a GPU and apply each tenant’s adapter on top of it. LoRA Hu et al. (2022), the most common method, adds a low-rank update to each adapted weight matrix, but other methods are also in use. Most of the methods below are implemented in the Hugging Face PEFT library Mangrulkar et al. [18], which we call the PEFT library and use as the reference implementation. DoRA Liu et al. [16] splits the update into a magnitude and a direction, (IA)3^{3} Liu et al. [14] rescales activations, SSF Lian et al. [13] rescales and shifts them, VeRA Kopiczko et al. [9] shares two frozen random matrices across all adapters and trains only two small vectors per layer, and OFT Qiu et al. [21]; ); Liu et al. [17] rotates the input of each layer. Which method works best can depend on the setting, including for adapters that a hypernetwork generates at inference time Walker & Baraniuk [26], so a platform may need to serve several methods side by side.

Most serving systems support only LoRA. Punica Chen et al. [2] batches requests for many LoRA adapters into one kernel launch, and S-LoRA Sheng et al. [24] pages thousands of LoRA adapters between host and GPU memory. The production engines vLLM Kwon et al. [10] and SGLang Zheng et al. [34] use this design, and they reject adapters of other methods at load.1

A platform can convert other adapters to LoRA, but conversion has two costs. The first cost is accuracy; (IA)3^{3} and OFT change the base weight by a matrix of full rank, so a LoRA adapter of practical rank can only approximate them, and the quality of the approximation depends on what the adapter is trying to represent. On adapters that we train, a rank-64 conversion of (IA)3^{3} keeps its accuracy on a mathematical reasoning task when it is fitted on the tenant’s own data, but loses it when it is fitted on generic text. The second cost is size. A VeRA tenant owns only a few vectors, but as LoRA it needs its own factors, 0.34 GB at rank 64, leaving less GPU memory for other tenants and for the attention state of running requests. An alternative to side-step these costs is to merge each adapter into its own copy of the model; this is exact but means requests for different adapters can no longer be run as a single batch on a shared base model.

To address this, we start from the kernel that serves LoRA, and understand how it can be adjusted to serve multiple tenants with different adapters. In this LoRA kernel, for each layer, a batch holds tokens from many adapters that share one base weight WW. It then computes h=xWh = xW once for the whole batch, and then computes each adapter’s low-rank update in two small steps, a shrink xAxA and an expand that multiplies the result by BB and adds it to hh. We generalize this structure with slots, the places in this computation where an adapter can multiply or add a vector or matrix of its own. These slots allow us to construct a kernel that can serve all the adapter shapes mentioned above. Our contributions are as follows.

A normal form and a fold rule (§ 3). One formula, the normal form, has six slots and covers LoRA, DoRA, (IA)3^3, SSF, VeRA and OFT. The fold rule makes explicit what serving cost these slots induce. A slot folds, or merges into the adapter’s own weights once at load time, if and only if the adapter owns a matrix or vector next to it. The slots that do not fold stay live and must be computed for every token.

Three routes and their cost (§ 4). Live slots induce a cost and can be paid for in one of three ways, which we call routes. A byte model predicts the time that each route adds when its passes are limited by memory bandwidth.

A compiler and a runtime (§ 5). A compiler turns a PEFT-library checkpoint of one of the five methods into the normal form, and a patch to vLLM, applied at run time, adds the routes to vLLM’s LoRA path. The resulting patched engine, Scute, can serve all five methods in one batch, with CUDA graphs and torch.compile on. The engine, with tests and examples, is available at https://github.com/ThomasWalker1/Scute-deploy.

Measurements (§ 6). Our engine matches the PEFT library with a mean error of at most 1.5% of each adapter’s effect, and to within 0.4 points of its task accuracy. On an RTX A6000, under a burst of requests that all arrive at once, a mix of four or five methods takes 5–17% more decode time and 4–18% more time to first token than the same number of LoRA adapters. Under the same burst and with the same base GEMM in both engines (§ 6.1), the best conversion that we tested, every tenant at rank 64, takes 12% more decode time than our engine. With 64 to 1024 (IA)3^3 and VeRA tenants on the same GPU, our engine sustains 1.5× to at least 2.5× the request rate of their conversions within a latency target, with up to 2.03× their throughput.

Background

Serving terms. A request first runs its whole prompt through the model in one step, called prefill, and then produces one output token per step, called decode. We measure the time to first token (TTFT) and the time per output token after the first (TPOT). An engine such as vLLM batches tokens from many requests in each step, and each running request needs memory for its attention state, the KVKV cache. An adapter is resident when its weights are in GPU memory. Other adapters wait in host memory, and the engine copies them in when a request needs them. We also use tenant to mean the tenant’s adapter, and we call the set of adapters that an engine serves its pool.

GPU cost terms. A kernel is a program that the host launches on the GPU, and each launch has a fixed cost. A pass is a kernel that reads tensors from GPU memory and writes a result. A GEMM is a general matrix multiplication. Adapter computations perform few arithmetic operations per byte that they move, so their time is set by memory bandwidth, and we count their cost in bytes. Unless stated otherwise, we measure on NVIDIA RTX A6000 GPUs (48 GB, 6 MiB L2 cache, PCIe) in fp16, and we repeat the main measurements on an NVIDIA A100-SXM4-80GB. A CUDA graph records a sequence of launches once and replays it with one call from the host. On the RTX A6000, a large copy within GPU memory reaches β=680\beta= 680 GB/s, a copy that fits in the L2 cache reaches βL2=3205\beta_{\mathrm{L2}} = 3205 GB/s, and one launch inside a CUDA graph costs λ=0.96 μs\lambda= 0.96\,\mu\mathrm{s}.

The LoRA schedule. LoRA (Hu et al., 2022) adds cxABc xAB to the base output xWxW. Here x∈RN×dinx \in\mathbb{R}^{N \times d_{\mathrm{in}}} holds one row per token, and W∈Rdin×doutW \in\mathbb{R}^{d_{\mathrm{in}} \times d_{\mathrm{out}}} is the base weight. The adapter’s factors A∈Rdin×rA \in\mathbb{R}^{d_{\mathrm{in}} \times r} and B∈Rr×doutB \in\mathbb{R}^{r \times d_{\mathrm{out}}} have a small rank rr, and cc is a scalar. Punica’s schedule (Chen et al., [2]) sorts the batch so that the rows of each adapter are contiguous, and calls each such group of rows a segment. A vendor library GEMM computes h=xWh = xW once for the whole batch. The shrink then computes V=xA∈RN×rV = xA \in\mathbb{R}^{N \times r} for each segment, and the expand reads hh, adds VBVB, and writes the result. A kernel splits its output into tiles, blocks that one group of GPU threads computes, and in this schedule each tile of rows belongs to one segment, so a tile reads the factors of one adapter. vLLM (Kwon et al., [10]) fuses projections that share an input, such as the query, key and value projections qq, kk, vv, into one layer with several output slices, and its expand covers every slice in one launch.

The methods. DoRA (Liu et al., [16]) rescales each output column of W+cABW + cAB to a learned magnitude mm. (IA)3^3 (Liu et al., [14]) multiplies the keys, the values and the intermediate activation of the feed-forward network by learned vectors. VeRA (Kopiczko et al., [9]) shares one frozen random pair AA, BB of rank 256 among all adapters, and then each adapter learns two vectors per layer, dd inside the term and ee after it. OFT (Qiu et al., [21]) multiplies the input by a learned block-diagonal, orthogonal matrix RR, with each block being computed from a skew-symmetric matrix using the Cayley transform.

Extending the LoRA Schedule

Every method of § 2 changes the output of some of the model’s linear projections, and it does so with a few small tensors of its own. This section writes all of these changes in one formula, which extends LoRA’s computation with a few extra parameters § 3.1. It then shows which of these parameters can be merged into the adapter’s own weights when the adapter loads, so that serving never computes them § 3.2. § 4 gives the cost of the parameters that remain.

The Normal Form

An adapter changes some of the model’s linear projections, such as the query projection of attention or the up projection of the feed-forward network, and leaves the others unchanged. We call a projection that an adapter changes a hook. At each hook, the base model computes xWxW, and we write the output of adapter aa as

Fa(x)=((xR)W)⊙s+b+∑k=1Kck(((xAk)⊙dk)Bk⊙ek).(1)F_a(x) = ((xR)W) \odot s + b + \sum_{k=1}^{K} c_k (((xA_k) \odot d_k)B_k \odot e_k). \tag*{(1)}

We call (1) the normal form. Besides the low-rank factors AkA_k and BkB_k, it has six slots. A slot is a parameter that an adapter can set, and that is the identity (a one, a vector of ones, a zero shift or the identity matrix) when the adapter does not use it. There are three slots on the base path and three on each low-rank term. On the base path, the input transform RR is a block-diagonal matrix with blocks of size bRb_R, the output scale s∈Rdouts \in\mathbb{R}^{d_{\mathrm{out}}} multiplies each output column, and the output shift b∈Rdoutb \in\mathbb{R}^{d_{\mathrm{out}}} is added to it. Each term kk, of rank rkr_k, has an inner vector dk∈Rrkd_k \in\mathbb{R}^{r_k}, an outer vector ek∈Rdoute_k \in\mathbb{R}^{d_{\mathrm{out}}} and a scalar ckc_k. A vector multiplies each row of the tensor on its left, element by element. Each tensor is either owned, which means that each adapter stores its own copy, or shared by all adapters in the pool, and the base weight WW is always shared. A method is a member of the normal form if (1) can express it. Figure 1 shows where each slot sits in the schedule.

The LoRA schedule and its six slots

Figure 1. The LoRA schedule and its six slots. The shaded GEMM is shared by all adapters. Thick circles mark slots that never fold. Thin circles mark slots that fold when the adapter owns the factor next to them. The expand already reads hh and writes h′h', so it can apply ss, bb and eke_k without an extra pass.

Table 1 shows how each method fits into the normal form. For example, (IA)3^3 scales the intermediate activation of a gated feed-forward network. With gate weight WgW_g, up weight WuW_u and activation function σ\sigma, this network computes σ(xWg)⊙xWu\sigma(xW_g) \odot xW_u. Since (σ(xWg)⊙xWu)⊙l=σ(xWg)⊙(xWu⊙l)(\sigma(xW_g) \odot xW_u) \odot l = \sigma(xW_g) \odot(xW_u \odot l), the vector ll of (IA)3^3 is the slot ss of the up projection. Similarly, OFT, as the PEFT library implements it, rotates xx and then calls the unchanged base layer, so its rotation is the slot RR.

methodowned matricesfoldslive slots
LoRAA,BA, Bc→Ac \to Anone
DoRAA,BA, Be→Be \to B, c→Ac \to Ass
(IA)3^3nonenoness
SSFnonenones,bs, b
VeRAnonec→dc \to dd,ed, e
OFTRRnoneRR

Table 1. Members of the normal form (1); we identify which matrices each adapter owns (VeRA’s AA, BB are shared), the folds that Proposition 1 allows, and the slots that stay live.

Other methods fit as well. LoHa Hyeon-Woo et al., [6] and LoKr Edalati et al., [3] build the update from products of small factors, and they are members once these factors are expanded at load into a low-rank term, whose rank can be larger than the adapter’s own rank. BOFT Liu et al., [17] is a variant of OFT whose rotation is a product of several permuted block-diagonal factors, so it needs one input transform for each factor, an extension of the single RR that we do not implement. SSF is a member, but the PEFT library (version 0.20) has no SSF method, so we do not consider it. HiRA Huang et al., [5] and full fine-tuning are not members, because they need a dense matrix for each adapter after the shared GEMM.

The Fold Rule

If a slot multiplies a matrix that belongs to the adapter, the engine can just multiply the two once, when the adapter loads. Thus, the slot disappears from serving. For example, (xA)⊙d=x(Adiag⁡(d))(xA) \odot d = x(A\operatorname{diag}(d)), so an inner vector dd folds into an owned AA. We say that such a slot folds, and we call a slot that does not fold live. A fold may change only tensors that the adapter owns, so it never changes the shared weight WW or factors that other adapters use (Appendix § 12 gives the exact definition).

Proposition 1 (Fold rule, informal). For adapters of low total rank and generic parameters, a slot folds if and only if the adapter owns a matrix or vector next to it in the product that applies it. In particular, the output scale ss, the output shift bb and the input transform RR never fold, because only the shared WW is next to them.

Proposition A1 in the appendix states the exact conditions, _久久爱 which are satisfied by every member in Table 1. The identities (xA)⊙d=x(Adiag⁡d)(xA) \odot d = x(A \operatorname{diag} d), ((xA)⊙d)B=(xA)(diag⁡(d)B)((xA) \odot d)B = (xA)(\operatorname{diag}(d)B) and (VB)⊙e=V(Bdiag⁡e)(VB) \odot e = V(B \operatorname{diag} e) prove the “if” direction. The “only if” direction is a rank argument, since a slot next to shared tensors changes the output by a matrix of high rank that the adapter’s owned low-rank factors cannot absorb. For the output scale ss, the same argument shows what an exact fold would need.

Corollary 1 (Folding ss, informal). To remove an output scale ss exactly, a member needs low-rank terms of total rank at least rank⁡(Wdiag⁡(s−1))\operatorname{rank}(W\operatorname{diag}(s-1)) minus the adapter’s own rank (Corollary A1).

For a trained DoRA adapter, the scale ss is close to one, with a root-mean-square deviation of 0.2%, but it differs from one in every column. Yet Wdiag⁡(s−1)W\operatorname{diag}(s-1) has full numerical rank on 60 of 60 adapted projections of Llama-3.1-8B, and even the best rank-16 approximation of it leaves at least 59.7% of its squared Frobenius norm (App. § 15). Hence, to fold DoRA’s scale exactly, an engine must merge the adapter into a copy of the base weight. This means that each DoRA tenant would then need its own copy of WW and its own GEMM, so its requests could no longer share a batch with other tenants. A batched engine must therefore keep ss live and apply it at serving time. In § 4, we show that the expand can do this at almost no extra cost.

VeRA’s scalar folds into its own vectors, but the vectors dd and ee stay live, because the factors next to them in (1) are shared. Thus, in contrast to DoRA, keeping the slots live is what makes VeRA cheap to serve. Folding dd and ee would otherwise give each tenant its own copy of AA and BB, which is equivalent to a conversion to a rank-256 LoRA adapter. With the slots live, all VeRA tenants share one copy of the factors, which a batch reads once per step (§ 4).

OFT’s rotation sits before the shared WW and cannot fold, but it still keeps the GEMM shared, because (xRa)W(xR_a)W for every adapter aa in a batch is one GEMM over the transformed rows.

Stacking. A request may also apply two adapters at one hook, one after the other. For example, a tenant may add its own LoRA adapter on top of an (IA)$3 adapter. Applying a member F2F_2 of (1) to the output of another member F1F_1 of (1) gives again a member of (1) (App. § 12). Its output scale is s1⊙s2s_1 \odot s_2 and its shift is b1⊙s2+b2b_1 \odot s_2 + b_2. It holds the terms of both members, where the scale s2s_2 of the second member also multiplies each term of the first, through its outer vector ee. The stack can therefore run as one segment with more terms, without an extra launch. However, it is important to note that the order of the two members matters. In the example, (IA)$3 after LoRA scales the LoRA update by s2s_2, while LoRA after (IA)$3 adds the update unscaled. In general, the two orders differ when one member has an output scale s≠1s \ne1 and the other has a shift or a term.

The Cost of Serving Methods Beyond LoRA

Section 3 showed which slots fold into an adapter’s own weights. A folded slot costs nothing at serving time, but a live slot must be computed for every token. This section asks how a LoRA schedule can compute each live slot, and what that costs. We first describe three ways to compute a live slot, which we call routes, and count the bytes that each one moves (§ 4.1). We then count the bytes of the alternative, converting each adapter to LoRA (§ 4.2). Finally, we turn these counts into a model of time and test it on the GPU (§ 4.3).

The Cost of Serving Through the Normal Form

A live slot must be computed inside the LoRA schedule of § 2, which runs the shared GEMM, the shrink and the expand for each layer. We call a way to do this a route. We introduce three routes to compute the live slots of the five members we are considering.

First, the output scale ss of DoRA and (IA)$3 multiplies hh, the output of the shared GEMM. The first route applies ss inside the expand. Second, the vectors dd and ee of VeRA sit next to factors that all VeRA tenants share. The second route computes this term with two GEMMs over the whole batch. Third, the rotation RR of OFT multiplies the input xx before the shared GEMM. The third route applies RR in a pass of its own.

Since adapter passes are limited by memory bandwidth (§ 2), the cost of a route can be quantified by the number of bytes that its passes read and write. Proposition 2 provides an overview of these bytes, with Propositions A2–A4 in the appendix giving the exact counts.

Proposition 2 (Cost of a live slot, informal). Consider a batch of NN tokens from kk adapters.

(i) A slot carried by a pass that already reads and writes its tensor adds one read of each adapter’s vector, at most kdoutkd_{\mathrm{out}} elements.

(ii) A slot on the output in a pass of its own adds 2Ndout2Nd_{\mathrm{out}} elements and one launch, and 4Ndout4Nd_{\mathrm{out}} elements if the pass first writes a copy of the slot for every row.

(iii) An input transform in a pass of its own adds 2Ndin2Nd_{\mathrm{in}} elements plus its blocks, and it does too little arithmetic per byte to be limited by anything but bandwidth.

(iv) A term on factors that all kk adapters share reads the factors once per step, while their LoRA conversions read them kk times.

Route one applies the scale ss in the expand. The expand reads hh, adds each adapter’s low-rank update, and writes the result, one tile of rows at a time. Each tile belongs to one adapter, so the expand can also multiply hh by that adapter’s ss before it writes the tile. This adds one read of ss per tile, at most kdoutkd_{\mathrm{out}} elements per step (Proposition 2(ii)), which is small next to the 2Ndout2Nd_{\mathrm{out}} elements that the expand already moves. The alternative is a pass of its own after the expand, which reads and writes the output a second time. By Proposition 2(ii), this pass doubles the expand’s traffic, and it triples that traffic if it first writes a copy of ss for every row.

Route two computes shared factors in GEMMs over row tiles. VeRA’s vectors dad_a and eae_a belong to each adapter aa, but the factors AA and BB next to them are shared, so the vectors stay live. For this term, the cost depends on the order in which a kernel visits the batch. The segmented expand visits the rows of one adapter at a time, so it reads BB once for each adapter, as kk conversions to LoRA would. A GEMM that splits the batch into tiles of rows, whatever their adapters, reads each tile of BB once for all the rows of a tile. The second route therefore runs two such GEMMs after the expand. The first computes xAxA for every row of the batch and multiplies each row by its adapter’s dad_a. The second multiplies the result by BB, multiplies each row by its adapter’s eae_a as each tile finishes, and adds the result to the output, skipping rows whose adapter has no such term. The factors are then read once per step, however many VeRA tenants the batch holds (Proposition 2(iv)), and the vectors add only k(r+dout)k(r+d_{\mathrm{out}}) elements.

Route three applies the rotation in a pass of its own. OFT’s rotation RR multiplies the input before the shared GEMM, and the expand runs after that GEMM, so the expand cannot carry RR. The third route applies RR in a pass of its own before the shared GEMM. This pass reads each row of the input, multiplies it by the blocks of that row’s adapter, and writes the result, which adds 2Ndin2Nd_{\mathrm{in}} elements plus the blocks (Proposition 2(iii)). Because RR is block diagonal, each output element needs only one small block, so the pass does little arithmetic per byte and stays limited by bandwidth. The rotated rows of all adapters then go through the shared GEMM, so OFT tenants still share WW with every other tenant.

The Cost of Serving Through Conversion

The alternative to a live slot is to convert the adapter to LoRA and serve with the existing kernel. The update of (IA)3(\mathrm{IA})^3, Wdiag⁡(s−1)W\operatorname{diag}(s-1), has full rank whenever WW has full rank and every entry of ss differs from one, and the update of OFT, (R−I)W(R-I)W, has full rank for a generic RR. A rank-rr LoRA conversion therefore only approximates them, and it reads r(din+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}}) elements per adapter per step. Table 2 gives these bytes for Llama-3.1-8B and a batch of 16 tenants, and for each method the route reads fewer bytes than the conversions. VeRA’s shared factors are the one large read, and that read happens once per step, however many VeRA tenants the batch holds. A route keeps each adapter exact and reads less than its conversion, and VeRA’s shared factors cost less than one exact conversion and less than rank-64 conversions from four tenants.

methodrouteexactGBms
LoRA, rank 16expand✓1.341.97
VeRA, rank 256route 2 (shared factors)✓1.191.75
as LoRA, rank 256✓21.4731.58
as LoRA, rank 64✗5.377.90
OFT, blocks of 32route 3 (pre-transform)✓1.281.88
as LoRA, rank 64✗5.377.90
(IA)³route 1 (expand)✓0.010.01
as LoRA, rank 64✗1.882.76

Table 2. The bytes that each route reads per decode step with 16 tenants in the batch, for Llama-3.1-8B (fp16, every adapted projection of 32 layers), and at that time at copy bandwidth. ✓: the route serves the adapter exactly. Bold: the fewest bytes for each method.

A Byte Model for Predicting Time Cost

We predict a route’s time as t0+ΔQ/β+ΔLλt_0+\Delta Q/\beta+\Delta L\lambda. Here t0t_0 is the measured time without the extra passes, and ΔQ\Delta Q and ΔL\Delta L are the bytes and launches that Proposition 2 adds. When the extra read and write fit in the L2 cache together, β\beta becomes βL2\beta_{L2}.

We test this model on OFT’s input transform and on four routes that apply DoRA’s ss outside the expand (Appendix § 19). Figure 2 shows every measured cell, and where the extra bytes dominate the launch cost, the median error on total time is 0.7%–14.0% per route. Where bytes dominate, the byte model predicts the time of a route from its bytes alone.

Figure 2. Predicted against measured time on the RTX A6000, for OFT’s input transform and for four routes that apply DoRA’s scale outside the expand (App. § 19). The line marks equality.

DoRA in vLLM. The byte counts also explain what DoRA costs in a real engine, where the choice is between the first route and a pass of its own after the expand. We measure both with vLLM’s own kernels, over every LoRA layer of Llama-3.1-8B. With DoRA’s ss carried in the expand (route one), the expand takes 0.985×–1.022× the time of vLLM’s LoRA expand for 1024 tokens or more, as Proposition 2(i) predicts. An open vLLM pull request instead writes a copy of the scale for every row and multiplies the output in a pass after the expand. This costs 2.38×–2.95× the time of the LoRA expand, close to the 3× that Proposition 2(ii) gives for such a pass. End to end, under a decode burst, four DoRA tenants served through our slot take 1.002× the time per output token of four LoRA tenants (Appendix § 20). Carried in the expand, DoRA’s scale costs about as much as plain LoRA.

Scute

We build these routes into vLLM, a production serving engine, without forking it. The system has two parts. An admission compiler turns each adapter into the normal form when the engine loads it (§ 5.1), and a runtime patch adds the routes to vLLM’s LoRA path (§ 5.2).

The Admission Compiler

Our compiler reads a checkpoint of the PEFT library and writes the adapter in normal form under vLLM’s LoRA tensor names, so that vLLM’s own loader, paging and scheduler handle it. LoRA and DoRA pass through unchanged. When the engine admits a DoRA adapter, it computes the slot s=m⊘∥W+cAB∥cs = m \oslash\lVert W + cAB \rVert_c from the magnitude mm, where ⊘\oslash is element-wise division and ∥⋅∥c\lVert\cdot\rVert_c is the norm of each output column. The engine computes ss at admission because it needs the base weight in the engine’s own layout. (IA)3^3 becomes the slot ss, on the up slice for the feed-forward network (§ 3.1). VeRA becomes the vectors dd and ee plus a reference to the shared pair, which is stored once. OFT’s parameters become explicit blocks RR, computed in the parametrization that the adapter was trained with. vLLM’s loader requires a term, so a method without one gets a zero rank-one term, which the expand skips.

The Runtime

The integration patches vLLM 0.19.1 at run time and leaves the scheduler, paging, attention and batch composition unchanged (App. § 18). Each adapted layer gets one buffer per slot, ss, dd, ee and RR, allocated with the LoRA weights and set to the identity. The layer’s forward pass runs four steps. First, the base GEMM runs, split by slice when an adapter has an RR, because qq, kk and vv each have their own rotation. Second, vLLM’s shrink runs, and it skips adapters without a term on the layer. Third, vLLM’s expand runs with ss applied as each tile finishes, which is a one-line change to the kernel, tiled_c += tiled_out * s. Fourth, the shared term of Proposition 2(iv) runs.

The input transform and the shared term are Triton ([25]) kernels that run every slice of a layer in one launch. We register them as custom PyTorch operators, which torch.compile treats as black boxes and CUDA graphs capture. Outside captured CUDA graphs, a step whose batch holds no OFT or VeRA adapter skips the split GEMM and the shared term. Tensor parallelism splits each layer across several GPUs. Each GPU then reads its own part of each factor, vector or block, and the layer’s existing all-reduce, a sum across GPUs, adds the partial terms.

Evaluating Scute

We now perform evaluations to answer the following. First, does Scute compute each method exactly, and how much accuracy would a conversion to LoRA lose (§ 6.2)? Second, how does the cost grow with the number of tenants of one method (§ 6.3)? Third, what does a pool of several methods cost, against LoRA-only and against the conversions (§ 6.4)? Fourth, what does the small size of the normal form give when many tenants share one GPU (§ 6.5)? Section 6.6 repeats the main results on a second GPU.

Setup

We use vLLM 0.19.1 and Llama-3.1-8B in fp16, with CUDA graphs and torch.compile on. vLLM’s LoRA kernels are tuned for the GPU at every rank that an engine serves. We train fifteen adapters with the PEFT library and its default settings, one for each of LoRA, DoRA, (IA)3^3, VeRA and OFT on each of GSM8K, MNLI and BoolQ (App. § 17). Serving cost depends only on an adapter’s shapes, so we measure it on synthetic tenants with the shapes of the trained adapters.

We compare four engines. Ours serves every adapter in normal form. Stock is vLLM serving every adapter converted to LoRA, VeRA at rank 256 and the other methods at rank 64. LoRA-only is vLLM serving the same number of rank-16 LoRA tenants, which shows the price of the other methods. The base model runs without adapters. vLLM runs every adapter at the largest rank that the engine serves, and pads smaller adapters with zeros.

Our slot buffers sit outside vLLM’s memory budget. We therefore give our engine 0.83 of device memory against 0.90 for the others, so that both use the same device memory, 44.8–44.9 GiB in the 20-tenant pool. Stock keeps every adapter resident, except in the 20-tenant pool with VeRA at rank 256, where 16 of the 20 fit.

A prefill-heavy load sends 256-token prompts for 8 output tokens, and a decode-heavy load sends 8-token prompts for 128 output tokens. Requests arrive at a fixed rate as a Poisson process, or all at once, which we call a burst. Tenants take turns, or, in one pool, follow a Zipf law, under which a few tenants receive most requests. Each measurement runs six repetitions with new prompts and arrival times, and each measurement with many tenants (§ 6.5) runs three. The engines of a repetition share these and run in every order equally often (App. § 20). We report the median of the per-repetition ratios, comparing median TPOT for decode-heavy load and mean TTFT for prefill-heavy load.

When an engine serves OFT, our runtime computes each fused base GEMM as one GEMM per slice (§ 5). On the RTX A6000, this split alone makes the base model faster, 26.8 against 30.6 ms per output token. For the pools that hold OFT, we therefore also run stock, LoRA-only and the base model with the same split. We call this the same GEMM path, and we call vLLM without the split as shipped.

Exactness and Accuracy

We first check that our engine computes what the PEFT library computes. Trained vectors are close to the identity, so a slot that is dropped, sharded wrongly or applied twice rarely changes the greedy tokens, and we therefore use amplified adapters, whose vectors move the log-probabilities by a clear margin. We serve one batch with 9 amplified adapters of all five methods and compare log-probabilities with the PEFT library in float32. For each method, the mean error is 0.3%–1.5% of the adapter’s effect, which is its change to the log-probabilities of the base model (App. § 18). The largest errors belong to the smallest effects, and they match the fp16 error of a plain LoRA adapter, and the same

holds on two GPUs with tensor parallelism. The fifteen trained adapters, served together, keep every task metric within 0.4 points of the PEFT library (Table 3 and Table A4). Here and below, a point is one percentage point of accuracy. Our engine serves all five methods in one batch and matches the PEFT library with a mean error of at most 1.5% of each adapter’s effect, and to within 0.4 points of its accuracy.

methodtaskPEFTScuteSVDASVDASVDg_gdist.
(IA)³GSM8K59.8+0.4−18.6−0.8−19.6−0.4
MNLI90.20.0−23.2+0.5−16.9−0.3
BoolQ87.9−0.1−4.8+0.8−1.6+0.3
OFTGSM8K63.0+0.2−5.6−1.2−4.6−2.0
MNLI90.90.0−2.9+0.3−0.9−0.1
BoolQ89.70.0−0.6−0.1−0.2−0.8
VeRAGSM8K58.4+0.2+0.8−0.8+3.4—
MNLI90.9+0.1−0.6+0.30.0—
BoolQ89.50.0−0.1+0.1−0.1—

Table 3. Task accuracy of the trained (IA)³, OFT and VeRA adapters. PEFT is the accuracy in percent when the PEFT library runs the adapter. The other columns give the difference from PEFT in points, in Scute (all fifteen adapters in one batch) and converted to a LoRA of rank 64. Bold marks the smallest absolute difference in each row. SVD is a truncated SVD of the update. ASVD is activation-aware SVD fitted on 128 examples of the tenant’s own task, and ASVDg_g is the same fitted on generic text. Dist. is distillation on the tenant’s task. VeRA was not distilled. App. § 16 gives other ranks and fits.

methodtaskPEFTScute1664256
(IA)3^3GSM8K59.8+0.4−29.8−18.2−6.2
MNLI90.20.0−35.8−23.2−7.1
BoolQ87.9−0.1−10.5−4.7+0.5
OFTGSM8K63.0+0.2−23.4−5.6+0.8
MNLI90.90.0−10.7−2.9+0.1
BoolQ89.70.0−2.2−0.6+0.2
DoRAGSM8K66.0−0.4+1.4+1.2+1.2
MNLI91.90.00.0−0.10.0
BoolQ88.70.00.0−0.10.0
VeRAGSM8K58.4+0.2——+0.2
MNLI90.9+0.1——+0.1
BoolQ89.50.0——0.0
LoRAGSM8K66.4+0.2−0.2——
MNLI90.80.00.0——
BoolQ90.00.00.0——

Table A4. Every trained adapter. PEFT is the accuracy in percent when the PEFT library runs the adapter. The other columns give the difference from PEFT in points, in Scute with all fifteen in one batch, and converted to LoRA by SVD and served by stock vLLM. Bold marks the smallest absolute difference in each row. For LoRA, the rank-16 column is the adapter as trained. Stock converts VeRA exactly, at its own rank of 256; Table A3 gives its truncated fits. The base model scores 26.2%, 36.8% and 74.0%.

Next, we convert the six (IA)³ and OFT adapters at ranks 16 and 64. Truncated SVD, which minimizes the error in the weights, falls more than 3 points below the PEFT library in 9 of the 12 cases. Activation-aware SVD ([32]) minimizes the error on sample inputs, and when it is fitted on the tenant’s own task data it stays within 1.2 points of the PEFT library at rank 64 on all six adapters. Distillation on the same data stays within 2.0 points. Fitted on generic text, activation-aware SVD falls more than 3 points below the PEFT library in 8 of the 12 cases, and (IA)3^{3} loses up to 19.6 points at rank 64. VeRA converts well by plain SVD at rank 64, within 0.6 points, but it loses up to 18.5 points at rank 16. DoRA converts within noise, because its magnitude changes little (App. D). Converted to a rank-64 LoRA without access to the original task data, (IA)3^{3} loses up to 23.2 points of accuracy, and a rank-16 conversion of VeRA loses up to 18.5 points. Our engine serves every adapter exactly from its checkpoint alone.

Cost per Tenant

Next, we isolate what each tenant costs. Figure 3 grows the number of tenants of one method, with every adapter resident in both engines.

Figure 3. Time per output token under a decode burst against the number of tenants of one method, RTX A6000. Scute serves each adapter exactly. Stock serves each adapter converted to LoRA, at rank 64 and, for OFT, also at rank 256, with every adapter resident. The base model, without adapters, is shown for reference. Stock and the base model run as shipped. With OFT, Scute runs the split base GEMM of § 6.1, which is faster than the fused base GEMM. Median of six repetitions.

Our engine reads VeRA’s shared factors once per step, so its time stays flat, 35.4 ms per token at one tenant and 36.1 ms at 16. Converted, each VeRA tenant brings its own rank-256 factors, so stock takes 36.7 ms at one tenant and 65.1 ms at 16, which is 1.80× our time.

OFT’s blocks are as small as the factors of a rank-16 LoRA, and (IA)3^{3} owns only vectors. At 16 tenants, the rank-64 conversion of (IA)3^{3} takes 1.23× our time, and that of OFT takes 1.23× with the same GEMM path (1.34× as shipped). Each conversion adds decode time per tenant within 10% of the byte model’s prediction, while our engine adds 0.05–0.16 ms per tenant (App. G). Under a prefill burst, stock takes 1.02×–1.04× our mean TTFT at one tenant and 1.05×–1.09× at 16.

Each tenant converted to LoRA adds the time to read its own factors, as the byte model predicts, while each tenant in our engine adds only 0.05–0.16 ms per token.

Pools of Methods

A platform serves several methods at once, so we now mix them in one engine. Table 4 gives the controlled comparison, under a burst, with every engine on the same GEMM path. Figure 5 then shows that the result holds at every arrival rate in vLLM as shipped.

poolloadScute / LoRA-onlyConversion / ScuteConversion / Scute (stock)
five methodsdecode1.171.651.73
five methodsprefill1.181.141.12
VeRA at 64decode1.111.12—
VeRA at 64prefill1.120.94—
no VeRAdecode1.051.141.25
no VeRAprefill1.041.010.99

Table 4. Here we measure serving times for pools of several methods in one engine, under a burst, RTX A6000. The five methods pool holds 20 tenants, four of each method, and stock keeps 16 of them resident. VeRA at 64 is a separate run of the same pool, with stock serving VeRA converted at rank 64 instead of 256. The no VeRA pool holds 16 tenants, four of each of LoRA, DoRA, (IA)3^{3} and OFT. We measure the ratios of serving times for Scute against serving the same number of plain rank-16 LoRA adapters, for the conversions against Scute on the same GEMM path, and for the conversions on vLLM as shipped against Scute. Ratios are the median over six repetitions. App. G gives every rate of vLLM as shipped.

Figure 5. Latency against arrival rate in the two pools, RTX A6000. Scute decodes faster than stock at every rate, and under prefill-heavy load stock’s TTFT grows fastest. Stock, LoRA-only and the base model run as shipped, so the gap between Scute and LoRA-only differs from the same-GEMM-path ratio of Table 4. Top, median TPOT under decode-heavy load. Bottom, mean TTFT under prefill-heavy load. Median of six repetitions.

The price of exactness. We compare our engine with LoRA-only, both on the same GEMM path. Under a burst, the pool of all five methods takes 1.17× the decode time and 1.18× the TTFT of LoRA-only, and the pool of four methods without VeRA takes 1.05× and 1.04×. The extra time comes from the routes, which are VeRA’s shared term, OFT’s input transform, and the vectors that each step reads.

Against the conversions. VeRA’s conversion is exact at rank 256 and keeps its accuracy at rank 64 (§ 6.2). With the rank-64 conversion, stock pads every adapter only to

rank 64 and keeps all 20 tenants resident, yet on the same GEMM path it still takes 1.12× our decode time under a burst and 1.13× at 16 requests per second, because each tenant brings its own factors. With VeRA at rank 256, the conversions take 1.65× our decode time.

At every rate. Figure 5 shows the two main pools from 2 to 16 requests per second. vLLM as shipped takes 1.21×–1.98× our decode time in every pool and at every rate, including under Zipf popularity and with two GPUs (App. G). In the five-method pool, most of this time comes from padding. vLLM runs every adapter at the largest rank in the engine, here 256, and it also runs zero factors on the projections that a tenant does not adapt. The bytes of the conversions at their own ranks would take 13.0 ms per step, and padding raises this to 31.6 ms, close to the measured 34.7 ms (App. G). Under prefill-heavy load, stock’s TTFT grows fastest once the engine saturates, to 5.96× ours at 16 requests per second.

Under a burst on one RTX A6000, exact serving of a mix of methods takes only 5–17% more decode time than serving LoRA adapters alone. Converted to LoRA, each mix takes at least 12% more decode time than our engine on the same GEMM path, and vLLM as shipped takes 1.21×–1.98× our decode time at every rate.

Many Tenants

The normal form keeps each adapter small. An (IA)3^{3} or VeRA tenant owns a few vectors, a few megabytes in total, and all VeRA tenants share one pair of factors, while the rank-64 LoRA that keeps VeRA’s accuracy is 0.34 GB. On one RTX A6000, our engine can hold all 1024 tenants resident with a KV cache of 98k tokens, while stock can hold 64 rank-64 conversions, with a cache of 28k tokens.

We serve 64, 256 and 1024 tenants, half (IA)3(\mathrm{IA})^3 and half VeRA, in turn under decode-heavy load. We limit our engine to 64 resident tenants, close to the 48 of stock, so that both engines copy adapters into GPU memory when a request needs them. Our engine keeps the other tenants in host memory, which holds all 1024 at a few megabytes each. Stock keeps 48 resident, which leaves it a cache of 75k tokens, keeps 256 more in host memory, and loads the rest from storage. Each of its rank-64 conversions is 0.34 GB. A request meets the latency target if its first token arrives within 1 s and each later token within 100 ms. The sustainable rate is the highest arrival rate at which at least 90% of requests meet the target. Figure 4 shows the share of requests that meet the target at each rate (App. § 20 gives the setup). Our engine sustains 10, 8 and 6 requests per second at 64, 256 and 1024 tenants, and stock sustains 4 at each, so our sustainable rate is 1.5× to at least 2.5× that of stock. With 64 tenants, our engine still meets the target at 10 requests per second, the highest rate that we tested, so 2.5× is a lower bound. At 1024 tenants and 8 requests per second, 82% of our requests meet the target, against 9% for stock. Our engine delivers 932 tokens per second, against 464 for stock, which is 2.01× the throughput (2.03× at 10 requests per second), and stock’s mean TTFT is 65× ours.

Figure 4. Share of requests that meet the latency target (first token within 1 s, then at most 100 ms per token) against arrival rate, under decode-heavy load on one RTX A6000. The pools hold 64, 256 and 1024 tenants, half (IA)3(\mathrm{IA})^3 and half VeRA. Scute serves them exactly, and stock serves their rank-64 conversions. Median of three repetitions.

Because each exact adapter is only a few megabytes, one RTX A6000 serves 1024 tenants at 1.5× the request rate that their LoRA conversions sustain within the latency target, and at 8 requests per second it delivers twice their throughput.

The A100

We repeat the measurements on an A100-SXM4-80GB (App. § 22), where the mixed batch of all five methods again matches the PEFT library to within 0.3%–1.6% of each adapter’s effect. With 80 GB, stock keeps every converted adapter resident, including all twenty at rank 256, and our engine is still faster in every decode-heavy cell. Stock takes 1.12×–1.69× our time per output token in the mixed pools at every rate, and 1.83× with 16 VeRA tenants. The conversions of OFFT and (IA)3(\mathrm{IA})^3 take 1.31× and 1.34× our time at 16 tenants. With many tenants, stock keeps 80 conversions resident, and at 64 tenants our engine sustains 20 requests per second against 8 for stock. At 1024 tenants, our engine sustains 8 requests per second, a rate at which stock meets the target for 32% of requests. At 256 tenants, both engines sustain 8 requests per second. At 24 requests per second, our engine delivers 1.2×–1.8× the throughput of stock. Exact serving costs more on the A100 than on the RTX A6000. In the mixed pools, ours takes 1.07×–1.89× the time of LoRA-only, and under prefill-heavy load stock’s mean TTFT is as low as 0.76×0.76\times ours (App. § 22).

On the A100, with 80 GB of memory, our engine still decodes faster than the conversions in every decode-heavy cell.

Related Work

Serving many adapters. PetS ([35]) first served several PEFT methods over one shared encoder model, batching the shared GEMM across tasks and running each task’s own operator on its part of the batch. In our engine, every method’s adapter work runs in the same batched kernels as LoRA. Punica ([2]) and S-LoRA ([24]) introduced the segmented LoRA schedule and adapter paging. FLoRA ([27]) batches a different LoRA adapter for each example. Later systems schedule, place, cache or compress LoRA adapters ([20, 8, 7, 1] (Wu et al. 2024; Li et al. 2024a)). DeltaZip ([30]) and BitDelta ([15]) serve full fine-tunes by compressing each one’s difference from the base model, which, like a conversion to LoRA, is an approximation at a chosen size. We ask which slots a LoRA schedule needs to serve each method exactly, which slots fold at load, and what each live slot costs.

Serving other methods. An open vLLM pull request2 computes DoRA’s scale at load, supports both tensor-parallel layouts, mixes LoRA and DoRA requests, and applies the scale after the LoRA expand. Zelenin & Zhuravlyova [33] fuse DoRA’s norm computation into the forward pass of a single adapter, while our engine computes DoRA’s scale once at load and so removes that computation from serving. Qiu et al. [22] make OFT training scale by rotating the inputs, and the same identity lets OFT keep the shared GEMM at serving time. VB-LoRA ((Li et al., 2024b)) and Tied-LoRA ([23]) also share factors across layers or adapters and train small parts per adapter. Where shared factors meet per-adapter vectors, as in VeRA, the shared-factor route applies.

Kernels and cost models. The expand is a Triton ([25]) kernel in the line of Punica’s SGMV and FlashInfer ((Ye et al., 2025)). The byte model is a roofline ([28]) argument applied to the passes that an adapter adds.

Conclusion

Serving engines batch requests for many LoRA adapters, but they reject adapters of other PEFT methods, and converting these adapters to LoRA costs accuracy or memory. This paper shows that modifications to the batched LoRA kernel can be made to facilitate the serving of the main PEFT methods exactly, in one batch, from each adapter’s checkpoint alone.

First, we provide a normal form with six slots that expresses LoRA, DoRA, (IA)3^3, SSF, VeRA and OFT, and a fold rule tells which slots an engine can merge into the adapter at load and which slots must stay live. The rule shows, for example, that an exact fold of DoRA’s scale needs a copy of the base weight for each tenant, so a batched engine must keep that scale live. Second, we introduce three routes to compute every live slot inside the LoRA schedule, and a byte model that predicts the time that a route adds from the bytes that it moves, with no fitted parameters, where bandwidth sets the cost. Third, we organize this into a compiler and a run-time patch to vLLM to serve all five methods in one batch, with CUDA graphs and torch.compile on.

As a result, a platform no longer has to choose between the methods that its tenants train and the efficiency of batched serving. Our engine matches the PEFT library with a mean error of at most 1.5% of each adapter’s effect. Under a burst on an RTX A6000, a mix of four or five methods costs only 5–17% more decode time than LoRA-only. Conversion to LoRA, in contrast, loses up to 23.2 points of accuracy for (IA)3^3 without the tenant’s data, and it gives each VeRA tenant its own copy of the shared factors. Because each adapter keeps the size of its own parameters, one RTX A6000 serves 64 to 1024 tenants at 1.5×1.5\times to at least 2.5×2.5\times the sustainable rate of their conversions. To serve a new method, an engine only needs to map it to the slots, and the fold rule and byte model tell in advance which slots stay live and how many bytes they move. Our measurements use Llama-3.1-8B on RTX A6000 and A100 GPUs, serving costs use synthetic tenants with the shapes of trained adapters, and accuracy uses the fifteen adapters that we trained.

Acknowledgments

This work was supported by ONR grant N00014-23-1-2714, DOE grant DE-SC0020345, DOI grant 140D0423C0076, and a Google Cloud Computing Award.

AI Usage Disclosure

We used Claude Code (Anthropic) to assist with code implementation, benchmarking and plotting scripts, bibliography verification, and drafting and editing the text of the paper under the authors’ direction. All system designs, mathematical proofs, kernel implementations, experimental results, and text were verified and revised by the authors, who take full responsibility for the content of the paper.

Proofs

We write the term matrix of adapter aa as M=∑kckAkdiag⁡(dk)Bkdiag⁡(ek)M = \sum_k c_k A_k \operatorname{diag}(d_k) B_k \operatorname{diag}(e_k). Then Fa(x)=xRWdiag⁡(s)+b+xMF_a(x) = x R W \operatorname{diag}(s) + b + xM and rank⁡M≤∑krk\operatorname{rank} M \le\sum_k r_k. Two members are equal when they agree for every xx. This section states the exact versions of the informal results of §3 and §4, each before its proof.

Folds. A slot vv of adapter aa that is not the identity folds if some member computes FaF_a for every xx with vv set to the identity, and this member differs from aa only in tensors that aa owns, each with the same shape and ownership.

Proposition A1 (Fold rule; exact version of Proposition 1). Let r=∑krkr = \sum_k r_k be the total rank. Assume that 4r<min⁡(din,dout)4r < \min(d_{\mathrm{in}}, d_{\mathrm{out}}), or, if every AkA_k is shared, that 3r<din3r < d_{\mathrm{in}} and 2r<dout2r < d_{\mathrm{out}}. Assume also that no two terms share a factor, that an owned RR is the adapter’s only owned tensor, and that a term with owned dkd_k and eke_k on shared AkA_k, BkB_k is the only term. Then, for all parameters outside a set of measure zero, a slot folds if and only if the adapter owns a tensor next to it in the product that applies it. In detail, ckc_k folds when any of AkA_k, dkd_k, BkB_k, eke_k is owned, dkd_k folds when AkA_k or BkB_k is owned, and eke_k folds when BkB_k is owned, while ss, bb and RR never fold.

Corollary A1 (Folding ss; exact version of Corollary 1). If R=IR = I, any member that equals FaF_a with s=1s = 1 has a total term rank of at least rank⁡(Wdiag⁡(s−1))−r\operatorname{rank}(W \operatorname{diag}(s - 1)) - r.

Every member of Table 1 meets the ownership conditions: OFT owns only RR, and VeRA has one term. The second size condition covers the key and value projections of VeRA, where dout=1024=4rd_{\mathrm{out}} = 1024 = 4r.

Proposition A1, the “if” direction. Each identity in §3.2 replaces one owned tensor by a tensor of the same shape and sets the slot to its identity. For ckc_k, use ckAk=(ckAk)⋅1c_k A_k = (c_k A_k) \cdot1, and likewise for BkB_k, dkd_k and eke_k. For dkd_k, use Akdiag⁡(dk)A_k \operatorname{diag}(d_k) or diag⁡(dk)Bk\operatorname{diag}(d_k)B_k. For eke_k, use Bkdiag⁡(ek)B_k \operatorname{diag}(e_k). Ownership and shapes do not change, so each is a fold.

Proposition A1, the “only if” direction. Let F′F' be a candidate fold of FaF_a that sets the slot under test to its identity. It may change every owned tensor, in every term, but no shared tensor. Equality for all xx separates the constant and linear parts. So b′=bb' = b, and

RWdiag⁡(s)−R′Wdiag⁡(s′)=M′−M.(2)R W \operatorname{diag}(s) - R' W \operatorname{diag}(s') = M' - M. \tag*{(2)}

The right side has rank at most 2r2r. We call the parameters generic when they avoid finitely many proper algebraic subsets of parameter space. Then every square submatrix of WW is invertible. When the AkA_k are shared, the columns of RWRW are also in general position with respect to their span. This excludes a set of measure zero.

This excludes a set of measure zero. The arguments below need nothing else, so they hold for every candidate at once.

The shift bb. bb is the constant part, and nothing else is constant in xx.

The transform RR. By assumption, an owned RR is the only owned tensor. So s′=ss' = s, M′=MM' = M, and a fold sets R′=IR' = I. Then (A1) reads (R−I)Wdiag⁡(s)=0(R - I)W \operatorname{diag}(s) = 0. R−IR - I is block-diagonal, so block jj gives (Rj−I)Wjdiag⁡(s)=0(R_j - I)W_j \operatorname{diag}(s) = 0, where WjW_j holds the bRb_R rows of WW that block jj mixes. For generic WW and bR≤doutb_R \le d_{\mathrm{out}}, WjW_j has full row rank. So, for ss without zeros, Rj=IR_j = I for every jj. The argument also holds when din>doutd_{\mathrm{in}} > d_{\mathrm{out}}, where WW has a left null space.

The scale ss. Here RR is not owned, so R′=RR' = R. The left side of (A1) is RWdiag⁡(s−s′)RW \operatorname{diag}(s - s'). For generic WW and invertible RR, its rank is min⁡(din,∣supp⁡(s−s′)∣)\min(d_{\mathrm{in}}, \lvert\operatorname{supp}(s - s')\rvert). So s′s' differs from ss on at most 2r2r columns. A fold sets s′=1s' = 1; therefore needs s−1s - 1 to be supported on at most 2r<dout2r < d_{\mathrm{out}} columns, which is false for generic ss. If every AkA_k is shared, the columns of MM and M′M' lie in the span of the AkA_k, of dimension at most rr. The left side spans the ∣supp⁡(s−s′)∣≤2r\lvert\operatorname{supp}(s - s')\rvert\le2r columns of RWRW on that support. For generic WW and 3r<din3r < d_{\mathrm{in}}, the two spans meet only in zero, so s′=ss' = s. The same equation gives Corollary A1. A candidate with s′=1s' = 1 has rank M′≥rank⁡(Wdiag⁡(s−1))−rank⁡MM' \ge\operatorname{rank}(W \operatorname{diag}(s - 1)) - \operatorname{rank} M.

The vectors and ckc_k. Let CC be the span of the coordinate vectors on supp⁡(s−s′)\operatorname{supp}(s - s'), of dimension at most 2r2r. The left side of (A1) has its rows in CC. So MP=M′PMP = M'P for PP the projection onto the orthogonal complement of V+CV + C, for any subspace VV. There are finitely many choices of CC, and C={0}C = \{0\} when every AkA_k is shared. No two terms share a factor. So, for generic factors and dim⁡V+dim⁡C+r<dout\dim V + \dim C + r < d_{\mathrm{out}}, the terms of MM have independent column spaces, and, after PP, independent row spaces. The rank of a sum of terms after PP is then the sum of their ranks. Every VV below has dimension at most rr, so either size condition of the proposition suffices.

eke_k with BkB_k shared. Take V=row⁡(Bk)V = \operatorname{row}(B_k). Term kk of M′M' has eke_k set to one, so its rows lie in VV, and PP removes it. Hence rank⁡M′P≤r−rk\operatorname{rank} M'P \le r - r_k. Term kk of MM has row space Vdiag⁡(ek)V \operatorname{diag}(e_k). For eke_k outside a set of measure zero, this space meets V+CV + C and the row spaces of the other terms only in zero. Hence rank⁡MP=r\operatorname{rank} MP = r, a contradiction. The exceptional set contains more than ek∝1e_k \propto1, whose scalar folds into ckc_k. An eke_k that differs from a constant in only a few entries leaves Vdiag⁡(ek)∩VV \operatorname{diag}(e_k) \cap V of dimension near rkr_k, and the proposition makes no claim there.

ckc_k with nothing owned in term kk. Take V={0}V = \{0\}. Then ∑j≠kTj′P=(∑j≠kTj+(ck−1)Fk)P\sum_{j \ne k} T_j'P = (\sum_{j \ne k} T_j + (c_k - 1)F_k)P, where TjT_j is term jj and FkF_k is the fixed term kk without ckc_k. The left side has rank at most r−rkr - r_k. The right side has rank rr whenever ck≠1c_k \ne1. So ck=1c_k = 1.

dkd_k with AkA_k and BkB_k shared. If eke_k is not owned, term kk of M−M′M-M' is Akdiag⁡(ckdk−ck′)Bkdiag⁡(ek)A_k\operatorname{diag}(c_kd_k-c'_k)B_k\operatorname{diag}(e_k). Its rank is the number of entries of ckdkc_kd_k that differ from ck′c'_k, which is at least one unless dk∝1d_k\propto1. The argument for ckc_k then gives a contradiction. If eke_k is owned and term kk is the only term, as in VeRA, then M′P=MPM'P=MP with AkA_k of full column rank gives ckdibijej=ck′bijej′c_kd_i b_{ij}e_j=c'_k b_{ij}e'_j for the entries bijb_{ij} of BkB_k outside CC. These entries are all nonzero for generic BkB_k, so did_i does not depend on ii. ■

Stacking. For members with R=IR=I, write Fa(x;h)=h⊙sa+ba+xMaF_a(x;h)=h\odot s_a+b_a+xM_a, so that Fa(x)=Fa(x;xW)F_a(x)=F_a(x;xW). A stack applies F2F_2 to the output of F1F_1, (F2∘F1)(x)=F2(x;F1(x))(F_2\circ F_1)(x)=F_2(x;F_1(x)). Then

F2(x;F1(x))=(xW⊙s1+b1+xM1)⊙s2+b2+xM2=xW⊙(s1⊙s2)+(b1⊙s2+b2)+xM1diag⁡(s2)+xM2.\begin{aligned} F_2(x;F_1(x))&=(xW\odot s_1+b_1+xM_1)\odot s_2+b_2+xM_2\\ &=xW\odot(s_1\odot s_2)+(b_1\odot s_2+b_2)+xM_1\operatorname{diag}(s_2)+xM_2. \end{aligned}

M1diag⁡(s2)M_1\operatorname{diag}(s_2) is M1M_1 with each eje_j replaced by ej⊙s2e_j\odot s_2, so the stack is a member. The rewritten eje_j folds into BjB_j when the stack owns a copy of BjB_j. When a server reuses the inner adapter’s BjB_j instead of copying it, eje_j is a live slot unless s2=1s_2=1. Exchanging the two members shows that the two orders agree if and only if b1⊙s2+b2=b2⊙s1+b1b_1\odot s_2+b_2=b_2\odot s_1+b_1 and M1diag⁡(s2−1)=M2diag⁡(s1−1)M_1\operatorname{diag}(s_2-1)=M_2\operatorname{diag}(s_1-1). If neither member scales, both sides are equal, so stacks of shifts and terms commute. If one member scales and the other has a shift or a term, the condition fails for generic parameters.

Byte counts

The following are the exact versions of Proposition 2. They count elements read from and written to GPU memory.

Proposition A2 (Live output slot). Let a live slot scale a tensor TT of shape [N,dout][N,d_{\mathrm{out}}]. If a pass that already reads and writes TT carries the slot, the slot adds one read of its vector per tile of rows, which for kk adapters in the batch is at most kdoutkd_{\mathrm{out}} elements. If the slot runs in a pass of its own, it adds at least 2Ndout2Nd_{\mathrm{out}} elements and one launch, and if that pass first writes a copy of the slot for every row, it adds 4Ndout4Nd_{\mathrm{out}} elements.

Proposition A3 (Live input slot). An input transform RR with blocks of size bRb_R, in a pass of its own, moves 2Ndin+kdinbR2Nd_{\mathrm{in}}+kd_{\mathrm{in}}b_R elements and performs bR/qb_R/q arithmetic operations per byte, where qq is the number of bytes per element. For the PEFT library’s default block size of 32, this is far below the point where the RTX A6000 or the A100 becomes limited by arithmetic.

Proposition A4 (Shared factors). Let kk adapters in a batch each have a term ((xA)⊙da)B⊙ea((xA)\odot d_a)B\odot e_a on the same factors A,BA,B of rank rr. Two GEMMs over the batch, with each row’s dad_a and eae_a read alongside, read r(din+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}}) factor elements per step for any kk, plus k(r+dout)k(r+d_{\mathrm{out}}) vector elements. Converted to LoRA, each adapter owns a copy of the factors, and the batch reads kr(din+dout)kr(d_{\mathrm{in}}+d_{\mathrm{out}}) elements.

Propositions A2–A4. A pass that applies a slot to TT must read every element of TT and write every element of the result. If the pass already reads and writes TT, only the slot’s elements are new. This is one read per row tile, at most kdoutkd_{\mathrm{out}} distinct elements from GPU memory, with the rest served from L2. Otherwise, the read and the write are both new. Writing a per-row copy of the slot first adds one more [N,dout][N,d_{\mathrm{out}}] tensor, written and then read. For RR, the pass reads and writes [N,din][N,d_{\mathrm{in}}] and reads the din/bRd_{\mathrm{in}}/b_R blocks of bR2b_R^2 elements of each of kk adapters. Each output element costs 2bR2b_R arithmetic operations for 2q2q bytes, which is bR/qb_R/q operations per byte with q=itemsizeq=\text{itemsize}. For the shared factors, each GEMM over the batch reads its factor once per step, and each row reads its own dad_a and eae_a, which gives r(din+dout)+k(r+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}})+k(r+d_{\mathrm{out}}). A conversion gives each of the kk adapters its own rank-rr factors, so the batch reads kr(din+dout)kr(d_{\mathrm{in}}+d_{\mathrm{out}}). ■

Tensor parallelism

We show how DoRA’s gg is computed and applied when a layer is split across GPUs. A column-parallel layer splits doutd_{\mathrm{out}}. The norm ∥W+cAB∥c\lVert W+cAB\rVert_c is taken over dind_{\mathrm{in}} for each output column, so each GPU holds complete columns and computes its own part of DoRA’s gg. A row-parallel layer splits dind_{\mathrm{in}}. Each GPU then holds a partial sum of squares for every column, and ∥⋅∥c2=∑t∥⋅∥c,t2\lVert\cdot\rVert_c^2=\sum_t\lVert\cdot\rVert_{c,t}^2 needs one all-reduce of doutd_{\mathrm{out}} values before the square root. This all-reduce is necessary, because a column norm depends on every part. At serving time, a row-parallel layer’s output is ∑tht\sum_t h_t over GPUs, and s⊙∑t(ht+VtB′)=∑ts⊙(ht+VtB′)s\odot\sum_t(h_t+V_tB')=\sum_t s\odot(h_t+V_tB'). So each GPU applies ss in its own expand. Table A1 compares each GPU’s gg with the value computed on one GPU, element by element, over 3328 slices. The largest row-parallel difference is two to four fp32 units in the last place, for values near one.

degreeshardingslicesmax ∣Δg∣\lvert\Delta g\rvert
2column-parallel5120
2row-parallel2562.4×10−72.4\times10^{-7}
4column-parallel5120
4row-parallel5122.4×10−72.4\times10^{-7}
8column-parallel5120
8row-parallel10242.4×10−72.4\times10^{-7}

Table A1. DoRA’s gg computed on real tensor-parallel shards against gg computed on one GPU, for four DoRA adapters.

Folding DoRA’s Scale on Real Weights

We take four public DoRA fine-tunes of Llama-3.1-8B and layers 0, 15 and 31. For each, we compute gg exactly as the engine does, and the singular values of Wdiag⁡(g−1)W\operatorname{diag}(g-1) in float64 (Table A2). Every one of the 60 of 60 matrices has full numerical rank. A trained gg deviates from one by only 0.2% in root mean square. The deviation is spread over every column, so even the best rank-16 approximation leaves at least 59.7% of the matrix’s squared Frobenius norm in every case.

proj.shaperankleft, r=16r=16rms g−1g-1
q4096×40964096 \times4096409661.6%0.0022
k4096×10244096 \times1024102459.7%0.0019
v4096×10244096 \times1024102483.2%0.0031
up4096×143364096 \times14336409692.8%0.0023
down14336×409614336 \times4096409695.9%0.0018

Table A2. Wdiag⁡(g−1)W\operatorname{diag}(g-1) for each projection, of shape din×doutd_{\mathrm{in}}\times d_{\mathrm{out}}: the minimum numerical rank, the minimum share of its squared Frobenius norm outside the best rank-16 approximation, and the median RMS deviation of gg, over four adapters and three layers.

Converting to LoRA

Each conversion approximates the exact update ΔW\Delta W of every adapted module. We compute ΔW\Delta W from the module of the PEFT library, and it matches the module’s own output to a relative error of at most 10−410^{-4} in float32. SVD keeps the top kk singular directions of ΔW\Delta W. ASVD, activation-aware SVD, minimizes ∥X(ΔW−L)∥F\lVert X(\Delta W-L)\rVert_F over rank-kk matrices LL. Here XX holds the module’s inputs on 128 calibration sequences, captured before the (IA)3^3 scale or the OFT rotation. We accumulate the input covariance in float64, regularise it by 10−610^{-6} times its mean diagonal, and take the minimizer in closed form from its eigendecomposition. The calibration sequences are the first 128 training examples of the adapter’s own task (ASVD), of another task (ASVDx_x: GSM8K from MNLI, MNLI from BoolQ, BoolQ from GSM8K), or 128 chunks of 160 tokens of WikiText-2 (ASVDg_g). Distillation starts from ASVD. It minimizes the KL divergence from the next-token distribution of the adapter in the PEFT library on the completion tokens of the adapter’s training data, for 300 steps at a learning rate of 10−410^{-4}, with the last 200 training examples held out. We do not distill VeRA, because own-task ASVD at rank 64 is already within 3 points of the PEFT library on every task. We evaluate conversions with the training harness on the examples of App. § 17. Its SVD results match the same conversions served by stock vLLM to within 0.4 points.

Table A3 gives every conversion at ranks 16 and 64. Table A4 gives every trained adapter, LoRA, DoRA and VeRA included, served by our engine and, converted by SVD, by stock vLLM.

methodtaskPEFTrank 16 SVDrank 16 ASVDrank 16 ASVDx_xrank 16 ASVDg_grank 16 dist.rank 64 SVDrank 64 ASVDrank 64 ASVDx_xrank 64 ASVDg_grank 64 dist.
(IA)3^3GSM8K59.8−29.8−24.6−34.4−32.8−3.2−18.6−0.8−37.8−19.6−0.4
MNLI90.2−35.9−0.9−23.2−25.6+0.3−23.2+0.5−15.4−16.9−0.3
BoolQ87.9−10.4−0.1−2.1−4.4+0.5−4.8+0.8−0.3−1.6+0.3
OFTGSM8K63.0−23.4−3.0−12.6−16.2−2.4−5.6−1.2−2.0−4.6−2.0
MNLI90.9−10.7+0.3−6.5−9.6−0.8−2.9+0.3−0.6−0.9−0.1
BoolQ89.7−2.3−0.2−0.5−2.5−0.5−0.6−0.10.0−0.2−0.8
VeRAGSM8K58.4−14.8+2.0−1.6−5.4—+0.8−0.8+3.2+3.4—
MNLI90.9−18.50.0−6.8−10.3—−0.6+0.30.00.0—
BoolQ89.5−6.9+0.1−1.0−1.9—−0.1+0.1+0.3−0.1—

Table A3. Every conversion of the trained (IA)3^3, OFT and VeRA adapters (VeRA not distilled): SVD, ASVD fitted on the task’s own data, on another task’s data (x) and on generic text (g), and distillation, evaluated with the training harness. PEFT is the accuracy in percent, and the other columns give the difference from PEFT in points.

Adapters, Training and DoRA in Practice

Training. We train fifteen adapters of Llama-3.1-8B with the PEFT library, version 0.20, one per method and task. LoRA and DoRA use rank 16 and α=32\alpha=32. VeRA uses rank 256, with its projections saved and the library’s default seed. OFT uses blocks of 32 in the library’s default Cayley form. These four methods adapt qq, kk, vv, oo, gate, up and down. (IA)3^3 adapts kk, vv and the input of the down projection, which is the library’s default. Training uses bf16 autocast, gradient checkpointing, sequences of at most 512 tokens, an effective batch of 16, AdamW without weight decay, and a cosine schedule with 3% warm-up. The loss covers the completion only, for min⁡(1000, one epoch)\min(1000,\text{ one epoch}) steps. For each method, we choose the learning rate with the lowest held-out loss after 100 steps on GSM8K. GSM8K is scored by exact match of the final number on 500 test problems, and MNLI and BoolQ by accuracy on 1000 validation examples, with greedy decoding and fixed prompts (Table A5).

methodtasklrstepsmetric
LoRAGSM8K0.00024670.664
LoRAMNLI0.000210000.908
LoRABoolQ0.00025880.900
DoRAGSM8K0.00024670.660
DoRAMNLI0.000210000.919
DoRABoolQ0.00025880.887
(IA)3^3GSM8K0.014670.598
(IA)3^3MNLI0.0110000.902
(IA)3^3BoolQ0.015880.879
VeRAGSM8K0.014670.584
VeRAMNLI0.0110000.909
VeRABoolQ0.015880.895
OFTGSM8K0.00014670.630
OFTMNLI0.000110000.909
OFTBoolQ0.00015880.897

Table A5. The trained adapters: learning rate, optimizer steps, and the task metric in the PEFT library.

DoRA in practice. The trained DoRA adapters have a scale gg that deviates from one by a median of 0.22% over columns. We also list every adapter on the Hugging Face Hub that is tagged as an adapter of a Llama-3.1-8B model or that matches a search for DoRA, and read its configuration. 149 distinct adapters from 10 authors use dora. For each, we compute g=m⊙∥W+cAB∥cg=m\odot\lVert W+cAB\rVert_c from the checkpoint and the base weights in float32. In 144 of them, the median column deviates from one by less than 1%. The median adapter’s median deviation is 0.16%, and the largest is 10.3%. This small magnitude is why DoRA converts to LoRA within noise.

Engine Integration

The patch. The integration patches vLLM 0.19.1 at run time, so the installed package stays unchanged. It patches vLLM’s check that rejects DoRA, the tensor-name parser, the tensor loader and the packer, adapter activation and weight creation, and the layer’s apply_ method. The loader and packer carry DoRA’s magnitude. Adapter activation computes DoRA’s ss, and weight creation allocates the slot buffers. The scheduler, paging and eviction, attention and batch composition are unchanged. The engine computes DoRA’s ss at admission, after it has fused qq, kk, vv, and gate, up, and sharded every tensor. This computation therefore uses the engine’s own layout.

CUDA graphs. A captured CUDA graph replays the same kernels for any batch. In an engine that serves OFT or VeRA, every decode step that runs a captured graph therefore runs the split GEMM and the shared term, whatever adapters its batch holds. Steps outside the captured sizes check the batch on the host, from the adapter mapping that vLLM already builds, and skip these kernels when they are not needed.

Amplified adapters. The PEFT library initializes (IA)3^3, VeRA and OFT at or near the identity. The agreement pool therefore draws each adapter’s vectors so that they move log-probabilities by a clear margin. For standard normal

zz, DoRA uses g=1+0.3zg = 1 + 0.3z, (IA)3^3 uses s=1+0.3zs = 1 + 0.3z, VeRA uses d=0.1(1+0.3z)d = 0.1(1 + 0.3z) and e=0.5ze = 0.5z, and OFT uses skew parameters 0.02z0.02z with blocks of 32. The pool has two adapters each of DoRA, (IA)3^3, VeRA and OFT, with different seeds, and one LoRA adapter (Table A6).

methodeffect (nats/token)error / effect, TP 1TP 2
LoRA0.201.27%1.51%
DoRA1.980.55%0.47%
(IA)3(\mathrm{IA})^31.090.54%0.55%
VeRA0.221.41%1.49%
OFT0.860.34%0.37%

Table A6. One batch of 9 amplified adapters against the PEFT library in float32. The effect is the mean change in log-probability per token that the adapter, run by the PEFT library, makes to the base model. The error is the engine’s largest mean error per method, as a share of that effect, on one GPU and on two GPUs with tensor parallelism.

The Byte Model

Constants. β\beta is the bandwidth of a 256 MiB device copy. βL2\beta_{\mathrm{L2}} is the fastest of copies sized to 1/16, 1/8 and 1/4 of the L2 cache. λ\lambda is the time per call of 200 one-block kernels chained in one CUDA graph.

Routes. We test the model on four routes that apply DoRA’s ss outside the expand, and on OFT’s input transform. A side kernel writes h⊙sh \odot s before our expand reads it, which adds 2Ndout2N d_{\mathrm{out}} elements and one launch. A gathered scale applies ss after the expand in one pass, which adds 2Ndout2N d_{\mathrm{out}} elements and one launch. An index scale first writes ss for every row and then multiplies, which adds 4Ndout4N d_{\mathrm{out}} elements and two launches. vLLM outside is vLLM’s expand followed by the open pull request’s index, copy and multiply, which adds 4Ndout4N d_{\mathrm{out}} elements and three launches. It is measured against vLLM’s LoRA expand. Table A7 gives the median error of each route. Below the bandwidth regime, the single constant λ\lambda stands for each kernel’s launch, tail and wait, and the error is larger.

routecellsincrementtotallaunch
side kernel273.2%1.4%2%
gathered scale277.4%3.4%23%
index scale2720.0%14.0%24%
vLLM outside281.2%0.7%21%
OFT transform12012.3%1.2%—

Table A7. Median absolute error of the byte model on the RTX A6000. Increment: error on the time that the extra passes add. Total: error on the route’s total time. Both are over the cells where ΔQ/β>λ\Delta Q/\beta> \lambda. Launch: error on total time over the other, launch-bound cells.

Measurement Protocol

Kernel timing. Issuing eight Triton launches from Python costs about 400 μ\mus, against roughly 95 μ\mus of kernel time. We therefore time every kernel inside a CUDA graph. We chain enough copies of a call into one graph that the timed interval reaches 2 ms, and divide. We rebuild the graph five times and report the median of the five.

Tuned kernels. vLLM ships no LoRA kernel configurations for the RTX A6000. With its defaults, the fused qkvqkv expand reaches 58%–60% of copy bandwidth. We search a grid for vLLM’s shrink and expand on every layer shape, choose by LoRA’s own time inside a CUDA graph, and write the configuration file that vLLM reads. Every engine in every serving measurement uses these files, tuned for ranks 16, 32, 64, 128 and 256. Our slot uses the configurations tuned for LoRA.

Serving

Serving. Each measurement cell runs one engine in a fresh process. The engine settings are fixed and recorded: 64 sequences, 8192 batched tokens, a maximum length of 1024, and fp16. vLLM’s on-disk compile cache is off in every engine, because its key does not include run-time patches. Adapters are admitted before timing. An untimed warm-up stream of the same shape runs first, because a kernel first compiled inside the timed stream costs about a second of TTFT. Prompts are random token ids of exact length, so prefix caching never applies. Arrival times are drawn in advance and submitted on an absolute clock. Prompts and arrivals change with the repetition, and the engines of one repetition share them. Over six repetitions, every order of the engines occurs equally often. Before each cell, its GPUs must be idle in two samples ten seconds apart. Each cell records its KV-cache capacity and the device memory in use. Ratios are medians of per-repetition ratios, with percentile-bootstrap intervals over repetitions. Across 102 ratios, whether an engine ran first moves the ratio by a median of 0.4%.

TTFT. Under a burst, prompts finish prefilling in waves of one batched step each. The median request sits on a wave boundary, so the median TTFT moves by a whole step between repetitions whose fastest, 90th-percentile and slowest requests agree. We therefore report mean TTFT for prefill-heavy load.

Pools. The single-method pools hold 1, 2, 4, 8 and 16 VeRA tenants, 1, 4 and 16 OFT tenants and 1, 4 and 16 (IA)3(\mathrm{IA})^3 tenants, with 128 requests each. For OFT, a second stock engine serves rank-256 conversions. The five-method pool holds 20 tenants, four of each method, with 8 requests per tenant. It runs at decode-heavy rates of 4, 8 and 16 requests per second and burst, and at prefill-heavy rates of 2, 4, 8 and 16 and burst. The same pool also runs with Zipf popularity and at tensor-parallel degree 2. The four-method pool holds 16 tenants, four each of LoRA, DoRA, (IA)3(\mathrm{IA})^3 and OFT. The many-tenant pools hold 64, 256 and 1024 tenants, half (IA)3(\mathrm{IA})^3 and half VeRA. Each tenant reuses the files of one of 16 synthetic adapters of its method, under its own name and id. The engine therefore loads, holds and swaps it as a separate adapter. Each many-tenant cell first admits every tenant with a one-token request. It then runs an untimed warm-up stream and 512 timed requests, with tenants in turn, and it has three repetitions. The DoRA comparison of §4 serves four DoRA adapters of rank 16 against four LoRA adapters, at 0.80 of device memory. Engine threads are capped (OMP_NUM_THREADS=16, eight compile threads).

Results. Table A8 gives every rate of the mixed pools with intervals. Table A9 sets the decode time that each tenant adds against the byte model. Table A10 breaks down stock’s decode time in the mixed pools.

poolloadreq/sstock / ScuteScute / LoRA-only
20 tenantsdecode41.92 [1.89, 1.94]1.13 [1.13, 1.14]
decode81.82 [1.78, 1.83]1.09 [1.06, 1.12]
decode161.66 [1.65, 1.67]1.09 [1.08, 1.09]
decodeburst1.73 [1.73, 1.75]1.05 [1.05, 1.06]
prefill21.36 [1.33, 1.43]1.23 [1.15, 1.24]
prefill41.59 [1.47, 1.71]1.29 [1.24, 1.37]
prefill84.91 [4.29, 5.84]1.38 [1.34, 1.46]
prefill165.96 [4.60, 6.49]1.52 [1.46, 1.57]
prefillburst1.12 [1.11, 1.13]1.18 [1.18, 1.19]
16, no VeRAdecode41.21 [1.20, 1.22]1.02 [1.02, 1.03]
decode81.31 [1.30, 1.32]0.99 [0.96, 1.02]
decode161.21 [1.21, 1.22]0.97 [0.97, 0.98]
decodeburst1.25 [1.24, 1.25]0.94 [0.94, 0.94]
prefill20.97 [0.95, 0.98]1.09 [1.07, 1.11]
prefill40.97 [0.95, 0.99]1.11 [1.09, 1.13]
prefill81.01 [0.99, 1.06]1.14 [1.12, 1.17]
prefill161.42 [1.27, 1.51]1.16 [1.15, 1.16]
prefillburst0.99 [0.99, 1.00]1.05 [1.05, 1.06]
20, Zipfdecode81.98 [1.94, 2.07]1.12 [1.07, 1.13]
decodeburst1.82 [1.78, 1.86]1.05 [1.05, 1.06]
prefill41.20 [1.14, 1.22]1.18 [1.16, 1.25]
prefillburst1.07 [1.05, 1.08]1.20 [1.19, 1.21]
20, TP 2decode161.75 [1.69, 1.76]1.26 [1.26, 1.28]
decodeburst1.69 [1.69, 1.69]1.24 [1.24, 1.24]
prefillburst1.23 [1.16, 1.26]1.19 [1.17, 1.26]

Table A8. Every rate of the mixed pools on the RTX A6000, vLLM as shipped: median per-repetition ratio and 95% interval. TPOT for decode-heavy load, mean TTFT for prefill-heavy load.

methodarmpredictedmeasured
VeRAScute0.000.05
rank 2561.971.94
OFTScute0.120.16
rank 640.490.53
rank 2561.971.94
(IA)³Scute0.000.12
rank 640.490.53

Table A9. Decode time added per tenant (ms) on the RTX A6000: the byte model’s prediction against the least-squares slope of TPOT over 1–16 tenants. Stock is counted on every LoRA layer at the padded rank, because vLLM runs its kernels there for every adapter.

20 tenants16, no VeRA
conversions at own rank13.05.1
+ zero factors elsewhere14.36.4
+ padding (vLLM)31.67.9
stock, measured34.710.5
Scute, predicted3.11.5
Scute, measured7.12.4

Table A10. Decode time over the base model under a decode burst (ms), vLLM as shipped. The byte model’s account of stock, built up step by step, against stock’s measured time and Scute’s.

The A100

Each run uses one A100-SXM4-80GB and the same source code as the RTX A6000 runs. vLLM’s kernels are tuned on the A100 for ranks 16, 64 and 256. Each cell has six repetitions, with every engine order equally often. Stock runs at 0.90 of memory with every adapter resident. Ours runs at 0.86, which on 80 GB uses about the same device memory. Table A11 gives the single-method pools and the mixed pools under a burst. Table A12 gives every rate of the mixed pools.

methodtenantsratioTPOTTTFT
burst
VeRA1stock / Scute1.061.04
4stock / Scute1.151.05
16stock / Scute1.831.10
OFT1rank 64 / Scute1.071.00
1rank 256 / Scute1.231.13
16rank 64 / Scute1.311.04
16rank 256 / Scute1.991.21
(IA)³1stock / Scute1.111.04
16stock / Scute1.341.05
mix20stock / Scute1.631.00
Scute / LoRA-only1.231.24
mix, no VeRA16stock / Scute1.140.96
Scute / LoRA-only1.071.08

Table A11. The A100: median per-repetition ratio under a decode burst (TPOT) and a prefill burst (mean TTFT), stock with every adapter resident.

poolloadreq/sstock / ScuteScute / LoRA-only
20 tenantsdecode81.69 [1.68, 1.71]1.34 [1.28, 1.36]
decode161.69 [1.64, 1.70]1.26 [1.25, 1.30]
decode burst1.63 [1.62, 1.63]1.23 [1.23, 1.23]
prefill40.77 [0.70, 0.81]1.38 [1.32, 1.65]
prefill160.95 [0.84, 1.43]1.89 [1.81, 2.02]
prefill burst1.00 [1.00, 1.01]1.24 [1.23, 1.27]
16, no VeRAdecode81.12 [1.11, 1.12]1.16 [1.14, 1.18]
decode161.13 [1.12, 1.14]1.10 [1.09, 1.10]
decode burst1.14 [1.14, 1.14]1.07 [1.07, 1.08]
prefill40.81 [0.79, 0.87]1.26 [1.19, 1.27]
prefill160.76 [0.73, 0.83]1.45 [1.38, 1.47]
prefill burst0.96 [0.96, 0.97]1.08 [1.07, 1.09]

Table A12. The mixed pools on the A100: median per-repetition ratio and 95% interval. TPOT for decode-heavy load, mean TTFT for prefill-heavy load.

For many tenants, stock keeps 80 rank-64 conversions resident and 256 in host memory, and ours keeps 64 tenants resident. The rates are scaled to the A100’s shorter step (Table A13).

tenants812162024
64 Scute10010010010047
stock10083211412
256 Scute10070121212
stock10024131212
1024 Scute1003312129
stock3210421

Table A13. The A100: requests that meet the latency target (%) against decode-heavy arrival rate (requests per second), median of three repetitions.

References

  1. [1]Brüel-Gabrielsson, R., Zhu, J., Bhardwaj, O., Choshen, L., Greenwald, K., Yurochkin, M., and Solomon, J. Compress Then Serve: Serving Thousands of LoRA Adapters with Little Overhead. In International Conference on Machine Learning, 2025.arxiv.org/abs/2407.00066
  2. [2]Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. Punica: Multi-Tenant LoRA Serving. In Conference on Machine Learning and Systems, 2024.arxiv.org/abs/2310.18547
  3. [3]Edalati, A., Tahaei, M., Kobzyev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. KronA: Parameter-efficient Tuning with Kronecker Adapter. In Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques. Springer, 2025.arxiv.org/abs/2212.10650
  4. [5]Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y. HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models. In International Conference on Learning Representations, 2025.
  5. [6]Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H. FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning. In International Conference on Learning Representations, 2022.
  6. [7]Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., and Torrellas, J. Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments. In IEEE/ACM International Symposium on Microarchitecture, 2025.DOI
  7. [8]Jaiswal, S., Arun, S., Paravil, A., Mallick, A., Mastorakis, S., Khare, A., Alverti, C., St Amant, R., Bansal, C., Rühle, V., and Torrellas, J. Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems. arXiv:2511.22880, 2025.arxiv.org/abs/2511.22880
  8. [9]Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-Based Random Matrix Adaptation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.11454
  9. [10]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In ACM Symposium on Operating Systems Principles, 2023.arxiv.org/abs/2309.06180
  10. [13]Lian, D., Zhou, D., Feng, J., and Wang, X. Scaling & Shift-ing Your Features: A New Baseline for Efficient Model Tuning. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2210.08823
  11. [14]Liu, H., Tam, D., Mueqeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.05638
  12. [15]Liu, J., Xiao, G., Li, K., Lee, J. D., Han, S., Dao, T., and Cai, T. BitDelta: Your Fine-Tune May Only Be Worth One Bit. In Advances in Neural Information Processing Systems, 2024a.arxiv.org/abs/2402.10193
  13. [16]Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight-Decomposed Low-Rank Adaptation. In International Conference on Machine Learning, 2024b.arxiv.org/abs/2402.09353
  14. [17]Liu, W., Qiu, Z., Feng, Y., Xu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng, S., Wen, Y., Black, M. J., Weller, A., and Schölkopf, B. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. In International Conference on Learning Representations, 2024c.arxiv.org/abs/2311.06243
  15. [18]Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., Bossan, B., and Tietz, M. PEFT: State-of-the-Art Parameter-Efficient Fine-Tuning Methods. https://github.com/huggingface/peft, 2022.
  16. [20]Predibase. LoRAX: Multi-LoRA Inference Server that Scales to 1000s of Fine-Tuned LLMs. https://github.com/predibase/lorax, 2023.
  17. [21]Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.07280
  18. [22]Qiu, Z., Liu, W., Weller, A., and Schölkopf, B. Orthogonal Finetuning Made Scalable. In Conference on Empirical Methods in Natural Language Processing, 2025.arxiv.org/abs/2506.19847
  19. [23]Renduchintala, A., Konuk, T., and Kuchaiev, O. Tied-LoRA: Enhancing Parameter Efficiency of LoRA with Weight Tying. In Conference of the North American Chapter of the Association for Computational Linguistics, 2024.arxiv.org/abs/2311.09578
  20. [24]Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J. E., and Stoica, I. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. In Conference on Machine Learning and Systems, 2024.arxiv.org/abs/2311.03285
  21. [25]Tillet, P., Kung, H. T., and Cox, D. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019.DOI
  22. [26]Walker, T. and Baraniuk, R. AdapterBench: Does Adapter Shape Matter for Inference-Time Adaptation? In Workshop on Towards Test-Time Continual Learning Agents at the Conference on Neural Information Processing Systems, 2026.
  23. [27]Wen, Y. and Chaudhuri, S. Batched Low-Rank Adaptation of Foundation Models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2312.05677
  24. [28]Williams, S., Waterman, A., and Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009.DOI
  25. [30]Yao, X., Hu, Q., and Klimovic, A. DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs. In European Conference on Computer Systems, 2025.
  26. [32]Yuan, Z., Shang, Y., Song, Y., Yang, D., Wu, Q., Yan, Y., and Sun, G. ASVD: Activation-Aware Singular Value Decomposition for Compressing Large Language Models. arXiv:2312.05821, 2023.arxiv.org/abs/2312.05821
  27. [33]Zelenin, A. and Zhuravlyova, A. Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels. arXiv:2603.22276, 2026.arxiv.org/abs/2603.22276
  28. [34]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2312.07104
  29. [35]Zhou, Z., Wei, X., Zhang, J., and Sun, G. PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In USENIX Annual Technical Conference, 2022.

Paper details

vlnc:W_kfndvqv4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code

View statement
Looking for related papers…
Contents