betaFeedback

WorksComputer Science

Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers

Abstract

Vision Transformers (ViTs) are bottlenecked by the length of their input token sequence. Standard patchification fixes this length with a single patch size applied across the image, regardless of regional content. Prior work on adaptive patch sizing reduces token count by assigning coarser patches to visually simple regions, but relies on noisy pixel-level heuristics such as local entropy to make patchification decisions. We argue that this compute allocation should be grounded in the model's interpretation of task relevant information, not in raw pixel statistics. We introduce Split Policy Learned Image Tokenization which learns to allocate token resolution based on what the model finds meaningful, producing shorter token sequences that are better matched to task-relevant content. SPLIT matches the accuracy of full-resolution ViT classification baselines while using 30% fewer tokens. The same gains hold on dense prediction: SPLIT achieves competitive accuracy on COCO and ADE20K, demonstrating that semantically aligned patchification generalises well beyond classification.

Introduction

Vision Transformers (ViTs) [9] have become the dominant architecture for various visual tasks, demonstrating strong scalability and effectiveness across a wide range of applications. However, their efficiency remains a major challenge: self-attention scales quadratically with sequence length, and because patch embedding uses a uniform grid, sequence length itself grows quadratically with input resolution. As a result, doubling the resolution of an image quadruples its token count and increases self-attention cost by roughly 16×16 \times, making high-resolution inputs prohibitively expensive to process.

To reduce the number of tokens processed while preserving task-relevant information, many prior works reduce the token count after the tokens have been fully computed. These methods typically leverage signals produced by the Vision Transformer itself, such as self-attention scores [19] or intermediate feature representations [1], to filter, merge, or refine tokens [28], aligning token allocation with the model’s learned semantics.

While promising, these methods are limited in their efficiency gains due to the cost of generating and processing dense tokens in the early stages. An alternative approach is to perform adaptive tokenization at the patch embedding stage itself. For example, Quadformer [29] uses the feature discrepancy between an image and its downsampled-then-upsampled counterpart as a proxy for regional importance, and APT [5] uses entropy as a heuristic to estimate task-relevant uncertainty, allocating higher resolution to regions that are more informative for dense visual tasks.

Although adaptive patchification has proven effective, existing methods still rely on hand-crafted or external proxies to estimate regional importance. Visual signals such as entropy provide a useful estimate of local uncertainty or information density, but they are not directly optimized for the downstream task and may overlook semantically important regions while mistakenly assigning high importance to visually complex but task-irrelevant areas. The natural alternative, using semantic importance derived from the ViT itself, is impractical, since it requires first processing the full dense token sequence to obtain reliable importance signals. This leads to a key question: can we obtain a lightweight yet semantic importance signal before processing with ViT layers?

Our key insight is that this dilemma is less rigid than it appears. Although processing the full dense token sequence is expensive, coarse tokens provide a lightweight alternative for obtaining model-aligned semantic cues. By optimizing the patch selection policy through the downstream task objective, the errors in the final prediction relative to different patch selection policies implicitly reveal which regions are semantically important, allowing the splitting decisions themselves to be learned in a task-aware manner, without requiring external proxies or costly dense ViT guidance.

Building on this insight, we propose Split Policy Learned Image Tokenization (SPLIT), an adaptive patchification framework that learns task-aware patch selection from coarse to fine. As shown in Figure 1, SPLIT uses coarse tokens as lightweight semantic anchors and selectively refines only the regions that are semantically important to the downstream task. To further encourage efficiency, we formulate patch refinement as a budget-constrained decision-making problem and optimize the network with reinforcement learning, learning effective region selection without exceeding the prescribed token budget.

Original Images and Split Policy Learned Image Tokenizer (Ours)

Figure 1. Split Policy Learned Image Tokenization. SPLIT is an adaptive patchification framework that learns to allocate fine tokens to semantically important regions and coarse tokens to homogeneous ones, accelerating ViT inference without sacrificing accuracy.

Our contributions can be summarized as follows:

  • We introduce SPLIT, an adaptive patchification framework that learns task-aware patch selection from coarse semantic anchors, rather than pixel-level heuristics.

  • We formulate adaptive patch refinement as a budget-constrained decision-making problem and optimize the decision policy with reinforcement learning, enabling effective region selection under a fixed token budget.

  • SPLIT is lightweight to train: it adapts pretrained ViTs to semantically aligned tokenization with minimal fine-tuning, and transfers cleanly across image classification, object detection, semantic segmentation, and VQA benchmarks.

Related Works

Efficient Vision Transformer

Vision Transformers (ViTs) [9, 34, 23] have become a standard architecture for visual backbones across diverse domains, including image classification [9], object detection [2], semantic segmentation [32], and vision-language tasks [27]. Despite their strong performance and scalability, ViTs typically rely on a fixed patchification protocol: an input image is uniformly divided into non-overlapping patches, each of which is linearly projected into a visual token. This protocol makes the input sequence length depend primarily on image resolution rather than image content, assigning the same token budget to redundant background regions, repetitive textures, and semantically important objects.

ViT Tokens Reduction

Due to the quadratic complexity of self-attention with respect to sequence length [35], reducing the number of visual tokens has become an important direction for improving ViT efficiency. Existing methods can be roughly divided into two categories. The first category reduces tokens after patchification, using intermediate model signals such as attention scores [19, 39], token similarity [1], or auxiliary importance predictors [28, 25, 26] as proxies for token relevance. These methods prune less informative tokens [28, 19, 39, 16, 33] or merge redundant token representations [1, 17, 4, 37], allowing later transformer layers to operate on shorter sequences. While effective, these methods reduce tokens only after dense processing and introduce dynamic indexing or gather-scatter operations that can limit practical gains from optimized attention kernels such as FlashAttention [7]. The other category reduces tokens before transformer processing by adapting the patchification process itself. APT [5] and related adaptive patch sizing methods [13, 29, 3] assign larger patches to visually simple regions and smaller patches to visually complex regions, producing shorter input token sequences without mid-layer token manipulation. However, these methods typically rely on hand-crafted pixel-level heuristics and task-specific threshold tuning, which increases adaptation cost for each downstream setting.

Semantic Token Patchification

Besides requiring task-specific thresholds for patchification, current adaptive patch sizing methods rely primarily on low-level pixel statistics to determine patch granularity. For example, local entropy or visual complexity is often used as a proxy for where fine-grained tokens should be allocated. However, such signals do not necessarily reflect semantic importance [15]. As illustrated in Figure 7, APT-style methods can over-tokenize noisy backgrounds or repetitive textures, while smooth but semantically meaningful regions can be compressed too aggressively. To address this limitation, our method learns a semantic patchification policy from the model’s own feature representations.

Patch allocation comparison showing Original Image, APT, and SPLIT (Ours) patch grids

Figure 7. Patch allocation comparison. APT (middle) over-tokenizes high-entropy textures regardless of their semantic content, while SPLIT (bottom) concentrates fine tokens around objects.

Method

SPLIT is a learnable adaptive patchification framework for efficient ViTs. Given an image I∈R3×H×WI \in\mathbb{R}^{3 \times H \times W}, where HH and WW denote the image height and width, a vanilla ViT uniformly partitions it into N=HW/p2N = HW/p^2 fixed-size patches, where pp is the patch size and NN is the resulting number of visual tokens. In contrast, SPLIT learns to generate only approximately ρN\rho N tokens under a token budget ratio ρ∈(0,1]\rho\in(0, 1].

Problem Definition

Following prior work [29, 5], we use a multi-scale patch embedding function that maps patches of different spatial sizes into a shared token embedding space, allowing adaptive patches of varying resolutions to be processed by the same ViT backbone.

For a tokenizer with SS stages and smallest patch size pp, we first divide the image into patches of size 2Sp×2Sp2^S p \times2^S p. A decision network then determines, based on each patch’s content, whether to split it into four sub-patches of size 2S−1p×2S−1p2^{S-1}p \times2^{S-1}p, and this process is applied recursively to every split patch over SS rounds. The result is a mixed-resolution tokenization that uses larger patches for homogeneous regions and smaller patches for content-rich ones.

The decision network must trade off two conflicting objectives: maximizing downstream performance (which favors more, finer tokens) and minimizing compute per sample (which favors fewer, coarser tokens). We formulate this as a constrained optimization problem and train the decision network with reinforcement learning, minimizing the unconstrained Lagrangian dual of the constraint.

Overview of SPLIT

Figure 2. Overview of SPLIT. An input image is first divided into coarse patches. A decision network then performs SS refinement stages, deciding whether to split each parent patch into its children or keep it as a single token. The example uses two splitting stages, yielding a mixed-resolution token sequence. For each parent-child group, a lightweight cross-attention module pools the four child embeddings with the parent embedding as the query. The pooled summary is concatenated with the parent embedding and passed to a two-layer MLP to predict the split probability pip_i.

Decision Network

Consider a parent patch Pi∈Rp×pP_i \in\mathbb{R}^{p \times p} with pre-positional embedding zip∈Rdembedz_i^p \in\mathbb{R}^{d_{\mathrm{embed}}}, and let zi,1c,…,zi,4c∈Rdembedz_{i,1}^c,\ldots,z_{i,4}^c \in\mathbb{R}^{d_{\mathrm{embed}}} be the embeddings of its four child sub-patches. All embeddings are computed using the multi-scale patch aggregation scheme of APT [5].

The decision network is a lightweight cross-attention MLP that uses the parent as a query and its children as keys and values, predicting the probability of splitting the parent into its children:

pi=σ(hθ([zip;CrossAttn⁡(zip,{zi,jc}j=14)])),(1)p_i = \sigma\left(h_\theta\left([z_i^p;\operatorname{CrossAttn}(z_i^p,\{z_{i,j}^c\}_{j=1}^{4})]\right)\right), \tag*{(1)}

where hθh_\theta is a two-layer MLP head and σ\sigma is the sigmoid. The attention pools the four child embeddings into a single summary, and the head decides whether that summary differs sufficiently from the parent to justify the finer tokens.

During training we sample the binary splitting decision si∼Bernoulli⁡(pi)s_i \sim\operatorname{Bernoulli}(p_i), where si=0s_i = 0 keeps the parent and si=1s_i = 1 replaces it with its four children. At inference we use the deterministic rule si=1[pi>0.5]s_i = \mathbb{1}[p_i > 0.5]. For a single splitting stage this yields the effective token set

T(x,s)={zip:si=0}∪{zi,jc:si=1, j=1,…,4}.(2)\mathcal{T}(x,s) = \{z_i^p : s_i = 0\} \cup\{z_{i,j}^c : s_i = 1,\ j = 1,\ldots,4\}. \tag*{(2)}

With two stages, patches are organized into three scales (top, mid, fine), and the same network is applied at both transitions to decide whether to split each parent into its children. The formulation extends naturally to any number of stages.

PPO Training with Lagrange Dual

Training the decision network with ordinary supervised learning is not straightforward. In SPLIT, each coarse image patch must decide whether it should remain a single token or be split into finer subpatch tokens. This indexing operation is non-differentiable, so the decision network cannot be directly optimized through the downstream ViT loss by standard backpropagation. This non-differentiability can be addressed through differentiable relaxations, MSViT [13] makes use of Straight-Through Estimators and Gumbel-Sigmoid, however such relaxations either need to materialize all coarse and fine tokens at training time or need to prune un-materialized tokens which results in the un-selected path not getting any gradients. Due to these limitations we approach this problem using policy gradients and avoid using biased differentiable relaxations. We provide more detail on this decision in Appendix section B. Following the RL formulation used in adaptive routing methods such as Ada-K [41], we formulate patch splitting as a one-step reinforcement learning problem.

Consider a single stage of splitting, let x∈Xx \in\mathcal{X} be an input image and let P(x)={pi}i=1N\mathcal{P}(x) = \{p_i\}_{i=1}^{N} denote the set of coarse patches. The decision network πθ\pi_\theta outputs a Bernoulli splitting distribution for each coarse patch, πθ(mi=1∣pi)\pi_{\theta}(m_i = 1 \mid p_i), mi∈{0,1}m_i \in\{0, 1\}, where mi=1m_i = 1 means that patch pip_i is split into its child subpatches, and mi=0m_i = 0 means that pip_i is retained as a coarse token. The full splitting mask is M=(m1,…,mN)∼πθ(⋅∣x)M = (m_1, \ldots, m_N) \sim\pi_{\theta}(\cdot\mid x). A sequence packing operation T(x,M)\mathcal{T}(x, M) uses the mask to create the final token sequence combining the fine tokens with the un-split coarse tokens retained by MM. T(x,M)\mathcal{T}(x, M) is now the input to the vision transformer backbone fϕf_{\phi}.

For a sampled mask MM, we compute the downstream error signal E(x,M)\mathcal{E}(x, M) and the fraction of sequence length relative to using all fine patches, P(x,M)=∣T(x,M)∣/∣Tfine(x)∣\mathcal{P}(x, M) = \lvert\mathcal{T}(x, M) \rvert/\lvert\mathcal{T}_{\mathrm{fine}}(x) \rvert, where Tfine(x)\mathcal{T}_{\mathrm{fine}}(x) denotes the fully split patchify. The decision network has two conflicting objectives: it should preserve downstream task performance while using as few tokens as possible. We write the constrained objective as

min⁡θ Jerr(θ)s.t.Jtok(θ)≤ρ,(3)\min_{\theta}\ J_{\mathrm{err}}(\theta) \quad\text{s.t.} \quad J_{\mathrm{tok}}(\theta) \le\rho, \tag*{(3)}

where Jerr(θ)=Ex,M[E(x,M)]J_{\mathrm{err}}(\theta) = \mathbb{E}_{x,M}[\mathcal{E}(x, M)] and Jtok(θ)=Ex,M[P(x,M)]J_{\mathrm{tok}}(\theta) = \mathbb{E}_{x,M}[\mathcal{P}(x, M)] are taken over x∼Dx \sim\mathcal{D}, M∼πθ(⋅∣x)M \sim\pi_{\theta}(\cdot\mid x), and ρ\rho is the target sequence length fraction. This is analogous to the reward-cost formulation in Safe RLHF [6], where the main objective is optimized subject to an expected cost constraint.

Using a Lagrangian relaxation, we optimize

L(θ,λ)=Jerr(θ)+λ(Jtok(θ)−ρ),λ≥0.\mathcal{L}(\theta, \lambda) = J_{\mathrm{err}}(\theta) + \lambda\left(J_{\mathrm{tok}}(\theta) - \rho\right), \qquad\lambda\ge0.

For policy optimization, the constant term −λρ-\lambda\rho does not affect gradients, we also normalize the combined cost by (1+λ)−1(1+\lambda)^{-1} to keep the scale of the cost stable as λ\lambda changes.

C(x,M;λ)=E(x,M)+λP(x,M)1+λ.C(x, M; \lambda) = \frac{\mathcal{E}(x, M) + \lambda\mathcal{P}(x, M)}{1+\lambda}.

For each image xx, we sample LL masks M(1),…,M(L)∼πθold(⋅∣x)M^{(1)}, \ldots, M^{(L)} \sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x). We use a leave-one-out advantage comparing each of the ℓ\ell rollouts to the other rollouts from the same image:

A^ℓ=−(Cℓ−1L−1∑j≠ℓCj),where,Cℓ=E(x,M(ℓ))+λP(x,M(ℓ))1+λ.\widehat{A}_{\ell} = -\left(C_{\ell} - \frac{1}{L-1}\sum_{j\ne\ell} C_j\right), \qquad\text{where}, \quad C_{\ell} = \frac{\mathcal{E}(x, M^{(\ell)}) + \lambda\mathcal{P}(x, M^{(\ell)})}{1+\lambda}.

Thus, a rollout has a positive advantage if its combined cost is less than that of the other sampled masks. We then optimize the decision network with a PPO clipped objective. Let rℓ(θ)=πθ(M(ℓ)∣x)πθold(M(ℓ)∣x)r_{\ell}(\theta) = \frac{\pi_{\theta}(M^{(\ell)} \mid x)}{\pi_{\theta_{\mathrm{old}}}(M^{(\ell)} \mid x)} be the importance ratio, the PPO loss is

LPPO(θ)=−Ex,ℓ[min⁡(rℓ(θ)A^ℓ,clip⁡(rℓ(θ),1−ϵ,1+ϵ)A^ℓ)].\mathcal{L}_{\mathrm{PPO}}(\theta) = -\mathbb{E}_{x,\ell}\left[\min\left(r_{\ell}(\theta)\widehat{A}_{\ell}, \operatorname{clip}\left(r_{\ell}(\theta), 1-\epsilon, 1+\epsilon\right)\widehat{A}_{\ell}\right)\right].

This objective allows the decision network to learn from discrete masks while preventing very large policy updates. Note that LPPO\mathcal{L}_{\mathrm{PPO}} does not explicitly guide the decision network towards ρ\rho, instead the multiplier λ\lambda decides the relative importance of the two conflicting objectives and this decides the optimal Ex∼D,M∼πθ(⋅∣x)[P(x,M)]\mathbb{E}_{x\sim\mathcal{D}, M\sim\pi_{\theta}(\cdot\mid x)}[\mathcal{P}(x, M)] corresponding to minimizing Ex∼D,M∼πθ(⋅∣x)[C(x,M;λ)]\mathbb{E}_{x\sim\mathcal{D}, M\sim\pi_{\theta}(\cdot\mid x)}[C(x, M; \lambda)]

Effectively λ\lambda is the only source of control on how aggressively our decision network chooses to split patches, if λ\lambda is too small, the decision network may split nearly all patches and ignore the token budget. If λ\lambda is too large, the network may retain too few fine tokens and collapse to an overly cheap representation. Therefore, following the dual descent update used in constrained RL [6], we update λ\lambda independently of the PPO step to ensure that λ\lambda pushes the decision network to respect our budget ρ\rho:

λt+1=[λt+ηλ(P‾t−ρ)]+,whereP‾t=1BL∑b=1B∑ℓ=1LP(xb,Mb(ℓ)).\lambda_{t+1} = \left[\lambda_t + \eta_{\lambda}\left(\overline{P}_t - \rho\right)\right]_+,\qquad\text{where} \quad\overline{P}_t = \frac{1}{BL}\sum_{b=1}^{B}\sum_{\ell=1}^{L}\mathcal{P}(x_b, M_b^{(\ell)}).

BB is the batch size, ηλ\eta_{\lambda} is the dual learning rate, and [⋅]+=max⁡(⋅,0)[\cdot]_+ = \max(\cdot, 0) enforces non-negativity.

Decision Network Warm-start

Arbitrary initialization of the decision network can make RL training unstable. To reduce this instability, we warm-start the decision network with deterministic pseudo-labels derived from patch-subpatch similarity. Let zicz_i^c be the embedding of a coarse patch pip_i and {zi,kf}k=14\{z_{i,k}^f\}_{k=1}^{4} be the embeddings of its four child subpatches. For each parent–child pair, we compute the cosine dissimilarity $d_{i,k} = (1−cos⁡(zi,kc,zi,kp))/2∈[0,1](1-\cos(z_{i,k}^{c},z_{i,k}^{p}))/2 \in[0,1]. The pseudo-label m~i=1[max⁡kdi,k>τsplit]\tilde{m}_i = \mathbb{1}[\max_k d_{i,k} > \tau_{\mathrm{split}}] marks a parent for splitting if at least one of its children is sufficiently dissimilar from it, where τsplit\tau_{\mathrm{split}} is a learnable threshold. The decision network is trained to imitate these pseudo-labels with a binary cross-entropy loss, and the threshold τsplit\tau_{\mathrm{split}} is updated by dual ascent on the same budget gap P‾t−ρ\overline{P}_t-\rho used during PPO training, so that the warm-start labels themselves track the target token budget.

During warm-start, these cosine-similarity masks are also used to fine-tune the ViT backbone, allowing the backbone to adapt to the adaptive token layout before the policy is optimized with PPO. In all subsequent decision network training, the ViT backbone is frozen.

Experiments

Image Classification

We provide results from training 1^1 with SPLIT for 1 epoch of warm-start followed by 1 epoch of PPO training initialized from a checkpoint already finetuned on ImageNet [8]. We provide a comparison against baselines including token merging methods like EViT, ToMe and DTEM on ViT-L/14 at a resolution of 224 in Figure 3. For a fair comparison in throughput, all the baselines in Figure 3 use flash attention. In Table 1 and Table 2, we show that our method is capable of matching performance to APT and vanilla ViT with significant speedup at an image size of 224 and 336.

ModelRes/PatchImg/sSpeedupAcc
ViT-B224/163310-85.1
Random224/163751+13%83.7
APT-B224/163540+7%85.1
SPLIT-B (Ours)224/163635+10%85.0
ViT-L224/14883-87.9
Random224/141049+19%86.9
APT-L224/14993+12%87.8
SPLIT-L (Ours)224/141107+25%87.8
ViT-H224/14441-88.3
Random224/14568+29%87.4
APT-H224/14542+23%88.3
SPLIT-H (Ours)224/14533+21%88.2

Table 1. Table 1: ImageNet Classification at 224. Comparison of ViT models with different patch reduction strategies at 224x224 resolution.

ModelRes/PatchImg/sSpeedupAcc
ViT-L336/14395-88.2
Random336/14550+39%87.3
APT-L336/14527+33%88.1
SPLIT-L (Ours)336/14570+44%88.1
ViT-H336/14175-88.5
Random336/14272+55%87.0
APT-H336/14263+50%88.4
SPLIT-H (Ours)336/14281+61%88.4

Table 2. Table 2: ImageNet Classification at 336. Comparison of ViT models with different patch reduction strategies at 336x336 resolution.

Comparison on ViT-L/14 at 224 × 224 resolution

Figure 3. Comparison on ViT-L/14 at 224 × 224 resolution.

Dense Visual Tasks

Object detection. Object detection is a strong stress test for adaptive patchification because localization depends on preserving fine spatial detail at high resolution. To evaluate this setting, we train EVA-02 backbones [10] with a ViTDet-style decoder head [18] under window attention, and benchmark on COCO [20] at 1536×15361536 \times1536. As shown in Table 3, SPLIT stays in the same high-accuracy regime as the fixed-grid baseline and APT while delivering the best throughput across both model scales. Notably, this throughput gain is achieved even when SPLIT and APT operate at a similar token-reduction level (about 30%): our runtime breakdown confirms that the remaining gap comes from lower decision making overhead in SPLIT, whereas APT incurs higher cost from entropy-based multi-scale scoring when extended beyond two patch sizes. These results indicate that input-level learned policy remains effective for high-resolution dense detection with window attention.

ModelResImg/sSpeedupmAPAP50
EVA-02-B15363.66-58.9377.85
Resizing15364.14+13%58.4377.22
APT15364.14+13%58.7977.65
SPLIT (Ours)15364.23+16%58.8477.80
EVA-02-L15361.61-62.2880.80
Resizing15361.91+19%61.7580.27
APT15361.91+19%62.0780.64
SPLIT (Ours)15362.02+25%62.2480.78

Table 3. Transfer to Object Detection. SPLIT can be scaled to high-resolution dense image tasks supporting window attention.

Semantic segmentation. We further evaluate SPLIT on semantic segmentation, which requires fine-grained boundary understanding in addition to region-level consistency. Following the EVA-02 segmentation protocol [10], we use an UperNet decoder [38] and report results on ADE20K [43, 42]. Table 4 shows that SPLIT closely matches baseline/APT quality on both aAcc and mIoU while consistently improving throughput. In this task, the speed gain aligns with stronger sequence reduction: SPLIT uses roughly 35% fewer tokens, compared with about 30% for APT, yielding additional runtime savings without materially affecting segmentation quality. Together, these results show that SPLIT preserves pixel-level performance while improving efficiency.

ModelResImg/sSpeedupaAccmIoU
EVA-02-L5126.48-86.6759.77
Resizing5127.19+11%86.0958.81
APT5127.19+11%86.6859.70
SPLIT (Ours)5127.43+15%86.5359.65
EVA-02-L6403.44-86.8360.05
Resizing6403.82+11%86.0658.83
APT6403.82+11%86.8260.01
SPLIT (Ours)6403.97+15%86.7559.99

Table 4. Transfer to Semantic Segmentation. SPLIT can handle pixel-level fine-grained tasks without compromising visual acuity.

Visual question answering. We further evaluate the transferability of SPLIT to multimodal visual question answering by applying it to the vision backbone of LLaVA-1.5 [21]. Following APT [5], we replace the original uniform patch embedding with our learnable adaptive patchification module, keeping the language model frozen while training the decision module and the vision-language projection layer. As shown in Table 5, SPLIT consistently improves inference throughput over vanilla LLaVA while matching or exceeding the baseline accuracy across a wide range of VQA benchmarks [12, 14, 24, 31, 30, 11, 22, 40]. Although APT achieves a slightly higher speedup at the same target token fraction, it falls short of the baseline on most metrics, whereas SPLIT recovers or surpasses the baseline on nearly every benchmark for both the 7B and 13B variants.

ModelImg/sSpeedupVQAv2^{v2}GQASQAI^{I}VQAT^{T}POPEMMEMMBMMBC^{C}MMV
LLaVA-1.5-7B3.70-78.561.067.858.286.91510.164.658.130.7
Resizing4.51+22%77.561.166.856.586.61473.863.258.130.2
APT4.51+22%77.961.467.556.986.41474.063.858.230.8
SPLIT (Ours)4.37+18%78.661.769.058.186.61497.365.258.931.3
LLaVA-1.5-13B2.22-80.063.272.761.287.11530.668.563.435.4
Resizing2.72+23%78.961.172.059.186.81496.965.862.533.9
APT2.72+23%79.463.072.459.587.21511.266.563.734.7
SPLIT (Ours)2.66+20%79.663.173.460.286.91504.967.564.036.3

Table 5. Transfer to VQA. SPLIT enables significant throughput increase while matching or exceeding performance to the baseline.

Ablation

Measuring SPLIT overhead. We next measure the computational overhead introduced by SPLIT. Adaptive patchification breaks the constant-input-shape assumption that GPUs are heavily optimized for, and the decision network itself adds a small forward pass on top of patch embedding. Both costs need to be accounted for before any throughput claims can be trusted. We report this analysis in Figure 4. With no sequence reduction applied, SPLIT incurs a minor 3% overhead relative to the vanilla ViT baseline due to multi-scale patch embedding computations, decision network and additional packing logic to join the multi-scale patch embeddings. Once sequence reduction is enabled, the savings from processing fewer tokens substantially outweigh the overhead.

Forward-pass timing breakdown

Figure 4. Forward-pass Timing Breakdown. SPLIT incurs minimal overhead from decision network and patch splitting operations while saving on transformer blocks, thus resulting in a significant throughput increase

Lagrange Multiplier Control. As discussed in § 3.3, the multiplier λ\lambda decides the relative weighting of the performance cost and compute cost. In Figure 6 we show that a continuously updated λ\lambda actively guides the model towards the specified token budget. Using a fixed value of λ\lambda for training involves laborious tuning of the value of λ\lambda to achieve the desired token budget. We also observe that even though we can achieve similar % tokens using fixed λ=0.01\lambda= 0.01 and adaptive λ\lambda, the gap in control error between train and val sets is much smaller when using an adaptive λ\lambda indicating that it is a better control mechanic.

Token-budget dynamics and control error

Figure 6. (a) Token-budget dynamics during training. (b) Train and validation control error. Adaptive λ\lambda more reliably controls the token budget than fixed-λ\lambda baselines. Left: percent tokens used during training, with the dashed line indicating the 50% target. Right: absolute deviation from the target token budget on train and validation sets.

Comparison against APT

We compare SPLIT directly against APT under matched conditions to isolate the contribution of our decision network. Since both methods use the same multi-scale patch embedding scheme, any performance gap must come from where each method chooses to allocate fine tokens. Figure 5a shows the Pareto frontier of accuracy against token retention on ViT-L at 336×336336 \times336. Across every token budget we evaluate, SPLIT achieves higher accuracy than APT, with the gap widening as the token budget tightens.

Comparison of validation accuracy and sequence length distribution

Figure 5. (a) Accuracy vs. % tokens for ViT-L 336/14 (b) Sequence length distribution at same mean % tokens. Comparison of our method against APT.

Figure 5b examines this allocation behavior more closely by plotting the per-image sequence length distributions of the two methods, with their mean sequence lengths matched. Both distributions are skewed toward maximum-length sequences, indicating that a substantial fraction of images are split fully, but APT does this noticeably more often than SPLIT. The mass that SPLIT pulls away from full splits is redistributed toward lower sequence lengths, meaning SPLIT identifies more images for which fewer tokens suffice. APT’s tendency to over-tokenize is consistent with its reliance on pixel entropy: visually busy images trigger fine splits regardless of whether the busy regions are semantically informative.

Figure 7 illustrates this difference qualitatively. SPLIT concentrates fine tokens around object boundaries and semantically meaningful structure, while leaving textured but task-irrelevant regions such as patterned walls, tiled floors and repeated backgrounds, at coarse resolution. In contrast, APT frequently allocates fine tokens to these high-entropy but semantically empty regions, consuming budget that SPLIT redirects to objects. Together, these results support our central claim: pixel-level heuristics mis-locate tokens in predictable ways, and a learned decision policy trained against the downstream objective improves patch size allocations.

Quantifying semantic alignment. To make this comparison concrete, we measure fine-token recall on ADE20K. We define object recall as the fraction of object-class patches that receive a fine (16×1616 \times16) token, and boundary recall as the fraction of class-boundary patches that receive a fine token, both at a matched token budget (full definition in Appendix section F). As Table 6 shows, SPLIT routes fine tokens to a much larger share of task-relevant patches than the entropy heuristic.

Method% TokensObject recallBoundary recall
Random0.7000.6970.698
APT (entropy)0.7190.8080.833
SPLIT (Ours)0.7140.9070.945

Table 6. Fine-token recall on ADE20K at a matched token budget (∼0.7\sim0.7). Object and boundary recall measure the fraction of object-class and class-boundary patches, respectively, that receive a fine token.

Conclusion

We presented SPLIT, an adaptive patchification framework that learns a content-based patch splitting policy. By treating coarse tokens as lightweight semantic anchors and selectively refining only the regions that matter for the downstream task, SPLIT produces shorter input sequences that are better matched to task-relevant content. We formulate patch refinement as a budget-constrained decision problem and train the splitting policy with reinforcement learning, allowing the model to learn where finer tokens are most useful under a fixed sequence-length budget.

Across ImageNet classification, COCO detection, and ADE20K segmentation, SPLIT matches or surpasses the accuracy of full-resolution ViT baselines while delivering consistent throughput gains over prior adaptive patchification methods. These results show that grounding patch splitting decisions in the model’s interpretation of task relevant information generalises well beyond classification and offers a practical path toward more efficient ViTs without sacrificing task performance.

Limitations. SPLIT delivers significant inference speedups by reducing the average token sequence length to a target reduction ratio that is fixed at training time. Evaluating SPLIT for a different desired speedup requires a short retraining run rather than a free inference-time hyperparameter. Conditioning the patchification policy on the desired ratio at inference would remove this constraint. Our second limitation is scope: we evaluate SPLIT only on image understanding tasks. Image generation typically operates at much higher resolutions and with much larger models, a regime where our method’s speedup is most pronounced. This remains untested, and we view extending SPLIT to generative settings as a promising direction for future work.

Future Work. Beyond the directions discussed in the limitations, using SPLIT for self-supervised tasks to create policies reusable across multiple tasks is a promising direction. On a similar note, a thorough analysis on the behavior of SPLIT trained on a task and re-used on another is yet to be characterized and may be useful in understanding how to develop a task-agnostic tokenization policy.

Acknowledgments and Disclosure of Funding

This work was supported by Fujitsu Research of America. We thank Souraja Kundu for helpful feedback and assistance with the writing of this paper.

Implementation Details

Hardware Setup. All inference-time results, including throughput analysis, were computed on a single NVIDIA A100. ImageNet classification training was conducted on a single node of 8x NVIDIA A100s. Object detection, semantic segmentation, and visual QA experiments were conducted on a single node of 8x NVIDIA H200s. We used a single node for all work in this paper.

Layer-level Merging Baselines. For comparisons against layer-level token reduction methods, we used the official repositories for EViT [19], ToMe [1], and DTEM [17]. We kept all experimental settings (training schedule, optimizers, input resolutions, and other hyperparameters) identical to the original baselines to ensure a fair comparison.

Image Classification. We implemented image classification models using the timm library [36], leveraging its pretrained checkpoints. The ImageNet-1K dataset [8] was used for training and evaluation, following prior work [29] (Havtorn et al. 2023). We perform 1 epoch of warmstart and 1 epoch of RL fine-tuning for all experiments. No augmentations are used. All training was conducted on 8 GPUs with a batch size of 512. For the decision network, we used a warm-start learning rate of 1e−31\mathrm{e}{-3}, a PPO learning rate of 1e−51\mathrm{e}{-5}, a Lagrangian dual learning rate of 5e−45\mathrm{e}{-4} and a warm start threshold update rate of ητ=1e−3\eta_{\tau} = 1\mathrm{e}{-3}. All image classification experiments involve a single stage of splitting.

Visual QA. For our Visual Question Answering (VQA) experiments, we utilized the official LLaVA-1.5 [21] implementation and its pretrained checkpoints. Unlike the original approach, which fine-tunes on the entire dataset for one epoch, we fine-tuned only 5% of the dataset and do one epoch for warmstart and one epoch for RL. To adapt to this setting, we reduced the learning rate by a factor of 10 while following all other fine-tuning procedures recommended by LLaVA. The base image resolution was set to 336 with a patch size of 14, as specified in LLaVA’s default configuration. We keep our retained token fraction to 0.7 and 0.001 for warm start threshold update rate.

Object Detection. For object detection, we used the official implementation of EVA-02 [10] along with its pretrained checkpoints, which utilize a window attention mechanism. Fine-tuning was conducted following the recommended procedures outlined in EVA-02. Consistent with our previous experiments, we fine-tuned for 5% of the total iterations, split evenly between the warm-start and PPO phases, while reducing the learning rate by a factor of 10. Following EVA-02’s settings, the image resolution was 1536 with a base patch size of 16. SPLIT additionally produces patches at sizes 128, 64, 32, and 16. For the decision network, we used a warm-start learning rate of 1e−31\mathrm{e}{-3}, a PPO learning rate of 1e−51\mathrm{e}{-5}, and a Lagrangian dual learning rate of 5e−35\mathrm{e}{-3}, with a target retained token fraction of ρ=0.7\rho= 0.7 and a warm start threshold update rate of ητ=1e−3\eta_{\tau} = 1\mathrm{e}{-3}.

Semantic Segmentation. We also utilized the official EVA-02 implementation along with its pretrained checkpoints for semantic segmentation. The ADE20K dataset [43, 42] was used for training and evaluation. Fine-tuning followed the recommended procedures outlined in EVA-02. In alignment with our previous experiments, we fine-tuned for 5% of the total iterations, split evenly between the warm-start and PPO phases, while reducing the learning rate by a factor of 10. According to EVA-02’s settings, the image resolution was either 512 or 640, with a base patch size of 16. SPLIT produces patches at sizes 64, 32, and 16. The decision network hyperparameters match the detection setting, except that the target retained token fraction is set to ρ=0.65\rho= 0.65.

Choice of Policy Gradients over Differentiable Relaxations

In this section, we provide more detail on our decision to use policy gradients which shape our core algorithm. As discussed in § 3.3, training the decision network requires gradients to flow through an indexing operation that selects whether a patch is represented by a coarse token or by 4 fine tokens in the final sequence passed to the transformer backbone. This operation is non-differentiable as the decision network makes a binary decision and that decision determines which tokens exist in the sequence. Using a differentiable relaxation possible in principle, MSViT [13] builds one for mixed-scale tokenization. However, since the difficulty is that our discrete decision is structural, a relaxation then has only two options: Option 1: Keep both scales. A soft mixture, or a masked union as in DynamicViT [28], lets gradients flow, but every scale must be materialized. For a single splitting stage, this means materializing 5N5N tokens (NN coarse +4N+ 4N fine), versus the 4N4N of the dense sequence. With attention cost quadratic in sequence length and activation memory growing with it, this becomes prohibitive at 1536×15361536 \times1536 resolution. Hierarchical splitting makes this worse: a fine token exists only if every ancestor split, so its relaxed weight is a product of gates across levels and gradients to each gate are filtered through the approximations of the others.

Option 2: Compute only the selected scale. Drop the un-selected tokens to keep the training costs from increasing beyond the dense sequence limit and use Straight-Through Estimators for the binary decision. In this case the gradients are biased and the unselected tokens produce no learning signal and the policy can never directly learn whether a coarse token would have sufficed and must rely on stochastic exploration for this.

A differentiable relaxation produces a biased gradient estimator and must choose between materializing sequences longer than dense sequences or losing signal on the unselected branch of the decision. Policy gradients avoid both problems and trains on exactly the hard, reduced sequences used at inference, with unbiased gradients. Ada-K routing [41] chose PPO for the same structural problem in MoE routing, stating "direct optimization . . . is precluded due to the non-differentiable nature of the sampling operation". Also, methods where differentiable gating was tried needed extra stabilizers: MSViT’s batch-shaping loss against gate collapse, DynamicViT’s distillation loss.

What the RL formulation contributes. The RL phase of training using policy gradients on hard sequences yields measurable gains. Table 9 shows the PPO phase improves object detection mAP monotonically over a warm-start-only policy (58.06 → 58.84 at a matched budget). Table 14 shows that it sharpens semantic allocation by +2.0 pp thing and +1.1 pp boundary recall. Also, since our policy gradients allows our training to use hard sampled masks, λ\lambda regulates the actual deployed token count rather than a soft proxy (Figure 6).

RL iterations0 (warm-start only)50010001500†1500^{\dagger}
COCO mAP58.0658.2058.5258.84

Table 9. RL-iterations ablation on object detection (EVA-02-B @ 1536, ρ=0.70\rho= 0.70) with a fixed 1500-iteration warm-start. Default configuration marked †\dagger.

MethodToken fractionThing recallBoundary recall
Random0.7000.6970.698
APT (entropy)0.7190.8080.833
SPLIT, warm-start only0.7080.8870.934
SPLIT, full0.7140.9070.945

Table 14. Fine-token recall on ADE20K val at a matched token budget (∼0.7\sim0.7). Thing recall is the fraction of object-class patches, and boundary recall the fraction of class-boundary patches, that receive a fine (16 × 16) token.

Sensitivity to Hyper-parameters

A natural concern for an RL-based method is how sensitive it is to training hyper-parameters, and whether the added optimization introduces brittle tuning. To address this, we sweep each of SPLIT’s PPO-specific and warm-start hyper-parameters, varying one at a time around the default configuration used in the main paper while holding all others fixed.

PPO hyper-parameter sweeps Table 7 varies the PPO clip ratio ϵ\epsilon, the number of rollouts KK, the number of PPO epochs, the warm-start balancer rate, and the backbone learning rate during the warm-start. The results in Table 7 correspond to the image classification task using the configuration: ViT-L, 224×224, 0.5 target token fraction. We observe that the accuracy is within ±0.2% across all settings except in the case of (i) running ≥4\ge4 PPO epochs over the same rollouts and (ii) backbone learning rates above 1e−61\mathrm{e}{-6} degrade the pretrained features (85.73% at 1e−51\mathrm{e}{-5}). Both failure modes have well-understood causes: (i) over-optimizing the PPO objective and (ii) over-writing pretrained features. These are avoidable pitfalls and do not reflect instability unique to SPLIT.

Hyper-parameter (default)Values sweptTest acc (%)
PPO clip ratio ϵ\epsilon (0.10)0.05 / 0.10 / 0.2086.49 / 86.49 / 86.48
Rollouts KK (4)2 / 4 / 686.49 / 86.49 / 86.33
PPO epochs (2)1 / 2 / 486.50 / 86.49 / 83.28
Warm-start balancer rate (0.005)0.001 / 0.005 / 0.0286.44 / 86.49 / 86.54
Backbone LR (1e−6)5e−7 / 1e−6 / 5e−6 / 1e−586.55 / 86.49 / 85.97 / 85.73

Table 7. Single hyper-parameter sweeps on ImageNet-1K (ViT-L, 224 × 224, target token fraction 0.5). A run using the default hyper-parameters scores 86.49 at token fraction 0.586.

Dual rate and policy learning rate are coupled. The Lagrangian dual rate ηλ\eta_\lambda controls how quickly the multiplier λ\lambda reacts to token-budget violations and the policy learning rate controls how fast the policy changes based on the combined cost weighted by λ\lambda. Due to this coupling of the two learning rates through λ\lambda, it makes sense to observe their joint behavior, which we present in Table 8 using the same configuration as Table 7 with a target token fraction of 0.5 and varying only the two learning rate hyper-parameters.

Policy learning rate
ηλ\eta_\lambda1e−61\mathrm{e}{-6}5e−6†5\mathrm{e}{-6}^{\dagger}1e−51\mathrm{e}{-5}
1e−41\mathrm{e}{-4}85.2 / 0.7286.3 / 0.8186.2 / 0.84
5e−4†5\mathrm{e}{-4}^{\dagger}86.1 / 0.6286.4 / 0.5886.1 / 0.51
1e−31\mathrm{e}{-3}86.3 / 0.5586.1 / 0.5286.1 / 0.51

Table 8. Joint sweep of the Lagrangian dual rate ηλ\eta_\lambda and the RL-phase policy learning rate (ViT-L, 224 × 224, target token fraction 0.5). Each cell reports test accuracy (%) / realized token fraction. Defaults (†\dagger) are ηλ=5e−4\eta_\lambda= 5\mathrm{e}{-4} and policy LR = 5e−65\mathrm{e}{-6}.

We observe that an under-reactive dual (ηλ=1e−4\eta_\lambda= 1\mathrm{e}{-4}) overshoots the target budget by 22–34% as the cost does not adjust fast enough to enforce the budget. For ηλ≥5e−4\eta_\lambda\ge5\mathrm{e}{-4} the two hyper-parameters substitute for one another, anti-diagonal cells match within 0.1% and effectively behave as a single degree of freedom.

Number of warm-start and RL iterations. We also study how performance depends on the iterations used during the warm-start and RL phases of training through two complementary experiments. In the first experiment, we fix a 1500-iteration warm-start on object detection and track performance across subsequent RL iterations. Table 9 shows that mAP increases monotonically with RL iterations (58.06 → 58.84). The PPO phase contributes +0.8 mAP over the warm-start decisions at a matched token budget. In the second, we vary the split of iterations between the two phases on semantic segmentation while holding the total iteration count fixed at 3000. As Table 10 shows, final mIoU stays within 0.05 mIoU across all three splits and during training we observe that shorter warm-starts only enlarge the transient token-fraction oscillation early in RL. The warm-start therefore mainly stabilizes the beginning of RL rather than setting final quality.

Warm-start itersRL itersTotal itersFinal mIoU
10002000300059.34
1500†1500^{\dagger}1500†1500^{\dagger}300059.39
20001000300059.38

Table 10. Warm-start/RL iterations split on segmentation (EVA-02-L @ 640, ADE20K, ρ=0.65\rho= 0.65) with the total iteration count fixed at 3000. Default configuration marked †\dagger.

Robustness to Distribution Shift

A desirable property of any adaptive tokenization method is that it must be robust to deviations from the data distribution on which the tokenization was trained or tuned. We run experiments to observe how the trained policy reacts to (i) change in distribution of test data and (ii) corruption of input images. We compare the behavior of SPLIT against APT [5] to show the benefit of using a learnable approach as compared to a heuristic driven approach.

Distribution shift. We applied the ImageNet-trained policy to five datasets at 224×224224 \times224 without retraining the policy: CIFAR-100, ImageNet-V2, -Sketch, -R, -A. For ImageNet →\rightarrow CIFAR-100 (resized), we learn a linear probe fit with the backbone frozen, since the label space of CIFAR-100 is different from that of ImageNet. For the four ImageNet variants, we use the same 1000-class head (frozen); specifically ImageNet-R/A are standard 200-class subsets, and are scored by restricting the logits to the subset. We compare with APT in two forms: fixed-τ\tau, a direct transfer condition, since recalibration needs target-domain data, and τ\tau recalibrated per dataset to SPLIT’s token fraction for a matched-budget accuracy comparison.

We run the experiments in Table 11 with a target token fraction of 0.85 while training. SPLIT maintains a token fraction within 0.74 −- 0.90 across all test datasets and is at most ∼\sim0.11 from its training target. Fixed-τ\tau APT swings over 0.50 −- 0.89, overspending on CIFAR-100, collapsing to 0.50 on ImageNet-Sketch. Even against per-dataset recalibrated APT, SPLIT is more accurate on the four ImageNet variants by +0.14 to +0.77, while staying within 0.1 pp of the dense baseline.

DatasetDenseSPLIT (tok. frac.)APT fixed-τ\tau (tok. frac.)APT recal.
CIFAR-100 (probe)89.8789.26 (0.795)89.60 (0.831)89.49
ImageNet-V279.1379.07 (0.90)78.93 (0.89)78.93
ImageNet-Sketch61.1461.22 (0.74)56.82 (0.50)60.73
ImageNet-R77.1977.06 (0.83)75.71 (0.74)76.42
ImageNet-A64.4564.37 (0.90)63.28 (0.89)63.60

Table 11. Transfer of the ImageNet-trained SPLIT policy to five datasets with no retraining (ViT-L, 224×224224 \times224; training target token fraction 0.85). For SPLIT and fixed-τ\tau APT we report accuracy with the realized token fraction in parentheses. “APT recal.” tunes τ\tau per dataset to SPLIT’s token fraction.

Input corruption. To isolate the behavior in a controlled setting, we add Gaussian pixel noise of increasing standard deviation σ\sigma to the ImageNet test data and evaluate at 336×336336 \times336 (ViT-L). In Table 12 we start with SPLIT and APT having the same token fraction under zero noise. As noise increases the entropy of every patch, APT progressively splits more patches and its token fraction climbs from 0.53 to 0.95, while SPLIT holds its budget (0.53 →\rightarrow 0.59).

Noise σ\sigmaSPLIT acc @ tok. frac.APT acc @ tok. frac.
0.0087.7 @ 0.5387.5 @ 0.53
0.0586.5 @ 0.5386.7 @ 0.62
0.1084.9 @ 0.5485.3 @ 0.75
0.1583.0 @ 0.5682.8 @ 0.92
0.2080.1 @ 0.5979.3 @ 0.95

Table 12. Gaussian pixel noise of standard deviation σ\sigma added to ImageNet test data (ViT-L @ 336×336336 \times336). Each entry is test accuracy (%) at the realized token fraction. APT τ\tau is calibrated to match SPLIT’s zero-noise token fraction.

Training Costs

We quantify SPLIT’s training overhead for EVA-02-L at 640×640640 \times640 on ADE20K, with two splitting stages (three patch scales). Table 13 uses identical hardware (single A100 GPU) and batch size, and a matched 3000-iteration schedule for all rows and reports per-iteration and per-image latency alongside total wall-clock training time. Even though SPLIT has two phases of training (warmstart and PPO), the training budget is matched, not doubled: SPLIT partitions APT’s 3000-iteration schedule into 1500 warm-start and 1500 PPO iterations. Because the backbone is frozen throughout the PPO phase, the backbone in SPLIT receives half the updates APT does. In total, SPLIT trains only about 12 minutes longer than APT, an overhead that comes almost entirely from the PPO phase.

Configms/iterms/imgItersTotal
Vanilla (dense)96360300048.2 min
APT finetune80951300040.45 min
SPLIT warm-start91057150022.8 min
SPLIT PPO phase118674150029.6 min
SPLIT total——300052.4 min

Table 13. Training cost on ADE20K (EVA-02-L @ 640, two splitting stages, 1×A100 GPU).

It is important to note that all methods, including heuristic driven ones like APT would require updating the backbone to adapt to the change in token distribution with a new tokenization scheme. The warm-start phase plays exactly this role for SPLIT, and in no configuration does SPLIT’s backbone receive more updates than APT’s. On classification, where recipes are epoch-based, the PPO phase adds one extra epoch with the backbone frozen (∼80\sim80 minutes total on 2×A100 for the full two-phase run). SPLIT’s gains are therefore attributable to policy learning, not additional finetuning.

Beyond training cost, this dependence on updating the backbone also constrains how an adaptive tokenizer can be used at inference. Changing the token budget at run time not only requires a tokenizer that allows for adjusting the budget but also needs the backbone to adapt to the new distribution of tokens for optimal performance. APT allows for budget control by adjusting the threshold τ\tau to change the token budget, but since the backbone was trained with a different budget, the backbone cannot perform optimally unless we separately train a backbone for the new τ\tau. SPLIT in its current formulation does not support changing the token budget without re-training, this can be approximately achieved without training by shifting probabilities of the decision network, but this still does not address the problem of the backbone not being tuned for a new budget. We note this per-budget retraining requirement as a limitation (§ 5) and as a direction for future work. The ideal resolution would train the backbone with a randomly sampled ρ\rho and a tokenizer that accepts ρ\rho as an input on which decisions are conditioned, resulting in a single model taking a desired token fraction directly as input and removing the need for any per-budget adaptation.

Quantitative Assessment of Semantic Alignment

To quantitatively show that SPLIT’s fine-token allocation is semantically aligned, we test whether the finest (16 × 16) patches land on semantically important regions. We measure fine-token recall on ADE20K val against SceneParse150 ground-truth masks (unlabeled / not-in-150 pixels are ignored). For each image we build (1) a binary map of which 16 × 16 cells the policy keeps at fine resolution, and (2) a patch-level ground-truth label by pixel majority vote per cell; thing patches are object-class cells, and boundary patches are cells whose majority label differs from that of a 4-neighbor. Thing (resp. boundary) recall is the fraction of such patches that receive a fine token, i.e., of all thing (resp. boundary) patches, what share is allocated fine resolution. We compare Random, APT (entropy), and SPLIT at matched budgets (∼0.7\sim0.7 token fraction) in Table 14.

As a sanity check, the Random baseline recalls objects and boundaries at approximately its token budget (0.697 and 0.698 at a 0.700 fraction), confirming that the metric is properly calibrated. At a matched budget, SPLIT allocates fine tokens to a substantially higher share of objects (+9.9 pp over APT) and boundaries (+11.2 pp), and the PPO phase further sharpens alignment beyond the warm-start policy (+2.0 pp thing, +1.1 pp boundary). This quantifies what our qualitative visualizations (Figure 7) shows: the entropy heuristic spends its budget on high-entropy but semantically empty texture, whereas the learned policy reallocates it toward task-relevant structure.

Decision Network Architecture

We also ablate various different decision network architectures with the same parameter count to justify our choice of the cross attention architecture. In Table 15 we test our choice against using 2 layer MLPs with input as (i) only the embedding of the un-split parent patch or (ii) embedding of the un-split parent concatenated with embedding of all the child patches. We find that using the embeddings of both the parent and child embedding as inputs to the decision network improves performance, and our cross attention design outperforms using simple MLP layers for this.

Decision Network ArchitectureParam Count% TokensVal Acc (%)
MLP(xparentx_{parent})1.115M89.5987.69
MLP(concat([xparentx_{parent}, xchild(1)x_{child}^{(1)} …\ldots xchild(4)x_{child}^{(4)}]))1.116M90.0887.74
Cross Attention 3.21.115M89.5387.81

Table 15. Comparison of different decision network architectures using ViT-L 224/14

Qualitative Results

SPLIT generalizes effectively to downstream visual tasks that require spatial precision, including object detection and semantic segmentation. As illustrated in Figure 9 and Figure 10, SPLIT reliably allocates larger patches to uniform background regions while preserving fine-grained structure with smaller patches around object boundaries and detail-heavy areas. The resulting predictions match the vanilla full-resolution baseline closely, supporting accurate bounding box regression and maintaining the pixel-level fidelity required for segmentation. To verify that mixed-resolution tokenization does not distort the model’s internal representations, Figure 11 compares backbone feature maps under a shared PCA basis: SPLIT’s features exhibit the same spatial structure as the vanilla baseline, indicating that the multi-stage splitting policy preserves the learned representation rather than disrupting it.

Object detection qualitative comparison (Vanilla vs Ours)

Figure 9. Object detection qualitative comparison (Vanilla vs Ours). We compare fixed-grid ViT inference (Vanilla) against our learned multi-stage patch splitting method. For each example, we visualize the patch assignment map and the corresponding detection output. Our method allocates finer patches to detail-heavy regions while preserving detection quality.

Semantic segmentation qualitative comparison (Vanilla vs Ours)

Figure 10. Semantic segmentation qualitative comparison (Vanilla vs Ours). We compare fixed-grid ViT inference (Vanilla) against our learned multi-stage patch splitting method. For each example, we visualize the patch assignment map and the corresponding segmentation output. Our method allocates finer patches to detail-heavy regions while preserving segmentation quality.

PCA visualization of backbone feature maps (Vanilla vs Ours)

Figure 11. PCA visualization of backbone feature maps (Vanilla vs Ours). We project high-dimensional backbone features to RGB using a shared PCA basis fit on Vanilla features, then apply the same projection to our method. Comparable spatial structure in the PCA maps indicates that mixed-resolution splitting does not corrupt the learned representation.

Figure 12 illustrates the failure mode that emerges when the token budget is pushed beyond the operating point used in our main experiments. At a 45% retention target, SPLIT begins to under-tokenize genuinely informative regions, producing coarse object boundaries and missing thin structures in the predicted segmentation. These cases mark the boundary of the budget–accuracy trade-off: while SPLIT preserves task performance across the budgets used in our main results, very aggressive budgets force the policy to drop fine tokens in regions where they are needed.

Failure cases under aggressive token budgets

Figure 12. Failure cases under aggressive token budgets. When the target sequence length is pushed below the level used in our main experiments (here, 45%), SPLIT begins to under-tokenize semantically important regions, leading to coarse boundaries and missed thin structures in the predicted segmentation.

Splitting decision at different token budgets

Figure 8. Splitting decision at different token budgets Visualizing the patch splitting decisions at various token budgets shows how the model progressively prioritizes patches based on content.

References

  1. [1]Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.09461
  2. [2]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.arxiv.org/abs/2005.12872
  3. [3]Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-to-fine method for vision transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 7042–7052, 2023a.
  4. [4]Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17164–17174, 2023b.arxiv.org/abs/2305.17997
  5. [5]Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, and Kris M. Kitani. Accelerating vision transformers with adaptive patch sizes. In International Conference on Learning Representations, 2026.arxiv.org/abs/2510.18091
  6. [6]Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.12773
  7. [7]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.14135
  8. [8]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  9. [9]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  10. [10]Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024.DOI
  11. [11]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https://arxiv.org/abs/2306.13394.
  12. [12]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  13. [13]Jakob Drachmann Haytorn, Amélie Royer, Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Msvit: Dynamic mixed-scale tokenization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 838–848, 2023.
  14. [14]Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.arxiv.org/abs/1902.09506
  15. [15]Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955, 2025.arxiv.org/abs/2507.07955
  16. [16]Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. SPvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pages 620–640. Springer, 2022.DOI
  17. [17]Dong Hoon Lee and Seunghoon Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. Advances in Neural Information Processing Systems, 37:54079–54104, 2024.arxiv.org/abs/2412.10569
  18. [18]Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280–296. Springer, 2022.arxiv.org/abs/2203.16527
  19. [19]Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations, 2022.arxiv.org/abs/2202.07800
  20. [20]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  21. [21]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024a.arxiv.org/abs/2310.03744
  22. [22]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024b.DOI
  23. [23]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.arxiv.org/abs/2103.14030
  24. [24]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022.arxiv.org/abs/2209.09513
  25. [25]Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309–12318, 2022.arxiv.org/abs/2111.15668
  26. [26]Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. IA-RED²: Interpretability-aware redundancy reduction for vision transformers. Advances in neural information processing systems, 34:24898–24911, 2021.DOI
  27. [27]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.arxiv.org/abs/2103.00020
  28. [28]Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.02034
  29. [29]Tomer Ronen, Omer Levy, and Avram Golbert. Vision transformers with mixed-resolution tokenization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4613–4622, 2023.arxiv.org/abs/2304.00287
  30. [30]Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer, 2022.arxiv.org/abs/2206.01718
  31. [31]Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.arxiv.org/abs/1904.08920
  32. [32]Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In IEEE/CVF International Conference on Computer Vision, 2021.arxiv.org/abs/2105.05633
  33. [33]Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12165–12174, 2022.arxiv.org/abs/2106.02852
  34. [34]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  35. [35]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
  36. [36]Ross Wightman. Pytorch image models, 2019. URL https://github.com/rwightman/pytorch-image-models.
  37. [37]Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint arXiv:2310.01812, 2023.arxiv.org/abs/2310.01812
  38. [38]Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418–434, 2018.arxiv.org/abs/1807.10221
  39. [39]Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10809–10818, 2022.arxiv.org/abs/2112.07658
  40. [40]Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vit: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.arxiv.org/abs/2308.02490
  41. [41]Tongtian Yue, Longteng Guo, Jie Cheng, Xuange Gao, Huang, and Jing Liu. Ada-K routing: Boosting the efficiency of MoE-based LLMs. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.10456
  42. [42]Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.DOI
  43. [43]Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019.arxiv.org/abs/1608.05442

Paper details

vlnc:W_mk85aznf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents