betaFeedback

WorksComputer Science

MATA: Motion-aligned Temporal Adapter

Abstract

Few-shot fine-grained video action recognition requires recognizing novel action classes from only a few labeled examples. The difficulty is that visually similar actions often share actors, scenes, poses, and objects, differing only in the order, timing, and short duration of motion changes. We study this problem under three practical constraints: few tunable parameters, low compute cost, and use of the video modality alone. We propose Motion-aligned Temporal Adapter (MATA), a parameter-efficient adapter on top of a frozen CLIP backbone. MATA specializes CLIP features along the temporal dimension using two components: temporal phase alignment through learnable temporal positional embeddings, and a local motion transition operator through depth-wise convolution. With only ~3.6M tunable parameters, MATA achieves state-of-the-art performance across multiple fine-grained benchmarks while remaining robust to both visual corruptions and temporal disruptions induced by frame dropping. Code is available at https://github.com/prakashchhipa/MATA-Motion-aligned-Temporal-Adapter

Introduction

Few-shot fine-grained action recognition requires separating novel actions given only a few labeled examples. The challenge is that visually similar actions often share actors, scenes, poses, and objects [20, 34], while differing in brief motion changes, timing, and action phases (e.g., preparation, execution, and completion stages). Thus, recognition is both low-data and temporally sensitive: support-query matching must align action phases and specialize temporal features to capture the motion cues that separate fine-grained classes.

Existing few-shot action recognition methods face two practical limitations. Metric- and alignment-based methods such as OTAM [3], TRX [29], STRM [37], and HyRSM [41] improve support-query matching, but lack the transferable priors of foundation models. CLIP-based methods address this limitation, yet often rely on full backbone fine-tuning, additional modalities as in CLIP-FSAR [43], MA-FSAR [45], or heavier meta-learning designs [43, 45]. On the other hand, parameter-efficient video adapters on CLIP reduce tunable parameters, but typically combine spatial and temporal adaptation through multi-branch designs (AIM [48], attention-based modules [2]–[28]), or 3D convolutions (ST-Adapter [26]). This can add computation and can dilute the motion-specific cues that are most critical in few-shot fine-grained action recognition. A third line of work improves temporal reasoning through explicitly extracted motion structure: Trokens [16] builds trajectory tokens in a separate offline stage before training, which adds preprocessing time and storage, and makes recognition sensitive to the quality of the extracted tokens.

This leads to the research question: Can CLIP be adapted for few-shot fine-grained video action recognition using a minimal, compute-efficient mechanism that focuses on temporal dynamics while using only the video modality? Therefore, we study few-shot fine-grained video action recognition under the accuracy–compute efficiency–unimodality objective: i) improving recognition performance with few tunable parameters, ii) being compute efficient, and iii) using only the readily available video modality. This objective follows directly from the challenges above. The model must preserve CLIP’s transferable spatial and semantic priors, recover discriminative temporal evidence from support and query videos, and avoid the cost of full backbone tuning, heavy spatio-temporal adaptation, or additional modality pipelines.

We propose Motion-aligned Temporal Adapter (MATA in Figure 1), a parameter-efficient adapter for a frozen CLIP visual backbone. MATA is motivated by two needs in few-shot fine-grained recognition: support-query action phases must be aligned, and brief motion transitions must be modeled explicitly. MATA addresses the first need through temporal phase alignment (i.e., representing each frame with its relative position in the action sequence), achieved by adding learnable temporal positional embeddings to patch-level bottleneck features so that each visual state is tied to its relative action phase. It addresses the second need through local motion transition modeling, achieved by depth-wise one-dimensional temporal convolution along the frame dimension to capture fine-grained details. The resulting temporal response is projected back to the CLIP token dimension through identity-preserving residual reconstruction, keeping adaptation compact while preserving CLIP’s pretrained priors.

Overview of the MATA few-shot inference pipeline

Figure 1. Overview of the MATA few-shot inference pipeline. Support and query videos are encoded by a shared frozen CLIP backbone, and query labels are predicted via prototype matching. Compared to full fine-tuning [43], MATA updates only a small set of parameters, improving accuracy while reducing latency.

With only ∼3.6M tunable parameters, MATA sets a new state of the art on fine-grained few-shot benchmarks under a CLIP backbone, reaching 80.12% and 81.28% 1-shot accuracy on Diving48 and FineGym288 and improving over D2ST [28] by 6.83 and +10.95 points while reducing per-episode latency by 4.10×4.10 \times. MATA also remains the most accurate method under all five test-time corruptions applied to query clips, showing that focused temporal adaptation can be accurate, compact, fast, and robust from the video modality alone. Our contributions are:

  • We show that temporal-only adaptation is sufficient for few-shot fine-grained action recognition, effectively capturing short-lived and rapid motion differences between visually similar actions without adapting spatial features.

  • We introduce MATA, a parameter-efficient temporal adapter that adapts a frozen CLIP visual backbone for few-shot recognition from short video clips, requiring neither additional modalities nor auxiliary cues such as pose or offline preprocessing, while substantially reducing tunable parameters and computational cost.

  • We establish MATA’s effectiveness and robustness through extensive evaluation across fine-grained action benchmarks, achieving state-of-the-art few-shot recognition while remaining robust to both visual corruptions and temporal disruptions induced by frame dropping.

Related Work

Vision–language pretraining for video actions. CLIP [31] provides open-vocabulary visual representations that have been extended to video by ActionCLIP [46], X-CLIP [25], ViFi-CLIP [32], and CAST [17] through video–text alignment, cross-frame attention, temporal alignment, and spatial–temporal cross-attention. These rely on full backbone fine-tuning or heavy video-specific modules, which is inefficient for fine-grained actions with rapid, subtle motion.

Parameter-efficient video adaptation. Following lightweight adapters in NLP [13], ST-Adapter [26], DUALPATH [27], D²ST [28], and AIM [48] adapt ViT backbones through spatial–temporal modules built from multi-branch or attention-based designs, while Task-Adapter [1] and Task-Adapter++ [2] add task-specific and order-aware alignment. All jointly model spatial appearance and temporal structure. We instead target fine-grained recognition, where short-duration motion is the main discriminative signal, through focused temporal adaptation without additional spatial modeling.

Few-shot action recognition. Classical FSAR methods such as ProtoNet [35], OTAM [3], TRX [29], PAL [55], STRM [37], and HyRSM [41] learn global similarity metrics or temporal alignments, while VPD [12] distills pose supervision into RGB features. Such cues are less suited to actions separated by brief motion evolution rather than coarse appearance. CLIP-based methods exploit stronger semantic priors: CLIP-FSAR [43] fine-tunes the full backbone with temporal transformers, and MA-FSAR [45] freezes it but adds multi-modal and meta-learning components. Others refine matching or temporal structure: TEAM [18] removes explicit temporal alignment, MPL [47] learns match-guided prototypes, TSAM [19] models frame order as a sequence, HMP-Net [5] builds hierarchical meta-prototypes, and EMP-Net [44] reduces cost via multi-level post reasoning. TATs [15] and Trokens [16] instead represent motion as trajectory tokens, which is effective on fine-grained data but requires an offline extraction stage and depends on tracker quality.

MATA retains a frozen CLIP visual backbone and introduces a minimal temporal-only adapter, avoiding attention-heavy or multi-branch architectures, auxiliary modalities, and offline preprocessing, while achieving strong performance on FineGym99, FineGym288, and Diving48.

Method

Problem setting

We consider NN-way KK-shot fine-grained action recognition under the episodic setting. Each episode contains a support set

S={(vi,yi)}i=1NK,yi∈{1,…,N},(1)S = \left\{(v_i, y_i)\right\}_{i=1}^{NK}, \qquad y_i \in\{1,\ldots,N\}, \tag*{(1)}

and a query set

Q={vjq}j=1M.(2)Q = \left\{v_j^q\right\}_{j=1}^{M}. \tag*{(2)}

The support set provides KK labeled videos for each of the NN action classes. For each query video vjqv_j^q, the recognition objective is to predict

y^jq∈{1,…,N}(3)\hat{y}_j^q \in\{1,\ldots,N\} \tag*{(3)}

using only the labeled support videos from the same episode. Videos are encoded by a shared frozen CLIP visual backbone with TT segment-sampled frames, so index tt denotes a normalized position within the action. A video is represented by the mean of its frame-level class tokens, z(v)=1T∑txcls(t)z(v) = \frac{1}{T}\sum_t x_{\mathrm{cls}}^{(t)}, which is ℓ2\ell_2-normalized. Each class prototype is the mean of its KK support representations, and a query is assigned to the prototype of highest cosine similarity. Training minimizes the cross-entropy of query predictions over episodes; the CLIP text encoder is not used.

Motion-aligned temporal adapter (MATA)

MATA formulates temporal adaptation (in Figure 1) in a low-dimensional bottleneck space. The key components of MATA are temporal phase alignment, a local motion transition operator, and residual reconstruction. Temporal phase alignment adds frame-order information, the local motion transition operator models short temporal changes, and residual reconstruction maps the temporal response back to the original CLIP token dimension.

Input tokens. Let BB be the batch size, TT the number of frames, H×WH \times W the spatial patch grid, and CC the token dimension. The token sequence entering a transformer block is

X∈R(HW+1)×(BT)×C,(4)X \in\mathbb{R}^{(HW+1)\times(BT)\times C}, \tag*{(4)}

where the first token is the class token and the remaining tokens are spatial patch tokens:

X=[xcls;Xp],Xp∈RHW×(BT)×C,(5)X = [x_{\mathrm{cls}}; X_p], \qquad X_p \in\mathbb{R}^{HW\times(BT)\times C}, \tag*{(5)}

so only XpX_p is processed in the bottleneck space and xclsx_{\mathrm{cls}} remains unchanged. Since the class token attends to patch tokens in the following block, the update applied to XpX_p reaches xclsx_{\mathrm{cls}}, and hence z(v)z(v), through the remaining blocks.

Three properties of the few-shot fine-grained setting motivate this design. First, CLIP provides strong spatial and semantic priors, and adaptation should refine them toward temporal discrimination rather than overwrite them, so MATA updates patch tokens through a residual correction

X^p=Xp+ΔXp.(6)\widehat{X}_p = X_p + \Delta X_p. \tag*{(6)}

Second, support-query comparison is reliable only when temporally corresponding action phases are represented consistently, so MATA assigns each frame a learnable phase coordinate before any temporal modeling. Third, fine-grained actions are distinguished by short temporal changes (eg., preparation, onset, release, reversal, delay, or completion), so MATA models transitions among neighboring latent frame states rather than relying on frame-wise appearance alone.

MATA Formulation

Bottleneck temporal projection. Patch tokens are first projected into a compact adapter space:

U=ϕ↓(Xp),U∈RHW×(BT)×Ca,Ca=⌊αC⌋,(7)U = \phi_{\downarrow}(X_{p}), \qquad U \in\mathbb{R}^{HW \times(BT) \times C_{a}}, \qquad C_{a} = \lfloor\alpha C \rfloor, \tag*{(7)}

where α\alpha is the bottleneck ratio. The tensor is then rearranged so that each spatial location forms an independent temporal sequence:

U→Uf∈R(BHW)×T×Ca.(8)U \rightarrow U_{f} \in\mathbb{R}^{(BHW) \times T \times C_{a}}. \tag*{(8)}

This operation isolates temporal evolution while keeping spatial locations structurally separated.

Temporal phase alignment. Let Uf∈R(BHW)×T×CaU_{f} \in\mathbb{R}^{(BHW) \times T \times C_{a}} denote the bottleneck patch sequence, where each index mm corresponds to the same spatial patch location observed across frames. MATA utilizes a learnable temporal positional embedding

Ptime∈RT×Ca.(9)P_{\mathrm{time}} \in\mathbb{R}^{T \times C_{a}}. \tag*{(9)}

The phase-aligned representation is

U~m,t,c=Uf,m,t,c+Ptime(t,c).(10)\widetilde{U}_{m,t,c} = U_{f,m,t,c} + P_{\mathrm{time}}(t,c). \tag*{(10)}

This phase conditioning is achieved through the learnable temporal positional embedding PtimeP_{\mathrm{time}}, which injects relative frame-order information directly into each patch-level bottleneck feature. Because PtimeP_{\mathrm{time}} is indexed by frame position, it is defined for a fixed TT and re-learned when the number of sampled frames changes. The novel design choice is that temporal phase is introduced before local transition modeling. Thus, MATA does not treat a patch feature only as a visual state; it represents it as a visual state tied to its relative action phase.

Local motion transition operator. After temporal phase alignment, MATA applies temporal specialization along the frame dimension. For fine-grained action recognition, this is the relevant adaptation axis: visually similar classes are often separated by short temporal changes rather than by global appearance differences. For each spatial location mm, channel cc, and time index tt, the temporal response is

Ym,t,c=∑δ=0k−1wc,δU~m,t+dδ−ρ,cpad,(11)Y_{m,t,c} = \sum_{\delta=0}^{k-1} w_{c,\delta}\widetilde{U}^{\mathrm{pad}}_{m,t+d\delta-\rho,c}, \tag*{(11)}

where kk is the temporal kernel size, dd is the dilation rate, wc,δw_{c,\delta} is the channel-specific temporal weight, and ρ\rho is the padding offset.

MATA formulates temporal specialization as depth-wise one-dimensional filtering in the bottleneck space, yielding only kCakC_{a} temporal parameters. Per block MATA adds 2CCa+kCa+TCa2CC_{a}+kC_{a}+TC_{a} parameters; with C=768C=768, α=0.25\alpha=0.25, k=3k=3, T=8T=8 over 12 blocks this gives the 3.6M reported in Table 1. The operation is also compute-efficient because it acts locally along TT frames at fixed spatial locations, avoiding dense temporal attention over all video tokens.

MethodAxisTemporal mechanismCostParams (M)
AIM [48]S+TReused temporal MSAO(T2C)O(T^{2}C)11.0
ST-Adapter [26]S+T3D depth-wise convO(k3TC)O(k^{3}TC)7.2
DualPath [27]S+TDual-branch attentionO(T2C)O(T^{2}C)10.0
D2^{2}ST [28]S+TDeformable ST attentionO(T2C)O(T^{2}C)7.3
MATATDepth-wise 1D convO(kTCa)O(kTC_{a})3.6

Table 1. Comparison with representative parameter-efficient video adapters. MATA performs temporal-only adaptation while maintaining linear temporal complexity. Cost is per patch for the temporal operator alone, at the width at which each operator acts. CaC_{a}: bottleneck width; kk: kernel size; S: spatial; T: temporal.

Residual reconstruction. The temporal response is mapped back to the original token dimension and added to the input patch tokens:

ΔXp=ϕ↑(σ(Y)),X^=[xcls;Xp+ΔXp].(12)\Delta X_{p} = \phi_{\uparrow}(\sigma(Y)), \qquad\widehat{X} = [x_{\mathrm{cls}}; X_{p}+\Delta X_{p}]. \tag*{(12)}

The up-projection ϕ↑\phi_{\uparrow} is zero-initialized, so the adapter starts as an identity mapping and gradually learns motion-specific corrections.

Compact form. The complete transformation can be summarized as

X^=[xcls;Xp+ϕ↑(σ(Ftemp(ϕ↓(Xp)+Ptime)))].(13)\widehat{X} = [x_{\mathrm{cls}}; X_{p}+\phi_{\uparrow}(\sigma(F_{\mathrm{temp}}(\phi_{\downarrow}(X_{p})+P_{\mathrm{time}})))]. \tag*{(13)}

Thus, MATA consists of three coupled operations: temporal phase alignment, local motion transition operator, and residual reconstruction.

Table 1 compares MATA among representative parameter-efficient video adapters. Where prior adapters jointly model spatial appearance and temporal structure through multi-branch, 3D-convolutional, or attention-based modules, MATA adapts only the temporal axis, and assigns each frame a learnable temporal phase before modeling local transitions, an ordering that ties each patch feature to its relative action phase and that we isolate in Table 8. Acting in the bottleneck space, its temporal cost is linear rather than quadratic in TT. MATA thus trades generality for focus, which we show is advantageous when the discriminative signal is short-lived motion rather than appearance.

ComponentDiving48FineGym99
1-shot5-shot1-shot5-shot
Local motion transition78.6988.7680.0289.26
+ Temporal phase alignment80.1290.0082.2191.29

Table 8. Component analysis of MATA on Diving48 and FineGym99. Each row cumulatively adds the indicated component.

Results and discussion

Implementation details

We evaluate MATA on six challenging few-shot action benchmarks: Diving48 [20], FineGym99, FineGym288 [34], Toyota SmartHome [7], HAA500 [6] and MPII Cooking2 [33]. These datasets cover fine-grained sports, daily activities, human-object interactions, and temporally sensitive action classes. We use the same class splits as M3M^3Net [36] where applicable for fair comparison.

We follow a 5-way episodic meta-learning protocol with 60,000 training episodes. Only the MATA adapter is optimized with query loss; the CLIP backbone remains frozen. Unless otherwise stated, each video is represented by 8 segment-sampled frames, and training uses batch size of 8, standard video augmentations, and NVIDIA H100 GPUs. At test time, we evaluate 10,000 episodes under 1-shot, 3-shot, and 5-shot settings and report Top-1 accuracy with single-view inference. All experiments use a frozen CLIP ViT-B/16 backbone and 8 input frames by default. MATA uses temporal kernel size k=3k=3, dilation d=2d=2, and bottleneck ratio 0.25.

Establishing state-of-the-art performance across benchmarks

MATA is the most accurate and the most parameter-efficient method in Table 2 in nearly all settings. It improves over the strongest non-foundation baseline by up to +12.62 points at 1-shot with 90% fewer parameters than M3M^3Net, and over D2D^2ST, the strongest prior adapter, by up to +10.95 points with half the tunable parameters. Gains are largest in the low-shot regime, where support-query temporal alignment matters most. Trokens is the closest competitor, but on unequal terms: it uses a stronger DINOv2 backbone and trajectory tokens built in an offline stage before training, paying in preprocessing cost, storage, and sensitivity to tracker quality. It leads on FineGym99 at 1-shot; MATA leads at 3- and 5-shot and Diving48 by +12.92 points, with 79% fewer parameters and a single pass over raw frames. Gains extend beyond sports motion. Table 3 shows consistent improvements on Toyota SmartHome, HAA500, and MPII Cooking2, where the cues are object manipulation and interaction order rather than body dynamics. Coarse-grained comparisons are in Sec. D of the supplementary.

MethodVenueTunable Params (M)Diving48FineGym99FineGym288
1-shot3-shot5-shot1-shot3-shot5-shot1-shot3-shot5-shot
TSN+Cosine [38]ECCV’1625M29.7235.6938.8136.6341.3542.7636.6842.8744.82
ProtoNet [35]NeurIPS’1725M59.8074.7477.6962.9974.2577.9958.9870.5674.87
OTAM [3]CVPR’2023.5M41.6447.9649.7848.9352.8855.6147.5853.6755.62
PAL [55]BMVC’2125M50.4057.5761.1758.8565.1368.6458.3263.6265.77
TRX [29]CVPR’2147.1M62.5377.8182.0166.5580.1583.8664.0376.7381.05
STRM [37]CVPR’2230M62.2977.7981.0368.0980.9384.3364.8176.9880.64
HyRSM [41]CVPR’2265.6M61.0476.0481.7467.0978.1582.9262.8673.1278.81
MoLo [42]CVPR’2389.6M---73.380.284.8---
TATs [15]ECCV’2410.8M---81.886.087.9---
M3Net [36]MM’2330M72.3582.5085.4472.7083.0386.7168.6679.7383.05
DINOv2 Backbone
Trokens [16]ϕICCV’2517.4M67.20 †77.14 †81.30 †84.4088.3089.90---
CLIP Backbone
D2ST [28]†ICCV’257.3M73.2981.6483.8576.8185.5287.5570.3376.2682.08
EMP-Net [44]†*`ECCV’247.5M71.00-69.20------
Task-Adapter [1]*MM’247.2M68.0472.4775.0964.4867.1270.54---
Task-Adapter++ [2]*PR’268.0M46.4560.6564.1760.8265.4368.1263.07-72.76
TEAM [18]†CVPR’2587.0M38.5050.4054.90------
MPL [47]†CVPR’2614.0M39.2944.6148.7------
MATA-3.6M80.12 ± 0.1588.31 ± 0.1490.00 ± 0.1282.21 ± 0.1888.97 ± 0.1691.29 ± 0.1381.28 ± 0.1789.39 ± 0.1590.13 ± 0.19

Table 2. Few-shot fine-grained action recognition on Diving48, FineGym99, and FineGym288 (%) under 5-way 1/3/5-shot settings. This table compares prior few-shot methods that do not use foundation models with MATA. Best results are in bold.† means results reproduced by us. “*” denotes multi-modal methods incorporating additional text input and a text encoder. ϕ denotes additional preprocessing.

MethodToyotaSHHAA500MPII Cooking 2
1-shot3-shot5-shot1-shot3-shot5-shot1-shot3-shot5-shot
Task-Adapter++ [2]†62.3973.4376.3097.0598.1499.1059.8568.7970.00
D2ST [28]†60.2571.0673.1896.5297.4698.7857.2368.0971.16
MATA64.8474.8577.7697.1898.2499.2860.0370.4872.50

Table 3. Few-shot fine-grained action recognition on additional datasets (%) under 5-way 1/3/5-shot settings.

Achieving the accuracy–compute efficiency–unimodality objective

The accuracy–compute efficiency–unimodality objective asks whether a few-shot video model can improve recognition while remaining compact, fast, and single-stream. Therefore, we compare episodic inference efficiency between MATA and D2D^2ST under an identical few-shot protocol on a single NVIDIA H100 80GB HBM3 GPU using FP16. FLOPs are measured with fvcore, and latency is measured with CUDA events over timed blocks of B=8B=8 consecutive meta-forwards. Each episode uses a 5-way 1-shot setting with one query per class and T=8T=8 frames, corresponding to 40 support and 40 query frame-rows per episode.

Table 4 shows that MATA improves both accuracy and runtime efficiency under the same episodic protocol. At comparable nominal FLOPs, MATA is more accurate and cuts per-episode latency from 111.26 to 27.16 ms, a 4.10×4.10\times speedup that raises throughput from 9.07 to 36.81 episodes/s and realized compute from 26.92 to 107.53 TFLOP/s.

Method1-shot Acc. (%) ↑GFLOPs Episode ↓Latency Episode (ms) ↓Throughput (episodes/s) ↑Effective Compute (TFLOP/s) ↑
D2D^{2}ST † [28]73.292990.0111.269.0726.92
Task-Adapter † [2]46.453335.9568.5414.6048.67
EMP-Net † [44]71.02964.24128.167.8023.12
MATA80.122921.327.1636.81107.53
Gain (MATA vs. D2D^{2}ST)↑ +6.83↓ 2.3%↓ 4.10×↑ 4.06×↑ 3.99×

Table 4. MATA delivers both higher accuracy and substantially faster few-shot adaptation. The results reveal a clear advantage in both recognition performance and practical inference efficiency, rather than a trade-off between the two. † denotes results reproduced by us; ↓ lower is better and ↑ higher is better.

FLOPs therefore do not predict few-shot inference cost. D2D^2ST’s deformable attention issues many small, irregular memory accesses, keeping GPU utilization low regardless of arithmetic volume (26.92 TFLOP/s realized). MATA’s temporal operator is a depth-wise convolution over a contiguous frame axis, which maps to dense kernels and leaves the frozen CLIP pathway untouched (107.53 TFLOP/s). The gain comes from what MATA does not add to that pathway, not from doing less arithmetic.

MATA is scalable and applicable to a fully-supervised setting.

MATA is not restricted to the few-shot regime. Under the standard fully-supervised protocol, it reaches 89.7% on Diving48 (Table 5) and 95.1% / 90.0% on FineGym99 and FineGym288 (Table 6), matching or exceeding AIM and DualPath with roughly a third of their tunable parameters and outperforming backbones an order of magnitude larger. MATA also scales with backbone capacity: Table 7 shows consistent gains from ViT-B/16 to ViT-L/14 across all three fine-grained benchmarks, indicating that temporal-only adaptation continues to benefit from stronger spatial priors rather than saturating.

MethodVenueTunable Params (M)FramesTop-1
Non-CLIP-based methods
ORViT TimeSformer [11]CVPR’22160.03288.0
BEVT [39]CVPR’2288.11686.7
VideoSwin-B [23]CVPR’2288.13281.9
SlowFast R101 [8]ICCV’1962.864+1677.6
TQN [50]CVPR’21–all81.8
CLIP-based methods ViT-B/16
BDC-CLIP † [24]ICML’25125.41676.6
PeVL [51]CVPR’2428.01684.5
DiST † [30]ICCV’2319.01676.85
AIM † [48]ICLR’2311.03288.9
DualPath † [27]CVPR’2310.03288.7
ST Adapter † [26]NeurIPS’227.21684.2
MATA (ours)–3.61685.8 ± 0.13
MATA (ours)–3.63289.7 ± 0.15

Table 5. Fully-supervised results on Diving48. Top-1 accuracy under the standard fully-supervised protocol, included to show MATA also applies beyond the few-shot setting.

MethodVenueFramesTunable Params (M)FineGym99FineGym288
Non-CLIP-based methods
TQN [50]CVPR’21all78.093.889.6
TSM [21]ICCV’1916+1648.688.483.1
TRNms [53]ECCV’188+836.687.882.0
SlowOnly [8]ICCV’193232.493.4686.8
I3D [4]CVPR’171628.075.677.3
CLIP-based methods ViT-B/16
PeVL† [51]CVPR’241628.091.285.4
AIM † [48]ICLR’233211.094.989.1
MATA (ours)–323.795.1 ± 0.1390.0 ± 0.10

Table 6. Fully-supervised results on FineGym99 and FineGym288. PeVL [51] results are reported using video modality.

MATAParamsDiving48FineGym99FineGym288
1-shot5-shot1-shot5-shot1-shot5-shot
ViT-B/163.680.1290.0082.2191.2981.2890.13
ViT-L/1412.582.6590.0083.0792.5383.0091.64

Table 7. Backbone scalability of MATA. Top-1 accuracy (%) under 1-shot and 5-shot settings with frozen CLIP backbones. Gains carry over to the larger backbone across all three benchmarks.

Ablation study

We ablate the core components of MATA to isolate the role of temporal phase alignment. Table 8 shows that adding temporal phase alignment to the local motion transition operator consistently improves both 1-shot and 5-shot accuracy on Diving48 and FineGym99. This indicates that local motion transition modeling alone is not sufficient: fine-grained support-query matching also benefits from knowing where a motion state occurs within the action sequence. Additional ablations on adapter depth, input frames, and supervision are provided in Sec. A in the supplementary. temporal stress test. Corruptions are applied in pixel space before CLIP normalization, and each method is evaluated on 2,000 episodes.

MATA remains robust against distribution shifts.

We evaluate 5-way 1-shot recognition under five test-time corruptions at severity levels {1,3,5}\{1, 3, 5\}. Support videos remain clean and only query clips are corrupted, isolating robustness to degraded test inputs. We consider Gaussian noise, Gaussian blur, JPEG compression, brightness, and frame drop; the latter removes 12.5%/37.5%/62.5% of the 8 frames and fills missing positions by hold-last, providing a

We report mean Corruption Accuracy (mCA) and mean Corruption Error (mCE). Following the mCE metric introduced with ImageNet-C [10], we adapt mCE to our video robustness setting. Let Aˉm,c\bar{A}_{m,c} denote the Top-1 accuracy of method mm on corruption cc, averaged over severities {1,3,5}\{1, 3, 5\}. We define

mCE⁡m=20∑c=15100−Aˉm,c100−Aˉref,c,(14)\operatorname{mCE}_{m} = 20 \sum_{c=1}^{5} \frac{100-\bar{A}_{m,c}}{100-\bar{A}_{\mathrm{ref},c}}, \tag*{(14)}

where Task-Adapter++ is the reference method. Thus, mCA measures absolute corrupted accuracy, while mCE measures corruption error relative to a common reference, with lower mCE indicating greater robustness.

Table Table 9 shows that MATA achieves the highest accuracy under every corruption, improving mCA over D2D^{2}ST by 8.0 points (72.8 vs. 64.8). MATA also obtains the lowest mCE, reducing normalized corruption error from 57.5 for D2D^{2}ST to 44.3, a 13.2-point improvement. These results show that MATA’s gains are not limited to clean inputs: its temporal specialization remains effective under both spatial corruptions and temporal degradations.

MethodClean Top-1 ↑Gaussian noiseGaussian blurFrame dropJPEG compr.BrightnessmCA (%) ↑mCE ↓
Task-Adapter++45.832.136.644.140.841.038.9100.0
D²ST73.855.462.966.968.570.164.857.5
MATA80.163.770.774.277.477.972.844.3

Table 9. Test-time robustness on Diving48 (5-way 1-shot); only queries are corrupted. Mean Top-1 (%) over severities {1,3,5}\{1, 3, 5\}; mCE is normalized by Task-Adapter++ (=100, lower is better).

Why MATA’s temporal modeling captures fine-grained details

We provide qualitative and statistical analyses to explain why MATA consistently outperforms D2D^{2}ST. We further include a single-frame blackout sensitivity analysis in Sec. C to measure temporal selectivity under targeted frame removal and nearest-neighbor retrieval across confusing classes in Sec. B in the supplementary.

Qualitative diagnostic on temporal saliency

We analyze temporal saliency to explain why MATA improves fine-grained discrimination through temporally localized adaptation. Fine-grained actions often differ by brief stage transitions and motion ordering, so effective adaptation should be temporally selective rather than uniformly modifying representations across the clip. MATA explicitly biases adaptation toward local temporal cues via a depth-wise 1D temporal convolution and learned temporal positions, aiming to spend limited capacity on the most discriminative frames.

We measure temporal saliency using the per-frame magnitude of the adapter-induced residual update. Let ΔXpatch∈RHW×(BT)×C\Delta X_{\mathrm{patch}} \in\mathbb{R}^{HW \times(BT) \times C} denote the adapter residual update in a transformer block (§ 3). Reshaping to isolate time, we define

s(t)=1HW∑n=1HW∥ΔXn,t∥2,(15)s(t) = \frac{1}{HW} \sum_{n=1}^{HW} \lVert\Delta X_{n,t} \rVert_2 , \tag*{(15)}

where nn indexes spatial tokens and t∈{1,…,T}t \in\{1, \ldots, T\} indexes frames. Intuitively, s(t)s(t) quantifies how strongly the adapter modifies the representation at time tt. We report the mean curve with variability (shaded region) across multiple episodes on randomly sampled FineGym classes, aggregating across examples to reflect a generalized trend.

Selective peaks vs. uniform updates. Figure 2 shows that MATA yields a lower baseline update magnitude with sharp temporal peaks concentrated around discriminative stages, consistent with fine-grained cues being short-lived. In contrast, D²ST [28] exhibits a higher and flatter response, suggesting more uniform perturbations across frames. These saliency patterns support the intended mechanism of MATA: it applies small, targeted modifications and activates strongly only near key transition moments, reducing unnecessary feature drift and thereby aiding few-shot generalization.

Temporal saliency via adapter update magnitude

Figure 2. Temporal saliency via adapter update magnitude: Per-frame update magnitude s(t)s(t) (Eq. (15)) for MATA and D²ST on randomly sampled FineGym clips (mean ± variability). MATA shows sharper temporal peaks, indicating more selective activation on discriminative motion stages. More comparison examples in Sec. E in the supplementary.

Conclusion and limitations

We presented MATA, a motion-aligned temporal adapter for few-shot fine-grained action recognition with a frozen CLIP backbone. By combining temporal phase alignment with local modeling of short motion transitions, MATA achieves state-of-the-art results on fine-grained benchmarks with only ∼3.6M tunable parameters, lower latency, and the best accuracy under every test-time corruption, without full backbone tuning or auxiliary modalities. MATA is specialized by design: it assumes a fixed frame count and does not target long-range or appearance-driven recognition, leaving long-duration benchmarks and streaming recognition for future work.

Acknowledgment: This work has emanated from research supported by the Kempe Foundations (Kempestiftelserna), JCSMK23-0109, Sweden.

Scaling behavior of MATA across capacity, context, and supervision

We analyze how MATA scales with adaptation capacity, temporal context, and supervision, using D2STD^{2}\mathrm{ST} as the strongest adapter-based baseline. Across all three axes, MATA consistently exhibits more favorable scaling behavior, indicating that its temporal adaptation mechanism makes more effective use of available resources.

Adaptation depth. Table 10a shows that MATA outperforms D2STD^{2}\mathrm{ST} even when adapters are placed only in the last few transformer blocks, and this margin persists as more blocks are adapted, indicating that MATA does not require deep or widespread architectural modification to model fine-grained temporal structure.

Temporal context. From Table 10b, both methods improve with more input frames, but MATA consistently gains more as clip length grows, suggesting it leverages extended temporal context more effectively to discriminate fine-grained actions.

MethodBlocksParamsDiving48 1-shotDiving48 5-shotFineGym99 1-shotFineGym99 5-shot
D2STD^{2}\mathrm{ST}63.670.0179.8473.8484.80
MATA61.878.4387.9579.8089.60
D2STD^{2}\mathrm{ST}84.871.0281.4375.1286.10
MATA82.479.8088.0280.7490.05
D2STD^{2}\mathrm{ST}106.172.7082.6476.3287.13
MATA103.080.0588.8381.6690.97

Table 10a. Comparison of MATA and D2STD^{2}\mathrm{ST} when inserting adapters into the last 6, 8, and 10 CLIP blocks. We report 1-shot and 5-shot accuracy on Diving48 and FineGym99.

MethodFramesDiving48 1-shotDiving48 5-shotFineGym99 1-shotFineGym99 5-shot
D2STD^{2}\mathrm{ST}873.2983.8576.8187.55
MATA880.1290.0081.3790.44
D2STD^{2}\mathrm{ST}1279.8887.8480.0089.98
MATA1284.6490.6884.1492.33
D2STD^{2}\mathrm{ST}1682.7589.8280.4189.38
MATA1685.9092.0388.5593.76

Table 10b. Effect of input frame count on MATA and D2STD^{2}\mathrm{ST}. We report 1-shot and 5-shot accuracy using 8, 12, and 16 frames on Diving48 and FineGym99.

Supervision. Table 11 shows that MATA maintains a stable advantage over D2STD^{2}\mathrm{ST} from 1-shot to 10-shot settings on both datasets, indicating smooth scaling with increased supervision without signs of overfitting.

MethodDiving48FineGym99
1-shot3-shot5-shot7-shot9-shot10-shot1-shot3-shot5-shot7-shot9-shot10-shot
D2^{2}ST73.2980.6483.8584.5984.5484.9676.8185.5287.5589.3090.0390.70
MATA79.3786.4688.4888.7689.3189.5081.3788.1290.4491.0592.3592.40

Table 11. Evaluation of MATA and D2^{2}ST across different shot settings. MATA consistently outperforms D2^{2}ST in all shot regimes.

Nearest-neighbor retrieval across confusing fine-grained classes

We probe how MATA reshapes representation structure using nearest-neighbor (NN) retrieval on confusable FineGym99 classes. We construct a small set of 80 videos from five confusable class pairs and, for each query, retrieve the top-KK nearest gallery videos under cosine similarity using three embeddings: (i) frozen CLIP (no adapter), (ii) D2STD^{2}\mathrm{ST}, and (iii) MATA. Each video is represented by a single clip embedding z(v)z(v) computed with the same 8-frame sampling and preprocessing as episodic evaluation; we ℓ2\ell_{2}-normalize z(v)z(v) so cosine similarity reduces to a dot product. Retrieval results are visualized as grids (Query | NN-1..NN-KK) and summarized in Figure 3.

Nearest-neighbor retrieval comparison for CLIP, D2ST, and MATA on FineGym99

Figure 3. Nearest-neighbor retrieval on confusable FineGym99 classes. For the same query and gallery, MATA yields cleaner top-KK neighborhoods than frozen CLIP and D2^{2}ST, consistent with improved fine-grained discrimination.

To quantify neighborhood quality, we report Top-5 label purity (Purity@5), defined as the number of same-label neighbors among the top-5 retrieved videos. Table 12 shows that MATA produces substantially cleaner neighborhoods than frozen CLIP and D2STD^{2}\mathrm{ST}: averaged over 20 queries (5 pairs ×\times 4 queries per pair), Purity@5 increases from 0.85/5 (frozen CLIP) to 4.00/5 for MATA, with D2STD^{2}\mathrm{ST} reaching 2.80/5, confirming MATA’s ability to discriminate among fine-grained confusing classes. This qualitative and statistical analysis supports our claim that MATA reshapes the local neighborhood structure to better align with discriminative action dynamics rather than appearance alone.

MethodPurity@5 ↑Purity rate ↑Δ\DeltaPurity@5 vs. CLIP ↑
Frozen CLIP0.850.17 (0.08–0.27)0.00
D2^{2}ST2.800.56 (0.39–0.73)+1.95 (1.20–2.70)
MATA4.000.80 (0.64–0.93)+3.15 (2.40–3.90)

Table 12. Nearest-neighbor retrieval quality on FineGym99. Purity@5 counts how many of the top-5 retrieved neighbors share the query label. Parentheses denote 95% bootstrap confidence intervals computed over the same 20 queries.

Single-Frame Blackout Sensitivity Analysis

To understand temporal selectivity, we perform single-frame blackout sensitivity analysis on 8-frame clips from FineGym99. For each frame t∈{1,…,8}t \in\{1,\ldots,8\}, we compute the margin drop Δ(t)=m(v)−m(v(t))\Delta(t) = m(v) - m(v^{(t)}), where m(v)=sy∗(v)−max⁡c≠y∗sc(v)m(v) = s_{y_*}(v) - \max_{c\ne y_*} s_c(v) is the decision margin for the original clip vv, m(v(t))m(v^{(t)}) is the margin when frame tt is replaced with a normalized black frame, sy∗(v)s_{y_*}(v) is the cosine similarity score for the ground-truth class y∗y_*, and sc(v)s_c(v) is the score for class cc. A large magnitude ∣Δ(t)∣|\Delta(t)| indicates strong temporal impact, while the sign indicates whether frame tt is helpful (positive Δ(t)\Delta(t)) or harmful (negative Δ(t)\Delta(t)) to the prediction. We evaluate both MATA and D2STD^{2}\mathrm{ST} on 20 randomly sampled 5-way 1-shot episodes from the test split, computing Δ(t)\Delta(t) for each frame and aggregating results across episodes.

Across 20 episodes, MATA demonstrates higher peak sensitivity (mean ∣Δ(t)∣max⁡=0.091±0.054|\Delta(t)|_{\max} = 0.091 \pm0.054) and higher temporal selectivity (standard deviation of Δ(t)\Delta(t): 0.036±0.0230.036 \pm0.023) compared to D2STD^{2}\mathrm{ST} (0.064±0.0380.064 \pm0.038 and 0.025±0.0140.025 \pm0.014, respectively). The aggregate analysis (Figure 4) shows MATA consistently identifies key frames (frames 3–4) with sharp peaks, while D2STD^{2}\mathrm{ST} exhibits a flatter temporal response across all frames. A representative example (Figure 4, Episode 1) demonstrates MATA’s sharp peak at frame 4 (∣Δ(4)∣=0.112|\Delta(4)| = 0.112), indicating strong temporal focus, while D2^{2}ST shows a flatter response with lower peak sensitivity (∣Δ(6)∣=0.070|\Delta(6)| = 0.070). These results demonstrate that MATA’s depthwise temporal convolution enables improved temporal selectivity by (1) identifying and focusing on discriminative frames with sharp sensitivity peaks, (2) efficiently distinguishing between critical and non-critical temporal phases, and (3) enabling cheap identification of key frames through margin drop analysis, thereby supporting more efficient temporal modeling compared to attention-based methods that process all frames uniformly.

Single-frame blackout sensitivity analysis showing an example episode and aggregate sensitivity across 20 episodes

Figure 4. Single-frame blackout sensitivity analysis. (a) Example episode showing MATA’s sharp peak at frame 4 (∣Δ(4)∣=0.112\lvert\Delta(4)\rvert= 0.112) vs D²ST’s flatter response (peak ∣Δ(6)∣=0.070\lvert\Delta(6)\rvert= 0.070). (b) Aggregate plot (mean ∣Δ(t)∣±std\lvert\Delta(t)\rvert\pm\mathrm{std} over 20 episodes). MATA shows higher peak sensitivity (0.091 vs 0.064) and higher temporal selectivity (0.036 vs 0.025), demonstrating MATA’s focused temporal modeling.

Comparison with other methods on Coarse-grained Datasets

Table 13 reports results on coarse-grained benchmarks, where MATA is competitive but does not lead. This is consistent with its design rather than a shortcoming of it. On HMDB51, UCF101, and Kinetics, classes are largely separable from appearance and scene context, and a single well-chosen frame is often sufficient; the discriminative signal these datasets reward is precisely the spatial and semantic content that MATA leaves untouched, since it adapts only the temporal axis. Methods that adapt spatially, or that recruit the CLIP text encoder as an additional supervisory signal such as CLIP-FSAR [43], MA-FSAR [45], and Task-Adapter++ [2], can exploit that signal directly and hold an advantage here. MATA nevertheless matches or exceeds several of these methods while operating from the video modality alone, without text prompts, pose, depth, or offline preprocessing, and with a fraction of their tunable parameters. Its advantage appears where the discriminative cue is short-lived motion: on the fine-grained benchmarks of Table 2 and Table 3, where classes share actors, scenes, and objects and differ only in the timing and ordering of motion, temporal-only adaptation is the more effective use of a limited parameter budget. We therefore position MATA as a fine-grained specialist rather than a general-purpose few-shot recognizer.

MethodReferencePretrainHMDB51 1-shotHMDB51 5-shotUCF101 1-shotUCF101 5-shotKinetics 1-shotKinetics 5-shotSSv2-Small 1-shotSSv2-Small 5-shotSSv2-Full 1-shotSSv2-Full 5-shot
CMN [54]ECCV’18ResNet50––––60.578.934.443.836.248.9
OTAM [3]CVPR’20ResNet50––––73.085.8––42.852.3
AmeFuNet [9]MM’20ResNet5060.275.585.195.574.186.8––––
TRX [29]CVPR’21ResNet50–75.6–96.163.685.9–59.1–64.6
SPRN [40]MM’21ResNet5061.676.286.595.875.287.1––––
HyRSM [41]CVPR’22ResNet5060.376.083.994.773.786.140.656.154.369.0
TA2N+Sampler [22]MM’22ResNet5059.973.583.596.073.686.2––47.161.6
MoLo [42]CVPR’23ResNet5060.877.486.095.574.085.641.956.255.069.6
MGCSM [49]MM’23ResNet5061.379.386.597.174.288.2––––
SA-CT [52]MM’23ResNet5060.478.385.496.471.987.1––48.969.1
CLIP-FSAR [43]IJCV’24ResNet5069.480.792.497.090.192.052.155.858.762.8
SOAP [14]MM’24ResNet5076.588.494.199.381.193.8––61.979.8
HMP-Net [5]MM’25ResNet5077.688.794.999.192.094.154.660.961.276.5
MPL [47]CVPR’26ResNet5063.378.190.197.577.489.5--43.556.6
Trokens [16]ICCV’25DinoV269.882.394.097.982.991.253.468.961.576.7
TEAM [18]CVPR’25ResNet5062.878.487.296.275.188.2----
CLIP-Freeze [31]ICML’21ViT-B51.471.084.694.568.285.329.542.526.836.3
CLIP-FSAR [43]*IJCV’24ViT-B77.187.797.099.194.895.454.6---
MA-FSAR [45]*PR’26ViT-B83.487.997.299.295.796.059.164.563.372.3
EMP-Net [44]*ECCV’24ViT-B76.885.894.398.289.193.557.165.763.173.0
Task-Adapter [1]*MM’24ViT-B83.688.898.099.495.096.860.270.271.374.2
Task-Adapter++ [2]*PR’26ViT-B86.592.098.999.796.297.563.673.272.875.5
SOAP [14]MM’24ViT-B79.389.896.899.589.995.5––66.781.2
HMP-Net [5]MM’25ViT-B83.889.998.799.595.797.560.570.468.479.5
AIM [48]ICLR’23ViT-B74.286.995.498.588.495.352.867.563.779.2
DUALPATH [27]CVPR’23ViT-B74.987.595.798.788.895.453.568.164.579.8
ST-Adapter [26]NeurIPS’22ViT-B74.187.395.998.988.595.153.168.064.279.5
D2ST [28]ICCV’25ViT-B77.188.296.499.189.395.555.069.366.781.9
TSAM [19]AAAI’25ViT-B84.588.998.399.396.297.160.566.765.874.6
MPL [47]CVPR’26ViT-B68.084.993.098.583.793.8--40.155.0
TEAM [18]CVPR’25ViT-B70.985.594.598.883.392.947.263.1--
MATA–ViT-B74.1086.5494.8298.50--55.273.258.774.8

Table 13. Comparison with current SOTA few-shot action recognition methods on HMDB51, UCF101, Kinetics, SSv2-Small, and SSv2-Full datasets. “*” denotes multi-modal methods incorporating additional text input and a text encoder.

More examples on Temporal Saliency of Adapter Updates

This section shows more examples for temporal saliency to further strengthen MATA’s temporal capacity.

Temporal saliency examples

Figure 5. Temporal saliency examples.

References

  1. [1]Congqi Cao, Yueran Zhang, Yating Yu, Qinyi Lv, Lingtong Min, and Yanning Zhang. Task-adapter: task-specific adaptation of image models for few-shot action recognition. In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 9038–9047, 2024. 3, 6, 15arxiv.org/abs/2408.00249
  2. [2]Congqi Cao, Peiheng Han, Yueran Zhang, Yating Yu, Qinyi Lv, Lingtong Min, and Yanning Zhang. Task-adapter++: Task-specific adaptation with order-aware alignment for few-shot action recognition. Pattern Recognition, page 113237, 2026. 3, 6, 7, 14, 15
  3. [3]Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10618–10627, 2020. 1, 3, 6, 15arxiv.org/abs/1906.11415
  4. [4]Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21–26, 2017. 7arxiv.org/abs/1705.07750
  5. [5]Xiaoyu Chen, Yigang Cen, Wanru Xu, Yue Zhang, Yi Jin, Yidong Li, and Linna Zhang. Hierarchical meta-prototypes network for few-shot action recognition. In Proceedings of the 33rd ACM International Conference on Multimedia, page 2643–2652, New York, NY, USA, 2025. Association for Computing Machinery. 3, 15DOI
  6. [6]Jihoon Chung, Cheng-hsin Wu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. HAA500: human-centric atomic action dataset with curated videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13465–13474, 2021. 5arxiv.org/abs/2009.05224
  7. [7]Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Giampiero Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 833–842, 2019. 5DOI
  8. [8]Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6202–6211, 2019. 7arxiv.org/abs/1812.03982
  9. [9]Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, and Yu-Gang Jiang. Depth guided adaptive meta-fusion network for few-shot video recognition. In Proceedings of the 28th ACM international conference on multimedia, pages 1142–1151, 2020. 15arxiv.org/abs/2010.09982
  10. [10]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019. 6
  11. [11]Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3148–3159, 2022. 7arxiv.org/abs/2110.06915
  12. [12]James Hong, Matthew Fisher, Michaël Gharbi, and Kayvon Fatahalian. Video pose distillation for few-shot, fine-grained sports action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9254–9263, 2021. 3arxiv.org/abs/2109.01305
  13. [13]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attarian, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning (ICML), pages 2790–2799, 2019. 3arxiv.org/abs/1902.00751
  14. [14]Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, and Lei Zhang. Soap: Enhancing spatio-temporal relation and motion information capturing for few-shot action recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4572–4580, 2024. 15DOI
  15. [15]Pulkit Kumar, Namitha Padmanabhan, Luke Luo, Sai Saketh Rambhatla, and Abhinav Shrivastava. Trajectory-aligned space-time tokens for few-shot action recognition. In European Conference on Computer Vision (ECCV), pages 474–493. Springer, 2024. 3, 6arxiv.org/abs/2407.18249
  16. [16]Pulkit Kumar, Shuaiyi Huang, Matthew Walmer, Sai Saketh Rambhatla, and Abhinav Shrivastava. Trokens: Semantic-aware relational trajectory tokens for few-shot action recognition. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13544–13556. IEEE, 2025. 1, 3, 6, 15arxiv.org/abs/2508.03695
  17. [17]Dongho Lee, Jongseo Lee, and Jinwoo Choi. Cast: cross-attention in space and time for video action recognition. Advances in Neural Information Processing Systems, 36: 79399–79425, 2023. 3arxiv.org/abs/2311.18825
  18. [18]SuBeen Lee, WonJun Moon, Hyun Seok Seong, and Jae-Pil Heo. Temporal alignment-free video matching for few-shot action recognition. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5412–5421. IEEE, 2025. 3, 6, 15arxiv.org/abs/2504.05956
  19. [19]Bozheng Li, Mushui Liu, Gaoang Wang, and Yunlong Yu. Frame order matters: A temporal sequence-aware model for few-shot action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18218–18226, 2025. 3, 15arxiv.org/abs/2408.12475
  20. [20]Yingwei Li, Yi Li, and Nuno Vasconcelos. RESOUND: towards action recognition without representation bias. In Proceedings of the European Conference on Computer Vision (ECCV), pages 513–528, 2018. 1, 5DOI
  21. [21]Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7083–7093, 2019. 7arxiv.org/abs/1811.08383
  22. [22]Huabin Liu, Weixian Lv, John See, and Weiyao Lin. Task-adaptive spatial-temporal video sampler for few-shot action recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6230–6240, 2022. 15arxiv.org/abs/2207.09759
  23. [23]Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3202–3211, 2022. 7arxiv.org/abs/2106.13230
  24. [24]Fei Long, Xiaoou Li, Jiaming Lv, Haoyuan Yang, Xianjun Cheng, and Peihua Li. BDC-CLIP: Brownian distance covariance for adapting CLIP to action recognition. In Proceedings of the International Conference on Machine Learning (ICML), 2025. 7
  25. [25]Bolin Ni, Hou wen Peng, Minghao Chen, Songyang Zhang, Gao feng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 1–18, 2022. 3arxiv.org/abs/2208.02816
  26. [26]Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hong-sheng Li. ST-adapter: parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462–26477, 2022. 1, 3, 4, 7, 15
  27. [27]Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image to video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2203–2213, 2023. 3, 4, 7, 15arxiv.org/abs/2303.09857
  28. [28]Wenjie Pei, Qizhong Tan, Guangming Lu, Jiandong Tian, and Jun Yu. D2ST-adapter: disentangled and deformable spatio-temporal adapter for few-shot action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11317–11326, 2025. 1, 2, 3, 4, 6, 7, 8, 15arxiv.org/abs/2312.01431
  29. [29]Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmethi, and Dima Damen. Temporal-relational crosstransformers for few-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 475–484, 2021. 1, 3, 6, 15arxiv.org/abs/2101.06184
  30. [30]Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Disentangling spatial and temporal learning for efficient image-to-video transfer learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13934–13944, 2023. 1, 2arxiv.org/abs/2309.07911
  31. [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748–8763, 2021. 2, 15arxiv.org/abs/2103.00020
  32. [32]Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned CLIP models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554, 2023. 3arxiv.org/abs/2212.03640
  33. [33]Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing fine-grained and composite activities using hand-centric features and script data. International Journal of Computer Vision, pages 1–28, 2015. 5DOI
  34. [34]Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. FineGym: a hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2616–2625, 2020. 1, 5arxiv.org/abs/2004.06704
  35. [35]Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017. 3, 6arxiv.org/abs/1703.05175
  36. [36]Hao Tang, Jun Liu, Shuanglin Yan, Rui Yan, Zechao Li, and Jinhui Tang. M3Net: Multi-view encoding, matching, and fusion for few-shot fine-grained action recognition. In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 1719–1728, 2023. 5, 6arxiv.org/abs/2308.03063
  37. [37]Anirudh Thatipelli, Sanath Narayan, Salman Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Bernard Ghanem. Spatio-temporal relation modeling for few-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19958–19967, 2022. 1, 3, 6arxiv.org/abs/2112.05132
  38. [38]Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 20–36, 2016. 6arxiv.org/abs/1608.00859
  39. [39]Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14733–14743, 2022. 7arxiv.org/abs/2112.01529
  40. [40]Xiao Wang, Weirong Ye, Zhongang Qi, Xun Zhao, Guangge Wang, Ying Shan, and Hanzi Wang. Semantic-guided relation propagation network for few-shot action recognition. In Proceedings of the 29th ACM international conference on multimedia, pages 816–825, 2021. 15DOI
  41. [41]Xiang Wang, Shiwei Zhang, Zhiwu Qing, Mingqian Tang, Zhengrong Zuo, Changxin Gao, Rong Jin, and Nong Sang. Hybrid relation guided set matching for few-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19948–19957, 2022. 1, 3, 6, 15arxiv.org/abs/2204.13423
  42. [42]Xiang Wang, Shiwei Zhang, Zhiwu Qing, Changxin Gao, Yingya Zhang, Deli Zhao, and Nong Sang. Molo: Motion-augmented long-short contrastive learning for few-shot action recognition. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18011–18021. IEEE, 2023. 6, 15arxiv.org/abs/2304.00946
  43. [43]Xiang Wang, Shiwei Zhang, Jun Cen, Changxin Gao, Yingya Zhang, Deli Zhao, and Nong Sang. CLIP-guided prototype modulating for few-shot action recognition. International Journal of Computer Vision, 132(6):1899–1912, 2024. 1, 2, 3, 14, 15arxiv.org/abs/2303.02982
  44. [44]Cong Wu, Xiao-Jun Wu, Linze Li, Tianyang Xu, Zhenhua Feng, and Josef Kittler. Efficient few-shot action recognition via multi-level post-reasoning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 38–56, 2024. 3, 6, 7, 15DOI
  45. [45]Jiazheng Xing, Jian Zhao, Chao Xu, Mengmeng Wang, Guang Dai, Yong Liu, Jingdong Wang, and Xuelong Li. Ma-fsar: Multimodal adaptation of CLIP for few-shot action recognition. Pattern Recognition, 169:111902, 2026. 1, 3, 14, 15
  46. [46]Mengmeng Xu, Han Zhang, Jian Xu, et al. ActionCLIP: a new paradigm for video action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2
  47. [47]Feng Yang, Jie Zhao, Fulin Luo, Anyong Qin, Tiecheng Song, Yue Zhao, Chenqiang Gao, and Junwei Han. Mpl: Match-guided prototype learning for few-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 34439–34448, 2026. 3, 6, 15
  48. [48]Taojian Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: adapting image models for effi- cient video action recognition. In International Conference on Learning Representations (ICLR), 2023. 1, 3, 4, 7, 15
  49. [49]Tianwei Yu, Peng Chen, Yuanjie Dang, Ruohong Huan, and Ronghua Liang. Multi-speed global contextual subspace matching for few-shot action recognition. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2344–2352, 2023. 15
  50. [50]Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine-grained video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4486–4496, 2021. 7arxiv.org/abs/2104.09496
  51. [51]Haosong Zhang, Mei Chee Leong, Liyuan Li, and Weisi Lin. PEVL: pose-enhanced vision-language model for fine-grained human action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18857–18867, 2024. 7DOI
  52. [52]Yilun Zhang, Yuqian Fu, Xingjun Ma, Lizhe Qi, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. On the importance of spatial relations for few-shot action recognition. In Proceedings of the 31st ACM international conference on multimedia, pages 2243–2251, 2023. 15arxiv.org/abs/2308.07119
  53. [53]Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 803–818, 2018. 7arxiv.org/abs/1711.08496
  54. [54]Linchao Zhu and Yi Yang. Compound memory networks for few-shot video classification. In European Conference on Computer Vision (ECCV), pages 782–797. Springer, 2018. 15DOI
  55. [55]Xiatian Zhu, Antoine Toisoul, Juan-Manuel Perez-Rua, Li Zhang, Brais Martinez, and Tao Xiang. Few-shot action recognition with prototype-centered attentive learning. arXiv preprint arXiv:2101.08085, 2021. 3, 6arxiv.org/abs/2101.08085

Paper details

vlnc:W_nwr33xmu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents