WorksComputer Science
CodecMotion-OAD: Efficient Online Action Detection with Compressed-Domain Motion
Abstract
Online action detection (OAD) identifies ongoing actions in streaming videos. Many existing OAD systems combine RGB appearance with optical flow, but dense-flow extraction remains a major computational bottleneck. Codec motion vectors (MVs) are available during decoding, avoiding a separate optical-flow computation. On THUMOS'14, optical flow exceeds MVs by 11.2 mAP points with motion-only inputs, whereas the gap narrows to 3.7 points after RGB fusion, with the largest reduction in action-category discrimination. We propose CodecMotion-OAD, which replaces optical flow with decoder-provided MVs while retaining existing temporal detector architectures. Its two-stage training first aligns MV features with a pretrained flow-feature interface without action labels and then trains RGB+MV detectors using ground-truth labels and soft predictions from a frozen RGB+Flow teacher. To the best of our knowledge, this is the first application of codec MVs to frame-level OAD. On THUMOS'14 with MiniROAD, CodecMotion-OAD achieves 68.3 mAP versus 70.9 mAP for matched-distillation RGB+Flow, while reducing decoder-to-prediction processing time by 11.48-fold on CPU and 2.04-fold on GPU.
Introduction
Online action detection (OAD) identifies ongoing actions in untrimmed video streams using observations available at each prediction step [1]. Many existing OAD systems combine RGB appearance with dense optical flow. Although recent research has improved the efficiency of temporal modeling [2, 3, 15, 4], dense optical-flow extraction remains a major computational bottleneck [3, 4]. Our measurements confirm this bottleneck: in the MiniROAD CPU pipeline, optical-flow extraction alone takes approximately 1008 ms per feature step, compared with 1098.1 ms for the complete decoder-to-prediction path.
Compressed videos contain motion vectors (MVs) generated during inter-picture prediction. These vectors are available during decoding, avoiding a separate optical-flow computation. Prior studies have exploited compressed-domain motion for action recognition, including approaches with optical-flow supervision [6, 7, 8]. Its use for frame-level OAD on untrimmed video streams remains less explored, where predictions are made sequentially from available observations. Codec MVs also differ from dense flow in spatial support and prediction statistics. We therefore examine the incremental discriminative value of optical flow over codec MVs when RGB appearance is already available.
We quantify this additional benefit using matched motion windows and without soft-label distillation. On THUMOS’14, optical flow exceeds codec motion by 11.2 mAP points when motion is used alone, but the gap narrows to 3.7 points after RGB fusion. Further analyses reveal a particularly large reduction in the action-category discrimination gap, while the action-versus-background gap decreases less. These findings suggest that motion-only performance can overstate the benefit of expensive flow in an RGB-based detector; § 3 provides the corresponding evidence.
These findings motivate CodecMotion-OAD, an RGB+MV OAD system that substitutes decoder-provided codec motion for dense optical flow while retaining existing temporal detectors. We develop Interface-Preserving Motion Transfer (IPMT) as its training procedure. IPMT separates label-free motion-feature interface alignment from detector-level soft-label supervision using a frozen RGB+Flow teacher. The resulting RGB+MV detectors preserve their original temporal architectures and require neither optical-flow estimation nor the teacher at inference. We evaluate CodecMotion-OAD with MiniROAD, CMeRT, LSTR, and TeSTra. Figure 1 summarizes the accuracy–processing trade-off for MiniROAD under matched soft-label distillation. Replacing TV-L1 optical flow with codec motion reduces CPU processing from 1098.1 to 95.6 ms/step (11.48×) and GPU processing from 77.9 to 38.2 ms/step (2.04×), with 68.3 versus 70.9 mAP. The measurements cover decoding, feature extraction, and detection; § 5.3 presents the full efficiency comparison.
Our contributions are summarized as follows:
Task-conditional analysis of motion utility. We characterize the incremental contribution of dense optical flow beyond RGB appearance in frame-level OAD through matched motion-only and RGB-fused comparisons, frame-wise probes, action-presence/category diagnostics, Flow-specific residual analysis, and controlled spatial coarsening.
Codec-motion OAD with interface-preserving training. To the best of our knowledge, CodecMotion-OAD is the first application of codec MVs to frame-level OAD. Its IPMT procedure separates label-free feature-interface adaptation from soft-label supervision of the temporal detector, supporting four established OAD architectures without changing their temporal models.
Measured efficiency with controlled accuracy comparisons. On MiniROAD, codec motion yields 11.48× CPU and 2.04× GPU decoder-to-prediction processing speedups, with a 2.6-point mAP difference relative to RGB+Flow under the same distillation supervision. We further evaluate performance across THUMOS’14 and Charades.
Related Work
Efficient Online Action Detection
The computational cost of OAD arises from both temporal inference and the acquisition of input features. OadTR, LSTR, and CMeRT develop temporal context and memory modeling [13, 2, 4], while TeSTra reuses attention computations to reduce the cost of streaming updates [3]. Colar, MiniROAD, CWC-Trans, and Backtrace Mamba explore complementary temporal modeling architectures [14, 15, 16, 17]. These studies focus on temporal processing, leaving the acquisition cost of motion features as a separate system-level concern.
A separate source of cost lies upstream of temporal detection: producing RGB and motion features. Some work accelerates flow extraction or avoids explicit flow through RGB-based representations, including GateHUB and end-to-end feature sharing in E2E-LOAD [18, 19]. Our study targets a different choice: retaining the standard RGB–motion interface while replacing the motion source itself. This allows motion acquisition to be examined independently of architectural changes to the temporal detector.
Motion Representations from Compressed Video
Compressed-domain video understanding has long exploited motion vectors and coding residuals to avoid expensive motion estimation [5]. EMV-CNN transfers knowledge from optical flow to MV-based recognition [6]; CoViAR combines I-frame appearance, MVs, and residuals [7]; and DMC-Net learns more discriminative motion cues [8]. EdgeOAR explores online recognition of trimmed actions on edge devices [9]. These studies establish the value of codec-derived motion, but their recognition-oriented objectives differ from frame-level OAD.
EMV-CNN also reports a reduced Flow–MV performance difference when appearance is included [6]. We investigate this observation in frame-level OAD, distinguishing action presence from action-category discrimination and examining the effect of spatially coarsening optical flow. Optical-flow supervision has also been used to improve compressed-domain motion representations [6, 8]. We deliberately use decoder-provided MVs alone in the deployed motion pathway: incorporating coding residuals would introduce additional extraction and feature-processing overhead.
Diagnosing Motion Utility in RGB-Based OAD
The additional value of dense optical flow in RGB-based OAD is determined by its contribution beyond RGB appearance. We characterize this contribution through controlled comparisons with codec MVs on THUMOS’14. All inputs follow a common 4-fps feature grid and matched five-field motion windows. To compare the motion sources within the same Flow-ResNet50 feature interface, codec-MV and coarsened-Flow encoders are adapted to frozen TV-L1 feature targets using training videos; the alignment procedure is detailed in § 4.2. No soft-label distillation is used in the analyses in this section.
Figure 2 compares the motion representations examined below. Spatial block averaging removes fine-scale variations from TV-L1 Flow, while codec MVs exhibit coarse, block-wise and often sparse motion patterns arising from inter-picture prediction. The following experiments examine how these differences affect detection with and without RGB appearance.
From Motion-Only to RGB-Fused Detection
With motion alone, flow reaches 59.6 mAP and codec motion 48.4 mAP, leaving an 11.2-point gap. When RGB is added, the cross-entropy-controlled RGB+Flow and RGB+MV models reach 68.1 and 64.4 mAP, respectively: the gap decreases to 3.7 points. The 68.1-mAP Flow result is the matched cross-entropy control for these analyses, not the 67.5-mAP standard Flow configuration reported in the system comparison. The contraction thus occurs before soft-label distillation and is not an artifact of Soft training.
The same trend holds without temporal modeling. Frame-wise linear probes on frozen features yield 31.9 mAP for Flow and 23.1 for MV (8.8 points apart); after concatenating RGB, they obtain 46.0 and 43.5 (2.5 points apart). The reduced gap is therefore observable at the feature level, independent of the recurrent detector. Probes are fitted on training videos and evaluated on held-out test videos.

Figure 3. Flow–MV performance-gap contraction after RGB fusion on THUMOS’14, without soft-label distillation. (a) Overall mAP gaps for MiniROAD and frame-wise linear probes. (b) MiniROAD gaps in action identity and action presence. MiniROAD identity macro-AP evaluates category discrimination on ground-truth action frames; presence AP evaluates action-versus-background discrimination. All gaps are expressed in percentage points.
Action Presence and Category Discrimination
We distinguish action presence from action-category discrimination. Presence treats all action classes as one positive class and scores each frame by . Identity evaluates category predictions only on ground-truth action frames, after renormalization over the 20 classes. Table 1 summarizes the Flow-minus-MV differences for the linear probe and MiniROAD.
| Metric | Linear probe Motion | Linear probe +RGB | MiniROAD Motion | MiniROAD +RGB |
| mAP | +8.8 | +2.5 | +11.2 | +3.7 |
| Presence AP | +5.3 | +3.2 | +4.1 | +3.1 |
| Presence AUROC | +3.1 | +1.7 | +3.1 | +1.7 |
| Identity macro-AP | +10.4 | +1.0 | +10.6 | +0.9 |
| Identity top-1 | +10.6 | +1.6 | +8.4 | +1.4 |
Table 1. Flow–MV performance gaps on THUMOS’14. Motion: Flow minus MV; +RGB: RGB+Flow minus RGB+MV. Presence measures action versus background; identity is evaluated on ground-truth action frames. Gaps are computed before rounding, and the MiniROAD RGB+Flow control uses the cross-entropy setting of Table 2.
For MiniROAD, the Flow–MV identity macro-AP gap decreases from 10.6 to 0.9 points after RGB fusion, whereas the presence AP gap changes from 4.1 to 3.1 points. Frame-wise linear probes show the same contrast: the identity gap contracts from 10.4 to 1.0 points, compared with 5.3 to 3.2 for presence (Table 1). As shown in Figure 3, RGB appearance reduces Flow’s additional value for action-category discrimination much more strongly than for action-versus-background discrimination.
A regression diagnostic helps interpret, but does not by itself explain, this effect. About 59% of Flow-feature variance is not linearly predictable from MV features; RGB explains little of that residual (). Yet adding the Flow-specific residual to MV features improves a motion-only linear probe by 8.4 mAP points, compared with only 1.9 points after RGB is included. Thus substantial representational differences can coexist with a much smaller incremental gain in the RGB-conditioned prediction task (Appendix A.4).
Coarsening the Flow Representation
To test whether reduced spatial detail produces a similar performance pattern, we block-average native Flow fields using block sizes of , , and , tile the block averages back to the original image size, and re-adapt each source encoder using the same feature objective. This intervention changes Flow’s spatial granularity while controlling for the encoder’s input distribution; it does not make the resulting Flow identical to codec motion (Appendix A.6).
At coarsening, motion-only MiniROAD falls by
8.1 mAP points relative to original Flow, while RGB+motion falls by only 3.8; at 32×32, the corresponding losses are 23.2 and 6.6 points. The 16×16 source also approaches codec MVs in motion-only presence and identity scores (76.7/75.4 versus 76.1/74.6), providing complementary evidence that spatial granularity contributes to the observed task profile. Together, the detector, probe, and coarsening results motivate evaluating a motion source by its incremental value with RGB, rather than by motion-only scores.
These analyses show that the incremental discriminative advantage of dense flow decreases substantially after RGB fusion. The qualitative comparisons and controlled coarsening results also indicate that codec MVs and flow retain some common coarse motion structure, although they are not interchangeable representations. The reduced RGB-fused performance gap and this partial representational commonality jointly motivate reusing and adapting a pretrained Flow encoder in § 4. The resulting system is evaluated across detectors and deployment settings in § 5.
Method
The diagnosis in § 3 supports codec motion as a practical input for RGB-based OAD. Figure 2 also illustrates motion patterns shared in part by codec MVs and dense flow, alongside their differences in spatial support and sparsity. We therefore reuse a pretrained Flow-ResNet50 as the starting point for codec-motion encoding. This is a deliberate interface-preserving design: we adapt the motion source while retaining the 2048-dimensional feature interface and the established temporal detector architectures. The shared motion structure motivates weight reuse, while the distinct codec-motion statistics make feature adaptation necessary.

Figure 2. Qualitative comparison of motion representations on THUMOS’14. The columns show RGB center frames, TV-L1 optical flow, spatially coarsened Flow ( block averaging, with block means tiled back to the original resolution), and codec motion vectors. Rows (a)–(d) show ThrowDiscus, HammerThrow, and two distinct Shotput examples. The three motion visualizations within each row share the same direction-to-hue mapping and magnitude-to-saturation scale, with a common magnitude cap for that row. Block averaging removes fine spatial detail, whereas codec MVs exhibit additional sparsity and prediction-dependent spatial structure.
Interface-Preserving Motion Transfer (IPMT) is the training procedure for CodecMotion-OAD and separates source adaptation from detection supervision. Stage I aligns MV features with a frozen Flow encoder without action labels. With the RGB and adapted MV encoders fixed, Stage II trains the original RGB+MV temporal detectors using ground-truth targets and soft labels from a shared CMeRT RGB+Flow teacher. At inference, only the RGB+MV pathway remains, without optical-flow extraction or the teacher. Figure 4 summarizes this design.

Figure 4. Overview of Interface-Preserving Motion Transfer (IPMT). (a) Stage I adapts a codec-MV encoder initialized from Flow-ResNet50 by matching its 2048-dimensional per-step features to those of a frozen Flow-ResNet50, without action labels. (b) Stage II freezes the RGB and adapted MV encoders and trains the original RGB+MV temporal detectors with ground-truth labels mixed with soft labels from a shared, frozen CMeRT RGB+Flow teacher. (c) Under the H.264 IPPP protocol, decoding supplies RGB frames and codec MVs to the student detector, without optical-flow extraction or the teacher at inference.
Codec-Motion Input and Preserved Interface
We follow the existing Flow-ResNet50 motion interface: five consecutive two-channel motion fields are stacked into a 10-channel input, and the encoder outputs a 2048-dimensional feature for the temporal detector. For a feature step centered at source frame , both optical flow and codec motion use five nominal field indices . RGB uses ; the motion fields are assembled with the same order and endpoint repetition. This matched construction isolates the choice of motion source rather than altering the detector’s temporal input definition.
The feature-step centers follow the existing RGB grid. Given a video with decoded frames and RGB feature steps, the center indices are formed by rounding to the nearest valid frame index. If is the center of step , its terminal motion index is ; the five motion fields are with endpoint clipping. This grid determines benchmark alignment, while the arrival of the terminal motion observations determines when a feature can be computed.
The decoder exposes block-wise displacement vectors from inter-predicted pictures. We convert them to pixel units, rasterize each vector across its destination block, and assign zero to intra-coded or invalid-motion regions. Prediction residuals are not used. The rasterized two-dimensional displacement fields are resized to , spatially sub-sampled, and cached as five two-channel motion fields for each feature step. This construction preserves vector displacement values during geometric resizing; the separate flow-style intensity mapping is applied only before the encoder. For the controlled THUMOS’14 experiments, we decode past-referenced IPPP streams to prevent coded-picture references from requiring later display-order frames. Exact indexing and source-frame dependencies appear in Sections C.2 and C.3 of the Appendix.
Stage I: Feature-Interface Alignment
Codec MVs and optical flow describe partially overlapping motion patterns, providing a basis for reusing flow-pretrained representations. We initialize the MV encoder from the Kinetics-pretrained Flow-ResNet50 and retain its architecture. Because codec MVs differ in spatial support, sparsity, and motion statistics, direct use of the flow-pretrained weights leaves a feature mismatch. We therefore adapt only the MV encoder to match the per-step features of a frozen TV-L1 Flow-ResNet50. Both encoders output the same 2048-dimensional representation expected by the temporal detector.
The MV displacements are mapped to the flow encoder’s numerical input convention as and , followed by the standard flow normalization. Let and denote the codec and frozen-flow features at step . We minimize a cosine-distance and mean-squared-error objective:
Here and . The factor matches the dimension-averaged MSE used in implementation. Stage I requires aligned motion inputs but no target-dataset action labels or detector predictions. The adaptation uses AdamW with learning rate and weight decay , a cosine schedule, batch size 24, and 12 epochs. BatchNorm statistics remain frozen, and 32 aligned feature steps are sampled per training video per epoch. The final-epoch encoder is used for evaluation rather than selecting a checkpoint from test-set similarity. Once adapted on a dataset’s training split, the encoder is frozen and reused across all evaluated temporal detectors on that dataset.
Stage II: Soft-Label Distillation
Stage I establishes compatibility with the pretrained Flow feature interface, whereas Stage II uses teacher predictions to supervise the downstream detection task. We train the original RGB+MV temporal detectors using soft-label distillation from a high-capacity CMeRT RGB+Flow teacher, combining its softened predictions with ground-truth labels. The motion encoder remains fixed during this stage, so the second objective acts on detector training rather than repeating feature adaptation. The contributions of the two stages are examined in the factorial ablation in § 5.4.2.
For THUMOS’14, all students use logits exported from the same IPPP-trained CMeRT RGB+Flow teacher (70.8 mAP). With the ground-truth target , teacher logits , and temperature , we define the soft teacher distribution and mixed training target as [20]
We use and . On single-label THUMOS’14, MiniROAD uses cross-entropy for both Hard (ground-truth targets only) and Soft (mixed targets) configurations; the other detectors use their corresponding soft-target objectives. On multi-label Charades and MultiTHUMOS, teacher logits are transformed by an elementwise sigmoid and optimized using the respective multi-label training objectives. The RGB and adapted codec-motion encoders remain frozen during Stage II, and the teacher is discarded after training. Accordingly, student optimization updates the temporal detector, not the spatial or motion feature extractors. The same adapted codec-motion encoder is reused across MiniROAD, CMeRT, LSTR, and TeSTra within each dataset, which keeps the motion input interface fixed when comparing temporal architectures.
Online Inference and Observation Availability
Inference uses one RGB+MV pathway. The RGB frame aligned to a feature step is buffered until its five-field motion observation is ready; the frozen RGB and MV encoders then provide features to the original temporal detector. At step , that detector only processes the current and earlier available feature steps. The five-field window is inherited from the Flow extractor, rather than newly introduced lookahead by IPMT. For example, the terminal optical-flow field can require the subsequent source picture, so feature availability is defined by observation arrival rather than the alignment-center timestamp.
Past-referenced IPPP coding removes extra dependencies on later display-order pictures during MV decoding. It does not eliminate the finite observation interval required to construct the inherited centered feature. We distinguish this observation availability from the measured decoder-to-prediction processing time, which excludes waiting for source frames (§ 5.1.4).
Experiments
We evaluate CodecMotion-OAD across four established OAD detectors and two datasets, measure decoder-to-prediction processing time, and analyze the contributions of feature adaptation and soft-label distillation.
Experimental Setup
Datasets and Metrics
We conduct the main evaluation on THUMOS’14 [21], which contains temporally annotated untrimmed videos covering 20 action classes. We use the common OAD subset of 200 validation videos for training and 211 test videos for evaluation (411 videos total). Ambiguous frames are excluded according to the CMeRT protocol. Predictions and targets are paired by feature-step index and performance is measured with frame-level mean Average Precision (mAP).
We additionally evaluate on Charades [22], which provides multi-label annotations for 157 indoor action categories. The Charades study includes CMeRT, MiniROAD, LSTR, and TeSTra with RGB-only, RGB+Flow, and RGB+codec-motion inputs. For Charades, decision targets come from a dataset-specific CMeRT RGB+Flow teacher, with logits converted by per-class sigmoid (, ).
Charades uses dataset-specific feature and target construction, described separately from the THUMOS’14 4-fps/IPPP protocol. We further report a MultiTHUMOS [23] experiment in Appendix § 7.§ 2 to test substitution under denser multi-label supervision.
OAD Detectors
On THUMOS’14, we evaluate MiniROAD, CMeRT, LSTR, and TeSTra without modifying their temporal architectures. CMeRT serves as the high-capacity reference and as the common RGB+Flow teacher. MiniROAD is the primary lightweight student because its sub-millisecond recurrent update allows the computational effects of motion-source substitution and temporal-model choice to be examined separately. LSTR and TeSTra provide additional cross-architecture evaluations using the same frozen codec-motion interface and teacher signal. Accordingly, the motion-only analysis, factorial ablation, and primary accuracy–processing comparison focus on MiniROAD, while CMeRT provides the higher-capacity reference. Throughout the experiments, Hard denotes RGB+codec-MV training with feature-interface adaptation and ground-truth supervision only, whereas Soft uses the same inputs together with mixed targets distilled from the CMeRT RGB+Flow teacher.
Feature and Coding Protocol
For the controlled THUMOS’14 comparisons, all modalities are derived from the same past-referenced IPPP videos and assigned to a common nominal 4-fps feature grid. RGB uses the alignment-center frame, while Flow and MV preserve the same five-field temporal support required by the adopted motion encoder, including identical field offsets, endpoint-repetition rules, and terminal motion-field index (§ 4.1 and § 4.4). As in § 4.4, each motion feature is released after its required source observations are available.
All source comparisons match the codec, feature centers, five-field offsets, boundary handling, and ResNet-50-scale backbone (approximately 23.5M parameters). Flow uses TV-L1 [24] with a Kinetics-pretrained Flow-ResNet50; the MV encoder is initialized from that model, interface-adapted on the training split, and then frozen. Because the released CMeRT configuration originally uses BN-Inception motion features, we re-extract its RGB and Flow features from the same IPPP videos to obtain a common ResNet-50-based reference. Appendix § 9 gives extraction and training details.
Efficiency Protocol
We benchmark decoder-to-prediction processing, including decoding, preprocessing, motion estimation when applicable, feature extraction, temporal detection, and CPU return of scores. The main GPU test uses 20 matched THUMOS’14 IPPP videos, fp32 batch-one inference on a Tesla V100-SXM2-32GB, and five warm-up plus 25 measured steps per video. TV-L1 runs as a persistent cuda::OpticalFlowDual_TVL1 implementation, without per-span process startup or intermediate JPG I/O. A separate fp32 CPU test uses an Intel Xeon Gold 6240 with eight threads and resident OpenCV TV-L1. The reported metric is processing time per step, excluding source-observation waiting, transport, and offline re-encoding. Appendix § 10 specifies the implementations, host conditions, and component timings.
Performance Across Detectors and Datasets
Table 3 evaluates the adapted codec-motion interface across four established temporal-detector architectures using the same frozen encoder within each dataset. It reports frame-level mAP for RGB-only, RGB+Flow, and CodecMotion-OAD, with training-supervision differences made explicit.
| Detector | THUMOS’14 RGB | THUMOS’14 RGB+Flow | THUMOS’14 RGB+MV (ours) | Charades RGB | Charades RGB+Flow | Charades RGB+MV (ours) |
| CMeRT | 59.7 | 70.8 | 68.6 | 14.7 | 15.4 | 15.5 |
| MiniROAD | 54.6 | 67.5 | 68.3 | 13.9 | 14.9 | 16.5 |
| LSTR | 59.1 | 69.6 | 67.9 | 13.3 | 13.7 | 14.9 |
| TeSTra | 60.1 | 70.5 | 68.3 | 12.1 | 12.5 | 13.8 |
Table 3. Frame-level mAP on THUMOS’14 and Charades. The RGB+Flow references use ground-truth supervision; CodecMotion-OAD uses both IPMT stages. The matched-distillation MiniROAD comparison is in Table 4.
With MiniROAD, CodecMotion-OAD achieves 68.3 mAP. The standard RGB+Flow model reaches 67.5 mAP, while applying the same distillation targets to RGB+Flow yields 70.9 mAP. We use the matched-distillation comparison (68.3 versus 70.9) when isolating the source-related accuracy trade-off in § 5.3.
The frozen codec encoder is reused by all four temporal detectors within each dataset. On THUMOS’14, CodecMotion-OAD reaches 67.9–68.6 mAP across architectures. On Charades, its scores range from 13.8 to 16.5 mAP, higher than the corresponding ground-truth-trained RGB+Flow references. The systems in Table 3 differ in training supervision as well as motion input; Table 4 presents the matched-supervision MiniROAD comparison that isolates the motion-source trade-off.
| Method | mAP | GPU ms/step | CPU ms/step |
| RGB-only | 54.6 | 13.0 | 42.8 |
| CodecMotion-OAD | 68.3 | 38.2 | 95.6 |
| RGB+RAFT (cost reference) | — | 99.7 | 1152.2 |
| RGB+Flow (TV-L1, Hard) | 67.5 | 77.9 | 1098.1 |
| RGB+Flow (TV-L1, Soft) | 70.9 | 77.9 | 1098.1 |
Table 4. MiniROAD accuracy and decoder-to-prediction processing time. GPU: Tesla V100; CPU: 8-thread Xeon Gold 6240. Hard uses ground-truth targets and Soft uses teacher targets. TV-L1 rows share the same runtime. RGB-only mAP follows Table 3.
Decoder-to-Prediction Processing Efficiency
With MiniROAD held fixed, codec MVs reduce decoder-to-prediction processing from 1098.1 to 95.6 ms/step on CPU.
(11.48×) and from 77.9 to 38.2 ms/step on a V100 GPU (2.04×). Under matched soft-label distillation, RGB+MV reaches 68.3 mAP versus 70.9 mAP for RGB+Flow. Figure 1 shows this matched-supervision comparison; Table 4 additionally reports RGB-only and RAFT processing costs.

Figure 1. Accuracy–processing trade-off of MiniROAD on THUMOS’14 under matched soft-label distillation. CodecMotion-OAD achieves 68.3 mAP compared with 70.9 mAP for RGB+TV-L1, while reducing decoder-to-prediction processing time by 2.04× on an NVIDIA Tesla V100 GPU and 11.48× on an Intel Xeon Gold 6240 CPU with eight threads. Filled markers denote GPU measurements, and hollow markers denote CPU measurements. The horizontal axis uses a logarithmic scale.
Relative to RGB-only, CodecMotion-OAD adds 25.2 ms/step on GPU and 52.8 ms/step on CPU. The two TV-L1 rows have identical runtime graphs and processing costs.
Table 5 separates motion-source and temporal-model choices. Replacing TV-L1 with codec motion reduces processing for both CMeRT and MiniROAD, while replacing CMeRT with MiniROAD provides an additional detector-side reduction.
| System | mAP | GPU ms | CPU ms |
| CMeRT + TV-L1 | 70.8 | 102.2 | 1148.3 |
| CMeRT + MV | 68.6 | 60.5 | 145.0 |
| MiniROAD + TV-L1 | 67.5 | 77.9 | 1098.1 |
| MiniROAD + MV | 68.3 | 38.2 | 95.6 |
Table 5. THUMOS’14 system-level accuracy and processing cost under matched GPU and CPU protocols. TV-L1 denotes persistent optical flow and MV denotes codec motion. The codec/TV-L1 processing speedups for CMeRT and MiniROAD are 1.69×/2.04× on GPU and 7.92×/11.48× on CPU, respectively.
Ablations and Motion-Source Sensitivity
We define four codec-motion configurations to test the two IPMT training stages independently (Stage I: feature-interface alignment; Stage II: soft-label distillation):
Raw: neither feature-interface adaptation nor soft-label distillation;
Decision-only: soft-label distillation without feature-interface adaptation;
Hard: feature-interface adaptation without soft-label distillation;
Soft: both mechanisms.
All four configurations use the same RGB and centered codec-MV inputs.
Soft-Label Distillation Across Temporal Detectors
Table 6 compares Hard and Soft students for each detector. Soft-label distillation yields the largest gain for MiniROAD (+3.9 mAP), with smaller improvements for LSTR (+1.2), CMeRT (+0.3), and TeSTra (+0.2) on THUMOS’14. These differences show that teacher supervision benefits the evaluated architectures unequally; they do not, by themselves, establish which architectural property causes the variation. Every Hard/Soft pair has the same inference graph.
| Detector | Stage II | THUMOS’14 | Charades |
| MiniROAD | Hard | 64.4 | 15.1 |
| MiniROAD | Soft | 68.3 (+3.9) | 16.5 (+1.4) |
| CMeRT | Hard | 68.3 | 15.1 |
| CMeRT | Soft | 68.6 (+0.3) | 15.5 (+0.4) |
| LSTR | Hard | 66.7 | 13.8 |
| LSTR | Soft | 67.9 (+1.2) | 14.9 (+1.1) |
| TeSTra | Hard | 68.1 | 12.6 |
| TeSTra | Soft | 68.3 (+0.2) | 13.8 (+1.2) |
Table 6. Effect of soft-label distillation after Stage I feature adaptation. Hard uses ground-truth targets; Soft uses mixed targets. Each Soft entry includes the mAP gain over its Hard counterpart in parentheses, calculated from displayed one-decimal results. All THUMOS’14 Soft rows share the same CMeRT RGB+Flow teacher (70.8 mAP).
Factorial Ablation of the Two Training Components
Table 7 presents a MiniROAD factorial ablation of feature-interface adaptation and soft-label distillation. All four configurations use the same cross-entropy formulation and differ only in whether the motion encoder is feature-adapted and whether mixed teacher targets are used.
| Feature adaptation | Hard target | Soft target |
| Disabled | Raw: 59.0 | Decision-only: 63.6 |
| Enabled | Hard: 64.4 | Soft: 68.3 |
Table 7. MiniROAD ablation of feature-interface adaptation and soft-label distillation on THUMOS’14. Hard uses ground-truth targets and Soft uses mixed teacher targets with the same cross-entropy objective.
Both training stages contribute independently. Feature-interface adaptation raises Raw from 59.0 to 64.4 mAP (+5.4), and soft-label distillation raises Raw to 63.6 (+4.6) without adaptation. Combining them achieves 68.3 mAP, a 9.3-point gain over Raw. Each stage remains beneficial with the other enabled (+4.7 for feature adaptation; +3.9 for decision transfer).
Source Dependence of Flow-Feature Adaptation
To examine how feature-interface adaptation depends on the motion source, we compare codec MVs with RAFT-Large.
Both use the same TV-L1-pretrained Flow-ResNet50 initialization, frozen TV-L1 feature targets, centered five-field input construction, and adaptation objective. Fixed denotes use of the pretrained encoder without further optimization on the source; Adapted denotes source-specific weight updates with the objective of § 4.2. Downstream CMeRT detectors use ground-truth supervision without soft-label distillation. This controlled setting measures how the same feature-interface reuse procedure behaves across two motion sources.
Higher uncentered feature similarity does not necessarily improve OAD accuracy. RAFT adaptation raises cosine from 0.578 to 0.721, yet CMeRT mAP falls from 64.2 to 60.2; its centered cosine and also decline. In contrast, codec-MV adaptation improves both feature agreement and CMeRT mAP (63.7 to 68.3). Motion-only probes corroborate these opposite task trends (Appendix § 7.§ 3). The source dependence reinforces the need to assess both feature compatibility and downstream detection, rather than relying on uncentered cosine similarity alone.
Conclusion
We presented CodecMotion-OAD, which incorporates codec motion vectors into existing OAD detectors while retaining their temporal architectures. The key empirical observation is that dense optical flow’s advantage over codec MVs shrinks substantially after RGB fusion: from 11.2 to 3.7 MiniROAD mAP points on THUMOS’14 before soft-label distillation. Action-category diagnostics and spatial-coarsening experiments further characterize when fine motion details contribute less to RGB-conditioned predictions.
IPMT uses label-free feature alignment to adapt the motion interface and soft-label distillation to transfer stronger teacher supervision to RGB+MV students. The approach supports four temporal detectors and is evaluated on THUMOS’14 and Charades. On MiniROAD, it achieves 68.3 mAP versus 70.9 mAP for a matched-distillation RGB+Flow system while reducing decoder-to-prediction processing time by 11.48× on CPU and 2.04× on GPU. These results demonstrate a practical accuracy–processing trade-off for codec-based motion acquisition in OAD; source-observation latency and offline re-encoding remain outside the measured processing metric.
Additional Experimental Analysis
Standalone Motion Discrimination and RGB Fusion
Table A1 examines motion-only discrimination by evaluating MiniROAD without RGB, using centered motion features for both sources. Both inputs use the same five-field offsets and boundary handling.
| Motion stream | mAP |
| Codec motion | 48.4 |
| Optical flow | 59.6 |
Table A1. MiniROAD motion-only evaluation on THUMOS’14 with common centered field-index windows.
Multi-Label Evaluation on MultiTHUMOS
We additionally evaluate on MultiTHUMOS, which provides denser multi-label annotations over the same underlying videos as THUMOS’14. Flow and MV retain the centered field-index construction used in the main experiments.
The implemented soft-label distillation uses the same mixing weights as on THUMOS’14, but the teacher is the multi-label CMeRT RGB+Flow model trained with BCE. Its logits are converted with a per-class sigmoid rather than a softmax:
with and , where denotes the elementwise sigmoid. The mixed target is optimized with MiniROAD’s standard BCE objective. Hard disables soft-label distillation and trains on ground-truth targets only. Table A2 reports the resulting MultiTHUMOS comparison.
| Method | mAP |
| RGB-only | 39.7 |
| RGB+Codec Motion (Hard) | 46.8 |
| CodecMotion-OAD (Soft) | 49.3 |
| RGB+Flow | 49.8 |
Table A2. MultiTHUMOS results using MiniROAD with standard BCE under the common IPPP and centered field-index protocol.
Without soft-label distillation, RGB+Codec Motion reaches 46.8 mAP, improving RGB-only by 7.1 points while remaining 3.0 points below RGB+Flow. Adding soft-label distillation raises performance to 49.3 mAP, a further 2.5-point gain. The Soft model exceeds RGB-only by 9.6 points and remains 0.5 point below RGB+Flow. MultiTHUMOS shares underlying videos with THUMOS’14, so this experiment tests the substitution under denser multi-label supervision.
Sensitivity to the Dense-Flow Estimator
We use RAFT-Large [25] as a supplementary dense-flow alternative under the THUMOS’14 centered five-field protocol. Table A3 reports the matched persistent-step-wise GPU processing costs from the remeasurement. Its accuracy figures reflect the feature-encoder and decision-supervision configurations specified in the table.
| Configuration | mAP | ms/step |
| TV-L1 (persistent); Original Flow-ResNet50; Ground truth | 70.8 | 102.2 |
| RAFT-Large; Fixed pretrained Flow-ResNet50; Ground truth | 64.2 | 120.5 |
| CodecMotion-OAD; Adapted codec encoder; Mixed teacher targets | 68.6 | 60.5 |
Table A3. THUMOS’14 CMeRT accuracy and GPU processing time across three source configurations. Each entry specifies the motion source, feature encoder, and decision supervision, respectively. Table 8 compares source adaptation under a common ground-truth CMeRT objective.
The fixed-encoder RAFT comparison measures RAFT inputs under the TV-L1-pretrained motion encoder. In CMeRT without soft-label distillation, its original fixed-encoder mAP was 64.18. For RAFT adapted with the cosine-plus-MSE objective, CMeRT mAP fell to 60.17; MV rose from 63.68 with the fixed encoder to 68.27 after adaptation.
Additional source-adaptation diagnostics. Table 8 records raw and centered cosine, along with ridge-regression prediction of teacher features. The fixed state uses the TV-L1-pretrained Flow-ResNet50 unchanged and the adapted state uses the representation-alignment loss of Section 4.2. Centered feature means and regression fits are computed using training data, then evaluated on the reported test feature set.
| Source/state | Raw cos. | Ctr. cos. | mAP | |
| RAFT fixed | 0.578 | 0.306 | 0.290 | 64.2 |
| RAFT adapted | 0.721 | 0.277 | 0.232 | 60.2 |
| MV fixed | 0.456 | 0.066 | 0.206 | 63.7 |
| MV adapted | 0.828 | 0.623 | 0.407 | 68.3 |
Table 8. Effect of source-dependent feature adaptation on THUMOS’14. Centered cosine subtracts training-set means; is the test-set ridge prediction score. CMeRT uses ground-truth supervision without soft labels.
Table A4 reports frame-wise linear-probe results on frozen features (30 epochs, AdamW; averaged over random initializations 0–2).
| Feature input | Fixed mAP | Adapted mAP | Change |
| RAFT-Large only | 17.02 | 13.35 | −3.67 |
| RGB + RAFT-Large | 41.01 | 40.28 | −0.73 |
| Codec MV only | 10.39 | 23.12 | +12.73 |
| RGB + Codec MV | 39.92 | 43.52 | +3.60 |
Table A4. Probe results before and after feature adaptation. TV-L1-only, RGB-only, and RGB+TV-L1 probe references are 31.89, 38.84, and 46.03 mAP, respectively. Decimal rounding can make a displayed change differ by 0.01.
Representation-Level versus Decision-Level Redundancy
All frame-wise analyses operate on frozen features and contain no temporal context. For THUMOS’14, fitting uses the 200 validation videos that form the OAD training split (167,037 feature steps, including 164,833 non-ambiguous steps), and evaluation uses the same 211 held-out test videos as detector evaluation (183,266 feature steps, including 180,400 non-ambiguous steps). Feature dimensions are standardized using statistics computed on the fitting split only.
The linear probe is a single 21-way linear classifier (20 action classes plus background). It is trained for 30 epochs with class-weighted cross-entropy, assigning the background class weight 0.5, using AdamW with learning rate , weight decay , and batch size 4096. Probe mAP follows the same frame-level evaluation used for the OAD detectors. and is averaged over three random initializations.
Linear predictability is measured with ridge regression on the standardized features. The ridge coefficient is set to , where is the number of fitting frames, and pooled is reported on held-out test frames. To isolate the Flow component not linearly predictable from MV, the MV-to-Flow ridge map is fit on training frames and then applied unchanged to the test set; the corresponding test residual defines the Flow-specific component used by the downstream probes.
The large motion-only Flow–MV gap contracts sharply even under this frame-wise protocol: MV and Flow obtain 23.1 and 31.9 mAP with motion alone, while RGB+MV and RGB+Flow obtain 43.5 and 46.0. The gap contraction thus also appears under frame-wise evaluation.
The regression analysis separates representational difference from task-level utility. Approximately 59% of Flow-feature variance is not explained linearly by MV. RGB has little linear predictive power for this Flow-specific residual (), so the reduced task advantage is not explained by substantial linear representational overlap with appearance.
The Flow-specific residual is nevertheless much less valuable after RGB is present. Adding it to MV raises the probe from 23.1 to 31.5 mAP (+8.4), whereas adding the same residual to RGB+MV raises performance from 43.5 to 45.4 (about two points). This task-conditional reduction in marginal utility is the decision-level redundancy discussed in § 3.1.
Per-Class, Presence/Identity, and Boundary Analyses
The per-class contraction is strongest for categories that suffer the largest Flow–MV gap with motion alone. Across the 20 THUMOS’14 classes, the motion-only Flow advantage and the amount by which this advantage contracts after adding RGB have Spearman correlation (). The contraction measure includes the original motion-only gap, so the correlation describes the observed per-class pattern with a shared term. Representative categories with large gap contraction include HighJump, JavelinThrow, FrisbeeCatch, CricketBowling, and SoccerPenalty, where coarse motion alone loses substantial category-discriminative evidence and RGB sharply reduces the dependence on fine motion structure.
We further decompose prediction into action presence and action identity. Presence uses as a binary action-versus-background score. Identity is evaluated only on ground-truth action frames after renormalizing probabilities over the 20 action classes. The Flow–MV differences are summarized in Table A5.
| Metric | Linear probe M | Linear probe +RGB | MiniROAD M | MiniROAD +RGB |
| mAP | +8.8 | +2.5 | +11.2 | +3.7 |
| Presence AP | +5.3 | +3.2 | +4.1 | +3.1 |
| Presence AUROC | +3.1 | +1.7 | +3.1 | +1.7 |
| Identity macro-AP | +10.4 | +1.0 | +10.6 | +0.9 |
| Identity top-1 | +10.6 | +1.6 | +8.4 | +1.4 |
Table A5. Flow–MV differences in detection, action presence, and identity on THUMOS’14. M denotes motion only; +RGB denotes RGB-fused input. Probe and MiniROAD entries compare Flow with MV under the same RGB condition. Positive values favor Flow; all differences are percentage points. Identity measures use ground-truth action frames.
The diagnostic difference is much larger for action identity than for action presence in the motion-only setting. After RGB fusion, about 90% of the identity gap disappears, whereas the presence gap contracts much less. The displayed mAP differences are computed from the unrounded run-level results as noted in Table 1, and the RGB+Flow comparison uses the same cross-entropy control as Table 2.
| Motion input | Feature cos. | Motion-only | With RGB |
| Original Flow | 1.000 | 59.6 | 68.1 |
| 8x8 block-averaged Flow | 0.915 | 57.6 | 66.6 |
| 16x16 block-averaged Flow | 0.860 | 51.5 | 64.3 |
| 32x32 block-averaged Flow | 0.812 | 36.4 | 61.5 |
| Codec MV | 0.828 | 48.4 | 64.4 |
Table 2. Controlled Flow spatial-coarsening intervention with re-adapted motion encoders. The RGB+original-Flow control uses the same cross-entropy training setting as the intervention models. Feature cosine is measured after training on all 211 held-out test videos and is used only as a representation diagnostic. Spatial coarsening reduces motion-only performance substantially more than RGB+motion performance, and the 16x16 intervention yields a task profile close to codec motion.
The boundary check shows that MV features are smoother (adjacent-frame cosine: 0.913 for MV, 0.837 for RGB, and 0.778 for Flow), yet the Flow-over-MV advantage is smaller near action boundaries than in action interiors (9.4 versus 11.7 points), both before and after RGB fusion (2.0 versus 3.1).
Controlled Flow Spatial-Coarsening Intervention
To test whether spatial granularity can reproduce the Flow–MV task signature, we construct block-averaged TV-L1 sources directly from the native 320 × 180 flow fields. For , each non-overlapping pixel block is replaced by its mean 2-D motion vector; partial blocks at the image boundary average only valid pixels. The block mean is then tiled back to the original spatial resolution before the standard resize and center crop. A control reproduces the original Flow features exactly (cosine similarity 1.0000), confirming that the intervention path itself leaves an unmodified source unchanged.
Directly applying the frozen original Flow encoder to coarsened fields creates a severe distribution shift: motion-only MiniROAD falls to 20.7–25.3 mAP and RGB+motion to 58.8–59.3. These frozen-encoder results are therefore not used to interpret spatial granularity. Instead, each coarsened source receives its own interface-adapted encoder. Every encoder starts from the same Kinetics Flow-ResNet50 initialization and follows the codec-motion adaptation recipe: frozen BatchNorm, the cosine-plus-MSE feature objective of Section 4.2, AdamW with learning rate , batch size 24, and 12 training epochs using 32 randomly sampled steps per training video. Only training videos contribute gradients, and the final-epoch encoder is used for downstream evaluation.
After adaptation, the intervention behaves smoothly with granularity (Table A6):
| Motion input | Cosine | Motion mAP | RGB+motion mAP |
| Original Flow | 1.000 | 59.6 | 68.1 |
| 8x8 block-averaged Flow | 0.915 | 57.6 | 66.6 |
| 16x16 block-averaged Flow | 0.860 | 51.5 | 64.3 |
| 32x32 block-averaged Flow | 0.812 | 36.4 | 61.5 |
| Codec MV | 0.828 | 48.4 | 64.4 |
Table A6. Adapted feature similarity and detection accuracy under spatial coarsening of TV-L1 flow, compared with codec motion on THUMOS’14. Feature cosine is evaluated on the held-out test set; mAP is reported for motion-only and RGB-fused MiniROAD.
The adapted feature cosines are post-hoc diagnostics evaluated on all 211 held-out test videos using the same procedure for all adapted encoders. The codec-motion cosine (0.828) is consistent with the adapted entry in Table 8. These test-set diagnostics are not used for training or checkpoint selection.
The mAP loss from spatial coarsening is much larger when motion must stand alone than when RGB is present. At 16x16, the two losses are 8.1 and 3.8 points; at 32x32 they are 23.2 and 6.6. The 16x16 intervention is especially close to codec motion in both settings.
The presence/identity decomposition reveals the same structure. For motion-only MiniROAD, original Flow obtains 80.1 Pres-AP and 85.2 Id-macroAP; 8x8 Flow obtains 79.4/82.9, 16x16 obtains 76.7/75.4, 32x32 obtains 71.2/59.3, and codec MV obtains 76.1/74.6. Thus identity degrades substantially faster than presence as Flow is spatially coarsened, and the 16x16 intervention nearly reproduces the codec-motion decomposition.
The frame-wise linear probe additionally shows how RGB suppresses this identity loss. Relative to original Flow, the 8x8, 16x16, and 32x32 interventions lose 1.1/3.4, 4.2/10.6, and 10.9/21.6 points in presence/identity with motion alone. After adding RGB, the corresponding losses shrink to 0.5/0.1, 2.3/0.9, and 5.7/2.1. Codec MV follows the same pattern: 5.3/10.4 with motion alone and 3.2/1.0 after RGB fusion.
These controlled interventions support spatial granularity as an explanatory factor: spatial coarsening reproduces much of the characteristic Flow–MV gap and its presence/identity signature after interface adaptation. Codec motion additionally differs in coding structure, missing-motion regions, partitioning, and reference dependencies.
Qualitative Analysis of Codec Motion
Figure B1 compares aligned TV-L1 and codec-MV fields on four THUMOS’14 examples. Codec motion captures actor or camera displacement at coarser spatial granularity, and I pictures provide no MV field. The examples illustrate the input differences handled by feature-interface alignment. For visualization, the codec reference-to-current displacement is sign-flipped to the same forward-displacement color convention as Flow; each row shares a magnitude cap determined by the 99th percentile of Flow magnitude.

Figure B1. Qualitative comparison of aligned RGB frames, TV-L1 optical flow, and codec motion vectors from THUMOS’14 under the shared IPPP protocol. The examples show (a) body motion (SoccerPenalty), (b) camera motion (CliffDiving), (c) localized motion (TennisSwing), and (d) an I picture without motion vectors (CleanAndJerk). For visualization, codec MVs are sign-flipped to match the forward-flow direction convention. Within each row, Flow and MV use the same magnitude cap, determined by the 99th percentile of Flow magnitude and labeled in pixels. The displayed motion fields are aligned at and each constitutes one field of the five-field encoder input.
Coding and Feature Extraction
Encoding Choice and Shared IPPP Configuration
All controlled comparisons use videos re-encoded once with H.264/AVC [10] (libx264). B-frames are disabled, the GOP/intra period is 25 frames, and the main experiments use CRF 23. RGB, Flow, and MV are derived from the same re-encoded stream, applying the same source coding to all modalities.
We choose past-referenced IPPP to preserve the observation range specified by the motion-feature extractor. A completed five-field MV input is ready once its selected pictures and their decoding references are available. Predictive coding with later display-order references can extend that requirement beyond the chosen motion-field window, adding source-picture waiting and potentially crossing a feature-interval boundary [10]. Using IPPP with only earlier display-order references removes this additional dependency and is compatible with low-delay video processing [11, 12].
Analysis of the original source streams for the common 411-video THUMOS’14 collection (200 training and 211 test videos) identifies 1,039,610 B pictures, approximately 41% of all source frames. Using the nearest subsequent I/P anchor in display order to estimate the forward dependency extent, the anchor lies 1, 2, or 3 frames ahead for 69.0%, 19.2%, and 11.8% of B pictures, respectively (mean 1.43 frames). Under the feature-interval assignment (, 19.3% of B pictures have an anchor in the next interval, and none have an anchor two or more intervals ahead. Membership in the next interval does not imply waiting until the next feature center or using the next feature vector. These statistics estimate dependencies through the subsequent I/P anchor.
Across the 350,303 feature steps of these original source streams, the analysis estimates that B-picture dependencies extend beyond the nominal centered MV window at approximately 41% of steps. The additional extent is 1–3 source frames, placing the latest required source picture as far as . Although 8.8% of steps extend beyond the midpoint between adjacent feature centers, none reach the next center. Past-referenced IPPP removes this codec-induced extension while retaining the explicitly specified centered-window support. The B-picture fraction and the affected-feature-step fraction use different denominators.
Appendix § 9.§ 3 formalizes source-frame availability; Appendix § 10 defines the associated decoder-to-prediction processing measurement.
Motion-Field Extraction and Feature-Grid Alignment
Codec MVs are extracted with PyAV and libavcodec motion-vector side data. Displacements are converted using the codec-reported scale and rasterized over destination blocks. Intra-coded regions and pictures without valid side data remain zero (Figure B1(d)); vectors are not propagated from previous frames into these regions. Decoded residuals are not used as model input, so the motion branch contains only MV side information.
Let a video contain source frames , and let be the number of RGB feature steps. We use the existing RGB feature length to define a shared nominal 4-fps grid:
where denotes round-to-nearest followed by integer conversion. RGB, Flow, and MV inputs share these sampling centers. The centers define feature alignment only: the corresponding feature becomes usable after the observations required by its motion extractor are available, as formalized in Appendix § 9.§ 3. For THUMOS’14, the released target_perframe has columns (background, 20 actions, Ambiguous) and uses nominal quarter-second label intervals . On the 410 videos with available annotation JSON within the 411-video collection, reconstruction agrees with 97.30% of non-ambiguous labels at zero shift (versus 94.35% at a −1-step shift and 96.64% at a +1-step shift). Residual discrepancies include overlapping CliffDiving/Diving labels and interval-boundary conventions. The evaluated IPPP RGB/Flow/MV feature lengths match the target axis, and the annotation reconstruction approximates the released label-generation procedure. This full-video grid defines offline benchmark alignment; live deployment would require timestamp-based sampling. Among the 411 training and test videos, 380 are 30 fps, 18 are 29.97 fps, 11 are 25 fps and two are 24 fps; the average selected step spacing exceeds 250 ms by 0.84 ms (range 0.04–8.82 ms), producing cumulative drift relative to regular 4-Hz label intervals. This grid construction is distinct from detector-internal long-memory sampling.
The exported CMeRT teacher logits are one-to-one indexed with labels and feature lengths. Teacher argmax/label agreement reaches its maximum at zero step shift (88.79%; offsets −1 and +1 give 87.78% and 88.50%).
Flow and codec-motion inputs use the same five centered field indices with ; out-of-range indices are clipped to legal endpoints, and RGB selects only the center frame .
Codec MVs are first rasterized at native decode resolution in pixel-displacement units using the codec-reported motion scale. The rasterized maps are bilinearly resized to and then spatially subsampled by a factor of two before caching, yielding a float16 cache of shape . Neither operation rescales displacement magnitudes by geometric resize factors.
Before the motion encoder, a scale factor of 16 maps displacement values to the flow-JPEG-style intensity convention, e.g.
followed by spatial interpolation/cropping to the network input size. This is an intensity encoding rather than geometric displacement rescaling.
Source-Frame Availability
We formalize the source availability of a feature step aligned at with terminal motion index . Let be the earliest source-frame index at which all observations needed for modality ’s motion window have arrived. Feature step can be constructed only at or after this availability point. This quantity is independent of target assignment, and decoding/inference incur additional processing time.
For Flow, is computed from . With valid Flow indices ,
An interior window therefore requires source frame . Endpoint repetition uses the source observations required by the clipped fields.
For codec motion, define the clipped field set
where . Let be the transitive set of source pictures required to decode picture , including itself. Then
Under the past-referenced IPPP configuration, all references precede the coded picture in display order. Since includes ,
Thus, codec references add no source-picture dependency beyond the explicitly selected MV window.
For modality , feature step becomes eligible for inference once the source observations through have arrived; the completed motion representation is then paired with the buffered RGB frame . Observations after that fall within the inherited five-field extractor therefore determine feature availability, not access to future feature steps by the temporal detector. Across the 411-video collection (350,303 feature steps), the codec terminal frame lies after the nominal label-interval end at 37.8% of steps (p95 offset +167 ms), compared with 49.2% for TV-L1 (p95 +200 ms). These statistics express source availability relative to the nominal label intervals; Appendix § 10 separately reports processing cost. The live timestamp-based sampling requirement follows from the offline grid construction in Appendix § 9.§ 2.
Training Details
The codec-motion ResNet-50 in the representation stage is trained with AdamW (learning rate , weight decay ) and a cosine annealing schedule, using batch size 24 for 12 epochs. BatchNorm statistics are frozen during adaptation. Each training video contributes 32 randomly sampled feature steps per epoch. Inputs are resized along the shorter side and center-cropped to ; no random augmentation is applied, and codec displacements use the flow-style intensity mapping of § 4.2. The final epoch is used for all reported codec-motion encoders; test-set feature cosine for codec motion is recorded only as a diagnostic and is not used for checkpoint selection. Detector training starts from the official CMeRT, MiniROAD, LSTR, and TeSTra configurations. The main changes are the input features and, for Soft models, the mixed soft-label targets of § 4.3. MiniROAD source analysis and the factorial experiment use the CE configurations specified in § 3.0.1 and § 5.4.2, distinct from the standard RGB+Flow reference in Table 3.
Timing Configuration and Component Analysis
Primary GPU Benchmark: Persistent Step-Wise Flow
The GPU measurements use a Tesla V100-SXM2-32GB with fp32 batch-one inference, the same 20 THUMOS’14 IPPP-re-encoded test videos, five warm-up steps and 25 measured steps per video. The evaluated pipelines are orchestrated for CMeRT and MiniROAD within one benchmark session, with per-video execution order rotated. Measured processing covers decoding, preprocessing, motion estimation when applicable, feature encoding, detector inference, softmax, and copy-back to CPU. Source-observation waiting and offline re-encoding are excluded. The reported GPU SM clocks ranged from 1327 to 1530 MHz, temperatures from 43 to 50°C, and one-minute system loads from 8.4 to 10.6. The historical DenseFlow path is retained only as an implementation reference and invokes a separate worker process per video span.
The persistent TV-L1 path uses the CUDA cuda::OpticalFlowDual_TVL1 implementation with the same defaults and bound-20 quantization as the historical denseflow source, computing five selected motion fields at each step. Its process remains resident and no intermediate JPG files are written or read. Across the full test set, the mean difference between the persistent flow and the legacy DenseFlow/JPG results is 0.176 quantization levels, approximately 60% of which is attributable to JPEG quantization.
Measured GPU Processing Time
In MiniROAD the persistent TV-L1/codec processing ratio is , and in CMeRT it is . Relative to RAFT, the corresponding ratios are and .
CPU Benchmark
The CPU benchmark uses an Intel Xeon Gold 6240 host with 16 vCPUs shared with other users. Eight threads are assigned consistently to PyTorch and OpenCV, with fp32 inference throughout. It follows the same 20-video protocol as the GPU benchmark, with five warm-up steps and 25 measured steps per video. TV-L1 uses the resident OpenCV 4.4 optflow::DualTVL1 CPU implementation. The one-minute host load varied from 8.5 to 16.8; the measurements characterize the evaluated shared-host CPU implementation. Table D2 reports the CPU processing results.
| Pipeline | CMeRT ms/step (CPU) | MiniROAD ms/step (CPU) |
| RGB-only | 85.4 | 42.8 |
| CodecMotion-OAD | 145.0 | 95.6 |
| RGB+TV-L1 (persistent CPU) | 1148.3 | 1098.1 |
| RGB+RAFT (CPU) | 1202.4 | 1152.2 |
Table D2. CPU processing measurements under the protocol above. Codec-vs-TV-L1 processing ratios are for CMeRT and for MiniROAD. Component measurements attribute approximately 1008 ms/step to CPU TV-L1 flow and 1075 ms/step to CPU RAFT; codec-motion decode/export and rasterization require approximately 13 ms and 7 ms per step, respectively, in addition to the motion encoder.
Historical DenseFlow Implementation
The earlier DenseFlow pipeline launched a separate process for each 25-step video span and used intermediate JPG files. Under the unified remeasurement it requires 140.8 ms/step with MiniROAD and 164.3 ms/step with CMeRT. The main comparisons use persistent TV-L1 from Table D1.
| Pipeline | CMeRT ms/step | MiniROAD ms/step |
| RGB-only | 33.6 | 13.0 |
| CodecMotion-OAD | 60.5 | 38.2 |
| RGB+TV-L1 (persistent, step-wise) | 102.2 | 77.9 |
| RGB+RAFT | 120.5 | 99.7 |
| RGB+DenseFlow (legacy one-wall) | 164.3 | 140.8 |
Table D1. Primary unified-remeasurement GPU pipeline means over the same 20 videos. Persistent TV-L1 is the main motion-source reference; legacy one-wall DenseFlow includes per-span startup and intermediate JPG I/O.
Historical Step-Wise Component Breakdown
The older (pre-remeasurement) step-wise runs separate decoding, preprocessing/transfer, encoders, and detection to identify implementation bottlenecks. Table D3 reports their component-level timings; main processing totals come from the unified remeasurement in Table D1.
| Detector / pipeline | decode | prep | encoder(s) | detector |
| CMeRT RGB-only | 3.1 | 1.8 | 7.7 | 21.2 |
| CMeRT CodecMotion | 13.5 | 9.2 | 16.3 | 23.0 |
| CMeRT RAFT | 3.4 | 79.7 | 15.7 | 22.5 |
| MiniROAD RGB-only | 3.0 | 1.6 | 7.3 | 0.6 |
| MiniROAD CodecMotion | 12.7 | 8.8 | 15.7 | 0.9 |
| MiniROAD RAFT | 3.4 | 79.4 | 15.7 | 0.9 |
Table D3. Component means (ms) from the historical step-wise runs. Python MV rasterization is a remaining implementation bottleneck.
References
- [1]Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees G. M. Snoek, Tinne Tuytelaars. “Online Action Detection.” In Computer Vision – ECCV 2016, pp. 269–284, 2016. DOI: https://doi.org/10.1007/978-3-319-46454-1_17.
- [2]Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, Stefano Soatto. “Long Short-Term Transformer for Online Action Detection.” In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2107.03377
- [3]Yue Zhao, Philipp Krähenbühl. “Real-Time Online Video Detection with Temporal Smoothing Transformers.” In Computer Vision – ECCV 2022, pp. 485–502, 2022. DOI: https://doi.org/10.1007/978-3-031-19830-4_28.
- [4]Zhanzhong Pang, Fadime Sener, Angela Yao. “Context-Enhanced Memory-Refined Transformer for Online Action Detection.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8700–8710, 2025. DOI: https://doi.org/10.1109/CVPR52734.2025.00813.
- [5]Chuohao Yeo, Parvez Ahammad, Kannan Ramchandran, Shankar Sastry. “High-Speed Action Recognition and Localization in Compressed Domain Videos.” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 8, pp. 1006–1015, 2008. DOI: https://doi.org/10.1109/TCSVT.2008.927112.
- [6]Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang. “Real-Time Action Recognition With Enhanced Motion Vector CNNs.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2718–2726, 2016. DOI: https://doi.org/10.1109/CVPR.2016.297.
- [7]Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, Philipp Krähenbühl. “Compressed Video Action Recognition.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6026–6035, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00631.
- [8]Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, Zhicheng Yan. “DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1268–1277, 2019.arxiv.org/abs/1901.03460
- [9]Wei Luo, Deyu Zhang, Ying Tang, Fan Wu, Yaoxue Zhang. “EdgeOAR: Real-Time Online Action Recognition on Edge Devices.” IEEE Transactions on Mobile Computing, vol. 24, no. 12, pp. 13426–13440, 2025. DOI: https://doi.org/10.1109/TMC.2025.3591188.
- [10]Thomas Wiegand, Gary J. Sullivan, Gisle Bjøntegaard, Ajay Luthra. “Overview of the H.264/AVC Video Coding Standard.” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, 2003. DOI: https://doi.org/10.1109/TCSVT.2003.815165.
- [11]NVIDIA. Using PyNvVideoCodec APIs. NVIDIA PyNvVideoCodec API Programming Guide, section “Latency Modes.” https://docs.nvidia.com/video-technologies/pynvvideocodec/pynvc-api-prog-guide/using_pynvvideocodec_apis.html. Accessed 2026-10-12.
- [12]Amazon Web Services. IVS Stream Ingest: Real-Time Streaming. Amazon IVS Real-Time Streaming User Guide. 2026. https://docs.aws.amazon.com/ivs/latest/RealTimeUserGuide/rt-stream-ingest.html. Accessed 2026-10-12.
- [13]Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, Nong Sang. “OadTR: Online Action Detection With Transformers.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7565–7575, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00747.
- [14]Le Yang, Junwei Han, Dingwen Zhang. “Colar: Effective and Efficient Online Action Detection by Consulting Exemplars.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3160–3169, 2022.arxiv.org/abs/2203.01057
- [15]Youngbin An, Hyolim Kang, Su Ho Han, Ming-Hsuan Yang, Seon Joo Kim. “MiniROAD: Minimal RNN Framework for Online Action Detection.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10341–10350, 2023.DOI
- [16]Shuqiang Cao, Weixin Luo, Bairui Wang, Ran Song, Wei Zhang, Lin Ma. “A Circular Window-based Cascade Transformer for Online Action Detection.” IEEE Transactions on Multimedia, pp. 1–12, 2026. DOI: https://doi.org/10.1109/TMM.2026.3676664.
- [17]Su Yan, Jiahua Li, Kun Wei, Cheng Deng. “Backtrace Mamba: Reviving Critical Temporal Contexts via Hierarchical Memory Compression for Online Action Detection.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 14, pp. 11559–11567, 2026. DOI: https://doi.org/10.1609/aaai.v40i14.38139.
- [18]Junwen Chen, Gaurav Mittal, Ye Yu, Yu Kong, Mei Chen. “GateHUB: Gated History Unit With Background Suppression for Online Action Detection.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19925–19934, 2022.arxiv.org/abs/2206.04668
- [19]Shuqiang Cao, Weixin Luo, Bairui Wang, Wei Zhang, Lin Ma. “E2E-LOAD: End-to-End Long-form Online Action Detection.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10422–10432, 2023.arxiv.org/abs/2306.07703
- [20]Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean. “Distilling the Knowledge in a Neural Network.” arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [21]Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, Mubarak Shah. “The THUMOS Challenge on Action Recognition for Videos in the Wild.” Computer Vision and Image Understanding, vol. 155, pp. 1–23, 2017. DOI: https://doi.org/10.1016/j.cviu.2016.10.018.
- [22]Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, Abhinav Gupta. “Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding.” In Computer Vision – ECCV 2016, 2016.arxiv.org/abs/1604.01753
- [23]Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, Li Fei-Fei. “Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos.” International Journal of Computer Vision, vol. 126, no. 2–4, pp. 375–389, 2018. DOI: https://doi.org/10.1007/s11263-017-1013-y.
- [24]Christopher Zach, Thomas Pock, Horst Bischof. “A Duality Based Approach for Realtime TV-L1 Optical Flow.” In Pattern Recognition (DAGM), pp. 214–223, 2007. DOI: https://doi.org/10.1007/978-3-540-74936-3_22.
- [25]Zachary Teed, Jia Deng. “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow.” In Computer Vision – ECCV 2020, pp. 402–419, 2020. DOI: https://doi.org/10.1007/978-3-030-58536-5_24.