WorksComputer Science
Real Time Motion Detection in Video Inspired by Photonic Time Stretch
Abstract
Motion perception is fundamental to computer vision and physical AI, and plays a central role in robotics, autonomous vehicles, human-computer interaction, and video compression. Traditional approaches such as background subtraction, frame-differencing and optical flow algorithms often struggle in complex or dynamic scenes, typically sacrificing spatial resolution for computational tractability. Here we introduce MoViD (Motion estimation via Virtual dispersion and phase Detection), a physics-inspired algorithm grounded in the principles that govern photonic time stretch - a highly successful ultrafast realtime data acquisition platform. Drawing inspiration from the optical physics therein, MoViD treats video as a collection of optical time series waveforms associated with pixels. Motion of an object through a pixel broadens the time series spectrum and, after group delay dispersion, induces a temporal phase that exists only if there is motion. Spatiotemporal group delay dispersion templates, corresponding to dielectric functions, classify specific motion trajectories. Unlike conventional empirical phase-based motion-analysis pipelines that require narrow-band spatial channelization and pyramid reconstruction, the physics-based MoViD operates directly on the broadband video signal, vastly increasing its computational speed. The resulting algorithm achieves unprecedented performance in both motion characterization and motion magnification, providing full-resolution motion mapping in real time at nearly 200 frames per second at 4K resolution and 70 frames per second for 8K video - orders of magnitude faster than classical optical flow approaches or learning-based methods. Notably, such extreme performance enables real time motion analysis at ultrahigh resolution for motion detection and compensation in robotics and other autonomous devices.
Introduction
The human eye is irresistibly drawn to motion–a survival instinct refined over millions of years, where noticing the faintest flicker could mean the difference between life and death [1]. Neuroscience shows that our sense of time is malleable: danger appears to stretch each instant, while moments of joy seem to vanish in a rush, reminding us that the brain’s internal clock is not a precise metronome but adapts to the rate of change and the information content of events [2]. This deep evolutionary interplay between motion and temporal perception offers a powerful model for machine perception: by attending selectively to change, the brain reduces overwhelming sensory input to what matters most [3].
As such, change detection and motion analysis are cornerstone tasks in computer vision, serving as the foundation for higher-level applications like object tracking, activity recognition, behavior analysis, and video summarization. In practical settings ranging from manufacturing to autonomous vehicles, Physical AI and robotics, the accurate identification of moving objects and dynamic elements is crucial for understanding events in real time. Over the years, various algorithms and models have been proposed to address these tasks, yet they often face challenges in balancing robustness, interpretability, and computational efficiency.
The Limitations of Traditional and Data-driven Motion Analysis
A longstanding family of traditional solutions for motion detection relies on rule-based techniques such as background subtraction and temporal differencing [4]. While effective in controlled environments or with static cameras, these approaches typically require excessive down-sampling and struggle in complex or cluttered scenes where motion is partially occluded or lighting changes rapidly. Such empirical algorithms have also investigated temporal Fourier transforms and spectral representation, often requiring a complex steerable pyramid decomposition to detect and magnify motion in video [5, 6, 7, 8].
In contrast, the past decade has witnessed significant advances driven by deep learning, where convolutional and recurrent neural networks can directly learn motion features from raw data. Data-driven approaches for optical flow calculation such as FlowNet [9] and optical-flow-based CNNs [10] have achieved state-of-the-art results in action recognition and object detection in videos. RAFT, or Recurrent All-Pairs Field Transforms for Optical Flow [11], represents one widely used approach that has demonstrated exceptional accuracy in optical flow prediction. More recent transformer-based models [12] have further accelerated vision tasks and contributed to improved performance in video understanding. Additional progress has been made in motion magnification with the adoption of deep learning, with learning-based techniques such as [13]. However, despite their success, deep learning methods frequently demand huge computational resources and large-scale labeled datasets, limiting their effectiveness in the real world.
Physics of Photonic Time Stretch Provides a Blueprint for Motion Analysis
Emerging in the gap between traditional and data-driven approaches, physics-inspired approaches represent a promising way to reconcile interpretability and performance in algorithm design. Building on the concept of leveraging established physical models as “blueprints” for algorithmic design [14] [15] [16], researchers have shown that encoding physical principles into computer vision methods can deliver both computational efficiency and generalizability.
Drawing from fundamental optics and wave propagation principles, we introduce here a novel motion detection, analysis, and magnification technique entitled MoViD (Motion estimation via Virtual dispersion and phase Detection). MoViD represents a completely novel perspective on analyzing and augmenting motion in video. Using simulated spatiotemporal dielectric dispersion to stretch motion in time, MoViD provides an elegant framework to detect, classify, and magnify motion signatures at the pixel-level in real time. As shown in Figure 1, MoViD produces dense, directionally encoded motion fields, enabling pixel-level representations suitable for downstream tasks such as anomaly detection, object tracking, and robotic perception.

Figure 1. Real-Time Full-Resolution Motion Analysis using the MoViD Algorithm. Three video frames of a quadruped robot (top row) and the corresponding MoViD outputs (bottom row). In the MoViD visualizations, motion directionality is encoded according to the color wheel shown in the bottom-left inset. Notably, visualization of translational and rotational motion is shown as the quadruped rotates and climbs the stairs in the original video. The MoViD algorithm characterizes motion at the pixel-level in full resolution. The resulting motion templates serve as a compact, label-free representation for Physical AI and robotic perception tasks, including anomaly detection, object association, throughput estimation, and long-term system monitoring.
The remainder of this paper presents the theoretical underpinnings of MoViD and demonstrates how the technique applies optical physics to motion analysis. We first delve into the physical equations that govern chromatic dispersion before outlining how these equations drive robust and interpretable motion detection and classification. We then describe how the approach naturally extends to motion magnification. By bridging established physical theory with modern computer vision needs, MoViD aims to provide a flexible, unified solution for motion analysis across a variety of domains.
Theory
MoViD’s Origins in Photonic Time Stretch
The physical processes and intuition underlying the MoViD algorithm are rooted in the photonic time stretch technique, a hardware approach for capturing ultrafast, non-repetitive events [17]. The essence of this method is to modulate time series and other data onto the spectrum of an optical carrier and transform it via spectral phase modulation realized using chromatic (group velocity) dispersion (these are the same steps utilized in the MoViD algorithm, as we will see later). In particular, a dispersion profile with quadratic dependence on frequency produces linear time dilation while higher order dependencies lead to engineered time warping. Over the past two decades, the time stretch framework has become a leading approach to ultrafast single-shot measurements and has spawned numerous variants across spectroscopy, imaging, vibrometry, velocimetry, radar, and lidar [18]. It has enabled the discovery of optical rogue waves; the first real-time observations of soliton-molecule dynamics; insights into the birth of laser mode-locking; direct measurements of relativistic electron beams in accelerators; and the capture of transient chemical dynamics during combustion [18].
Time-stretch imaging extends these capabilities to high-throughput, blur-free quantitative phase imaging and flow cytometry, while time-stretch lidar achieves record-speed four-dimensional (4D) imaging [19].
MoViD adopts the same principle as photonic time stretch but in the numerical domain. Rather than imposing a spectral phase using chromatic dispersion in the physical domain, MoViD applies a virtual spectral phase in the Fourier domain on the time series representing each pixel’s intensity in the video. Motion of an object through a pixel broadens the time series spectrum, and group delay dispersion induces a temporal phase that exists only if there is motion. In this way, MoViD mirrors the time-stretch pipeline while remaining lightweight and programmable in the digital domain.
Reimagining Video as a 2D Array of Temporally Modulated Waveforms
Departing from the conventional frame-based perspective for video analysis wherein video is considered a collection of two-dimensional arrays in time, MoViD considers video data as a collection of single-pixel time series arranged across two spatial dimensions. We define a single-pixel time series as the intensity recorded at a specific discrete pixel location across time, which we treat as a one-dimensional “lightfield”. As motion occurs in the broader scene, a single-pixel in space experiences temporal pulses of variation. Variation in a lightfield corresponds to a spectral bandwidth, as the bandwidth is proportional to the magnitude of the change that a time series experiences. This phenomenon is described in Figure 2. The core conceptual flow is outlined here:

Figure 2. Operating and physical principles of MoViD. Together, panels (a–c) illustrate how MoViD converts intensity variations to spectral phase kinematics, enabling pixel-level motion detection. (a) Conceptual illustration: two objects that move at different velocities (blue = fast, red = slow) modulate the blue and red pixels time series via their motion. The gray pixel remains static and therefore produces a constant pixel time series. (b) Each time series is modulated via the moving object. Faster motion generates a narrower intensity pulse and broader spectrum. Virtual chromatic dispersion encodes motion in spectral phase. The broader spectrum is stretched more strongly in time, producing a larger temporal-phase modulation ; conversely, slower motion yields a narrower bandwidth and a weaker phase response. The constant gray pixel time series has essentially zero temporal bandwidth and therefore produces zero phixel response. Spectral power remains unchanged. (c) Without dispersion, the intensity pulse width, spectral power, and temporal phase are unaltered, so motion cannot be discriminated.
We re-imagine video as a 2D array of time series signals, each corresponding to a single pixel in the video.
The time series signals are modulated by motion of an object through that pixel, as shown in Figure 2. The modulation of the time series broadens its spectral bandwidth as dictated by the Fourier transform.
When the time series is subjected to chromatic (group delay) dispersion, it experiences a time-dependent phase, i.e. a chirp. The amount of induced phase is proportional to the spectral bandwidth.
This phase is the signature of motion experienced at that pixel and is at the heart of the proposed motion detection technique.
In the absence of motion, there will be no spectral broadening and hence no temporal phase.
This process transforms the real-valued single-pixel time series into a complex, phase-modulated output. Examining the temporal phase of the dispersed signal provides a measure of the motion occurring at each pixel location over time. We introduce the term “phixel” to describe this temporal phase at each pixel location. This methodology allows for a detailed and nuanced understanding of temporal changes at the pixel level. Chromatic dispersion is numerically implemented via a Fourier transform followed by multiplication by a spectral phase kernel.
Through this process, signals with sharper intensity changes (higher bandwidth) experience more aggressive temporal stretching. Conversely, constant signals undergo no stretching at all. This change-sensitive operation is conceptually analogous to biological visual attention, in which motion and other salient changes receive prioritized processing [1, 3]. This process reveals a natural mapping between signal change and signal stretch, producing a bandwidth-dependent motion response: the greater the motion in the original signal, the greater the chirp. This fundamental relationship will be mathematically formalized below.
Group Velocity Dispersion Converts Motion to Chirp
To formalize the mapping of motion to chirp, we explore the propagation of an initially real-valued Gaussian signal through a linear dispersive medium. In this case, the Gaussian pulse in the pixel time series is caused by motion of an object through that pixel, as described in Figure 2. For a given 1D Gaussian temporal pulse with standard deviation , propagation in the absence of attenuation and nonlinearity is governed by the following partial differential equation:
The different order contributions correspond to dispersive modes [20], which represent the expansion terms in the Taylor series of the mode-propagation constant . We can more elegantly describe this transformation as a phase filter in the frequency domain defined by a spectral phase kernel
where represents the frequency with carrier frequency and represents the spectral transfer function of the dispersive medium. This is a material property that is defined by the frequency-dependent dielectric function of the medium. Considering only second-order dispersion , Equation (1) is trivially solved in the Fourier domain. After propagation over length , the pulse acquires a temporal phase given by,
where and we assume . Given this temporal phase, the instantaneous frequency, or chirp of the system, is given by
Algorithm 1: MoViD: Motion Detection via Virtual Dispersion and coherent Detection
Data: $E_i[n,m,t]$, input intensity video
Result: $\psi[n,m,t]$, phixel motion video
Parameters: $S$, $\phi[\omega]$
with $S \le 2\pi$
$H[\omega] \leftarrow \exp(-i \cdot S \cdot \phi[\omega])$
for $n \in \{1,\ldots,N\}$ do
for $m \in \{1,\ldots,M\}$ do
$E_i[t] \leftarrow E_i[n,m,t]$
$\widetilde{E}_i[\omega] \leftarrow \mathrm{FFT}\{E_i[t]\}$
$\widetilde{E}_o[\omega] \leftarrow \widetilde{E}_i[\omega] \cdot H[\omega]$
$E_o[t] \leftarrow \mathrm{IFFT}\{\widetilde{E}_o[\omega]\}$
$\psi[n,m,t] \leftarrow \tan^{-1}\left(\frac{\operatorname{Im}\{E_o[t]\}}{\operatorname{Re}\{E_o[t]\}}\right)$
end
end
The chirp in the pixel time series is highly dependent on , which is inversely proportional to the velocity of the object that passed through the pixel. In other words, the chirp is proportional to the object velocity. When approaches infinity, which corresponds to the case of no motion, the chirp vanishes. Conversely, as approaches 0, indicating ultrafast motion, the chirp grows infinitely large. The output chirp therefore acts as a motion detector, wherein motion is mapped to the instantaneous frequency of the pixel time series. This mapping is controlled by the applied spectral phase profile, whose shape determines how the temporal spectrum is converted into a complex phase response. Concrete kernel examples and their mathematical constraints are provided in the Methods. In particular, the section 6.2.1 of Methods describes important symmetry requirements related to the Hermitian properties. As discussed later, by designing the dispersive properties we can look for and classify specific spatio-temporal features in the video.
Figure 3 presents the algorithmic flow of MoViD for motion detection. Each pixel of the input video is treated independently as a one-dimensional temporal signal and passed through a virtual chromatic-dispersion operator defined by a user-specified spectral phase kernel. Coherent detection of the resulting complex signal yields a pixel-wise phase field that directly encodes motion. This formulation enables dense, full-resolution motion extraction without spatial filtering, training data, or explicit scene modeling, producing a compact motion representation that preserves fine spatial structure.

Figure 3. Algorithmic Flow of MoViD. Pseudocode of MoViD and block diagram of the physical pipeline. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by coherent detection to extract the phixel phase map that forms the output motion video. Inputs to the algorithm are a single-channel input video as data and parameters phase strength and normalized spectral phase kernel . Performed independently for every pixel: a 1-D FFT applies a user-defined spectral phase ; an inverse FFT returns the dispersed signal; coherent phase detection yields the phixel field .
Dielectric Function as a Template for Motion Filtering
The mapping of motion to temporal phase is governed by the dielectric dispersion function, which serves as a bridge between the temporal phase modulation and the material-like response of the system. Although our prior analysis focuses on the second-order dispersion term to capture temporal group velocity dispersion (GVD), the general framework naturally admits higher-order dispersive modes that define arbitrary frequency-dependent phase relationships.
In our virtualization scheme, we employ a spectral phase kernel as an intermediary for the equivalent dielectric function of a medium applied through the phase-only transfer function , where controls the accumulated phase strength. For kernels that admit a physical interpretation, the dielectric function can be inverted directly from the chosen spectral phase kernel, providing a direct link between computational filters and their physical analogs. The detailed mathematical derivation of this relationship is provided in the Supplementary Information (SI), where the mapping is shown to follow from equating the spectral phase accumulated through propagation in a dispersive medium with the spectral phase kernel. The resulting expression for the relative permittivity is given by
where is the spectral phase strength, a scaling term that determines the strength of the dispersion, is the speed of light in vacuum, is the propagation length over which the phase accumulates, is the angular frequency of the temporal signal, and is the user-defined spectral phase kernel applied in the frequency domain to achieve motion filtering.
Consequently, the spectral phase kernel serves as the motion-filtering template in MoViD, while the equivalent dielectric response provides a physical analogy for kernels that satisfy the relevant material constraints. Concrete kernel implementations and physical constraints are described below and in the Methods.
Spectral Phase Considerations in the Near- and Far-Field
A key point of the MoViD framework is that motion selectivity is not imposed primarily by one particular kernel shape or hand-tuned thresholds. Instead, it follows from the underlying physics of the transformation. The framework therefore works with a broad family of dispersive phase profiles, although the kernel shape influences which temporal or spatiotemporal motion characteristics receive higher attention.
The spectral phase is written as , where is the spectral phase strength. In this work, distinguishes two nominal operating regimes. For motion detection and classification, we use , corresponding to a “nominal near-field” regime. In this regime, the virtual dispersion modifies the complex phase of the signal, but the temporal intensity envelope is not meaningfully reshaped. The motion-dependent phase is read out directly as the pixel field with no required unwrapping. Further discussion is provided in [21]. This differs from conventional (incoherent) photonic time stretch, which typically operates in the temporal far field where stationary phase approximation is satisfied. Here, a stronger dispersion produces a substantial change in the temporal intensity envelope. In this regime, greater accumulated phase allows small motion-induced phase variations to be amplified and visualized. In this far-field operating regime where , MoViD performs motion magnification, amplifying imperceptible changes in video. Thus, the same virtual-dispersion operator supports phase-domain motion detection at lower and intensity-domain motion magnification at higher .
Spatiotemporal Phase Kinematics for Motion Classification
While the single-pixel perspective yields a mapping of intensity change to phase, incorporation of spatial dependence into our spectral phase kernels generalizes the framework beyond the purely temporal domain. An incoming video signal as a two-dimensional lightfield propagating in time has corresponding spatial and temporal frequency values defined by . The incoming lightfield is a real-valued sequence of two-dimensional intensity signals .
When motion occurs within this signal, it manifests as propagation across the transverse spatial axes, and , of the lightfield. This propagation is a superposition of many spatiotemporal traveling waves, each representing a specific position in Fourier space. The unique combination of these spectral components results in a distinct motion signature. Much of this motion signature is contained within the spectral phase of the signal, and by modifying and filtering by spectral phase profile we can adjust the spectral phase kinematics of the data, thereby detecting or even magnifying motion on the pixel-level. In other words, the dispersion-induced spatiotemporal phase specified by the applied spectral phase template probes specific patterns of motion that produce a MoViD output. By defining spectral phase kernels as functions in this space, we can stretch specific portions of the incident signal associated with a given motion class. Our spatiotemporal spectral phase kernels are therefore crafted as motion templates, motion feature extractors that are designed to activate, i.e. impart spatiotemporal chirp, for specific classes of motion.
The chirped output, the complex-valued dispersed signal , contains a spatiotemporal phase produced by the applied motion template. A spatiotemporal phase is produced when the spatiotemporal motion in the video matches that of the applied template. The process decomposes the motion within the lightfield into its primitive spatiotemporal components, enabling efficient filtering and classification for motion characteristics such as direction, magnitude, and periodicity. The explicit angular and speed spectral phase kernels are provided in the Methods.
Broadband Nature of Time Stretch Eliminates Channelization and Reconstruction
Given its basis in spectral phase modulation and measurement, the MoViD algorithm also offers a direct approach for motion magnification – the amplification of minor shifts in video that are imperceptible to the human eye. Traditional phase-based motion magnification methods [6] rely on the Fourier shift theorem, which states that a spatial shift in the time domain corresponds to a phase shift in the frequency domain. Such approaches first decompose the input signal using complex steerable pyramids, which break down the image into multiple frequency bands and orientations. This decomposition is necessary because the Fourier shift theorem is fundamentally constrained to single-tone signals, which require the video to be processed in narrow frequency bands. This constraint leads to significant computational overhead, band-limiting artifacts, and reduced spatial resolution.
MoViD introduces a fundamentally different paradigm through its broadband temporal dispersion approach. Rather than requiring complex decomposition into narrow frequency bands, MoViD operates directly on the broadband signal. This approach enables simultaneous processing across all frequencies, preserving the full spectral content of the motion. Figure 4 illustrates the corresponding MoViD formulation for motion magnification. As in the detection case, each pixel time series is independently dispersed using the same spectral phase framework; however, the quadrature component is selectively amplified before recombination with the in-phase signal. The resulting magnitude encodes motion-magnified intensity while maintaining spatial alignment with the original video. This per-pixel formulation enables controlled motion amplification at full resolution, without spatial pyramids or scale-specific tuning, allowing subtle motions to be enhanced while preserving fine image detail.

Figure 4. Algorithmic Flow of MoViD Motion Magnification. Pseudocode of MoViD motion magnification. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by magnification of quadrature information. After summation with the in-phase component, the magnitude of the signal serves as the output. Inputs to the algorithm are an input video as data and parameters phase strength which should be greater than , gain term , and normalized spectral phase kernel .
Figure 5 contrasts the MoViD processing pipeline with conventional motion-magnification approaches based on steerable spatial pyramids. Existing methods require multi-scale decomposition, phase-specific temporal filtering, spatial Fourier shifting, and reconstruction, resulting in multiple computationally expensive serial passes over the video. In contrast, MoViD replaces this workflow with a single virtual chromatic-dispersion operation followed by coherent detection, producing in-phase and quadrature channels that are combined to form the output. By eliminating pyramid decomposition and reconstruction, MoViD collapses the pipeline to two lightweight stages, significantly reducing computational complexity while preserving full spatial resolution.

Figure 5. Algorithmic Block Diagram of MoViD vs. Existing Motion-Magnification Pipelines. (a) Current state-of-the-art (steerable-pyramid) workflow. Video frames are first decomposed into a multi-scale spatial pyramid; every spatial bin then undergoes phase-specific temporal filtering, spatial Fourier shifting, and a final pyramid reconstruction, resulting in multiple serial passes over the data cube and high memory traffic. (b) MoViD processing chain. A single broadband chromatic-dispersion operator generates in-phase (I) and quadrature (Q) channels; coherent gain is applied to the Q-channel and the two streams are summed to produce the output video. Eliminating pyramid decomposition/reconstruction collapses the pipeline to two lightweight stages, greatly reducing computational complexity.
By leveraging spectral phase modulation in the virtual dispersion domain, MoViD achieves motion magnification that is extremely computationally efficient while maintaining high spatial and temporal resolution. The elimination of the complex steerable pyramid stage significantly reduces processing requirements, making real-time applications possible even at high resolutions, while still providing precise control over which motion signatures to amplify through tailored spectral phase kernels.
Results
Pixel-Level Motion Segmentation
A core advantage of the MoViD algorithm is its ability to isolate and represent motion at the resolution of individual pixels while preserving fine-grained structural detail. In Figure 6, we demonstrate MoViD’s performance in a real-world industrial conveyor sorting environment, where only dynamically changing regions are retained and static background structure is suppressed. Despite operating without any training data, MoViD produces high-contrast motion representations of moving packages while preserving subtle object details at the pixel level, including package edges and printed barcodes. This level of detail retention enables precise object-level segmentation and tracking directly in the motion domain, even in visually cluttered scenes. By effectively acting as an intrinsic, physics-driven attention mechanism, MoViD supports downstream tasks such as automated inspection, sorting, throughput estimation, and long-term system monitoring without explicit scene modeling or large labeled datasets.

Figure 6. Motion Segmentation Capabilities of MoViD in a Real-World Industrial Sorting Scenario. Three frames from an overhead camera of a manufacturing conveyor sorting system (top row) and the corresponding MoViD outputs (middle row), demonstrating single-pixel-level motion segmentation. MoViD isolates only dynamically changing regions while suppressing any static background, yielding high-contrast representations of moving packages. The highlighted zoom-in (bottom) shows that fine object details are preserved by MoViD, including package edges and printed barcodes, which are clearly visible in the motion domain. This capability enables precise object-level segmentation, sorting, and tracking for industrial inspection, sorting, and automation tasks.
To quantify foreground-motion localization, we evaluated MoViD on all four sequences in the baseline category of CDnet2014: PETS2006, Highway, Office, and Pedestrians [22]. We compared MoViD with KNN and MOG2 background subtraction as they process standard intensity video without supervised training, semantic object labels, or scene-specific learned models, and are intended for computationally efficient motion localization. As shown in Table 1, MoViD achieved the highest macro-averaged F1 score and IoU among the methods tested. Details are provided in the Supplemental Information section S1.
| Method | F1 score | IoU |
| KNN | 0.772 | 0.649 |
| MOG2 | 0.688 | 0.557 |
| MoViD | 0.821 | 0.707 |
Table 1. Motion Localization on the CDnet2014 Baseline Category. Mean F1 score and IoU are unweighted averages across the four sequence-level values. Higher values are better. Bold values mark the best result.
Classifying Pixel-Level Motion by Direction and Magnitude
Figure 7 demonstrates MoViD’s ability to segment and classify motion based on user-defined templates encoding specific directions or magnitude ranges. In this example, the middle row contains the color-coded directionality of motion while the bottom row contains the color-coded magnitude of velocity. As the turbine blades rotate, the local motion direction continuously changes, causing the color-coded output to transition in accordance with the prevailing motion orientation. To filter for this directionality, each MoViD motion filter corresponds to a “virtual dielectric” whose dispersive properties target a particular class of spatiotemporal features.

Figure 7. Directional and magnitude-based motion classification demonstrated on wind turbine footage. (a) Original video frames showing turbine rotation. (b) Directional classification of motion components where color represents directionality. (c) Speed-based classification where colors represent different velocity ranges, revealing how blade tips move faster than regions closer to the hub.
Beyond direction, MoViD can isolate motion according to magnitude. In the bottom row of Figure 7, regions of differing speeds are encoded using distinct color channels. Given the turbine is rotating at a constant angular velocity, the different portions of the turbine blade will travel at different linear velocities given their distances from the axis of rotation. MoViD reveals this difference in linear velocity, encoding the linear speed of the object in color. By designing phase kernels that amplify or attenuate certain ranges of velocity, MoViD becomes a highly selective motion detector capable of parsing out subtle variations within a single scene.
Comparative Performance Against Optical Flow
To highlight MoViD’s capability to capture motion details, Figure 8 contrasts two common dense optical flow algorithms with MoViD for the rotating turbine example.

Figure 8. Comparison with Traditional and State-of-the-Art Optical Flow. The left result is generated via the classical Farnebäck Method [23], the center is generated with the learning-based approach RAFT [11], and the right is MoViD. Whereas the classical approach is too coarse and low-resolution, the learning-based approach introduces artifacts in the background. MoViD shows the highest resolution result. Frame-per-second metrics are shown for each approach, with MoViD running at double the traditional approach and an order of magnitude faster than the deep learning approach.
The two approaches, the Farnebäck algorithm and RAFT, represent two of the most commonly used dense optical flow approaches from the traditional and deep-learning families, respectively.
The Farnebäck approach, while lightweight, relies on extensive downsampling to provide accurate optical flow. In doing so, it loses out on a great deal of detail in high definition scenes and can also exhibit noisy motion outputs, as can be seen in the magnified inset. Alternatively, RAFT as a data-driven approach provides much higher resolution optical flow, but with the added cost of potential hallucination. It can be seen in this example that this approach leads to background artifacts that are not tied to true motion. These hallucinations and artifacts are an added risk in many data-driven approaches.
In contrast, MoViD yields crisp segmentation and motion boundaries, even for subtle changes, without requiring training or parameter tuning. Regions with minimal motion remain devoid of phase artifacts, whereas regions undergoing large or abrupt motion exhibit correspondingly higher phixel magnitudes. In Figure 8, MoViD retains the shape of the turbine segment with high fidelity and low noise owing to its pixel-level resolution.
Real-Time High-Definition Motion Magnification
A distinctive feature of MoViD is its ability to amplify very small, often imperceptible, motions and vibrations in a scene. By tailoring the dispersion kernel to respond strongly to a specific frequency band corresponding to small displacements or periodic oscillations, MoViD can “stretch” faint signals in time and render them visible. Mathematically, the amplitude of the phase pixel (phixel) is increased. This “phixel amplification” is demonstrated in the Supplementary Information videos, where minor vibrations on thin structures (e.g., a swaying tower or a rotating shaft with small-amplitude oscillations) become perceptible.
Notably, owing to its broadband nature, MoViD performs motion magnification in real-time at up to 8K resolution. In comparison, common phase-based [6] and deep-learning-based approaches [13] are orders of magnitude slower. To characterize this speed comparison, we design a dataset composed of six motion amplification videos resized to seven different resolutions. We then compare the runtime of each approach over a collection of frames for each video and average to compute their speed in frames per second. Speed results from a phase-based approach, a deep learning approach, and MoViD are shown in Figure 9, with comparisons on CPU and GPU hardware. MoViD is over two orders of magnitude faster than the comparative approaches for motion magnification and capable of running at 874 frames per second at 1080p resolution.

Figure 9. Runtime Advantage of MoViD over Existing Motion Magnification Pipelines. (Top) Runtime comparison (FPS) across various resolution scales up to 8K resolution. (Bottom) Visualization of runtime comparison (FPS) at 1080p resolution. Throughput benchmarking on an Intel i9-12900K CPU and NVIDIA TITAN RTX GPU. Across seven resolutions (240p → 8K), MoViD vastly outperforms both the steerable-pyramid CPU baseline [6] and a learning-based GPU baseline [13]. At 1080p, MoViD reaches 21 FPS on CPU (vs. 0.09 FPS) and 874 FPS on GPU (vs. 6.2 FPS); at 8K it still sustains 68 FPS on GPU, whereas competing methods fail to achieve real-time performance. These gains exceed two orders of magnitude at every tested resolution and stem directly from the streamlined architecture. Results represented by a × correspond to the situation in which the algorithm is not able to complete the task on the data given memory or timing constraints of the machine.
| Resolution | CPU | GPU | ||
| Resolution | Pyramid-Based | MoViD | Learning-Based | MoViD |
| 426 × 240 | 1.84 | 443.8 | 125.2 | 20,981 |
| 640 × 360 | 0.88 | 200.3 | 56.2 | 9,784 |
| 854 × 480 | 0.44 | 113.7 | 31.3 | 4,285 |
| 1280 × 720 (HD) | 0.21 | 48.6 | 14.1 | 1,932 |
| 1920 × 1080 (Full HD) | 0.09 | 21.2 | 6.2 | 874 |
| 3840 × 2160 (4K) | × | 4.1 | 1.52 | 194 |
| 7680 × 4320 (8K) | × | 1.2 | × | 68 |
Table 2. Performance comparison across motion magnification methods and hardware (frames per second).
Discussion
Application in Low-latency Motion Analytics in High Resolution Video
MoViD is well positioned for monitoring and autonomous-perception applications that require wide-field coverage, fine spatial detail, and high temporal sampling under latency, bandwidth, or onboard-computation constraints. This regime arises in civil-infrastructure monitoring, industrial automation, and robotics, where prior work has demonstrated video-based vibration measurement, drone-based displacement sensing, and low-latency visual representations for resource-constrained navigation [24, 25, 26, 27, 28].
MoViD represents temporal intensity modulation as a full-resolution phixel field. In the conveyor example (Figure 6), this highlights moving packages and operators while suppressing much of the static production line. Because the output indicates where motion occurs rather than assigning semantic identities, it can serve as a pre-semantic input for container tracking, counting, throughput estimation, anomaly detection, or updating a digital-twin [29]. With such front-end video processing, subsequent inference can focus on dynamically active regions. In high resolution videos (e.g. 8k) this can unlock significant computation resources for other tasks.
The motion-magnification mode addresses small structural displacements that would otherwise go undetected. As shown in the Supplementary Video, the virtual-dispersion kernel can amplify minute vibrations, motivating use in non-contact screening of bridges, buildings, and industrial machinery. In this setting, MoViD should be viewed as a motion-localization and visualization framework rather than a calibrated structural-diagnostic system; deployment will require validation under camera motion, illumination changes, and environmental loading [30].
Limitations
Several limitations follow directly from MoViD’s per-pixel formulation. Regions that are textureless or locally uniform in image coordinates may produce weak phixel responses even when they are semantically part of a moving object. The rotating turbine hub in Figure 7 illustrates this limitation. Near the center, the linear velocity and hence the lateral displacement in the image-plane is small, so the local pixel time series has little spectral broadening and produces a weak phixel response. Additionally, because MoViD operates in image coordinates, camera motion also contributes to the phixel response. In moving-camera scenes, MoViD measures motion relative to the camera unless global motion is compensated. Established solutions for camera motion compensation include digital image stabilization (DIS), homography- or feature-based registration, camera movement detection using an inertial measurement unit (IMU), or static background anchors [31, 32, 33].
Responsible Use and Dual-Use Considerations
MoViD is presented here as a motion-amplification and physical motion-analysis method, not as a system for identifying, re-identifying, tracking, or profiling individuals. The present work does not develop, evaluate, or deploy identity recognition, re-identification, biometric classification, person tracking, or behavioral profiling. The reported experiments did not involve any human-subject data collection.
Nonetheless, a fast, full-resolution motion-analysis method could be misused for non-consensual human tracking, surveillance, workplace monitoring, or other privacy-infringing applications. If future applications involve people, they should require appropriate institutional and ethical review and privacy safeguards. When used responsibly, such an ultrafast, high resolution motion analysis technique could unlock a wealth of new applications across the domains of Physical AI, robotics and structural health monitoring.
Conclusion
In this work, we have introduced MoViD, a novel physics-inspired algorithm for motion detection and classification in video sequences. Departing from traditional approaches and the prevailing deep learning paradigm, MoViD takes inspiration from the fundamental physics of temporal group velocity dispersion and coherent detection. By virtualizing these phenomena and mapping them to the digital domain, MoViD offers a unique perspective on motion analysis, interpreting video data as a spatiotemporal lightfield and operating in the spectral phase domain.
The core strength of MoViD lies in its ability to leverage spectral phase kinematics, a qualitatively new representation that captures the intrinsic dynamics of motion at the pixel level. Through the application of tailored spectral phase kernels, or “motion templates,” MoViD can effectively classify motion signatures based on attributes such as direction, magnitude, and higher-order dynamics. This approach not only provides robust and interpretable motion segmentation but also enables hierarchical motion classification, paving the way for a wide range of applications in areas such as object tracking, activity recognition, traffic analysis, and video summarization.
References
- [1]Franconeri, S. L. & Simons, D. J. Moving and looming stimuli capture attention. Perception & Psychophysics 65, 999–1010 (2003).pubmed.ncbi.nlm.nih.gov/14674628
- [2]Eagleman, D. M. Human time perception and its illusions. Current opinion in neurobiology 18, 131–136 (2008).DOI
- [3]Bruce, N. D. & Tsotsos, J. K. Saliency, attention, and visual search: An information theoretic approach. Journal of vision 9, 5–5 (2009).DOI
- [4]Horn, B. K. P. & Schunck, B. G. Determining optical flow. Artificial Intelligence 17, 185–203 (1981).
- [5]Wu, H.-Y. et al. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics 31 (2012).DOI
- [6]Wadhwa, N., Rubinstein, M., Durand, F. & Freeman, W. T. Phase-based video motion processing. ACM Transactions on Graphics 32 (2013).DOI
- [7]Fuchs, M. et al. Real-time temporal shaping of high-speed video streams. Computers & Graphics 34, 575–584 (2010).
- [8]Becker, A., Erbach, J., Narnhofer, D. & Schindler, K. Continuous space-time video super-resolution with 3d fourier fields. International Conference on Learning Representations (2026). URL https://openreview.net/forum?id=bLmImy7g1w.arxiv.org/abs/2509.26325
- [9]Dosovitskiy, A. et al. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE international conference on computer vision, 2758–2766 (2015).arxiv.org/abs/1504.06852
- [10]Caldelli, R., Galteri, L., Amerini, I. & Bimbo, A. D. Optical flow based CNN for detection of unlearnt deepfake manipulations. Pattern Recognition Letters 146, 31–37 (2021).DOI
- [11]Teed, Z. & Deng, J. RAFT: Recurrent all-pairs field transforms for optical flow. European Conference on Computer Vision (ECCV), 402–419 (2020).arxiv.org/abs/2003.12039
- [12]Huang, Z., Shi, X., Zhang, C., Sun, N. & Li, H. Flowformer: A transformer architecture for optical flow. European Conference on Computer Vision (ECCV), 44–61 (2022).arxiv.org/abs/2203.16194
- [13]Oh, T.-H. et al. Learning-based video motion magnification. European Conference on Computer Vision (ECCV), 633–648 (2018).arxiv.org/abs/1804.02684
- [14]Zhou, Y., MacPhee, C., Suthar, M. & Jalali, B. PhyCV: The first physics-inspired computer vision library. arXiv preprint (2023).arxiv.org/abs/2301.12531
- [15]Jalali, B., Zhou, Y., Kadambi, A. & Roychowdhury, V. P. Physics-AI symbiosis. Machine Learning: Science and Technology 3, 041001 (2022).arxiv.org/abs/2109.05959
- [16]Asghari, M. H. & Jalali, B. Edge detection in digital images using dispersive phase stretch transform. International Journal of Biomedical Imaging (2015).DOI
- [17]Jalali, B., Coppinger, F. & Bhushan, A. Photonic time-stretch offers solution to ultrafast analog-to-digital conversion. Optics and Photonics News 9, 31–32 (1998).DOI
- [18]Mahjoubfar, A. et al. Time stretch and its applications. Nature Photonics 11, 341–351 (2017).DOI
- [19]Jiang, Y., Karpf, S. & Jalali, B. Time-stretch lidar as a spectrally scanned time-of-flight ranging camera. Nature photonics 14, 14–18 (2020).DOI
- [20]Jalali, B. & Mahjoubfar, A. Tailoring wideband signals with a photonic hardware accelerator. Proceedings of the IEEE 103, 1071–1086 (2015).DOI
- [21]Jalali, B. & MacPhee, C. Vevid: vision enhancement via virtual diffraction and coherent detection. eLight 2, 24 (2022).
- [22]Wang, Y. et al. Cdnet 2014: An expanded change detection benchmark dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 393–400 (2014).DOI
- [23]Farnebäck, G. Two-frame motion estimation based on polynomial expansion. Scandinavian Conference on Image Analysis (SCIA), 363–370 (2003).DOI
- [24]Chen, J. G. et al. Video camera–based vibration measurement for civil infrastructure applications. Journal of Infrastructure Systems 23, B4016013 (2017).DOI
- [25]Yang, Y. et al. Blind identification of full-field vibration modes from video measurements with phase-based video motion magnification. Mechanical Systems and Signal Processing 85, 567–590 (2017).DOI
- [26]Ri, S., Ye, J., Toyama, N. & Ogura, N. Drone-based displacement measurement of infrastructures utilizing phase information. Nature communications 15, 395 (2024).DOI
- [27]Schoepe, T. et al. Finding the gap: neuromorphic motion-vision in dense environments. Nature Communications 15, 817 (2024).DOI
- [28]Gehrig, D. & Scaramuzza, D. Low-latency automotive vision with event cameras. Nature 629, 1034–1040 (2024).DOI
- [29]Ward, R., Souliotantork, P., Finneran, S., Hughes, R. & Tiwari, A. Real-time vision-based multiple object tracking of a production process: Industrial digital twin case study. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 235, 1861–1872 (2021).DOI
- [30]Kim, J.-W., Choi, H.-W., Kim, S.-K. & Na, W. S. Review of image-processing-based technology for structural health monitoring of civil infrastructures. Journal of Imaging 10, 93 (2024).DOI
- [31]Roberto e Souza, M., Maia, H. d. A. & Pedrini, H. Survey on digital video stabilization: Concepts, methods, and challenges. ACM Computing Surveys (CSUR) 55, 1–37 (2022).DOI
- [32]Guilluy, W., Oudre, L. & Beghdadi, A. Video stabilization: Overview, challenges and perspectives. Signal Processing: Image Communication 90, 116015 (2021).DOI
- [33]Zitova, B. & Flusser, J. Image registration methods: a survey. Image and vision computing 21, 977–1000 (2003).DOI