DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization

Introduction

Temporal Action Localization (TAL) [1, 2, 3] aims to detect actions of interest in untrimmed videos by identifying their temporal boundaries and action categories, which is a fundamental task in video understanding. Fully-supervised TAL methods [1, 2, 3] have achieved remarkable performance with dense and precise temporal boundary annotations. However, obtaining such annotations is time-consuming. As a more annotation-efficient alternative, Weakly-Supervised Temporal Action Localization (WTAL) [4] has been widely studied, where only video-level class labels are available. Nevertheless, without instance-level temporal cues, WTAL methods often struggle to distinguish actions from background contexts. To better balance annotation cost and localization performance, SF-Net [5] introduces the Point-Supervised Temporal Action Localization (PTAL) task, where each action instance is annotated with only a single timestamp.

Existing PTAL methods [6, 7, 8, 9] commonly follow a Multiple Instance Learning (MIL)-based paradigm and derive localization cues mainly from frame- or snippet-level classification responses. Although these methods have achieved promising performance, response-driven localization tends to emphasize locally discriminative regions, while point annotations provide supervision only at isolated temporal positions, leaving the broader temporal evolution of an action instance largely unconstrained. As a result, existing approaches are effective at identifying salient action evidence, but remain limited in modeling how different action-related regions jointly form the structure of an action instance.

This limitation is particularly important because an action often evolves through multiple temporal stages exhibiting distinct motion patterns. For example, a long jump typically involves a run-up, take-off, and landing, which jointly constitute the complete action instance. As illustrated in Fig. 1, these different stages are manifested as multiple action-related regions distributed over time. We further observe that the feature characteristics, temporal extents, and relative configurations of these regions vary considerably across instances. Specifically, (1) different action-related regions exhibit distinct semantic and motion characteristics; (2) their temporal positions, durations, and relative intervals show substantial instance-dependent variations; and (3) these regions, together with their relative temporal configurations, form an instance-specific temporal structure that can be obscured when an action is modeled as a single entity. These observations suggest that effective PTAL should go beyond locally discriminative responses and explicitly model deformable temporal structures that can adapt to instance-specific variations.

To this end, we propose DeformStruct, a deformable structure modeling framework for point-supervised temporal action localization. Instead of directly treating an action as a single entity, DeformStruct progressively constructs the structure of each action instance through three stages: Structural Anchor Mining (SAM), Deformable Part Alignment (DPA), and Relational Structural Organization (RSO). The key idea is to first discover multiple action-related structural nodes from sparse supervision, then adapt their temporal supports to variations, and finally model their relative temporal relations. In this way, DeformStruct captures both multiple action-related evidence regions and their relational structure.

Specifically, we first introduce Structural Anchor Mining to discover multiple anchors from sparse action evidence. Starting from the annotated point, we identify additional highly responsive temporal regions and apply temporal suppression to avoid repeatedly selecting neighboring responses. The resulting anchors correspond to complementary action-related regions within the current instance, providing a sparse set of structural nodes for subsequent modeling. Compared with representing an action using only the annotated point, these anchors expose richer action evidence across the action instance.

Given these anchors, we further introduce Deformable Part Alignment to establish adaptive correspondences between latent structural parts and action anchors. Since instances of the same action category may vary considerably in duration, execution speed, and temporal composition, fixed temporal assignments are often inadequate. We therefore employ one-to-one deformation-aware matching to associate latent structural parts with the mined anchors, allowing their temporal supports to adapt to variations. This process produces a set of adaptively aligned parts that characterize the structural composition of each action instance. Building on these aligned parts, we introduce Relational Structural Organization to further model how they are organized within the instance. RSO maps the aligned part features into a shared structural coordinate space and derives their pairwise temporal relations through coordinate differences. By representing all parts within a common coordinate system, RSO captures their instance-specific relative temporal organization.

As a result, our approach shows significant improvements compared to previous PTAL methods and achieves competitive performance with state-of-the-art methods on four popular benchmark datasets: THUMOS14 [10], ActivityNet-v1.3 [11], GTEA [12] and BEOID [13].

To summarize, our contributions are as follows:

• We introduce DeformStruct, an instance-level deformable structure modeling framework for point-supervised temporal action localization. It models an action instance as a set of structurally related parts, providing a new perspective for exploiting sparse point supervision beyond independent classification responses.

• We propose Structural Anchor Mining and Deformable Part Alignment to discover complementary action-related evidence and adaptively instantiate latent parts at instance-specific temporal locations, accommodating structural variations without relying on fixed temporal partitions.

• We introduce Relational Structural Organization to model the relative temporal relations among aligned parts, allowing the model to characterize both multiple action-related evidence regions and their relational structure.

• Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate the effectiveness of DeformStruct. Detailed analyses further validate the benefits of anchor discovery, deformable alignment, and relational organization.

We note that a conference version of this work appeared at CVPR 2025 [14]. The conference version mainly exploited sparse point supervision through semantic query construction and optimal transport-based pseudo-segment supervision. Building upon this foundation, the present manuscript extends the original formulation from point-level semantic exploitation to deformable structure modeling, enabling the model to capture not only action-related evidence but also how multiple action-related parts are relatively organized within each action instance. The major extensions over the conference version are summarized as follows:

• We extend the modeling target from isolated point-level semantics to the structural organization of an entire action instance, explicitly considering multiple motion parts and their instance-dependent variations.

• We introduce a new structural modeling pipeline that discovers complementary action evidence, establishes adaptive part-to-evidence correspondences, and further captures the relations among these parts.

• We substantially expand the experimental study with structural analyses, mechanism-level ablations, robustness evaluations, and efficiency comparisons to better understand the effectiveness of the extended framework.

This paper is organized as follows: Section II briefly reviews related work. We present the technical details of the proposed method in Section III. Experimental results and discussions are presented in Section IV. Finally, we conclude in Section V.

Related Work

We briefly review related work in fully-supervised temporal action localization, weakly-supervised temporal action localization, point-supervised temporal action localization and part-based learning.

Fully-Supervised Temporal Action Localization. relies on accurately annotated temporal coordinates, assigning precise labels to each snippet during the training phase. This complete supervision enables high performance in both classification and localization tasks due to the availability of action boundaries. There are three primary approaches in this domain. (1) Proposal-based methods [15, 16, 17, 18, 19, 20] generate a temporal window that marks the start and end snippets of an action. These methods then refine the boundaries through a regression head to enhance accuracy. (2) Proposal-free methods [21, 22, 23, 24, 25, 26] focus on predicting the likelihood of each snippet containing an action. By identifying the snippet with the highest probability, they determine the action boundaries without relying on pre-defined proposals. (3) Transformer-based [27, 28, 29, 30] methods utilize the Transformer architecture to facilitate action localization, achieving impressive results on various Temporal Action Localization benchmarks. Despite the effectiveness of these approaches, the substantial effort required for annotating each snippet poses a significant challenge.

Weakly-Supervised Temporal Action Localization operates under a framework where only video-level labels are provided during training. This approach significantly reduces the annotation burden since frame-level labels are unnecessary. Existing methods can be broadly grouped into three categories. (1) The attention-based methods [31, 32, 33, 34] assess frame-level features to compute probabilities and utilize the resulting attention maps to guide the classification process. (2) The MIL-based methods [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45] generate frame-level class scores, i.e., the class activation sequence (CAS). These scores are then aggregated to form a video-level label, allowing for a holistic assessment of the action. (3) The graph-based methods [46, 47, 48, 49] model each video snippet as a node within a graph, where edges are weighted according to the similarity between snippets. In this framework, related snippets are drawn closer together in the feature space, while unrelated ones are pushed apart, enhancing the model’s ability to differentiate actions.

Point-Supervised Temporal Action Localization serves as a bridge between fully supervised and weakly supervised methods by requiring only a single label for each action instance. Miltosanti et al. [50] first proposed point-supervised learning in temporal action localization. SF-Net [5] expands the point annotation to adjacent snippets to obtain the pseudo-label for training. Ju et al. [51] propose a novel two-stage framework, which divides the entire video into multiple clips and sequentially conquers these clips. LACP [6] utilizes a greedy algorithm to find the optimal sequence of actions and learns the completeness of the action for the optimal sequence. HR-Pro [7] proposes that the high reliability of point annotations can be propagated at both snippet and instance levels. RCTSI [8] presents a novel proposal-level plug-in framework to relearn the aligned confidence of proposals generated by the base locators. SMBD [9] proposes to search for action boundaries to obtain complete action segments.

However, these methods mainly focus on propagating point-level responses or recovering temporal boundaries, while the internal organization of multiple action-related regions within an action instance remains insufficiently explored.

Deformable Alignment Mechanisms. Early works such as deformable template models [52] and manifold part-based models [53] achieved remarkable success in handling appearance variance by designing meaningful part patterns in a bottom-up manner. In particular, Pictorial Structure Models [54] and the seminal Deformable Part Models [55] explicitly captured the geometric arrangement of parts through a set of springs connecting part pairs, utilizing deformation costs to penalize spatial deviations from the root position. Recently, this structural paradigm has been reinvigorated within deep learning architectures. DCN [56] augmented convolutional kernels by learning spatial offsets to accommodate irregular object layouts. Deformable DETR [57] integrated this concept into Vision Transformers to enhance feature representativeness.

In this study, we explore deformable structural modeling under point-supervised temporal action localization. Unlike conventional part-based formulations that rely on predefined structures or stronger temporal supervision, our setting provides only a single annotated timestamp for each action instance. We therefore use instance-specific evidence anchors as temporal references and adaptively instantiate latent parts through deformation-aware matching. The resulting aligned parts are further organized through their relative temporal relations, enabling structural modeling without predefined partitions.

Proposed Method

Preliminary

Problem Definition. Point-supervised temporal action localization (PTAL) aims to accurately retrieve the start and end frames of the action of interest in an untrimmed video with only a single frame annotation per action. Given a labeled video, each action is annotated with a timestamp $t$ and its corresponding action category $y$. The PTAL model outputs predictions for each action, represented as $(\hat{s}, \hat{e}, \hat{y}, \hat{p})$, where $\hat{s}$ and $\hat{e}$ are the predicted start and end times, $\hat{y}$ is the predicted action category, and $\hat{p}$ denotes its confidence score.

Feature Embedding. Following conventions [6], [7], we extract snippet-level features from consecutive fixed-length frames using a pre-trained model (i.e., I3D [58]). The feature representation for each video is formulated as $X \in\mathbb{R}^{T \times D}$, where $T$ is the number of snippets and $D$ is the feature dimension. The Transformer encoder maps the input features $X$ to context-enhanced representations $X_E \in\mathbb{R}^{T \times D}$.

Overview. We build upon our conference version [14] and introduce DeformStruct as a structural supervision branch on top of the encoded video representation $X_E$. Given $X_E$, DeformStruct progressively encourages the encoder to learn structure-aware representations through Structural Anchor Mining, Deformable Part Alignment, and Relational Structural Organization. The resulting structural modeling complements the original localization framework by capturing multiple action-related parts, adapting them to temporal variations, and modeling their relative temporal relations. For training supervision, we retain the pseudo-label generation mechanism from our conference version, where the generated pseudo segments are used to supervise the localization decoder. The overall framework is illustrated in Fig. 2.

DeformStruct

Point supervision provides only sparse local cues for each action instance, making it difficult to characterize its complete structure. In practice, an action instance often evolves through multiple temporal stages with distinct motion patterns. For example, a long jump typically involves the run-up, take-off, and landing stages, which jointly constitute the complete action instance. The temporal locations, durations, and relative configurations of these stages may vary considerably across different instances. To explicitly capture such structural variations, we propose DeformStruct, which models each action instance as a deformable composition of latent parts. DeformStruct progressively constructs the structure through three stages: Structural Anchor Mining, Deformable Part Alignment, and Relational Structural Organization. Structural Anchor Mining discovers multiple action-related anchors from sparse point supervision to provide structural evidence. Deformable Part Alignment adaptively associates latent parts with these anchors, allowing their temporal supports to accommodate variations across action instances. Relational Structural Organization further models the relative temporal relations among the aligned parts to characterize their structural configuration. The resulting structure is jointly optimized through the structural consistency objectives described below. These structural components jointly regularize the shared encoder representation, enabling the localization branch to benefit from instance-level structural cues.

1) Structural Anchor Mining: Traditional point-supervised paradigms are often constrained by “Single-frame View Bias”, where the feature at the sparse annotation point is treated as the sole training sample. Relying solely on such transient temporal snapshots inevitably overlooks diverse action-relevant patterns distributed along the temporal dimension, which is essential for comprehensively characterizing the internal temporal composition of the action. To rectify this limitation, we propose Structural Anchor Mining. Operating via a Query-and-Expand mechanism, we leverage the annotated point as a Semantic Query to explore the temporal interval bounded by its preceding and succeeding annotations, identifying latent regions that exhibit high semantic consistency with the annotated action. This effectively leverages the internal statistics of the video itself to expand the “single view” into a comprehensive set of candidate anchors. To prevent anchor redundancy (i.e., multiple anchors collapsing onto the same discriminative region), we further incorporate an iterative temporal NMS [59] mechanism. Specifically, let $\{t_g\}_{g=1}^{G}$ denote the temporally ordered point annotations in a video. For the current annotated point $t_g$, we extract its feature as the instance prototype $\mathbf{p}_{\mathrm{inst}} = \mathbf{X}_{E,t_g}$. To reduce interference from other action instances, anchor mining is restricted to the temporal interval bounded by its preceding and succeeding annotations: $\mathcal{R}_g = [t_{g-1}, t_{g+1}]$. For the first and last annotations, the corresponding video boundary is used when one neighboring annotation is unavailable.

To facilitate global semantic interaction between the prototype and the video representation, we construct a Semantic Resonance Map $\mathbf{S}_r \in\mathbb{R}^{T}$ by measuring the dot-product similarity between the prototype and the video feature sequence,

High activation values in $\mathbf{S}_r$ indicate that the snippets are semantically resonant with the annotated core. However, simply selecting the top-$K_a$ values often yields redundant anchors clustered around the most discriminative frames, failing to cover temporally distributed action-related evidence. To capture temporally diverse action-related evidence, we employ an NMS strategy to mine a temporally diverse candidate set $\mathcal{A}$.

Formally, we initialize $\mathbf{S}_r^{(0)} = \mathbf{S}_r$ and denote the number of candidate anchors by $K_a$. The mining process is conducted iteratively for $k = 1, \ldots, K_a$. In each step, we identify the timestamp $t_k$ with the maximum response and subsequently suppress its temporal neighborhood to encourage diversity,

where $\delta$ is a predefined suppression radius. By leveraging the internal self-similarity of the video, this strategy effectively mines semantically responsive anchors that span various temporal regions, preventing them from collapsing into a single action phase. These anchors serve as a solid foundation for subsequent deformable part alignment (§ 3-B2), enabling the model to perceive the action holistically rather than locally.

Deformable Part Alignment

Given the set of candidate anchors $\mathcal{A} = \{a_1, \ldots, a_{K_a}\}$ (as described in § 3-B1), where each anchor $a_k = \mathbf{X}_{E,t_k}$ represents the feature vector extracted at the mined timestamp $t_k$, we aim to model the instance-specific temporal structure of the action. Specifically, we formulate the action instance as a composition of $N_{\mathrm{part}}$ latent parts, which are then optimally aligned with the mined anchors under deformable geometric constraints.

We employ a set of learnable part-specific classifiers to produce response sequences for different latent parts. For the $i$-th latent part, we utilize a dedicated linear projection $\phi_i(\cdot)$ to generate the raw response sequence $\mathbf{S}_i \in\mathbb{R}^{T}$,

where $\mathbf{w}_i$ and $b_i$ are the learnable weight and bias for the $i$-th part classifier. The activation scores are then normalized via a sigmoid function $\sigma(\cdot)$.

To model the temporal deformation of actions, we introduce a deformation cost that penalizes parts for deviating from their reference anchors. For the $i$-th part and a candidate anchor $a_k \in\mathcal{A}$, the deformation penalty $\Psi_i$ at time $t$ is defined as a quadratic function of the temporal distance $d = |t-t_k|/T$,

where $\gamma_{i,1}$ and $\gamma_{i,2}$ are non-negative learnable stiffness parameters for the $i$-th part, parameterized as $\gamma_{i,r} = \operatorname{Softplus}(\tilde{\gamma}_{i,r})$, $r \in\{1,2\}$. This allows the model to adaptively control the penalty for temporal deviations from the matched anchor.

We then determine the alignment score $V_{i,k}$ for assigning the $i$-th part to the $k$-th anchor by maximizing the deformation-aware objective over the temporal dimension,

Here, $\lambda_{\mathrm{deform}}$ is a hyperparameter controlling the trade-off between classification score and temporal deformation penalty. This operation effectively searches for the optimal temporal position $t^*$ that balances semantic confidence and geometric consistency relative to the anchor.

Finally, to resolve ambiguities where multiple parts might compete for the same anchor, we formulate the assignment as a maximum weight bipartite matching problem. The optimal assignment $\hat{\pi}$ that maps each latent part to a unique anchor is obtained via the Hungarian algorithm,

The resulting assignment grounds the abstract latent parts onto concrete temporal regions according to their alignment quality.

Relational Structural Organization

Although deformable matching adaptively associates individual latent parts with temporal regions, these aligned parts are still modeled independently. An action instance is jointly characterized by multiple motion parts, whose relative temporal configuration provides complementary information beyond the evidence of each individual part. Therefore, we introduce Relational Structural Organization (RSO) to explicitly characterize the relative organization among the deformably aligned parts.

After Hungarian matching, the $i$-th latent part is assigned to an anchor and obtains its optimal response position $t_i^*$ through deformation-aware matching. We extract the encoder feature at this aligned position as,

where $\mathbf{X}_E \in\mathbb{R}^{T \times D}$ denotes the encoded video feature sequence, $T$ denotes the temporal length, and $D$ denotes the feature dimension. Accordingly, $\mathbf{z}_i \in\mathbb{R}^{D}$ represents the motion evidence captured by the $i$-th aligned part.

Directly predicting the relation of every part pair independently may produce mutually inconsistent pairwise relations. To obtain a unified relational representation, we instead map all aligned parts into a shared one-dimensional structural coordinate space. Specifically, the structural coordinate of the $i$-th part is formulated as,

where $g : \mathbb{R}^{D} \to\mathbb{R}$ denotes a lightweight two-layer MLP shared by all latent parts, and $\sigma(\cdot)$ denotes the sigmoid function. The resulting $q_i \in[0,1]$ serves as a latent structural coordinate that characterizes the relative position of the $i$-th part in the learned structural space. Importantly, the same mapping $g(\cdot)$ is applied to all parts, so that their structural coordinates are represented in a common coordinate system.

To characterize the temporal configuration of the aligned parts, we use their deformation-adjusted temporal positions as the structural reference. Specifically, the normalized temporal coordinate of the $i$-th aligned part is defined as,

where $t_i^*$ denotes the optimal temporal position obtained for the $i$-th latent part through deformation-aware alignment. The normalization reduces the influence of different video durations and provides a comparable temporal coordinate across action instances.

For each part pair $(i,j)$, we define the observed relative temporal relation as,

where the sign of $\Delta_{ij}$ indicates the relative temporal order between the two aligned parts, while its magnitude characterizes their normalized temporal separation. Since $t_i^*$ and $t_j^*$ are obtained through instance-specific deformable alignment, $\Delta_{ij}$ directly describes the relative temporal configuration of the instantiated parts without assuming a predefined ordering or fixed temporal partition.

Correspondingly, the relation predicted from the learned structural coordinates is defined as,

Rather than independently regressing every pairwise relation, this formulation derives all relations from the shared structural coordinates $\{q_i\}_{i=1}^{N_{\mathrm{part}}}$. Therefore, the predicted relations naturally satisfy the antisymmetric property,

and the transitive consistency,

These properties encourage different pairwise relations to form a coherent structural configuration. In this way, RSO maps the deformably aligned parts into a shared latent structural coordinate space and characterizes their organization through relative coordinate differences. Consequently, DPA determines where individual motion parts are grounded, while RSO further describes how these parts are organized relative to one another.

Structural Consistency Optimization

To jointly optimize the discovered action evidence and the deformably aligned parts, we introduce three complementary structural objectives. The part classification loss promotes part-specific discrimination, the anchor-prototype alignment loss preserves the semantic reliability of the mined anchors, and the relational consistency loss regularizes the relative temporal organization of the parts.

First, we enforce a Part Classification Loss to encourage each latent part to establish a distinctive correspondence with its matched structural anchor. Let $\hat{\pi}$ denote the optimal assignment obtained by bipartite matching, where $\hat{\pi}(i)$ indicates the anchor assigned to the $i$-th latent part. Since $V_{i,k}$ jointly measures the part response and temporal deformation with respect to the $k$-th anchor, we directly optimize the deformation-aware matching scores as follows,

This objective encourages the anchor assigned to the $i$-th part to exhibit higher deformation-aware compatibility with that part than with the other parts. During back-propagation, the discrete assignment $\hat{\pi}$ is treated as fixed, while the matching scores $V_{i,k}$ remain differentiable. Therefore, $\mathcal{L}_{\mathrm{part}}$ jointly optimizes the part classifiers and the learnable deformation stiffness.

Simultaneously, to guarantee that the selected anchors are semantically grounded to the action instance, we introduce an Anchor-Prototype Alignment Constraint. We minimize the distance between the prototype and the features of the anchors assigned to the latent parts. Formally, let $\mathbf{x}_{\mathrm{anchor}}^{(i)}$ denote the feature vector of the anchor matched to the $i$-th part. The alignment loss is formulated as follows,

This term encourages the mined anchors to remain semantically associated with the annotated action instance, providing reliable action-related evidence for deformable part alignment.

Finally, we impose a relational consistency constraint on the aligned parts. For each part pair $(i,j)$, the relative temporal relation $\Delta_{ij}$ defined in Eq. (11) is used as the structural supervision. Given the relation prediction $\widehat{\Delta}_{ij}$ produced by RSO, we define the relational consistency loss as,

By constructing the relational target directly from the deformable part positions, $\mathcal{L}_{\mathrm{rel}}$ encourages the learned structural coordinates to preserve the instance-specific organization of the aligned parts.

The final structural objective is formulated as,

where $\lambda_{\mathrm{part}}$, $\lambda_{\mathrm{align}}$, and $\lambda_{\mathrm{rel}}$ are the weighting parameters for part classification, anchor alignment, and relational consistency, respectively.

Query Reformation

Following our conference version [14], we retain the Query Reformation mechanism to provide semantic priors for decoder initialization under point supervision. Specifically, point-labeled features from all training videos are collected according to their action categories and clustered to obtain a set of semantic consensus prototypes. These prototypes are then projected into decoder queries, providing class-aware semantic cues that guide the localization process. Since the prototypes are constructed from sparse point annotations, this module mainly captures point-level semantic information and serves as the semantic initialization for the localization decoder.

Pseudo-label Generation via Optimal Transport

Following our conference version [14], we employ an Optimal Transport (OT)-based pseudo-label generator to provide segment-level supervision under point annotations. Specifically, annotated action points and mined background points are treated as suppliers, while unlabeled snippets serve as consumers. The transportation cost between supplier $i$ and consumer $j$ jointly considers semantic compatibility and temporal distance:

where $d^{s}_{i,j} = \lvert p^{s}_{i}(y_i) - p^{s}_{j}(y_i) \rvert$ measures their semantic discrepancy, $d^{p}_{i,j} = \lvert t(i)-t(j) \rvert$ denotes their temporal distance, and $T$ is the video length.

We then solve the OT problem as follows,

where $\mathbf{b}$ denotes the supplier capacities. As in [14], the capacities are dynamically refined according to the snippet assignments and their updated temporal centers. Finally, each consumer inherits the label of the supplier receiving the largest transport mass, and the resulting snippet-wise assignments are merged into pseudo temporal segments for training the localization model.

Training

Label Assignment. The predicted action set is denoted as $\hat{\Psi} = \{\hat{\phi}_n = (\hat{s}_n,\hat{e}_n,\hat{y}_n,\hat{p}_n)\}_{n=1}^{N_q}$, where $\hat{s}_n$ and $\hat{e}_n$ represent the start and end times of the $n$-th predicted action instance, while $\hat{p}_n$ and $\hat{y}_n$ denote the corresponding class probability distribution and predicted category, respectively. The pseudo-label set $\Psi= \{\phi_i = (s_i,e_i,y_i)\}_{i=1}^{N_q}$ includes a padded no-action placeholder, denoted by $\varnothing$. For a padded no-action label, $y_i = \varnothing$, and its temporal interval $(s_i,e_i)$ is ignored during temporal matching and regression. The cost associated with a permutation $\sigma\in\mathcal{S}_{N_q}$ of the query set is defined as follows,

where $y_i$ denotes the class label of the $i$-th pseudo-label, and $(s_i,e_i)$ denotes an OT-generated pseudo temporal segment rather than a ground-truth boundary annotation. Focal loss [60] is utilized as $\mathcal{L}_{\mathrm{cls}}$ following [6], [9], while $\mathcal{L}_{\mathrm{iou}}$ and $\mathcal{L}_{L_1}$ represent the temporal IoU loss and the $L_1$ distance, respectively. The bipartite matching between two sets aims to find the permutation $\sigma^*$ with the lowest cost,

where $\mathbb{I}(\cdot)$ denotes the indicator function. The classification cost is computed for all pseudo-labels, including the padded no-action labels, whereas the temporal IoU and $L_1$ costs are computed only for non-empty pseudo-labels. The weights $\alpha_{\mathrm{cls}}$, $\alpha_{\mathrm{iou}}$ and $\alpha_{L_1}$ of the matching costs are set to 2, 2, and 5, respectively, as in [61].

Loss Function. After obtaining the best permutation $\sigma^*$, the optimization goal can be expressed as follows,

where $\mathbb{I}(\cdot)$ is the indicator function and the weights $\lambda_{\mathrm{cls}}$, $\lambda_{\mathrm{iou}}$ and $\lambda_{L_1}$ are set to 2, 2 and 5, respectively, as in [61].

For the optimization of the pseudo-label generation process, we adopt the focal loss [60] to facilitate the training process. Let $\Omega= \Omega_{\mathrm{act}} \cup\Omega_{\mathrm{bg}}$ denote the supervised snippet set, where $\Omega_{\mathrm{act}}$ contains the annotated action snippets and $\Omega_{\mathrm{bg}}$ contains the selected background seeds. For each snippet $t \in\Omega$, let $y^p_t \in\{0,1,\ldots,N_c\}$ denote its pseudo-label, where $y^p_t = 0$ represents the background class and $y^p_t \in\{1,\ldots,N_c\}$ represents an action class. The snippet-level classification loss is defined as follows,

where $p^s_t(y^p_t)$ denotes the predicted probability that snippet $t$ belongs to its pseudo-labeled class $y^p_t$, and $\beta$ is the focusing factor, which is set to 2 following [60].

The overall loss function for training our model is as follows,

where $\lambda_{\mathrm{detr}}$, $\lambda_{\mathrm{act}}$ and $\lambda_{\mathrm{struct}}$ are the weighting parameters to balance the losses, which are set to 1, 0.5 and 1, respectively.

Experiments and Discussion

In this section, we first introduce four standard benchmarks, the evaluation metrics, and the implementation details. Then, we compare the proposed DeformStruct with the SOTA methods, followed by a set of ablation studies. Note that only point-level labels are leveraged to train the proposed DeformStruct.

Dataset and Evaluation

THUMOS14. The training data in THUMOS14 [10] is based on the UCF101 [70] action dataset. There are a total of 200 validation videos and 213 test videos that belong to 20 action classes. On average, each video in the dataset includes 15 action instances, making the task of recognizing all actions quite challenging. Similarly to [5], we use the validation set for training and the test set for evaluation.

ActivityNet-v1.3. The ActivityNet-v1.3 [11] dataset includes 200 different types of activities, totaling 849 hours of YouTube video. In terms of activity categories and video quantity, ActivityNet-v1.3 is currently the largest benchmark for temporal activity detection. ActivityNet-v1.3 contains a total of 19,994 untrimmed videos, divided into training, validation, and test subsets in a 2:1:1 ratio. Each activity category includes an average of 137 untrimmed videos, with each video containing an average of 1.41 activities.

GTEA. The GTEA [12] comprises 28 videos and contains seven different types of everyday activities. Each video lasts approximately one minute and contains around 20 annotated action instances. For our experiments, we used 21 videos for training and 7 for testing.

BEOID. The BEOID, as described in [13], comprises 58 videos representing 30 action classes, and each video in the dataset contains an average of 12.5 actions. Following [5], 80% of the videos are used for training, while the remaining 20% are reserved for testing.

Evaluation Metrics

Following previous PTAL works [5, 6, 7, 8, 9], [51], we adopt the standard evaluation protocol for all datasets. For THUMOS14, we report mAP at IoU thresholds from 0.3 to 0.7 and the average mAP over both [0.1 : 0.1 : 0.5] and [0.3 : 0.1 : 0.7]. For GTEA and BEOID, the average mAP is calculated over [0.1 : 0.1 : 0.7]. For ActivityNet-v1.3, we report mAP at 0.5, 0.75, and 0.95, together with the average over [0.5 : 0.05 : 0.95].

Implementation Details

We use a two-stream I3D network [58] pretrained on Kinetics-400 [58] as the feature extractor. We divide each video into non-overlapping 16-frame snippets and apply the TV-L1 algorithm [71] to extract optical flow. After obtaining the RGB and optical-flow features, we employ the two-stream fusion operation as described in [72] to integrate the RGB and optical-flow branches, resulting in a 2048-dimensional vector for each snippet. The number of non-overlapping snippets is denoted as $T$. For all datasets, we optimize our method using Adam [73] with a learning rate of $10^{-4}$ and a batch size of 8. The numbers of encoder and decoder layers, $L_E$ and $L_D$, are set to 4 and 6, respectively. The number of attention heads $M$ is set to 8, and the number of sampling points $K_s$ is set to 4. All results for our framework use the same settings as QROT [14], including I3D features, point-label protocol, data splits, and evaluation code. We adopt TE-TAD [61] as the base framework and adapt it to point supervision as our base model. The SOTA comparison on THUMOS14 is reported as mean $\pm$ standard deviation over three runs, whereas the ablation studies report the mean performance over three runs.

Method	mAP@tIoU (%)			AVG
	0.5	0.75	0.95	[0.5:0.05:0.95]
SF-Net [5]	–	–	–	–
Ju et al. [51]	–	–	–	–
LACP [6]	40.4	24.6	5.7	25.1
HR-Pro [7]	42.8	27.2	8.0	27.1
RCTSI [8]	–	–	–	–
SMBD [9]	43.2	27.8	8.3	27.5
QROT [14]	43.3	28.2	8.5	27.7
DeformStruct w/o RSO	43.8	28.6	8.9	28.1
DeformStruct (Ours)	44.1	29.2	9.3	28.5

Method	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	[0.1:0.1:0.7]
SF-Net [5]	37.9	19.3	11.9	31.0
Ju et al. [51]	38.3	21.9	18.1	33.7
LACP [6]	55.7	33.9	20.8	43.5
HR-Pro [7]	61.1	37.3	17.5	47.3
RCTSI [8]	60.9	39.5	16.6	49.0
SMBD [9]	61.3	41.1	14.2	47.4
QROT [14]	61.5	37.9	18.2	47.9
DeformStruct w/o RSO	64.2	40.1	18.5	49.1
DeformStruct (Ours)	64.7	40.9	19.3	49.8

Comparisons with the state-of-the-art methods

THUMOS14. Table I compares DeformStruct with fully-supervised, weakly-supervised, and point-supervised methods on THUMOS14. Under point supervision, our conference version QROT already achieves competitive localization performance. Building upon it, introducing deformable part modeling

Supervision	Method	0.3	0.4	0.5	0.6	0.7	[0.1:0.1:0.5]	[0.3:0.1:0.7]
Fully supervised	BMN [1] (ICCV’19)	56.0	47.4	38.8	29.7	20.5	–	38.5
Fully supervised	G-TAD [19] (CVPR’20)	54.5	47.6	40.2	30.8	23.4	–	39.3
Fully supervised	TCANet [62] (CVPR’21)	60.6	53.2	44.6	36.8	26.7	–	44.4
Fully supervised	React [3] (ECCV’22)	69.2	65.0	57.1	47.8	35.6	–	55.0
Fully supervised	ASL [63] (ICCV’23)	83.1	79.0	71.7	59.7	45.8	–	67.9
Fully supervised	MCBD [20] (TPAMI’23)	68.8	62.2	53.5	41.6	29.5	–	51.1
Fully supervised	TE-TAD [61] (CVPR’24)	83.3	78.4	71.3	60.7	45.6	–	67.9
Fully supervised	AD-IDP [64] (CVPR’24)	84.9	81.5	76.5	63.0	48.0	–	70.8
Weakly supervised	CMCS [37] (CVPR’19)	41.2	32.1	23.1	15.0	7.0	40.9	23.7
Weakly supervised	DGAM [65] (CVPR’20)	46.8	38.2	28.8	19.8	11.4	45.6	29.0
Weakly supervised	CoLA [66] (CVPR’21)	51.5	41.9	32.2	22.0	13.1	50.3	32.1
Weakly supervised	BackTAL [40] (TPAMI’21)	54.4	45.5	36.3	26.2	14.8	–	35.4
Weakly supervised	STCL-Net [39] (TPAMI’23)	58.2	49.7	41.8	28.7	16.0	57.9	38.9
Weakly supervised	DDG-Net [67] (ICCV’23)	58.2	49.0	41.4	27.6	14.8	57.8	38.2
Weakly supervised	PseudoFormer [68] (CVPR’25)	65.4	55.6	44.1	31.3	17.1	62.9	42.7
Weakly supervised	MLLM4WTAL [69] (CVPR’25)	61.8	52.3	43.0	30.8	16.6	60.2	40.9
Point supervised	SF-Net [5] (ECCV’20)	53.2	40.7	29.3	18.4	9.6	51.5	30.2
Point supervised	Ju et al. [51] (ICCV’21)	58.1	46.4	34.5	21.8	11.9	55.3	34.5
Point supervised	LACP [6] (ICCV’21)	63.3	55.2	43.9	33.3	20.8	61.6	43.3
Point supervised	HR-Pro [7] (AAAI’24)	74.3	64.3	52.2	39.8	24.8	71.6	51.1
Point supervised	RCTSI [8] (CVPR’24)	70.1	60.0	49.4	37.6	24.5	67.9	48.3
Point supervised	SMBD [9] (ECCV’24)	66.0	57.9	47.0	36.0	22.0	64.2	45.7
Point supervised	CRC [44] (TPAMI’25)	65.5	56.9	45.4	34.9	21.2	63.2	44.8
Point supervised	QROT [14] (CVPR’25)	73.1	64.4	54.3	41.3	27.4	72.3	52.1
Point supervised	DeformStruct w/o RSO	$75.6 \pm0.7$	$67.8 \pm0.8$	$57.4 \pm0.5$	$42.8 \pm0.4$	$27.8 \pm0.7$	$73.5 \pm0.6$	$54.3 \pm0.5$
Point supervised	DeformStruct (Ours)	$76.4 \pm0.5$	$68.6 \pm0.6$	$57.9 \pm0.8$	$43.2 \pm0.9$	$28.0 \pm0.4$	$74.1 \pm0.7$	$54.8 \pm0.8$

Method	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	[0.1:0.1:0.7]
SF-Net [5]	40.6	16.7	3.5	30.9
Ju et al. [51]	46.8	20.9	5.8	34.9
LACP [6]	61.4	42.7	25.1	51.8
HR-Pro [7]	72.1	55.3	26.1	59.4
RCTSI [8]	73.0	51.1	23.8	59.6
SMBD [9]	71.0	52.5	25.2	57.4
QROT [14]	73.0	55.9	26.7	59.9
DeformStruct w/o RSO	74.9	56.7	27.3	60.7
DeformStruct (Ours)	75.7	57.8	28.1	61.4

consistently improves the overall performance, demonstrating the effectiveness of capturing multiple action-related parts. Further incorporating RSO brings additional gains, indicating that modeling the relative organization among aligned parts provides complementary structural information beyond independent part alignment. Overall, DeformStruct achieves the best average performance among the compared point-supervised methods, while the results over three runs further characterize the stability of the model across different runs.

ActivityNet-v1.3. As shown in Table II, DeformStruct consistently outperforms previous point-supervised methods, including QROT, across all reported tIoU thresholds. Further incorporating RSO leads to additional improvements, indicating that relative temporal organization provides complementary cues for action localization.

GTEA. On GTEA, DeformStruct achieves the best average performance among the compared point-supervised methods, as reported in Table III. The gain over the variant without RSO further supports the usefulness of relational structure modeling for capturing the organization of aligned action parts.

BEOID. Table IV shows that DeformStruct improves upon existing PTAL approaches across the reported tIoU thresholds. Moreover, the full model consistently surpasses DeformStruct w/o RSO, suggesting that relational modeling contributes structural information beyond deformable part alignment. Overall, DeformStruct consistently outperforms state-of-the-art point-supervised methods across all four datasets, including

caption: Qualitative visualization of the DPA module on THUMOS14. The colored markers indicate the locations of anchors matched with latent action parts. The bars at the bottom indicate the ground-truth segments. Notably, the mined anchors align well with the ground-truth action intervals and capture complementary action-relevant regions distributed over time.

our conference version QROT [14]. These results demonstrate the general effectiveness of the proposed deformable structural modeling across diverse action datasets.

Ablation Study

In this subsection, to better analyze the contribution of each component, we conduct ablation studies on THUMOS14. For clarity, we highlight the conference version in green and the final setting used for comparison with SOTA methods in blue. Ablation study of different modules on THUMOS14. Table V first compares different pseudo-label generation strategies under the same DETR architecture. Specifically, NPE denotes no pseudo-label expansion, FW and GW represent fixed-window and Gaussian-window expansion, respectively. SF-Net follows the adaptive pseudo-label mining strategy of SF-Net [5], GS denotes greedy search in LACP [6], and OT denotes optimal transport. Among these strategies, OT achieves the best overall performance. Building upon OT, both QR and DPA improve localization performance, and their combination further raises the average mAP to 54.3%, showing their complementary benefits. Adding RSO further improves the average mAP to 54.8% and yields gains across all reported tIoU thresholds. These results verify that modeling the relative

Setting	QR	DPA	RSO	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
NPE	–	–	–	4.5	1.9	0.4	2.2
FW	–	–	–	35.9	13.9	2.4	16.4
GW	–	–	–	12.2	5.4	1.3	5.9
SF-Net	–	–	–	54.3	31.4	10.7	31.6
GS	–	–	–	71.9	50.6	25.0	49.4
OT	–	–	–	72.0	53.4	25.2	50.9
OT	–	–	–	72.0	53.4	25.2	50.9
OT	✓	–	–	73.1	54.3	27.4	52.1
OT	–	✓	–	74.9	56.7	22.8	53.0
OT	✓	✓	–	75.6	57.4	27.8	54.3
OT	✓	✓	✓	76.4	57.9	28.0	54.8

temporal organization among aligned parts provides additional structural cues beyond deformable part alignment.

Visualization of the Part-Anchor Alignment Score Matrix.

In Fig. 3, we provide a visualization of the Part-Anchor Alignment Scores in the DeformStruct module. The rows represent the learned deformable parts (Part 1 to Part 3), and the columns correspond to the candidate anchors mined from the video. The color intensity indicates the matching score (negative cost), where darker colors represent higher compatibility. The red boxes highlight the optimal assignment results obtained by the Hungarian algorithm. This matrix demonstrates how our model jointly considers appearance similarity and temporal deformation to align parts with anchors.

Ablation study of the loss weights in DeformStruct. In Table VI, we present an ablation study of the loss weights. Rows 1 to 4 correspond to different settings of $\lambda_{\mathrm{part}}$, rows 5 to 8 report the results of varying $\lambda_{\mathrm{align}}$, and the final four rows evaluate different settings of $\lambda_{\mathrm{rel}}$ for RSO.

Loss	Weight	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
$\lambda_{\mathrm{part}}$	0.1	73.7	51.9	20.4	49.2
	0.5	75.2	56.1	25.5	53.4
	1	76.4	57.9	28.0	54.8
	2	75.5	56.0	24.0	53.3
$\lambda_{\mathrm{align}}$	0.1	75.1	57.3	26.0	53.9
	0.5	76.4	57.9	28.0	54.8
	1	75.0	56.7	22.3	52.7
	2	75.5	53.8	21.5	51.4
$\lambda_{\mathrm{rel}}$	0.1	76.7	55.6	23.8	53.1
	0.5	76.5	57.0	25.6	54.0
	1	76.4	57.9	28.0	54.8
	2	76.4	55.6	25.5	53.3

Effect of Structural Consistency Objectives. We evaluate the contribution of each objective in Table VII. The performance consistently decreases when any individual objective is removed, demonstrating that the three losses provide complementary supervision for learning the action structure.

$\mathcal{L}_{\mathrm{part}}$	$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{rel}}$	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
$\checkmark$			75.6	57.4	27.8	54.3
	$\checkmark$	$\checkmark$	75.0	57.1	26.8	53.1
$\checkmark$		$\checkmark$	75.1	56.9	27.1	53.5
$\checkmark$	$\checkmark$	$\checkmark$	76.4	57.9	28.0	54.8

Effect of Relational Structure Modeling. To examine whether the improvement of RSO is merely due to additional temporal-position supervision, we compare several structural formulations while keeping the DPA framework unchanged. As shown in Table VIII, directly regressing the absolute positions of aligned parts does not improve the overall performance over DPA, indicating that absolute temporal-position supervision alone is insufficient to characterize the structural organization of action parts. Modeling only pairwise temporal distances also leads to degraded performance, suggesting that separation without ordering cannot adequately describe their temporal organization. Moreover, randomized relational targets clearly degrade localization performance, showing that incorrect structural relations can be harmful. In contrast, RSO achieves the best overall performance by jointly preserving the temporal ordering and relative separation among aligned parts.

Variant	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
DPA	75.6	57.4	27.8	54.3
+ Absolute Position	75.8	57.7	26.6	53.8
+ Distance Relation	76.1	55.2	22.6	52.2
+ Randomized Relation	74.1	54.1	21.5	51.1
+ RSO	76.4	57.9	28.0	54.8

Effect of Relational Representation. We further compare different strategies for representing the relations among aligned parts. MAE measures the error in predicting pairwise temporal differences, Order Acc. evaluates whether the relative temporal order between parts is correctly preserved, and Spearman $\rho$ measures the consistency of the overall relational ranking. As shown in Table IX, Shared Coord. achieves a slightly lower MAE, while substantially improving Order Acc. from 55.7% to 69.5% and Spearman $\rho$ from 0.201 to 0.397, together with a higher average mAP. These results indicate that the shared-coordinate formulation is more effective at preserving the global temporal organization among action parts, rather than independently fitting individual pairwise relations.

Relation	Param.	MAE ↓	Order Acc. ↑	$\rho$ ↑	AVG
Pairwise Rel.	197,121	0.1294	55.7	0.201	53.5
Shared Coord.	66,049	0.1226	69.5	0.397	54.8

Part Consistency and Diversity. We further analyze whether the learned latent parts exhibit meaningful structural specialization. As shown in Table X, the same latent part across different instances achieves a higher cosine similarity than different parts (0.595 vs. 0.459), yielding a similarity gap of 0.136 and indicating consistent part-specific representations across instances. Meanwhile, only 1.57% of part pairs collapse to the same temporal position, while 24.38% fall within 5% of the local search interval. The mean normalized pairwise distance reaches 0.373, showing that different latent parts are generally instantiated at distinct temporal regions rather than concentrating on the same discriminative location.

Semantic Consistency	Temporal Diversity
Same: 0.595 ↑	Coll.: 1.57% ↓
Diff.: 0.459 ↓	$CR_{0.05}$: 24.38% ↓
Gap: +0.136 ↑	Dist.: 0.373 ↑

Anchor Consistency Analysis. Table XI shows that 95.90% of the mined anchors belong to the target action class, and 90.45% are located within the current action instance. Only 5.45% come from other same-class instances, while different-class and background anchors account for 1.11% and 2.99%, respectively. These results indicate that the mined anchors are both class-consistent and largely instance-specific, while a small ambiguity remains among repeated same-class instances.

Anchor Category	Fraction (%)
Current action instance	90.45
Other same-class instance	5.45
Other different-class instance	1.11
Background	2.99
Target action class	95.90
Other classes / background	4.10

Ablation study of DPA. We conduct ablations of anchor construction and deformation modeling, as shown in Table XII. Replacing the mined anchors with uniform or random anchors decreases the AVG to 47.8 and 45.4, confirming the importance of informative instance-level anchor mining. Removing deformation modeling reduces the AVG to 52.2, confirming the contribution of deformation-aware matching to adapting latent parts to temporal variations.

Anchor	Deformation	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
Mining	–	73.9	54.1	23.8	52.2
Mining	✓	75.6	57.4	27.8	54.3
Uniform	✓	68.4	50.9	21.8	47.8
Random	✓	68.3	47.1	19.1	45.4

Cross-instance Structural Consistency. Table XIII evaluates the consistency of GT-normalized part configurations across action instances. Compared with DPA, DPA+RSO yields a higher Inside rate and lower position, gap, and span dispersions, showing that RSO promotes more stable relative temporal organization among aligned parts.

Method	Inside ↑	Pos. Disp. ↓	Gap Disp. ↓	Span Disp. ↓
DPA	87.6	0.1842	0.1425	0.1693
DPA + RSO	90.8	0.1517	0.1098	0.1326

Efficiency comparison. As shown in Table XIV, DeformStruct introduces only modest additional computational overhead during training, while maintaining comparable overall efficiency. Robustness analysis under different point distributions. We further evaluate DeformStruct under different point-annotation distributions. As shown in Table XV, the model maintains competitive overall performance under random, center-biased, and discriminative-frame annotations, although the perfor-

Method	Params. (M)	FLOPs (G)	Train Runtime (ms)	Memory (MB)
QROT	18.717	10.04	30.46	797.53
DeformStruct w/o RSO	18.930	10.04	31.10	798.04
DeformStruct	18.996	10.04	33.53	798.29

Setting	mAP@tIoU (%) 0.3	0.5	0.7	AVG [0.3:0.1:0.7]
Dataset-distribution	76.4	57.9	28.0	54.8
Random Point	71.1	56.7	28.3	53.4
Center-biased Point	71.5	61.6	30.7	56.5
Discriminative-frame Point	76.0	58.3	26.3	54.7

mance varies across different annotation distributions and tIoU thresholds. In particular, center-biased annotations achieve higher average performance, while random and discriminative-frame annotations remain reasonably close to the original dataset setting. These results indicate that DeformStruct can accommodate different point-annotation distributions without relying on a specific annotation pattern.

Qualitative visualization of the DPA module. In Fig. 4, we provide visualizations of the score distributions of different parts in the DeformStruct module. The colored markers indicate the locations of anchors matched with latent action parts. The bars at the bottom indicate the ground-truth segments, while the three curves correspond to the response scores of different latent part classifiers. Notably, the mined anchors align well with the ground-truth action intervals and capture complementary regions distributed over time.

$\delta$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
0	75.3	56.9	23.5	53.4
0.01	76.4	57.9	28.0	54.8
0.05	74.3	56.3	25.7	53.6
0.1	74.6	54.3	24.5	52.0

$\lambda_{deform}$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
0.5	75.5	53.4	24.1	51.6
1	76.4	57.9	28.0	54.8
2	75.3	57.5	23.3	53.5
4	74.6	55.6	23.3	52.3

$q$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
0	74.6	55.5	25.4	52.7
2	73.7	55.4	22.5	51.8
3	76.4	57.9	28.0	54.8
5	73.6	55.2	22.4	51.7

$N_{part}$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
2	75.1	55.9	23.5	52.5
3	76.4	57.9	28.0	54.8
4	75.5	57.3	25.2	53.0
5	75.2	56.1	26.3	53.5

$K_a$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
3	73.2	54.6	24.5	52.3
5	75.5	57.8	26.1	53.7
8	76.4	57.9	28.0	54.8
10	75.8	56.4	26.0	53.8

$\lambda_{OT}$	mAP@tIoU (%)			AVG
	0.3	0.5	0.7	0.3:0.1:0.7
0.1	73.5	53.0	20.1	50.3
0.5	76.4	57.9	28.0	54.8
1	75.1	56.9	22.7	53.0
2	74.3	53.7	20.6	50.7

Visualization of Part Deformation. Figure 5 visualizes the deformation performed by DPA. The matched anchors are not directly used as the final part locations; instead, each latent part can shift according to its response and deformation cost. As shown in the examples, different parts exhibit different deformation directions and magnitudes, while some remain close to their anchors when further adjustment is unnecessary. This qualitatively shows that DPA adaptively refines the mined anchors rather than simply inheriting their temporal positions.

Visualization of Learned Structural Coordinates. Figure 6 compares the normalized aligned-part coordinates $p_i$ with the structural coordinates $q_i$ predicted from aligned part features. After mean-centering, the learned structural coordinates exhibit configurations consistent with the anchor structure while allowing flexible global shifts. The pairwise difference plots further show that RSO preserves the relative ordering and spacing among parts. These results demonstrate that RSO captures the underlying relational structure without forcing the learned coordinates to exactly match the temporal anchors.

Sensitivity analysis of the hyperparameters. We further study the sensitivity of DeformStruct to its main hyperparameters in Table XVI, including the suppression radius $\delta$, deformation regularization weight $\lambda_{deform}$, cluster number $q$, number of latent parts $N_{part}$, mining size top-$K_a$, and OT balancing weight $\lambda_{OT}$. The results show that the model achieves relatively stable performance within moderate ranges of these hyperparameters, whereas excessively small or large values may lead to performance degradation.

Qualitative Results

To further validate the effectiveness of the proposed method, we provide some qualitative results using HR-Pro [7], QROT [14] and our model on test videos #46 and #188 in THUMOS14 [10] for comparison. Several representative results are visualized in Figure 7, illustrating the improved localization quality of our approach.

Conclusion

In this work, we propose DeformStruct, an end-to-end framework for point-supervised temporal action localization that focuses on instance-level structural modeling under sparse supervision. DeformStruct progressively discovers complementary action-related evidence, instantiates latent parts through deformation-aware alignment, and explicitly models their relative temporal organization. Semantic query construction and OT-based pseudo-segment generation are retained to provide point-level semantic priors and segment-level supervision, respectively. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate that DeformStruct consistently outperforms existing point-supervised methods, while detailed ablations and structural analyses further validate the effectiveness of part discovery, deformable alignment, and relational organization.

Mengnan Liu received the B.S. degree in Artificial Intelligence from Shandong University, Jinan, China, in 2022. She is currently pursuing the Ph.D. degree at the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. Her research interests include computer vision, multimodal video understanding and weakly-supervised learning.

Sanping Zhou (Member, IEEE) received his Ph.D. degree from Xi’an Jiaotong University, Xi’an, China, in 2020. From 2018 to 2019, he was a Visiting Ph.D. Student with the Robotics Institute, Carnegie Mellon University. He is currently a Professor with the Institute of Artificial Intelligence and Robotics at Xi’an Jiaotong University. His research interests include machine learning, computer vision and embodied intelligence, with a focus on meta-learning, multitask learning, object detection, multi-target tracking, trajectory prediction, medical image segmentation, visual navigation, and visual grasping.

Kun Xia received the Ph.D. degree in Control Science and Engineering from Xi’an Jiaotong University, Xi’an, China, in 2024. From 2022 to 2023, he was a visiting Ph.D. student at the University of Illinois Chicago, IL, USA. He is currently an Assistant Professor with the School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China. His research interests include computer vision, image/video processing, analysis and understanding.

Yonghao Dong received the Ph.D. degree in Electronic Information from Xi’an Jiaotong University, Xi’an, China, in 2025. He received the B.S. degree from East China University of Science and Technology, Shanghai, China, in 2018, and the M.S. degree from the University of Illinois at Chicago, Chicago, IL, USA, in 2019. He is currently an Assistant Professor with the School of Software Engineering, Xi’an Jiaotong University, Xi’an, China. His research interests include computer vision, image/video processing and autonomous driving.

Jie Wang (Senior Member, IEEE) received the BSc degree in electronic information science and technology from the University of Science and Technology of China, Hefei, China, in 2005, and the PhD degree in computational science from the Florida State University, Tallahassee, FL, in 2011. He is currently a professor with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. His research interests include reinforcement learning, knowledge graph, large-scale optimization, deep learning, etc.

Gang Hua (Fellow, IEEE) received the B.S. and M.S. degrees in Automatic Control Engineering from Xi’an Jiaotong University (XJTU), Xi’an, China, in 1999 and 2002, respectively. He received the Ph.D. degree in Electrical Engineering and Computer Science at Northwestern University, Evanston, Illinois, USA, in 2006. He is currently the Director of Applied Science at Amazon Alexa AI. He is an associate editor of TPAMI and MVA. He is a general chair of ICCV’2025, a program chair of CVPR’2019&2022, and an area chair of CVPR’2015&2017 and ICCV’2011&2017. He is the author of more than 200 peer-reviewed publications in prestigious international journals and conferences. He holds 19 US patents and has 15 more US patents pending. He is the recipient of the 2015 IAPR Young Biometrics Investigator Award. He is an IEEE Fellow, an IAPR Fellow, and an ACM Distinguished Scientist.