betaFeedback

WorksComputer Science

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Abstract

Recently, point-supervised temporal action localization has attracted increasing attention due to its low annotation cost (only one frame per action instance) and competitive detection performance. However, since point annotations are less complete than proposal annotations, existing methods mainly rely on multiple instance learning for snippet-level classification, which leads to a performance gap compared to fully supervised models. To bridge this gap, we propose P2P-Former, a novel point-to-proposal Transformer-based framework that effectively connects point-level supervision with proposal-level learning for temporal action localization. In this work, we identify two key challenges in transitioning from point-level supervision to full supervision: (1) how to generate reliable proposals as pseudo ground-truth, and (2) how to effectively integrate point-level cues into the fully supervised paradigm. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy that refines the confidence of proposals produced by a base locator using a point-aware importance estimation function, and then performs optimal matching between point annotations and candidate proposals to select reliable proposals as pseudo ground-truth. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module that aggregates prototypes from point-annotated features and injects them into snippet features through Transformer-based interactions. Moreover, we design a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to include the annotated point, thereby encouraging more accurate localization. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our method outperforms several existing state-of-the-art approaches.

Introduction

Temporal action localization is one of the fundamental tasks in video understanding [35], aiming to locate and categorize actions within untrimmed videos. It plays an essential role in various applications, such as highlight detection, security monitoring, and action spotting. Although traditional fully supervised methods [38, 39] have achieved impressive performance, they rely on precise temporal boundary annotations, which are expensive, time-consuming to annotate, and difficult to scale. To reduce annotation costs, weakly supervised temporal action localization methods [37] typically rely only on video-level labels. However, the absence of accurate boundary information results in a significant performance gap compared to fully supervised methods.

To address the above issue, point-supervised temporal action localization [34] has been introduced as a compromise solution, requiring only one labeled timestamp per action instance while achieving a favorable balance between annotation cost and detection performance. SF-Net [1] pioneers single-frame supervision for temporal action localization by iteratively mining pseudo action and background frames. Building on this paradigm, LACP [2] models action completeness through greedy proposal generation, TSPNet [3] exploits the point-center prior, and SMBD [4] progressively detects ambiguous action boundaries. HR-Pro [5] combines snippet-level discrimination with instance-level refinement, whereas QROT [31] incorporates query reformation and optimal transport. However, as shown in Figure 1(a), these methods mainly rely on sparse point-level labels for classification supervision, generate class activation sequences, and obtain action proposals through thresholding, without explicitly learning temporal localization. As a result, they struggle to capture the complete temporal structure of action instances. Therefore, leveraging more complete action instances as supervision is crucial, as they allow the model to perform temporal boundary regression in addition to action classification. By directly predicting the boundary offsets of action intervals, the model can explicitly learn temporal localization, as illustrated in Figure 1(b).

Figure 1

Figure 1. Comparison. (a) Existing methods rely on sparse point-level annotations for classification and derive proposals by thresholding class activation sequences, without explicitly learning temporal localization. (b) P2P-Former discovers high-quality action instances as pseudo labels, enabling joint classification and boundary regression for more complete action localization.

To this end, we propose P2P-Former, a novel point-to-proposal Transformer-based framework for point-supervised temporal action localization, which enables instance-level learning to narrow the performance gap with fully supervised methods. Specifically, our framework is motivated by two key challenges: (i) how to generate reliable proposals for supervision, and (ii) how to effectively integrate point-level information into the proposal-based temporal action localization framework. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy. Specifically, we construct a relevance matrix based on the location relationship between the generated proposals and point annotations, and then perform optimal matching to assign each point to its most relevant proposal. This yields a set of reliable proposals that serve as supervisory signals. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module, which aggregates fine-grained prototypes from point-annotated features and leverages them to enhance original snippet features via Transformer-based interactions. Furthermore, we introduce a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to cover the annotated reference point, thereby guiding the model toward more accurate proposal generation.

In summary, the main contributions of this work are threefold:

Fully-Supervised Temporal Action Localization

Fully-Supervised Temporal Action Localization (FTAL) depends on precise temporal boundary annotations, providing accurate labels for every snippet during training. Similar to the categorization of object detection methods [36], mainstream FTAL methods can be divided into one-stage and two-stage approaches based on the localization workflow. The one-stage methods [18] utilize an end-to-end framework to locate and classify action instances in untrimmed videos. In contrast, the two-stage methods [19, 20] first densely generate proposals, then predict the categories of these proposals and adjust their boundaries. While complete supervision delivers superior performance in classification and localization, it suffers from very high annotation costs.

Weakly-Supervised Temporal Action Localization

Weakly-Supervised Temporal Action Localization (WTAL) relies solely on video-level labels to localize action instances. Some studies employ multiple instance learning (MIL) to aggregate top-k snippet-level predictions into video-level classification outcomes, while others adopt attention-based mechanisms to improve detection performance. For example, UntrimmedNets [21] utilizes a soft attention mechanism to effectively retrieve relevant video segments. HAM-Net [28] introduces a hybrid attention mechanism to model action completeness, and ASCN [29] similarly integrates hybrid attention with frame-difference features to capture local context. However, the absence of frame-level annotations still results in a notable performance gap from fully supervised methods.

Point-Supervised Temporal Action Localization

Point-Supervised Temporal Action Localization (PTAL) serves as a middle ground between weak and full supervision, requiring only one-sixth of the annotation time (50s vs. 300s per 1-min video) [1, 2] compared to fully supervised methods [1, 2]. SF-Net [1] pioneers the use of single-frame supervision for temporal action localization, where pseudo action and background frames are iteratively mined to enrich the training supervision. Building on this paradigm, subsequent studies have explored diverse optimization strategies. For example, LACP [2] employs a greedy algorithm to generate densely optimized sequences, thereby learning action completeness from point annotations. TSPNet [3] exploits the prior that annotations are usually placed near the center of action instances, and proposes a center-scoring mechanism to enhance localization accuracy. To mitigate the semantic ambiguity in action boundary determination, SMBD [4] introduces a stepwise multi-granularity boundary detection method for precise localization. Moreover, by combining snippet-level discrimination with instance-level refinement, HR-Pro [5] generates more reliable proposals. Recently, QROT [31] combines query reformation and optimal transport, achieving enhanced performance in point-supervised temporal action localization. Beyond label-derived semantic priors, TRA [33] exploits point-guided text refinement and multimodal contrastive learning to incorporate textual semantics into visual representations. Meanwhile, another study [34] explicitly models temporal consistency through three self-supervised tasks: action completion, action order understanding, and action regularity understanding. Unlike previous methods that mainly rely on sparse point annotations for training, we propose a proposal-based PTAL approach to better model the temporal completeness of action instances.

Figure 2. Overview of the proposed P2P-Former framework. A base action locator first generates coarse proposals, which are then evaluated using point-aware scoring and matched with point annotations via the Hungarian algorithm to obtain reliable pseudo labels. Point-labeled features are clustered to construct category prototypes, which are injected into a prototype-guided semantic enhancement block to enrich video tokens. The enhanced tokens are processed by a Transformer encoder and FPN, followed by classification and regression heads for temporal action localization. The model is trained with Lcls\mathcal{L}_{cls}, Lreg\mathcal{L}_{reg}, and the point-to-proposal constraint Lp2p\mathcal{L}_{p2p}.

Methodology

Preliminaries

Problem Definition. Point-supervised temporal action localization (PTAL) aims to localize the temporal boundaries of action instances in untrimmed videos using sparse supervision, where only a single annotated frame is provided for each action instance. For a given video, every action instance is annotated with a timestamp tt along with its associated category yy. During inference, given an input video, the well-trained PTAL model produces a set of predicted action proposals, denoted as R={(si,ei,y^i,coni)}i=1KR = \{({s}_i, {e}_i, \hat{y}_i, con_i)\}_{i=1}^K, where KK is the number of output proposals, si{s}_i and ei{e}_i represent the predicted start and end times, y^i\hat{y}_i is the predicted action category, and conicon_i represents the associated confidence score.

Feature Embedding. Following previous works, we divide each video into snippets, each comprising a fixed number of contiguous frames. We then employ a pre-trained video classification model (i.e., I3D) to extract snippet-level features from these frames. The feature representation for each video is formulated as X∈RT×D\mathbf{X} \in \mathbb{R}^{T \times D}, where TT and DD denote the number of snippets and the feature dimension, respectively.

Overview. The overall framework is illustrated in Figure 2. We adopt a Transformer-based temporal detector equipped with classification and regression heads. Under point supervision, reliable proposals are first generated as pseudo labels in Section subsection:PPG. Point-derived semantic cues are then incorporated into the proposal-based framework in Section subsection:PSE, while Section subsection:PPCL introduces an additional positional constraint based on point annotations. Finally, Section subsection:training_inference describes the training and inference procedures.

Point-aware Reliable Proposal Generation

  Proposal-based temporal action localization methods typically rely on precise action boundary annotations as supervision. However, such proposal-level supervisory signals are not directly available under the point-annotation setting. A straightforward solution is to leverage the predictions of existing PTAL methods as pseudo proposals. Despite the relatively strong performance of existing point-supervised methods, they predominantly rely on snippet-level classification, which limits their ability to model complete action instances. Moreover, these methods tend to produce a large number of redundant proposals, resulting in a notable discrepancy from the ground-truth setting where each action instance corresponds to a single proposal. To generate proposals that are both reliable and non-redundant, two key challenges need to be addressed: (i) how to evaluate the importance of candidate boxes, and (ii) how to effectively filter redundant candidates. For the first issue, a simple strategy is to use the confidence scores of proposals generated by existing point-supervised methods to assess their importance; however, these scores still rely on snippet-level classification. Motivated by the observation that point annotations tend to lie near action centers [3], we introduce a point-aware candidate box importance evaluation function based on point locations. Specifically, for the ii-th proposal, it can be formulated as:

 ϕi(t)=(1−(t−μiσi)2)ke(−(t−μi)22σi2),\begin{equation} ~ \phi_i(t)=\left(1-\left(\frac{t-\mu_i}{\sigma_i}\right)^2\right)^{k} e^{\left(-\frac{(t-\mu_i)^2}{2\sigma_i^2}\right)}, \tag*{(1)} \end{equation}

where σi=ei−si2\sigma_i = \frac{e_i - s_i}{2} denotes the half length of the ii-th proposal, μi=ei+si2\mu_i = \frac{e_i + s_i}{2} is its temporal center, tt denotes the temporal position of the point annotation, and kk controls the sensitivity to the point–center deviation. The function adaptively assigns higher scores to proposals whose centers are close to the annotation point, while suppressing those whose annotations are located near the boundaries. Specifically, ϕi(t)\phi_i(t) reaches its maximum positive value when t=μit = \mu_i, while ϕi(si)=ϕi(ei)=0\phi_i(s_i) = \phi_i(e_i) = 0 at the boundaries. Moreover, ϕi(t)\phi_i(t) is explicitly set to zero when t<sit < s_i or t>eit > e_i, thereby suppressing locations outside the proposal. As the distance between the point annotation and the proposal center increases, the corresponding importance score decreases accordingly, naturally down-weighting proposals whose annotations are close to boundary regions.

To jointly consider the original proposal confidence and the positional consistency between the point annotation and the proposal, we refine the confidence score of the ii-th proposal with respect to the nn-th point annotation as

 Sn,i=coni⋅ϕi(tn),\begin{equation} ~{S}_{n,i} = con_i \cdot \phi_i(t_n), \tag*{(2)} \end{equation}

where conicon_i denotes the original confidence score of the ii-th proposal, tnt_n denotes the temporal position of the nn-th point annotation, and ϕi(tn)\phi_i(t_n) is the point-aware importance score defined in Eq. (1). A high adjusted confidence score therefore indicates that the proposal has both high prediction confidence and strong positional consistency with the point annotation. Next, we describe how these importance scores are utilized to perform effective candidate box selection.

Since each point annotation corresponds to a single action instance, we aim to assign exactly one proposal to each point annotation. To this end, we formulate the selection process as an optimal matching problem between NN point labels and KK predicted boxes. We first compute a similarity matrix S∈RN×K\mathbf{S}\in\mathbb{R}^{N\times K}, where each element Sn,iS_{n,i} is computed using Eq. (2). To ensure a globally optimal one-to-one matching, we adopt the Hungarian algorithm as the assignment solver. Since the Hungarian algorithm minimizes the overall cost, we transform the similarity into a cost matrix by C=1−S\mathbf{C} = 1 - \mathbf{S}, where higher similarity corresponds to lower cost. The Hungarian algorithm is then applied to C\mathbf{C} to obtain the optimal one-to-one assignment, which maximizes the total similarity between labels and boxes. Finally, through this global matching process, we obtain NN optimal proposals as pseudo labels.

Point-guided Semantic Enhancement

  To further exploit the prior information provided by point annotations, we design a Point-guided Semantic Enhancement module that effectively incorporates point supervision cues into the proposed framework. Since point annotations may fall on different temporal positions within an action instance, different point annotations of the same category can capture diverse semantic information across the temporal dimension. To this end, we seek to extract representative semantic information from each action category and further integrate it into the model to enhance feature representation. The proposed PSE module comprises two components: category-specific prototype construction and a Prototype-guided Semantic Enhancement block. Specifically, we collect all features corresponding to point-labeled samples from the training set. Since features from the same action category are similar, we group them by category and perform clustering separately for each category. To implement this process, we adopt the K-means clustering algorithm, whose optimization objective is defined as

O∗=arg⁡min⁡O∑j=1L∑xi∈Oj∥xi−μj∥22,\begin{equation} \mathcal{O}^* = \arg\min_{\mathcal{O}} \sum_{j=1}^{L} \sum_{x_i \in \mathcal{O}_j} \| x_i - \mu_j \|_2^2, \tag*{(3)} \end{equation}

where xix_i denotes the ii-th labeled feature in a specific category, LL is the number of clusters, and O={O1,…,OL}\mathcal{O} = \{\mathcal{O}_1, \dots, \mathcal{O}_L\} represents the cluster partition. The centroid of each cluster Oj\mathcal{O}_j is computed as

μj=1∣Oj∣∑xt∈Ojxt.\begin{equation} \mu_j = \frac{1}{|\mathcal{O}_j|} \sum_{x_t \in \mathcal{O}_j} x_t. \tag*{(4)} \end{equation}

We collect the features closest to the LL cluster centroids as sub-prototypes, and stack them within the same category to form a semantically richer category prototype. This design enables the prototype to capture diverse semantic information within the same action, such as the “long jump” action, which consists of sub-actions like run-up and jump. Formally, given CC action categories, we construct CC category prototypes, each defined as

mc∈RL×D,c=1,2,…,C,\begin{equation} \mathbf{m}_c \in \mathbb{R}^{L \times D}, \quad c = 1, 2, \dots, C, \tag*{(5)} \end{equation}

where LL is the number of sub-prototypes per category, which is identical to the number of clusters used in the K-means process, and DD denotes the feature dimension, consistent with that of the extracted video features.

Based on the prototypes constructed from point-annotated features, we employ a Prototype-guided Semantic Enhancement block to inject representative action semantics into the original video features. Specifically, a linear projection layer fqf_q is applied to the video features to obtain the query embeddings. The video features X\mathbf{X} are then concatenated with the prototype features mc\mathbf{m}_c, where [ ; ] denotes the concatenation operation. Two independent linear layers, fkf_k and fvf_v, are used to project the concatenated features into the key and value spaces, respectively:

Q=fq(X),K=fk(X,m1,…,mC),V=fv(X,m1,…,mC).\begin{align} \mathbf{Q} &= f_q(\mathbf{X}), \nonumber\\ \mathbf{K} &= f_k(\mathbf{X}, \mathbf{m}_1, \dots, \mathbf{m}_C), \nonumber\\ \mathbf{V} &= f_v(\mathbf{X}, \mathbf{m}_1, \dots, \mathbf{m}_C). \tag*{(6)} \end{align}

The attention map attn∈RT×(T+CL)\mathbf{attn} \in \mathbb{R}^{T \times (T + CL)} is computed as the scaled dot-product between the query and the transposed key.

 attn=Softmax(Q⋅K⊤D)\begin{equation} ~ \mathbf{attn} = \text{Softmax}\left(\frac{\mathbf{Q} \cdot \mathbf{K}^\top}{\sqrt{D}}\right) \tag*{(7)} \end{equation}

The attention output is obtained by weighting the value embeddings with attn\mathbf{attn}, and then passed through a Feed-Forward Layer, which consists of FC–ReLU–FC blocks, with LayerNorm and residual connections for normalization and information preservation. Finally, the enhanced video features are fed into the downstream layers for temporal action localization.

Point-to-Proposal Constraint Loss

  Although selected reliable proposals are employed as pseudo labels, they inevitably contain noise. As a result, incorporating precise ground-truth point annotations as supervisory signals remains essential. To this end, we design a point-to-proposal constraint loss Lp2p\mathcal{L}_{p2p}, which exploits the positional information of point annotations to explicitly constrain predicted proposals. Specifically, Lp2p\mathcal{L}_{p2p} consists of a point inclusion term and a center alignment term.

First, we introduce the point inclusion loss to ensure that each predicted proposal contains its associated annotated point:

Linc=1Npos∑i=1Npos[ReLU(si−ti)+ReLU(ti−ei)],\begin{align} \mathcal{L}_{inc} = \frac{1}{N_{pos}} \sum_{i=1}^{N_{pos}} \Big[ &\mathrm{ReLU}(s_i-t_i) \nonumber\\ &+ \mathrm{ReLU}(t_i-e_i) \Big], \tag*{(8)} \end{align}

where NposN_{pos} denotes the number of positive samples, (si,ei)(s_i,e_i) represents the predicted temporal proposal, and tit_i denotes its associated point annotation. When tit_i falls outside the predicted proposal, the corresponding ReLU term produces a positive penalty and guides the violated boundary toward the annotated point. Once tit_i is contained within the proposal, both terms become zero, avoiding unnecessary further expansion of the proposal boundaries.

However, merely satisfying the inclusion constraint does not guarantee that the annotated point is well aligned with the predicted proposal. Therefore, we further introduce a center alignment loss:

Lalign=1Npos∑i=1NposIi SmoothL1(ci−ti),\begin{align} \mathcal{L}_{align} = \frac{1}{N_{pos}} \sum_{i=1}^{N_{pos}} \mathbb{I}_i \,\mathrm{SmoothL1}(c_i-t_i), \tag*{(9)} \end{align}

where

ci=si+ei2,Ii=I(si≤ti≤ei).\begin{equation} c_i=\frac{s_i+e_i}{2}, \qquad \mathbb{I}_i=\mathbb{I}(s_i\leq t_i\leq e_i). \tag*{(10)} \end{equation}

Here, cic_i denotes the center of the predicted proposal, and Ii\mathbb{I}_i is an indicator function that equals one when the annotated point lies inside the proposal and zero otherwise. Therefore, the center alignment constraint is activated only after the point inclusion condition is satisfied, encouraging the proposal center to be closer to the annotated point without interfering with the initial inclusion process.

Finally, the overall point-to-proposal constraint loss is formulated as

Lp2p=Linc+Lalign.\begin{equation} \mathcal{L}_{p2p} = \mathcal{L}_{inc} + \mathcal{L}_{align}. \tag*{(11)} \end{equation}

In this way, Lp2p\mathcal{L}_{p2p} first ensures point inclusion and then further improves proposal–point positional consistency, while the precise proposal extent is jointly optimized by the boundary regression objective.

Training and Inference

 

Training. At the training stage, the overall loss function can be formulated as follows:

 L=Lcls+Lreg+λLp2p,\begin{equation} ~ \mathcal{L} = \mathcal{L}_{cls} + \mathcal{L}_{reg} + \lambda \mathcal{L}_{p2p}, \tag*{(12)} \end{equation}

where Lcls\mathcal{L}_{cls} and Lreg\mathcal{L}_{reg} denote the focal loss for multi-label classification and the IoU loss for boundary regression, following previous temporal action localization methods [18, 30]. The focal loss alleviates the imbalance between positive and negative samples by down-weighting easy negatives, while the IoU loss is differentiable and applied only when the current time step corresponds to a positive sample. λ\lambda controls the relative importance of the point-to-proposal constraint loss Lp2p\mathcal{L}_{p2p}.

Inference. In the inference phase, the well-trained P2P-Former alone is used to directly generate action proposals, eliminating the need for the base locator. Finally, we use soft-NMS [17] to filter redundant proposals and obtain the final predictions.

Experiments

Datasets and Evaluation

THUMOS14 [9] is a widely used temporal action localization benchmark containing 413 untrimmed sports videos across 20 action categories. Each video contains an average of 15 action instances, with substantial variation in video length. Following the standard split, 200 videos are used for training and 213 for testing.

ActivityNet-1.3 [22] is a large-scale benchmark covering 200 action categories, with 10,024 training, 4,926 validation, and 5,044 test videos. Each video contains an average of 1.6 action instances, and approximately 36% of the frames correspond to ambiguous action contexts or background. Following previous studies, we report results on the validation set.

Evaluation Metrics. Following previous works, we evaluate the performance of temporal action localization using mean Average Precision (mAP) under different Intersection over Union (IoU) thresholds. For THUMOS14, the IoU thresholds are set to [0.1:0.1:0.7], while for ActivityNet-1.3 they range from [0.5:0.05:0.95]. Additionally, we report results on THUMOS14 for the ranges [0.1:0.1:0.5] and [0.3:0.1:0.7] for comprehensive comparison.

Implementation Details

We use the snippet-level model of HR-Pro [5] as the base locator and adopt the backbone and detection head of ActionFormer [18]. Following prior studies, two-stream I3D [10] pre-trained on Kinetics-400 is used to extract 1024-dimensional RGB and optical-flow features, resulting in an input dimension of D=2048D=2048. For THUMOS14, we extract one feature every 16 frames with a stride of 4, while the base locator uses a stride of 16. For ActivityNet-1.3, features are extracted from non-overlapping 16-frame snippets. We train the model using AdamW with a weight decay of 0.050.05. The learning rates are set to 1×10−41\times10^{-4} and 1×10−31\times10^{-3} for THUMOS14 and ActivityNet-1.3, respectively. The model is trained for 35 and 15 epochs on the two datasets, with a 5-epoch warm-up. We set kk=2 in Eq. (1), use two sub-prototypes per class and set λ=1\lambda=1 for Lp2p\mathcal{L}_{\mathrm{p2p}}. All experiments are implemented in PyTorch and conducted on a single NVIDIA RTX 3090 GPU.

SupervisionMethodmAP@IoU (%)AVGAVGAVG
0.10.20.30.40.50.60.7(0.1:0.5)(0.3:0.7)(0.1:0.7)
tabularcFully
supervisedtabular & TCANet (CVPR'21) & - & - & 60.6 & 53.2 & 44.6 & 36.8 & 26.7 & - & 44.3 & -
GCM (TPAMI'21)72.570.966.560.851.9--64.5--
VSGN (ICCV'21)--66.760.452.441.030.4-50.2-
AFSD (CVPR'21)--67.362.455.543.731.1-52.0-
tabularcWeakly
supervisedtabular & P-MIL (CVPR'23) & 71.8 & 67.5 & 58.9 &49.0 & 40.0 & 27.1 & 15.1 & 57.4 & 38.0 & 47.0
PivoTAL (CVPR'23)74.169.661.752.142.830.616.760.140.849.6
ISSF (AAAI'24)72.466.958.449.741.825.512.857.837.646.8
PVLR (MM'24)74.969.961.453.145.130.517.160.9-50.3
tabularcPoint
supervisedtabular &ARST (CVPR'19) &24.3 &19.9 &15.9 &12.5 &9.0 &- &- &16.3 &- &-
SF-Net (ECCV'20)68.362.352.842.230.520.612.051.231.641.2
DCM (ICCV'21)72.364.758.247.135.923.012.855.635.444.9
LACP (ICCV'21)75.771.464.656.545.334.521.862.744.552.8
BackTAL (TPAMI'21)--54.445.536.326.214.8-35.4-
CRRC-Net (TIP'22)77.873.567.157.946.633.719.864.645.153.8
SMBD (ECCV'24)--66.057.947.036.022.064.245.7-
SNPR (TIP'24)77.973.966.659.448.636.722.765.346.855.1
AAPL (AAAI'25)64.3-54.6-35.2-14.0--42.8
TSPNet (CVPR'24)82.377.670.160.049.437.624.567.948.357.4
HR-Pro (AAAI'24)85.681.674.364.352.239.824.871.651.160.3
QROT (CVPR'25)--73.164.454.341.327.472.352.1-
Ours\textbf{Ours}85.282.978.971.661.448.732.276.058.665.8

Comparison with State-of-the-Art Methods

In Table table:thumos_benchmark, our method achieves state-of-the-art performance on the THUMOS14 dataset, attaining an average mAP@[0.1:0.7] of 65.8% and outperforming all existing point-supervised and video-level weakly supervised temporal action localization methods. In particular, compared with existing point-supervised approaches, our method achieves substantial improvements in average mAP@[0.1:0.7], ranging from 5.5% over HR-Pro to 24.6% over SF-Net. Furthermore, compared with PVLR, the best-performing video-level weakly supervised method in the table, our approach achieves a 15.5% improvement in average mAP@[0.1:0.7]. Notably, our method achieves an average mAP@[0.3:0.7] of 58.6%, surpassing several fully supervised approaches, such as AFSD (58.6% vs. 52.0%).

We further report the comparison results on ActivityNet-1.3 in Table table:anet_benchmark. Our method achieves an average mAP@[0.5:0.95] of 31.1%, outperforming all existing point-supervised methods. Specifically, it improves the average mAP@[0.5:0.95] by 3.4% over QROT and by 7.1% over CRRC-Net. The consistently strong performance on both benchmarks demonstrates the effectiveness and generalization ability of our method.

MethodmAP@IoU (%)AVG
0.50.750.95
LACP (ICCV'21)40.424.65.725.1
CRRC-Net (TIP'22)39.824.15.924.0
HR-Pro (AAAI'24)42.827.28.027.1
SMBD (ECCV'24)43.227.88.327.5
SNPR (TIP'24)41.330.94.826.5
AAPL (AAAI'25)39.624.35.624.7
QROT (CVPR'25)43.328.28.527.7
Ours50.730.06.931.1

Row
Index & 4c|Setup & 3cAVG mAP@IoU(%)
BaselinePPGPSEPPCL[0.1:0.5][0.3:0.7][0.1:0.7]
173.254.762.6
274.456.664.1
375.356.964.6
474.657.364.7
576.058.665.8

Ablation Study

Contribution of each component. Table table:ablation_studies presents the ablation results of the proposed components. Row 1 denotes our baseline, which constructs pseudo proposals by sequentially matching each point annotation with the highest-scoring candidate proposal containing that point, while preventing repeated proposal assignments. This baseline achieves an average mAP@[0.1:0.7] of 62.6%. Introducing PPG improves the performance to 64.1%, corresponding to a 1.5% gain over the baseline (Row 2 vs. Row 1), demonstrating the benefit of using more reliable pseudo proposals for supervision. Adding PSE further increases the average mAP to 64.6% (Row 3 vs. Row 2), validating the effectiveness of point-guided semantic enhancement. Similarly, PPCL improves the performance from 64.1% to 64.7% (Row 4 vs. Row 2), showing that explicitly exploiting the positional information of point annotations provides additional supervision for proposal prediction. When all three components are jointly employed, the model achieves the best performance of 65.8%, outperforming the baseline by 3.2% (Row 5 vs. Row 1), which verifies their effectiveness and complementarity.

MethodmAP@IoU(%)
0.10.30.50.7AVG
LACP∗^*75.364.345.120.652.4
LACP + Ours86.776.759.331.564.7
HR-Pro1∗_{1}^*82.069.448.221.956.2
HR-Pro1_{1}+ Ours85.278.961.432.265.8

Impact of the base locator. To evaluate the applicability of our method across different base locators, we conduct experiments using LACP and the snippet-level model of HR-Pro. As shown in Table table:Generalization, our method improves the average mAP of LACP from 52.4% to 64.7% and that of HR-Pro from 56.2% to 65.8%, yielding gains of 12.3% and 9.6%, respectively. Notably, the performance gap between the two base locators is reduced from 3.8% to 1.1% after incorporating our method, further demonstrating its effectiveness across different PTAL frameworks.

LLmAP@IoU(%)AVG
0.10.30.50.7[0.1:0.7]
1185.178.261.031.965.4
2285.278.961.432.265.8
3385.078.360.831.965.3
4484.778.060.331.765.1

Effect of the number of clusters. We study the effect of the number of clusters constructed for each action category, and Table table:ablation_cluster_num shows that 2 is a better choice than other numbers. Specifically, setting the number of clusters to 2 yields the best performance, with an mAP gain of approximately 0.4%–0.7% over other configurations. This may be because too few clusters fail to provide sufficient action-semantic information, while too many clusters introduce additional noise.

λ\lambdamAP@IoU(%)AVG
0.10.30.50.7[0.1:0.7]
0.50.585.178.360.831.965.3
1185.278.961.432.265.8
1.51.585.278.761.131.665.5
2284.877.860.231.265.0

Impact of the balance weight. In Table table:ablation_lambda, we compare the performance of our method under different values of the loss balance weight λ\lambda. The results show that λ=1\lambda=1 is the optimal balance weight, achieving an average mAP@[0.1:0.7] of 65.8%. A lower λ\lambda (e.g., 0.5) leads to a slight performance drop, likely due to insufficient point-to-proposal constraint. Conversely, a higher λ\lambda (e.g., 1.5) also results in degraded performance, as the overly strong constraint may weaken the influence of the base loss. As λ\lambda further increases to 2, the performance continues to decline, suggesting that excessive emphasis on the point-to-proposal constraint may compromise the optimization of the base loss.

Inference Efficiency Analysis. Table tab:p2pformer_cost compares the localization performance, model complexity, and inference efficiency of different PTAL methods. P2P-Former achieves the best average mAP of 65.8% with 46.0M parameters, outperforming SF-Net, LACP, and HR-Pro by 25.6%, 13.4%, and 5.6%, respectively. Although P2P-Former requires more computation due to its proposal-level Transformer architecture, its inference latency remains acceptable at 49.3 ms, making it practical for offline video understanding applications.

MethodmAP (%)Params (M)FLOPs (G)Time (ms)
SF-Net∗^*40.216.81.71.0
LACP∗^*52.412.61.31.2
HR-Pro∗^*60.249.48.214.1
P2P-Former65.846.084.449.3

Figure 3. Qualitative comparison between our proposed method, HR-Pro [5], and LACP [2] on ActivityNet-1.3. We provide four cases of temporal action localization from different action classes, including “River tubing”, “Applying sunscreen”, “Paintball”, and “Scuba diving”. Prediction errors are highlighted with red dashed boxes. The dark blue region represents the overlap between different predicted instances. The IoUs between our detection results and the ground truths are notably higher.

Qualitative Results

Qualitative comparison. In Figure 3, we present a qualitative comparison of our method with LACP and HR-Pro on the test videos of ActivityNet-1.3. The results clearly indicate that our method achieves more precise localization of action instances. Taking Figure (a) as an example, LACP predicts an incomplete action instance on the left side of the video, while HR-Pro generates redundant instances with overlapping segments. In contrast, our method not only predicts complete action instances but also avoids generating redundant ones. In Figure (b), LACP detects only one incomplete action instance, in contrast to the two complete instances present in the ground truth. Although HR-Pro identifies two action instances, the left one remains partially missing. By comparison, our approach successfully localizes both action instances with full temporal extent. In Figure (c), both LACP and HR-Pro generate imprecise boundaries that almost span the entire video and yield redundant detections, whereas our method correctly identifies complete action instances with accurate temporal boundaries. Figure (d) shows that the detection produced by our method closely matches the ground-truth annotation, whereas the two comparison methods only capture part of the action instance.

Error Analysis. We further perform a false positive analysis averaged over tIoU thresholds 0.1–0.7 (step 0.1), as shown in Figure 4. Compared to the base locator, our P2P-Former outputs a greater proportion of beneficial positive detections. Moreover, in terms of false positives, our method yields a significantly lower rate of localization errors. This demonstrates the benefit of using complete action instances for supervision, allowing the model to capture action completeness instead of relying solely on snippet-level classification.

Figure 4. False Positive Profiling on THUMOS14 using the diagnostic tool proposed by [32]. The left panel shows the analysis of the base locator results, whereas the right panel presents the analysis of our P2P-Former.

Visualization of Pseudo Ground Truth Generation. Figure 5 visualizes the matching between labeled points and candidate proposals. Through one-to-one optimal matching, the proposals that are most consistent with the point annotations are selected as pseudo ground-truth, while the remaining proposals are discarded. The selected proposals show better temporal alignment with the ground-truth action intervals.

Figure 5. Visualization of the optimal matching between labeled points and candidate proposals. The successfully matched proposals are selected as pseudo ground-truth, while unmatched proposals are discarded. Ground-truth intervals are provided only for reference.

Visualization of clustering results. We illustrate the sub-prototype construction process in Figure 6. For visualization, we randomly select six categories from the THUMOS14 dataset and cluster the point-annotated features of each category into two groups. For each category, the features closest to the two cluster centroids are selected as representative sub-prototypes. Since THUMOS14 contains 20 action categories, our model constructs a total of 20×220 \times 2 sub-prototypes.

Figure 6. Visualization of the clustering results for six randomly selected categories from the THUMOS14 dataset. The point-annotated features of each category are grouped into two clusters, yielding two representative sub-prototypes per category.

Conclusion

In this paper, we identify that existing point-supervised methods mainly rely on snippet-level classification under sparse supervision, limiting their ability to model complete action instances. To address this issue, we propose P2P-Former, an instance-level framework for point-supervised temporal action localization. P2P-Former first evaluates proposals generated by a base locator using a point-aware importance function and performs optimal matching to select reliable pseudo proposals. It further introduces a point-guided semantic enhancement module to enrich snippet features with point-derived prototypes, together with a point-to-proposal constraint loss that enforces proposal–point consistency. Experiments on two widely used benchmarks demonstrate the effectiveness of the proposed method.

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China under Grants No. 62272438 and 62521007, in part by the Beijing Natural Science Foundation under Grant No. L25700, and in part by the Fundamental Research Funds for the Central Universities under Grant No. E2ET1104.

CRediT authorship contribution statement

Yunchuan Ma: Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Methodology, Formal analysis, Project administration, Data curation. Laiyun Qing: Writing – review & editing, Resources, Supervision, Funding acquisition. Guorong Li: Resources, Formal analysis. Qingming Huang: Resources.

References

  1. [1]Ma, F., Zhu, L., Yang, Y., Zha, S., Kundu, G., Feiszli, M., & Shou, Z. (2020). SF-Net: Single-Frame Supervision for Temporal Action Localization. ECCV, 12349, 420–437.
  2. [2]Lee, P., & Byun, H. (2021). Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization. ICCV, 13628–13637.
  3. [3]Xia, Z., Cheng, J., Liu, S., Hu, Y., Wang, S., Zhang, Y., & Dang, L. (2024). Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization. CVPR, 18440–18450.
  4. [4]Liu, M., Wang, L., Zhou, S., Xia, K., Wu, Q., Zhang, Q., & Hua, G. (2024). Stepwise Multi-grained Boundary Detector for Point-Supervised Temporal Action Localization. ECCV, 15065, 333–349.
  5. [5]Zhang, H., Wang, X., Xu, X., Qing, Z., Gao, C., & Sang, N. (2024). HR-Pro: Point-Supervised Temporal Action Localization via Hierarchical Reliability Propagation. AAAI, 7115–7123.
  6. [9]Idrees, H., Zamir, A. R., Jiang, Y.-G., Gorban, A., Laptev, I., Sukthankar, R., & Shah, M. (2017). The THUMOS challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 155, 1–23.
  7. [10]Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. CVPR, 4724–4733.
  8. [17]Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017). Soft-NMS - Improving Object Detection with One Line of Code. ICCV, 5562–5570.
  9. [18]Zhang, C.-L., Wu, J., & Li, Y. (2022). Actionformer: Localizing moments of actions with transformers. ECCV, 492–510.
  10. [19]Lin, T., Zhao, X., Su, H., Wang, C., & Yang, M. (2018). Bsn: Boundary sensitive network for temporal action proposal generation. ECCV, 3–19.
  11. [20]Lin, T., Liu, X., Li, X., Ding, E., & Wen, S. (2019). Bmn: Boundary-matching network for temporal action proposal generation. ICCV, 3889–3898.
  12. [21]Wang, L., Xiong, Y., Lin, D., & Van Gool, L. (2017). Untrimmednets for weakly supervised action recognition and detection. CVPR, 4325–4334.
  13. [22]Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015). ActivityNet: A large-scale video benchmark for human activity understanding. CVPR, 961–970.
  14. [28]Islam, A., Long, C., & Radke, R. J. (2021). A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization. AAAI, 1637–1645.
  15. [29]Zhao, Y., Zhang, H., Gao, Z., Gao, W., Wang, M., & Chen, S. (2023). A Novel Action Saliency and Context-Aware Network for Weakly-Supervised Temporal Action Localization. TMM, 25, 8253–8266.
  16. [30]Shi, D., Zhong, Y., Cao, Q., Ma, L., Li, J., & Tao, D. (2023). TriDet: Temporal Action Detection with Relative Boundary Modeling. CVPR, 18857–18866.
  17. [31]Liu, M., Wang, L., Zhou, S., Xia, K., Sun, X., & Hua, G. (2025). Boosting Point-Supervised Temporal Action Localization through Integrating Query Reformation and Optimal Transport. CVPR, 13865–13875.
  18. [32]Alwassel, H., Heilbron, F. C., Escorcia, V., & Ghanem, B. (2018). Diagnosing Error in Temporal Action Detectors. ECCV, 11207, 264–280.
  19. [33]Ma, Y., Qing, L., Li, G., Liu, Y., Qi, Y., & Huang, Q. (2027). Boosting point-supervised temporal action localization via text refinement and alignment. Expert Systems with Applications, 334, 134333.
  20. [34]Ma, Y., Qing, L., Li, G., Liu, Y., Qi, Y., & Huang, Q. (2027). Exploring the temporal consistency for point-level weakly-supervised temporal action localization. Pattern Recognition, 183, 114948.
  21. [35]Ma, Y., Qing, L., Li, G., Qi, Y., Beheshti, A., Sheng, Q. Z., & Huang, Q. (2026). RETTA: Retrieval-enhanced test-time adaptation for zero-shot video captioning. Pattern Recognit., 171, 112170.
  22. [36]Liu, Y., Ma, Y., Qi, Y., Qing, L., & Li, G. (2025). Boosting UAV Detection via Memory-Enhanced Attention and Contrastive Learning. IEEE Signal Process. Lett., 32, 3132–3136.
  23. [37]Zou, Y., Zhao, Q., Sarker, P. K., Li, S., Wang, L., & Liu, W. (2025). Diffusion-based framework for weakly-supervised temporal action localization. Pattern Recognit., 160, 111207.
  24. [38]Kim, J., Lee, M., & Heo, J.-P. (2026). Long-term pre-training for temporal action detection with transformers. Pattern Recognit., 171, 112144.
  25. [39]Chen, G., Zheng, Y.-D., Zhu, W., Wang, J., & Lu, T. (2026). Feature matters: Revisiting channel attention for Temporal Action Detection. Pattern Recognit., 169, 111846.

Paper details

vlnc:W_d9cyraq3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents