WorksComputer Science
Structure-Guided Multi-Granularity Spatiotemporal Transformer for Efficient Monocular 3D Human Pose Estimation
Abstract
Existing multi-granularity methods for monocular 3D human pose estimation emphasize spatial joint features while underusing skeletal structure, including global connectivity and temporal continuity. Moreover, modeling entire frame sequences incurs high computational costs. We propose a Structure-Aware Multi-Granularity Spatiotemporal Transformer that integrates structural priors across spatial and temporal representations for accurate and efficient pose estimation. A three-level feature extractor captures human topology at the joint, bone, and limb levels. Gated Adaptive Spatial Attention (GASA) embeds structural constraints into spatial attention, while Multi-Granularity Feature Enhancement (MGFE) uses hierarchical pooling and fusion to strengthen local details and global semantics. Temporal Segment Fusion (TSF) aligns temporal contexts across granularities to capture both short- and long-range motion patterns. Hierarchical Cross-Attention (HCA) promotes spatiotemporal interaction between granularities, and Keyframe Attention (KFA) selects informative motion segments to reduce redundancy and computational cost. Experiments demonstrate improved accuracy and efficiency over conventional Transformer-based approaches, highlighting the effectiveness of structure-aware multi-granularity modeling for monocular 3D human pose estimation.
Introduction
Three-dimensional human pose estimation (3D HPE) aims to recover the positions of key skeletal points of the human body in three-dimensional space from images or videos, and is widely used in human-computer interaction, behavior recognition, virtual reality, and other fields [1, 2, 3]. This task is essentially an inverse problem of inferring three-dimensional structure from two-dimensional image information, which is significantly ill-posed and easily affected by factors such as occlusion, depth blurring, and cluttered backgrounds. Therefore, fully leveraging auxiliary clues from temporal sequences, spatial structures, or multi-view perspectives is crucial for improving estimation accuracy.
In practical applications, monocular 3D pose estimation (Monocular 3D HPE) has garnered particular attention. Compared to multi-camera systems, monocular systems are simpler to deploy and more cost-effective, making them more suitable for real-time tasks in open environments [3]. However, due to the lack of parallax information, relying solely on a single frame image to recover 3D structure faces the issue of severe information deficiency. Although numerous methods have attempted to introduce temporal consistency or structural constraints [4] in recent years, it remains challenging to extract both global and local motion semantics from a single granularity, thereby affecting the stability and generalization capability of pose prediction.
To address these issues, researchers have gradually begun to explore multi-granularity modeling mechanisms. By abstracting human body structure at different levels—such as from overall anatomical structure, major trunk joints, to basic motion joints—the system can systematically model topological and semantic information of the human body in both spatial and temporal dimensions. This not only enhances representational capabilities but also better leverages the complementary nature of local details and global context[5], but the interaction and fusion between features of different granularities are often insufficiently deep, preventing the model from fully leveraging the complementary information between features of different granularities [5, 6]. For example, in multi-granularity algorithms based on GCN, the fusion of local joint features and global skeleton features is often merely a simple concatenation or weighted summation which fails to effectively capture the complex relationships between local joint movements and changes in the global skeleton structure [7]. For instance, during jumping, local joint changes are closely linked to global motion trends, but existing fusion methods fail to capture this, limiting pose estimation robustness. High-precision monocular 3D human pose estimation algorithms often require a large number of frames as input, which to some extent leads to the issue of excessive redundant frames [8, 9]. However, current multi-granularity algorithms fail to effectively identify and discard redundant frames, treating all frames equally and thereby increasing computational overhead.
The current main methods for monocular 3D human pose estimation can be broadly categorized into the following types: first, image-to-3D regression methods, which directly regress 3D keypoint coordinates from a single RGB image, typically using convolutional neural networks (CNNs) or their variants to establish the mapping relationship between images and 3D poses [10, 1]. Second, 2D-to-3D pose lifting methods adopt a two-stage strategy of “2D first, then 3D.” They first use a mature 2D pose detector to obtain 2D keypoints of the human body, then map them to 3D poses. This approach can leverage large-scale 2D annotated data to improve the accuracy of the front-end detection. However, due to the inherent projection ambiguity of 2D keypoint information (a single 2D pose may correspond to multiple 3D solutions), this method inherently remains an ill-posed problem [11]. Third, temporal modeling methods that incorporate temporal information aim to address the limitations of single-frame image data by utilizing consecutive frames from videos to obtain more stable 3D pose estimates through temporal modeling [12, 13]. However, information between video frames is often highly redundant, and modeling with short time windows has limited effectiveness; modeling with long sequences significantly increases computational costs. Fourth, Transformer methods leverage their strong ability to model long-range dependencies and global perception for pose estimation tasks. For example, methods like PoseFormer and LiftFormer use spatio-temporal self-attention modeling on 2D keypoint sequences to achieve more accurate pose recovery [14, 15].
Although Transformers demonstrate strong global modeling capabilities in 3D human pose estimation, they still have many shortcomings in practical applications. First, traditional Transformers mainly rely on unified modeling of all joints and lack a detailed distinction between human structural levels, resulting in insufficient utilization of structural information and difficulty in modeling complex movement structures with precision [16]. Second, Transformers typically fuse information of different granularities using simple concatenation or weighting methods, ignoring semantic alignment between granularities, which hinders the collaborative expression of multi-level motion information [5]. Third, Transformers typically process entire video segments uniformly, unable to effectively identify key action frames, leading to redundant information interfering with modeling and reducing computational efficiency [17, 18].
To address these issues, this paper proposes a multi-granularity spatiotemporal Transformer network that integrates skeletal structural information, achieving efficient and accurate monocular 3D pose estimation. This paper introduces a progressive three-level granularity modeling framework, which hierarchically represents the human body structure at the levels of the full body, torso, and motion joints, enabling fine-grained modeling of human topology. Specifically, to tackle the insufficient utilization of structured information, a Gated Adaptive Spatial Attention (GASA) mechanism is proposed to explicitly incorporate skeletal connectivity constraints into the attention computation, thereby achieving precise modeling of spatial structural information. To overcome the lack of semantic alignment in multi-granularity feature fusion, a Multi-Granularity Feature Enhancement Module (MGFEM) is designed into the Spatial Transformer. It performs hierarchical pooling and fusion to integrate global contextual information at each granularity level, enhancing the representation of local details and global semantics. In the Temporal Transformer, a Temporal Segment Fusion (TSF) mechanism is proposed to align the temporal context and improve the modeling of local motion dynamics. Furthermore, a Hierarchical Cross-Attention (HCA) mechanism is designed to enable spatiotemporal semantic alignment and collaborative modeling across granularities. Finally, to reduce redundancy and improve efficiency in sequence processing, a Keyframe Attention (KFA) mechanism is proposed to dynamically select key motion frames, effectively reduce computational cost while preserving temporal modeling of essential motion cues. The contributions of this paper are:
Addressing the limitations in structure-aware modeling and multi-granularity semantic alignment in existing Transformers for 3D human pose estimation, we establish a unified framework that integrates structure awareness, multi-granularity alignment, and efficient temporal modeling, thereby improving the accuracy and computational efficiency.
We propose a structure-aware Spatial Transformer that adaptively embeds structural information into multi-granularity spatial attention, together with a feature enhancement mechanism. Meanwhile, a Temporal Segment Fusion mechanism is designed in the temporal Transformer to align temporal context.
A hierarchical cross-attention mechanism is designed to achieve semantic alignment and fusion of multi-granularity features, overcoming the semantic drift issues associated with traditional fusion.
Key Frame Attention is proposed to dynamically select temporal key segments for attention modeling, effectively reduce computational costs while maintaining the ability to model temporal dependencies in long sequences.
Related Work
Multi-granularity Feature Modeling
The human skeleton naturally has a layered anatomical structure, and joints at different levels have different semantic meanings when expressing movement, posture, and overall dynamics. Therefore, multi-granularity modeling has gradually become an important direction for improving posture estimation performance. The core idea of multi-granularity modeling is to abstract and model the human skeleton at multiple scales and structural levels, enabling the model to capture both local details (such as fine movements of the arms and legs) and understand overall movement semantics (such as the orientation of the torso and the overall dynamic trends of the body).
Current multi-granularity modeling methods can be broadly categorized into three types: first, hierarchical structural modeling methods, which divide the skeleton into multiple semantic layers based on human anatomical structure. For example, SemGCN [19] uses GCN to perform structural modeling on groups of joints and fuses whole-body information through hierarchical graphs. Second, multi-scale feature extraction methods extract features at different spatial scales to achieve both local and global semantic coverage. For instance, PoseGTAC [20] designs graph dilated convolutions and graph Transformer modules to extract local fine-grained and global coarse-grained information, respectively. LiftFormer [14] uses multi-scale attention mechanisms to learn cross-temporal granularity structural representations. Third, partition-based modeling and feature enhancement methods divide the human body into functional regions. FLEX [21] fuses 2D joint coordinates and confidence scores, using partition-based modeling to extract stable local features.
Existing methods still have some limitations: for example, the granularity division is fixed, making it difficult to adapt to dynamic changes in action semantics; the interaction mechanisms between granularities are simple, often involving concatenation or weighting, lacking deep interaction modeling; and there is no systematic integration of granularity structure and temporal dimension information, making it difficult to capture the continuity of dynamic structures [3]. To address these issues, we achieves effective interaction between granularities through dynamic gated fusion and cross-attention mechanisms.
Skeletal Sequence Structural Information
In 3D human pose estimation, skeletal sequences not only carry the geometric positions of human joints but also implicitly contain rich structural information such as topological connections and motion patterns. Existing methods can be categorized as follows: Graph-based Structural Modeling methods model the human skeleton as a graph structure. ST-GCN [22] was the first to introduce spatiotemporal graph convolutions. SemGCN [19] further introduced semantic priors to enhance contextual modeling. PoseGraphNet [23] explores different graph structure designs. Prior-constrained modeling methods explicitly utilize physical constraints. Fang et al. [24] proposed a method based on pose grammar. FLEX [21] enhances structural stability by jointly predicting bone length and angle information. Temporal-structural modeling methods focus on the continuity of joint movements. Pavllo et al. [12] proposed a framework based on temporal convolutions. Strided Transformer [25] introduces a hierarchical Transformer to model long-range temporal dependencies. Cai et al. [26] unify spatial structure and temporal information modeling.
The above methods still have several issues: First, most methods rely on static topology or artificial priors, making it difficult to dynamically adapt to diverse human motion structures; Second, graph structures often model joint connections at a single granularity; Third, spatio-temporal structure modeling is mostly handled separately. To address this, this paper designs a dynamic gated fusion mechanism to explicitly encode structural information into attention calculations and a multi-granularity cross-attention mechanism to achieve alignment across granularities.
Spatio-Temporal Transformers in Pose Estimation
Transformers have been widely adopted in 3D human pose estimation due to their powerful global and sequence modeling capabilities. Current Transformer-based methods can be categorized as follows: First, Temporal Transformer from 2D Keypoints methods focus on using 2D keypoint sequences as input. A typical example is PoseFormer [15]. MHFormer [27] introduces a multi-hypothesis modeling mechanism. Second, Spatiotemporal Transformer methods jointly model spatial structures and temporal dynamics. LiftFormer [14] implements an efficient conversion from 2D to 3D pose. Strided Transformer [25] introduces strided convolution to progressively aggregate temporal information. Third, Multi-view and Temporal Fusion Transformer methods address high redundancy in multi-view and video scenarios. MTF-Transformer [28] designs multi-view and temporal fusion modules. TransFusion [29] utilizes Transformers to establish cross-frame semantic associations.
Existing methods still have issues: First, most do not fuse structured topological information; Second, the fusion of spatio-temporal information is typically modeled sequentially; Third, models often uniformly process video sequences without a keyframe selection mechanism. To address these issues, our proposed network designs a progressive multi-granularity feature extraction structure, combines feature enhancement and cross-attention mechanisms, introduces a dynamic gating fusion module, and uses a keyframe attention mechanism to balance efficiency and accuracy.
Method
Problem Analysis and System Design
Although Transformers have been widely used in monocular 3D human pose estimation, existing methods still lack structure-aware modeling, often rely on static priors, and fail to align multi-granularity features across skeletal hierarchies[30][5]. Moreover, uniform temporal modeling introduces redundancy and reduces computational efficiency[31, 17].
To address the lack of structured information utilization, we introduce GASA to dynamically embed skeletal connection relationships into the attention calculation for multi-granularity, Compared to existing GCNs that rely on fixed adjacency matrices or Transformers that are fully data-driven, GASA can dynamically adapt to skeletal dependencies under varying actions and pose changes. To address the issue of insufficient granularity alignment, We propose MGFE and HCA for better feature alignment. Unlike approaches that use only single-granularity features or simple concatenation, our approach fully leverage the complementary strengths of joints, bones, and limbs in capturing fine details and understanding structural context. To address the inefficiency in temporal modeling, we proposed TSF and KFA to improve efficiency. Unlike conventional methods that uniformly process entire video sequences or rely on fixed-length temporal windows, our approach adaptively aligns temporal context and selectively focuses on key motion frames.
As shown in Figure 1, the overall process is as follows: first, multi-granularity features are extracted via grouped pooling. Then, spatial and temporal modeling are performed sequentially using our proposed GASA, MGFE, TSF, and KFA mechanisms. Subsequently, HCA is used for feature interaction. Finally, inverse pooling restores fine-grained features for pose estimation.

Figure 1. The overall architecture of our proposed network (Structure-Aware Aligned Multi-Granularity Spatiotemporal Transformer). The model processes a 2D pose sequence through three parallel, multi-granularity streams. Features are progressively refined and then fused via a Hierarchical Cross-Attention mechanism before being up-sampled for the final 3D pose estimation.
Multi-scale Skeleton Topological Representation
Based on the anatomical structure of the human skeleton [23], we divide it into six groups : head, torso, left arm, right arm, left leg, and right leg. Figure 2 shows our designed three-scale skeleton topology. The number of keypoints at each level is .

Figure 2. Representation of our multi-scale skeletal topology. The skeleton is progressively simplified from 17 joints (J1) to 11 joints (J2) and finally to 6 joints (J3) through a hierarchical pooling strategy.
After inputting the monocular feature sequence, the grouped skeleton pooling layer extracts coarse-grained skeleton features from the fine-grained ones using 2D convolution. The group-wise pooling operation is defined as:
where the skeleton is partitioned into groups based on human body parts. The input features of the -th skeleton pooling layer can be denoted as , where represents the feature of group at level , is a feature matrix for group with joints and -dimensional features per joint, and is the learnable parameter matrix that reduces the number of keypoints in group from to .
Following the multi-granularity spatiotemporal alignment by HCA at the final stage of the pipeline, The unpooling layer applies 2D transposed convolution to each group individually, restoring coarse-grained skeletal features to fine-grained high-resolution representations for pose estimation:
In the -th skeleton unpooling layer, the input features represented as which passed through the unpooling operation to restore the structural resolution. Specifically, is a learnable parameter that upsamples group from keypoints to keypoints.
Gated Adaptive Spatial Attention (GASA)
To integrate skeletal topology priors, we propose a Gated Adaptive Spatial Attention (GASA) mechanism that incorporates multi-scale skeletal connectivity into attention computation to enhance spatial relationship modeling between keypoints.
We construct a multi-level skeletal adjacency matrix , where if joint and are physically connected, otherwise . Based on this, we design a dynamic gating mechanism to adaptively fuse prior knowledge and original attention. A gating matrix is generated via a fully connected layer followed by a Sigmoid function , This gating matrix adaptively fuses the two attention branches to produce the final attention weights:
where denotes the original attention matrix, and is the adjacency matrix at layer. This formulation allows each keypoint to adaptively balance between prior structural knowledge and local attention cues. The final attention matrix is used to update features by multiplying with the value matrix .
Multi-Granularity Feature Enhancement (MGFE)
To improve the model’s ability to extract spatial features, we propose a Multi-Granularity Feature Enhancement (MGFE) mechanism, as shown in Figure 3. From left to right are the first to third levels of feature extraction. For the second and third levels, we fuse the pooled features from the raw input with the output features from the previous level, as shown in Equation (4):

Figure 3. Illustration of the multi-granularity feature enhancement mechanism (MGFE). At the bone and limb levels, features from the original input are pooled and concatenated with the output of the previous, finer-grained encoder.
Here, represents the output features from the -th level, denotes the pooled features from the original input , and denotes feature concatenation.
After extracting joint-level spatial correlations via the joint-level Spatial Transformer, to enhance the utilization of raw input information, we apply pooling to the original input and concatenate the resulting features with the joint-level outputs from the previous layer. This combined feature is then fed into the organ-level Spatial Transformer to extract organ-level structural information. At the third level, we again pool the original input, concatenate it with the output from the previous level, and feed it into the limb-level Spatial Transformer to model spatial dependencies at the limb level.
The pooled features from the raw input contain global skeletal structure information, which serves as prior knowledge and enriches the spatial representation. Fusing these global features with local features from the previous level allows the model to capture not only localized spatial dependencies (e.g., joint and bone structures) but also global contextual information. This combination improves the model’s understanding of the spatiotemporal nature of the human skeleton across different levels. For example, in multi-step or compound actions, local features help capture detailed movement patterns, while global features provide broader structural context. Their combination enables the model to more accurately estimate the 3D pose.
Temporal Segment Fusion (TSF)
To enhance temporal feature capture, we introduce a Temporal Segment Fusion (TSF) mechanism. By explicitly introducing temporal context through feature concatenation, the model achieves faster training and improved temporal sequences understanding. Before the temporal Transformer, for each time step , we concatenate the first half of the feature vector from the previous time step with the second half of the current time step’s feature vector:
where and denote the first and second halves of the feature vector, respectively, and denotes concatenation.
The temporal segment fusion mechanism enables the model to explicitly perceive the relationships between adjacent time steps during training. Instead of learning temporal continuity from scratch, the model can directly leverage the concatenated features, thereby accelerating training and improving its ability to model temporal dependencies.
Keyframe Attention Mechanism (KFA)
We propose a temporal Transformer module based on sparse attention across blocks, called the Keyframe Attention (KFA) mechanism. First, the temporal dimension is divided into equal-length sub-blocks of length in the previous step. Then, block-level features are generated through average pooling. To filter key temporal segments, we use a gating mechanism to compute attention weights for each block:
where and are trainable parameters. We select the top- blocks, reducing computational complexity from to . The selected blocks are then treated as patches and processed by a standard Transformer encoder with position embeddings [32].
The selected sparse blocks are passed through the temporal Transformer module to extract inter-block dependencies. Specifically, the key temporal block features are flattened along the spatial dimension and treated as patches. These patches are then processed sequentially by Multi-Head Self-Attention (MSA), a Multi-Layer Perceptron (MLP), and Layer Normalization (LN), resulting in the final output representation of the same dimension.
Hierarchical Cross-Attention (HCA)
Conventional methods typically employ simple feature concatenation or weighted averaging when modeling multi-granularity human pose features, failing to fully exploit the intrinsic relationships between features at different granularities. To fuse multi-level spatiotemporal features, we design a Hierarchical Cross-Attention (HCA) mechanism. The model extracts three levels of features . After upsampling and to the dimension of (denoted as ), the HCA module, composed of triple cross-attention units, performs the interaction as follows:
Each attention unit uses the standard multi-head attention mechanism:
Multi-granularity Loss Function (MLF)
The proposed multi-granularity loss combines intermediate and final representations, offering stronger supervision than traditional joint-error losses by adaptively weighting each granularity, as experimentally validated.
The network designed in this paper adopts a three-level progressive feature extraction structure. After processing human skeleton topologies of different granularities through the spatial and temporal Transformer modules, the feature tensors output at each level can be expressed as , , and . To fully utilize multi-granularity feature information, the intermediate features are mapped to the target space through the previously mentioned inverse-pooling operation and linear transformation, as shown in (11):
where represents the group feature of the output features at each level, represents the learnable parameters of the inverse-pooling operation, represents the dimension mapping operation implemented by a fully connected layer, and represents the inverse-pooling operation that restores the number of joint nodes to 17.
In terms of the weight coefficient design for the hierarchical supervision strategy, this paper proposes weight settings based on clear physical significance. Since the sets of keypoints corresponding to the three-level structure of the network are , the weight coefficient is determined by the proportional relationship between the number of joint nodes at each level and the base level, as shown in (12):
From this, the weights for each layer are . The improved loss function is as follows:
where is the MPJPE loss for the prediction .
Experiments
Datasets and Experimental Settings
We evaluate on Human3.6M [33] and HumanEva-I [34], following standard protocols from previous works [12, 15, 27]. For Human3.6M, we train on S1, S5, S6, S7, S8 and test on S9, S11. For HumanEva-I, we evaluate on three actions from S1, S2, S3. The model is implemented in PyTorch [35] and trained on a single NVIDIA Tesla V100 GPU. We use the Adam optimizer [36] with a learning rate of 0.0001, weight decay of 0.1, and stochastic depth [37] with a drop rate of 0.1. The batch size is 1024, and training runs for 200 epochs.
Ablation Study
We conduct ablation experiments to analyze the contribution of each module, using CPN as the 2D pose detector with an input sequence length of . As shown in Table 1, the full model achieves an MPJPE of 46.0 mm. Removing the Multi-granularity Loss Function (MLF), Hierarchical Cross-Attention Module (HCA), Multi-granularity Spatial Transformer Module (MSTM) (include GASA and MGFE), and Multi-granularity Temporal Transformer Module (MTTM) (include TSF and KFA) progressively degrades performance, demonstrating the effectiveness of each component.
| Model | MSTM | MTTM | HCA | MLF | MPJPE (mm) |
| 1 | 54.9 | ||||
| 2 | 47.7 | ||||
| 3 | 47.0 | ||||
| 4 | 46.0 |
Table 1. Effect of different components on model performance.
We also verify the effectiveness of the MGFE and GASA mechanisms. As shown in Table 2, the full method achieves the lowest MPJPE of 46.0 mm. Removing either mechanism results in a performance drop, demonstrating their synergistic effect without significant computational overhead.
| Method | Params (M) | FLOPs (M) | MPJPE (mm) |
| Ours w/ MFE & GASA | 6.39 | 43.40 | 46.0 |
| Ours w/o MFE | 6.39 | 43.36 | 47.4 |
| Ours w/o GASA | 6.39 | 43.36 | 47.5 |
| Ours w/o MFE & GASA | 6.39 | 43.32 | 47.8 |
Table 2. Effect of MFE and GASA mechanism.
This study validates the TSF and KFA. As shown in Table 3, their combination achieves the lowest MPJPE. Notably, removing KFA significantly increases FLOPs and degrades performance, highlighting its critical role in improving efficiency.
| Method | Params (M) | FLOPs (M) | MPJPE (mm) |
| Ours w/ TSF & KFA | 6.39 | 43.40 | 46.0 |
| Ours w/o TSF | 6.39 | 43.40 | 46.7 |
| Ours w/o KFA | 6.39 | 165.29 | 54.1 |
| Ours w/o TSF & KFA | 6.39 | 165.29 | 55.3 |
Table 3. Effect of TSF and KFA mechanism.
Table 4 presents the impact of different parameter configurations in the spatial Transformer. The configuration of 2 layers and (16,24,32) channels offers the best trade-off.
| Params (M) | FLOPs (M) | MPJPE (mm) | ||
| 2 | 16,16,16 | 3.99 | 26.62 | 47.3 |
| 2 | 16,24,32 | 6.39 | 43.40 | 46.0 |
| 2 | 16,32,48 | 9.92 | 68.71 | 46.5 |
| 4 | 16,16,16 | 4.00 | 30.64 | 48.7 |
| 4 | 16,24,32 | 6.42 | 51.03 | 47.8 |
| 4 | 16,32,48 | 9.98 | 81.88 | 47.0 |
Table 4. Effect of layers and feature lengths of the spatial transformer.
Table 5 shows that for the temporal Transformer, using 4 layers and selecting key blocks achieves the best performance.
| Params (M) | FLOPs (M) | MPJPE (mm) | ||
| 2 | 4 | 3.48 | 20.19 | 49.1 |
| 2 | 5 | 3.48 | 23.10 | 48.3 |
| 2 | 6 | 3.48 | 26.00 | 48.1 |
| 2 | 7 | 3.48 | 28.90 | 47.4 |
| 4 | 4 | 6.39 | 31.79 | 47.8 |
| 4 | 5 | 6.39 | 37.60 | 46.8 |
| 4 | 6 | 6.39 | 43.40 | 46.0 |
| 4 | 7 | 6.39 | 49.21 | 47.5 |
Table 5. Effect of layers and blocks of the temporal transformer.
Comparative Experiments
Table 6 shows a comparison on Human3.6M under Protocol 1. Our method, with frames, achieves an average MPJPE of 46.0 mm, outperforming previous methods like Pavllo et al. [12] (48.8 mm) and Zheng et al. [15] (47.0 mm).
| Method (CPN) | Pub. | Dir. | Disc. | Eat. | Greet | Phone | Photo | Pose | Purch. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg. |
| Pavllo et al. [12] () | CVPR’19 | - | - | - | - | - | - | - | - | 56.5 | 64.5 | 47.4 | 43.0 | 48.1 | 33.0 | 35.1 | 48.8 |
| Lin et al. [38] () | arXiv’19 | 42.5 | 44.8 | 42.6 | 44.2 | 48.5 | 57.1 | 52.6 | 41.4 | 56.5 | 64.5 | 47.4 | 43.0 | 48.1 | 33.0 | 35.1 | 46.6 |
| Li et al. [39] () | CVPR’20 | 43.5 | 44.8 | 43.9 | 44.1 | 47.7 | 56.5 | 44.0 | 44.2 | 55.8 | 67.9 | 47.3 | 46.5 | 45.7 | 33.4 | 33.6 | 46.6 |
| Zheng et al. [15] () | ICCV’21 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 47.0 |
| Ours () | 42.9 | 46.8 | 47.8 | 42.9 | 46.2 | 54.4 | 47.8 | 41.8 | 47.7 | 61.7 | 47.2 | 44.5 | 52.4 | 31.6 | 35.0 | 46.0 |
Table 6. Quantitative Comparison on Human3.6M with detected 2D poses as inputs under Protocol 1 (MPJPE in mm).
To evaluate generalization, we tested on HumanEva-I. As shown in Table 7, our method achieves an average P-MPJPE of 17.2 mm, demonstrating strong cross-dataset generalization.
| Method | Pub. | Walk | Jog | Avg. | |||||
| S1 | S2 | S3 | S1 | S2 | S3 | ||||
| Pavlakos et al. [40] | CVPR’18 | 18.8 | 12.7 | 29.2 | 23.5 | 15.4 | 14.5 | 19.0 | |
| Lee et al. [41] | ECCV’18 | 18.6 | 19.9 | 30.5 | 25.7 | 16.8 | 17.7 | 21.5 | |
| Pavllo et al. [12] | CVPR’19 | 13.9 | 10.2 | 46.6 | 20.9 | 13.1 | 13.8 | 19.8 | |
| Liu et al. [42] | CVPR’20 | 13.1 | 9.8 | 26.8 | 16.9 | 12.8 | 13.3 | 15.5 | |
| Liu et al. [43] | ICRA’21 | 13.7 | 9.2 | 46.2 | 20.1 | 12.5 | 12.7 | 19.1 | |
| Zheng et al. [15] | ICCV’21 | 14.4 | 10.2 | 46.6 | 22.7 | 13.4 | 13.4 | 20.1 | |
| Li et al. [25] | TMM’22 | 14.0 | 10.0 | 32.8 | 19.5 | 13.6 | 14.2 | 17.4 | |
| Ours | 13.2 | 12.4 | 31.3 | 19.7 | 13.1 | 13.5 | 17.2 |
Table 7. Quantitative Comparison on HumanEva-I under Protocol 2 (P-MPJPE in mm).
Figure 4 illustrates qualitative results of our method on various in-the-wild videos, showcasing its robustness and practical applicability.

Figure 4. Qualitative results of our method on in-the-wild video datasets. Our method successfully reconstructs 3D poses for various complex actions, including (a) playing table tennis, (b) practicing Tai Chi, and (c) riding a bicycle.
Discussion
This paper addresses insufficient utilization of structured information, inadequate multi-granularity feature alignment, and inefficient temporal modeling in monocular 3D HPE. On Human3.6M, our method achieves an MPJPE of 46.0 mm, outperforming PoseFormer (47.0 mm) [15] and MHFormer (46.5 mm) [27]. On HumanEva-I, our method achieves an average error of 17.2 mm, superior to competitors, validating its cross-scene generalization. Crucially, our method maintains a significantly lower computational cost (FLOPs: 43.40M)
compared to MHFormer (210M+) or PoseFormer (150M+), leading to a substantial improvement in inference speed.
The MSTM, enhanced with GASA and MGFE, improves spatial awareness without increasing computational burden. The MTTM, with TSF and CSA, significantly reduces computational cost while improving accuracy by focusing on key temporal segments. Future work could explore learnable skeletal topologies, dynamic granularity selection, and cross-domain validation to further improve adaptability and generalization.
Conclusion
This paper proposed a multi-granularity spatiotemporal Transformer network that integrates structured skeletal information to address key challenges in monocular 3D HPE. Our method is designed along the lines of structure-aware modeling, multi-granularity hierarchical collaboration, and efficient keyframe selection. In spatial modeling, GASA and MGFE achieve adaptive fusion of structural priors. In temporal modeling, TSF and KFA enhance dynamic capture and efficiency. HCA facilitates collaborative feature alignment. Experimental results demonstrate that our method achieves state-of-the-art performance on public datasets like Human3.6M and HumanEva-I, significantly outperforming prior works while maintaining low computational cost. Future work will explore learnable topology optimization and dynamic granularity scheduling to further advance real-world applications.
Acknowledge
This work was supported in part by the National Natural Science Foundation of China (Grant No. 32271996).
References
- [1]J. Martinez, R. Hossain, J. Romero, J. J. Little, A simple yet effective baseline for 3d human pose estimation, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2640–2649.arxiv.org/abs/1705.03098
- [2]P. Wang, Z.-Q. Li, C.-G. Li, G. Lu, Deep 3d human pose estimation: A review, Computers & Visual Media 7 (2021) 331–356.DOI
- [3]W. Sun, J.-X. Li, L.-Z. Ren, C.-G. Li, G. Lu, A survey of the state of the art in monocular 3d human pose estimation: Methods, benchmarks, and challenges, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (4) (2022) 4552–4570.
- [4]J.-X. Li, W. Sun, C.-G. Li, G. Lu, A survey on depth ambiguity of 3d human pose estimation, IEEE Access 9 (2021) 111746–111759.
- [5]L. Li, X. Liu, Y. Xu, J. Liu, L. Xu, MSTFormer: Multi-granularity spatial-temporal transformers for 3d human pose estimation, in: European Conference on Computer Vision, Springer, 2022, pp. 311–327.
- [6]Z. Luo, B. Zhu, J. Li, X. Liu, L. Xu, GraFormer: Graph convolution transformer for 3d pose estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12971–12980.
- [7]B. Duan, J. Li, GLA-GCN: Global-local adaptive graph convolutional network for 3d human pose estimation from monocular video, Neurocomputing 489 (2022) 238–249.
- [8]W. Chen, M.-H. Zhang, C. Wu, Spectral compression transformer with line pose graph for monocular 3d human pose estimation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 455–463.
- [9]W. Li, H. Liu, C. Li, C.-Y. Lin, Kinematics-aware spatial-temporal feature transform for 3d human pose estimation, Pattern Recognition 138 (2023) 109378.
- [10]J. Martinez, Y. Raaj, J. Romero, J. J. Little, Lifting from the deep: Convolutional 3d pose estimation from a single image, arXiv preprint arXiv:1701.00295 (2017).
- [11]Z. Zhang, J.-W. Chen, J.-F. Zhang, Towards alleviating the modeling ambiguity of unsupervised monocular 3d human pose estimation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 13589–13598.DOI
- [12]D. Pavllo, C. Feichtenhofer, D. Grangier, M. Auli, 3d human pose estimation in video with temporal convolutions and semi-supervised training, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 7753–7762.arxiv.org/abs/1811.11742
- [13]C.-H. Chen, D. Ramanan, Exploiting temporal information for 3d pose estimation, in: European conference on computer vision, Springer, 2016, pp. 835–850.
- [14]Y. Li, J. Shi, H. Zhang, LiftFormer: 3d human pose estimation using attention models, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11270–11279.
- [15]C. Zheng, S. Zhu, M. Mendieta, T. Yang, C. Chen, Z. Ding, 3d human pose estimation with spatial and temporal transformers, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 11656–11665.arxiv.org/abs/2103.10455
- [16]W. Zeng, Z. Cai, Y. Ma, S. Li, HSTFormer: Hierarchical spatial-temporal transformers for 3d human pose estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11280–11289.
- [17]Q. Li, C. Wang, Y. Zhang, Y. Chen, L. Wang, Y. Liu, PoseFormerV2: Exploring frequency domain for efficient and robust 3d human pose estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11259–11269.
- [18]J. Yuan, H. Chen, Z. Zheng, Y. Yu, W. Li, Three-dimensional human pose estimation with spatial–temporal interaction enhancement transformer, Applied Sciences 13 (10) (2023) 6164.
- [19]Z. Yan, P. Peng, X. Sun, X. Sun, Z. He, Semantic graph convolutional networks for 3d human pose regression, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9007–9016.DOI
- [20]W. Shan, R. Zhang, Y. Zhou, PoseGTAC: Graph transformer encoder-decoder with atrous convolution for 3d human pose estimation, in: European Conference on Computer Vision, Springer, 2022, pp. 262–278.
- [21]T. Zeng, X. Sun, X. Sun, W. Liu, Flexible graph convolutional network for 3d human pose estimation, in: European Conference on Computer Vision, Springer, 2020, pp. 313–329.
- [22]S. Yan, Y. Xiong, D. Lin, Spatial temporal graph convolutional networks for skeleton-based action recognition, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 32, 2018.arxiv.org/abs/1801.07455
- [23]H. Ci, C. Wang, X. Wang, Y. Sun, PoseGraphNet++: Enriching 3d human pose with orientation estimation, in: European Conference on Computer Vision, Springer, 2020, pp. 346–362.
- [24]H.-S. Fang, G. Xie, Y. Wang, C. Lu, Learning pose grammar to encode human body configuration for 3d pose estimation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, 2018.arxiv.org/abs/1710.06513
- [25]W. Li, H. Liu, R. Ding, H. Tang, Exploiting temporal contexts with strided transformer for 3d human pose estimation, IEEE Transactions on Multimedia 25 (2022) 1282–1293.arxiv.org/abs/2103.14304
- [26]Y. Cai, L. Ge, J. Liu, J. Cai, T.-J. Cham, J. Yuan, Y. He, Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4052–4061.DOI
- [27]W. Li, H. Liu, C. Li, C.-Y. Lin, MHFormer: Multi-hypothesis transformer for 3d human pose estimation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 13147–13156.arxiv.org/abs/2111.12707
- [28]J. He, J. Yan, Y. Yu, Multiple view geometry transformers for 3d human pose estimation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11290–11299.
- [29]H. Zhang, C. Wang, Y. Sun, TransFusion: Cross-view fusion with transformer for 3d human pose estimation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11300–11309.
- [31]K. Cibrario, J.-M. Odobez, Hourglass tokenizer for efficient transformer-based 3d human pose estimation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 5550–5559.
- [32]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008.arxiv.org/abs/1706.03762
- [33]C. Ionescu, D. Papava, V. Olaru, C. Sminchisescu, Human3.6M: Large scale datasets and predictive methods for 3d human sensing in natural environments, IEEE transactions on pattern analysis and machine intelligence 36 (7) (2013) 1325–1339.DOI
- [34]L. Sigal, A. O. Balan, M. J. Black, HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion, International journal of computer vision 87 (2010) 4–27.DOI
- [35]A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, in: Advances in neural information processing systems 32, 2019.arxiv.org/abs/1912.01703
- [36]D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).arxiv.org/abs/1412.6980
- [37]G. Huang, Y. Sun, Z. Liu, D. Sedra, K. Q. Weinberger, Deep networks with stochastic depth, in: European conference on computer vision, Springer, 2016, pp. 646–661.arxiv.org/abs/1603.09382
- [38]J. Lin, G. H. Lee, Trajectory space factorization for deep video-based 3d human pose estimation, arXiv preprint arXiv:1908.08289 (2019).arxiv.org/abs/1908.08289
- [39]W. Li, R. Du, S. Chen, Skeleton-based spatio-temporal u-network for 3d human pose estimation in video, Sensors 22 (7) (2022) 2573.DOI
- [40]G. Pavlakos, X. Zhou, K. G. Derpanis, K. Daniilidis, Coarse-to-fine volumetric prediction for single-image 3d human pose, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7025–7034.arxiv.org/abs/1611.07828
- [41]K. Lee, I. Lee, Propagating lstm: 3d pose estimation based on joint interdependency, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 119–135.DOI
- [42]R. Liu, J. Shen, H. Wang, C. Chen, Z. Ding, Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 5064–5073.DOI
- [43]J. Liu, J. Rojas, Y. Li, Y. Guan, A graph attention spatio-temporal convolutional network for 3d human pose estimation in video, in: 2021 IEEE international conference on robotics and automation (ICRA), IEEE, 2021, pp. 3374–3380.arxiv.org/abs/2003.14179