DragonMoCap: Monocular Video-based Motion Capture for Diverse Dragon Topologies

Authors’ Contact Information: Zhongfei Qing, [email protected], Tencent, Shenzhen, China; Xiaoxiang Zhao, [email protected], Tencent, Shenzhen, China; Ling-Hao Chen, [email protected], Tsinghua University, Shenzhen, China; Zhongang Cai, [email protected], Nanyang Technological University, Singapore, Singapore; Wenjia Wang, [email protected], The University of Hong Kong, Shenzhen, China; Yifan Wu, [email protected], Tencent, Shenzhen, China; Qianying Liu, [email protected], University of Glasgow, Shanghai, China; Wei Zeng, [email protected], Tencent and Southern University of Science and Technology, Shenzhen, China; Bo Hu, [email protected], Tencent, Shenzhen, China; Songnan Li, [email protected], Tencent, Shenzhen, China. CCS Concepts: • Computing methodologies → Motion capture.

Introduction

Dragons occupy a pivotal position in modern digital content, ranging from AAA games to feature films. Despite their ubiquity, dragon animation remains a labor-intensive manual craft. Established pipelines that have commoditized human animation, such as large-scale mocap datasets and parametric models like SMPL(-X), fail to generalize to these complex fantasy creatures. This gap stems from two primary challenges. First, there is a critical scarcity of high-quality dragon motion data. Second, the lack of a canonical skeleton or consistent bone naming creates a significant topological barrier. Production rigs often feature highly diverse configurations, with joint counts ranging from 20 to over 60 and varying wing or leg structures. Consequently, a universal video-based motion capture system for dragons remains an elusive goal.

Existing monocular motion capture methodologies are intrinsically tied to fixed structural priors, such as the SMPL model, and typically regress a fixed-size output vector. This structural rigidity prevents these frameworks from being redirected to characters with different joint counts or hierarchical topologies. Specifically, the field lacks a video-based regressor capable of recovering high-fidelity motion directly from monocular frames when no clean 3D source is available. Furthermore, such a system must generalize to arbitrary dragon rigs without the need for expensive per-case finetuning or labor-intensive manual correspondences. As a result, introducing a topology-agnostic and flexible framework for dragon motion capture and synthesis is fundamental and meaningful.

To address these challenges, we introduce DragonMoCap, the first video-based motion capture framework designed for dragon-type characters. Given a monocular input video and a user-provided rig, defined by its bone names and hierarchy, our method predicts per-joint rotations independently for each frame and concatenates the predictions into a motion sequence. To ensure high-fidelity results and rigorous evaluation, we deliberately scope the framework to the three most prevalent production topologies: winged quadrupeds, wingless quadrupeds, and winged bipeds (wyverns). This targeted approach allows the model to be trained and validated on a high-quality dataset of professional animations while maintaining broad applicability across the most common fantasy morphologies.

The core design principle of DragonMoCap is to treat the target rig as a dynamic input rather than a hard-coded architectural constraint. Inspired by the success of open-vocabulary detectors, we represent each target joint as a language-grounded query token. Specifically, we encode each bone name using a frozen CLIP text encoder and combine the semantic embedding with topological and directed kinematic descriptors. The three embeddings are concatenated and projected, then processed by a skeletal encoder with topology-aware attention. The resulting joint queries, augmented with character-specific residuals, enter a shared transformer decoder

together with a global position token and image tokens from the current frame.

This formulation enjoys several key advantages. First, the decoder naturally handles rigs with arbitrary joint counts since the output is determined by the input query set. Second, the language-grounded nature of the queries ensures semantic consistency across diverse naming conventions, effectively bridging the gap between varying production standards. Empirically, we demonstrate that the combination of CLIP-based semantics and topological priors produces tokens that are discriminative within a single rig yet consistent across different characters. This property provides the essential foundation for robust, cross-rig motion synthesis without the need for manual correspondence authoring.

Building upon this semantic foundation, our framework further incorporates character-specific motion priors to achieve robust zero-shot generalization. While language-grounded queries effectively capture the universal semantic role of each bone, we enhance this representation during training by attaching learnable residuals that account for joint-specific range-of-motion and local kinematic coupling. As the model trains, these residuals naturally organize into a structured bone function space, where functionally equivalent joints seamlessly align. During inference on unseen rigs, we leverage this latent structure through a novel, non-parametric query-token retrieval mechanism. For each joint in a novel topology, the system ranks training bones using a weighted combination of semantic and topological similarity, then interpolates their residuals with softmax weights to augment the novel rig’s shared query. This strategy effectively closes the performance gap on unseen characters without requiring any retraining or incurring additional inference-time parameter costs.

We summarize our core contribution as follows.

• We introduce a novel and real-time system, DragonMoCap, capturing dragon motions from a video to diverse dragon-type characters, which is integrated as a UE add-on for practical animation workflows.

• We propose language-grounded joint query tokens that fuse CLIP-based semantics with hierarchical topological descriptors, which ensures intra-rig discriminability and cross-rig similarity, establishing a robust foundation for cross-topology generalization.

• We introduce query-token retrieval, a non-parametric mechanism that enables zero-shot generalization to unseen dragon rigs. This approach synthesizes high-quality query tokens for novel topologies without requiring any retraining or per-case optimization.

Related Work

Monocular Human Motion Capture

Monocular human motion capture has been largely driven by parametric human body models such as SMPL [20] and SMPL-X [24]. Early HMR-style methods directly regress human pose and shape from a single image [10], and later works improve robustness through optimization-in-the-loop training [13], part-aware attention [12], mesh-alignment feedback [Zhang et al. 2021], hybrid inverse kinematics [14], and full-frame location cues [17]. Video-based methods further exploit temporal context for more stable reconstruction [3, 11], while transformer-based systems such as HMR2.0/4DHumans improve image-based reconstruction and tracking [7]. TokenHMR improves human body recovery through a tokenized pose representation [5]. Recent whole-body methods extend the output space from body-only SMPL to expressive SMPL-X recovery, including one-stage whole-body regression [18] and large-scale expressive human pose estimation [1, 33]. Despite these advances, existing HMR pipelines remain tied to human-specific priors and fixed SMPL/SMPL-X parameterizations, making them difficult to apply to dragon rigs with diverse joint counts, hierarchies, and naming conventions. Synthetic datasets provide complementary supervision for human reconstruction. SynBody [32] combines layered, clothed human models with a scalable rendering pipeline to provide accurate 3D annotations for human pose and shape estimation and neural rendering. Our UE5-rendered dragon corpus similarly supplies paired images and motion labels, but targets heterogeneous production rigs rather than a human parametric body model.

Non-human Motion Capture, Synthesis, and Transfer

For non-human subjects, several works reconstruct articulated animals or deformable objects from images and videos, including template-free articulated shape reconstruction [30], neural animatable models from casual videos [31], category-level deformable 3D object learning [26], and articulated animal reconstruction in the wild [27]. These methods demonstrate the importance of moving beyond human templates, but they often focus on reconstructing shape, appearance, or category-level articulation rather than producing production-ready rotations for a user-specified rig.

In character animation, motion transfer and synthesis methods address heterogeneous skeletons from another direction. GenMM synthesizes motion from limited examples using generative motion matching [16], Motion2Motion transfers animation across different skeletal topologies with sparse correspondences [2], and AnyTop generates motions for arbitrary skeletons using skeleton-conditioned diffusion [6]. More directly related to monocular arbitrary-skeleton capture, MoCapAnything reformulates category-agnostic motion capture with a reference asset and factorized pose-to-rotation recovery [8], while MoCapAnything V2 learns an end-to-end pose-to-rotation pipeline [9]. CAST [23] is directly related to video-based capture for arbitrary target skeletons. It combines learned local rotations with predicted global bone directions and uses reliability-gated geometric correction in parent-to-child order to reduce hierarchical rotation errors. Our focus is language-grounded rig queries and retrieval of learned bone residuals for unseen dragon rigs; our decoder directly regresses rotations without a geometric correction stage.

Automatic Rigging and Representations

A parallel line of work studies how to make arbitrary 3D assets animatable by predicting skeletons and skinning weights. Classical and neural auto-rigging methods estimate animation skeletons and skinning for articulated characters [28], incorporate motion cues for rigging [29], or learn skeletal articulations with neural blend shapes [15]. Recent large-scale approaches further improve category generality through autoregressive or tokenized representations, including RigAnything [19], UniRig [34], Anymate [4], and SkinTokens [35]. These works are complementary to ours: they create or refine rigs, whereas DragonMoCap assumes a user-provided rig and estimates its motion from video.

Method

Overview and Problem Formulation

Setup. Given a monocular video $V = \{I_t\}_{t=1}^{T}$ with $T$ frames and a target character $c$, DragonMoCap recovers the motion of $c$ on its own rig. Each character is associated with a single rig, so we use $c$ to index both. The rig is a kinematic tree $\mathcal{G}^{(c)} = (\mathcal{J}^{(c)}, \mathcal{E}^{(c)})$ with joint set $\mathcal{J}^{(c)} = \{1, \ldots, N_c\}$, where $j = 1$ denotes the root joint. Each non-root joint $j$ has a parent $\pi(j)$, and the edge set $\mathcal{E}^{(c)} = \{(\pi(j), j)\}_{j=2}^{N_c}$ encodes the parent–child hierarchy. We denote by $\mathcal{A}(j)$ the set of ancestors of joint $j$, by $\ell_{jk}$ the number of edges on the tree path between joints $j$ and $k$ (so that $\ell_{\pi(j)j} = 1$), and by $n_j$ the semantic bone name attached to joint $j$.

For every frame $t \in\{1, \ldots, T\}$, DragonMoCap predicts the root translation $\hat{\mathbf{r}}_t \in\mathbb{R}^{3}$ in camera space and a rotation $\hat{\boldsymbol{\theta}}_{t,j} \in\mathbb{R}^{6}$ for every joint $j \in\mathcal{J}^{(c)}$. The root rotation $\hat{\boldsymbol{\theta}}_{t,1}$ is the global orientation of the character in camera space, while $\hat{\boldsymbol{\theta}}_{t,j}$ for $j \ge2$ is the local rotation of joint $j$ relative to its parent $\pi(j)$. All rotations use the continuous 6D representation. For brevity, we omit the superscript $(c)$ whenever the rig is clear from context.

Rig-conditioned prediction. To support different dragon rigs with one network, we treat the rig as a conditioning input. A shared decoder predicts each joint from a rig-derived joint query, so its output cardinality follows the target rig rather than a fixed joint vocabulary. Each joint query is the sum of two parts: a shared query $\mathbf{q}_j$, computed from the bone name, skeletal topology, and hierarchical relations of joint $j$ (§ 3.3), and a character-specific residual $\boldsymbol{\delta}_j$. Shared queries are computed in the same way for any rig. Residuals are learned for training rigs and, for an unseen rig, transferred from training rigs through non-parametric retrieval (§ 3.6).

Per-Frame Image Encoder

Following the common paradigm of image-based human mesh recovery [5, 7, 33], we estimate motion by processing each video frame independently. For each frame $I_t$, a frozen DINOv2 ViT-L/14 [22] extracts spatially distributed visual patch features $\mathbf{M}_t \in\mathbb{R}^{P \times d_v}$, where $P$ is the number of patch tokens and $d_v$ is the feature dimension. These features serve as visual memory for the pose decoder (§ 3.4), where the joint queries gather image evidence through cross-attention.

Language-Grounded Joint Queries

Query design principles. Our query design serves two goals. First, intra-rig discriminability allows the model to distinguish joints with different roles, such as a finger and a leg, and to learn their respective motion patterns. Second, cross-rig similarity allows joints in an unseen rig to relate to similar joints encountered during training, providing cues for predicting their motion. To achieve both goals, we combine bone-name semantics for functional similarity, topological and hierarchical-relation features for structural context, and learned residuals for character-specific motion patterns.

Semantic embedding. Bone names vary across rigs in naming conventions and descriptive detail. Before text encoding, we use an LLM to rewrite each bone name $n_j$ into a more consistent and descriptive form, making its semantic information more explicit. A frozen CLIP text encoder [25], denoted $E_{\mathrm{text}}$, then extracts the semantic descriptor

where $\mathbf{e}^{\mathrm{sem}}_j$ is the semantic embedding. These features help the model learn shared motion patterns among joints with similar semantic functions across different rigs. For an unseen character, bone names provide cues about the likely motion roles of its joints, supporting generalization to new skeletons.

Topological and hierarchical-relation descriptors. Character rigs differ in skeletal topology. For each joint $j$, we construct a topological descriptor $\mathbf{f}_j$ that collects its hierarchy depth, child count, chain position, leaf status, subtree size, and ordering cues, together with laterality and role indicators parsed from the bone name. These features enable the model to learn repeatable patterns across similar skeletal structures. To further describe how joint $j$ is situated in the kinematic tree, we encode each of its directed ancestor and descendant relations, together with the corresponding path length $\ell_{jk}$, using a shared entry-wise encoder, and mean-pool the results into a relation descriptor $\mathbf{f}^{\mathrm{rel}}_j$. Two MLPs then produce the topological and relation embeddings:

Shared query construction. For each joint $j$, we concatenate the three embeddings and fuse them with a shared learned projection $\mathbf{W}_{\mathrm{in}}$ followed by GELU:

A skeleton encoder then refines these features with local attention over the kinematic tree, allowing each query to capture its structural context. To make attention aware of the direction of skeletal relationships, we add a hierarchy bias $\mathbf{H} \in\mathbb{R}^{N_c \times N_c}$, whose rows index the attending joint $j$ and whose columns index the attended joint $k$:

Let $\Omega(j) = \{k : \ell_{jk} \le\rho\}$ denote the local neighborhood of joint $j$ within $\rho$ hops. Each attention head $h$ learns a scale $\gamma_h$ for the bias, and the attention weights are

and $A_{h,jk} = 0$ otherwise. Here, $Z_{h,jk}$ is the scaled dot-product score between the query of joint $j$ and the key of joint $k$ in head $h$. After $L$ stacked attention and feed-forward layers, the skeleton encoder outputs the shared queries $\mathbf{q}_j = \mathbf{x}^{(L)}_j \in\mathbb{R}^{d}$. The same construction is used for training and unseen rigs.

Learnable residuals. We further augment the shared queries with learnable tokens to increase query expressiveness and capture motion patterns specific to each character. Let $\mathcal{C}_{\mathrm{train}}$ denote the set of training rigs. Each joint $j$ of a training rig $c \in\mathcal{C}_{\mathrm{train}}$ has a learnable residual $\boldsymbol{\delta}^{(c)}_j \in\mathbb{R}^{d}$, initialized with small random values and added to its shared query:

We refer to $\tilde{\mathbf{q}}^{(c)}_j$ as the adapted query. After training, the residuals are collected into a bone bank for inference on unseen rigs (§ 3.6).

Transformer Pose Decoder

For a rig with $N_c$ joints, the decoder operates on a query sequence of $N_c + 1$ tokens,

consisting of a learned, rig-independent translation token $q_{\mathrm{tr}} \in\mathbb{R}^d$ and the $N_c$ adapted joint queries, including that of the root joint. Each transformer layer applies self-attention among the tokens of $Q$, cross-attention from these tokens to the image features $M_t$, and a feed-forward block. Let $h_{t,\mathrm{tr}}$ and $h_{t,j}$ denote the final-layer outputs of the translation token and joint token $j$ for frame $t$. Two prediction heads produce

where $\operatorname{MLP}_{\theta}$ is shared across all joints. Thus, the root-joint token predicts the root orientation $\hat{\theta}_{t,1}$, and the remaining joint tokens predict local rotations. Because the joint queries form a variable-length set and the rotation head is shared, the decoder parameters are independent of joint count and identity, allowing one decoder to operate on different dragon rigs without changing its output head.

Model Training

We train the projection MLPs, the skeleton encoder, the residuals $\{\delta_j^{(c)}\}$, the translation token, the decoder, and the prediction heads with a per-frame pose loss. Both the image and text encoders remain frozen. Given ground-truth joint rotations $\theta_{t,j}$, converted to the same 6D representation, and ground-truth root translation $r_t$, the loss for frame $t$ is

which is averaged over all frames in a mini-batch. The rotation term includes the root joint and therefore supervises the root orientation together with the remaining joint rotations. The weights $\lambda_{\theta}$ and $\lambda_r$ balance rotation and translation supervision. Normalizing the rotation term by $N_c$ prevents rigs with more joints from dominating the batch objective.

Model Inference on Unseen Rigs

Bone bank and candidate scoring. After training, we index every training bone by $b = (c,i)$ with $c \in\mathcal{C}_{\mathrm{train}}$ and $i \in\mathcal{J}^{(c)}$, and denote the index set by $\mathcal{I}_{\mathrm{train}}$. The bone bank stores the semantic descriptor, the topological descriptor, and the learned residual of each training bone,

and remains fixed during inference. Here, $s$ and $f$ are semantic embedding and the descriptors defined in Eq. (1) and § 3.3. For an unseen rig $u$, we first compute its shared queries $q_j^{(u)}$ with the same semantic, topological, and hierarchical-relation encoding used for training rigs. For each target joint $j \in\mathcal{J}^{(u)}$, every bank entry $b$ receives a weighted semantic and topological similarity score

where $\operatorname{sim}$ denotes cosine similarity, and $\alpha$ and $\beta$ weight the semantic and topological terms. Semantic similarity identifies bones with related meanings, while topological similarity helps distinguish similarly named bones that occupy different positions in the skeleton hierarchy. Let $\mathcal{N}_m(j) \subset\mathcal{I}_{\mathrm{train}}$ contain the $m$ highest-scoring bank entries for joint $j$.

Residual interpolation. We transfer the selected residuals with temperature-scaled softmax weights:

where the temperature $\tau$ controls how strongly the transfer favors the highest-scoring entries. The unseen rig retains its own shared queries and receives only the interpolated residuals from training rigs. Mirroring Eq. (6), its adapted queries are

which are fed to the decoder in place of the training-rig queries in Eq. (7). The procedure requires no retraining or per-rig optimization, only candidate scoring and interpolation over the fixed bone bank. Sec. 4.6 examines the semantic structure of the learned residuals that supports this transfer.

Experiments

Implementation Details

Model implementation. We use a frozen DINOv2 ViT-L/14 [22] as the per-frame image encoder and a frozen CLIP ViT-L/14 [25] text tower for bone-name grounding. The transformer decoder has 8 layers, a hidden dimension of 512, and 8 attention heads, and is applied independently to each frame. All rig-dependent quantities, including CLIP bone-name embeddings, topological and hierarchical-relation descriptors, and shared queries, are computed once per rig and cached. Excluding the CLIP text tower, whose outputs are cached and never evaluated at inference time, the model comprises approximately 420M parameters, of which 115M are trainable. We train on 4× NVIDIA A100 GPUs with an effective batch size of 512 and input resolution 308 × 308, using AdamW [21] with an initial learning rate of $2 \times10^{-4}$ for 200 epochs.

Appearance augmentation. To reduce overfitting to the limited visual diversity of UE5 dragon assets, we apply Gemini-based appearance edits (skin texture, color palette, and background) to a small set of pose-clustered frames per rig, preserving the original pose and rotation labels.

Experimental Setup

Data. We use an internal dataset of professionally animated dragon clips rendered in Unreal Engine 5, with ground-truth joint rotations and root transforms exported directly from the engine. It contains 8 training rigs and 7 held-out rigs with 20–60 joints each, covering three topology classes in roughly balanced proportions: winged quadrupeds, wingless quadrupeds, and winged bipeds (wyvern-style). Each training rig provides 40K–120K animator-authored frames, which we augment with SLERP-interpolated poses to about 150K frames per rig (~ 1.2M in total). All quantitative evaluation is conducted on UE5-rendered held-out data; AI-generated inputs without ground truth are evaluated through a user study (§ 4.5). Evaluation protocol. We evaluate in two settings: (1) seen rigs, where the 8 training rigs are evaluated on held-out clips with unseen motions; and (2) unseen rigs, where the 7 held-out rigs share topology classes with the training rigs but differ in rig definitions, bone counts, and naming conventions, serving as the main cross-rig generalization setting. We report MPJPE (cm), the world-space mean per-joint position error, and PA-MPJPE (cm), its Procrustes-aligned variant, which factors out a per-frame similarity transform and focuses on articulation quality. Each rig is normalized to a 1.6–2 m bounding size to control for scale differences.

Baselines and variants. Since prior work on dragon motion reconstruction is limited, we adopt two representative SMPL-X-based human motion reconstruction methods, SMPLest-X [[33]] and OSX [[18]], and retrain both on our dragon training data. For fairness, we replace their image encoders with ours and adjust the remaining capacity to match our number of trainable parameters. Their output heads are defined over the joints of the training rigs, so they can only be evaluated on seen rigs. MoCapAnything (V1 & V2) [[8 b]] also targets non-humanoid motion capture, but official code and weights were not available at the time of our experiments. We further compare three query-design variants that differ from the full model only in how joint queries are constructed during training: learned embedding uses a free learnable embedding per joint, while CLIP only and topology only build the shared query from the semantic or the topological and hierarchical-relation embeddings alone. On unseen rigs, all variants use the same residual retrieval (§ 3.6), so their differences reflect query construction rather than the transfer mechanism.

Main Results

As shown in Table 1, DragonMoCap achieves the best performance in both settings. On seen rigs, it reduces MPJPE from 10.57 to 9.46 (−10.5%) relative to the strongest retrained baseline, SMPLest-X, showing that a unified, rig-conditioned query representation does not sacrifice accuracy on familiar rigs. On unseen rigs, residual retrieval transfers learned residuals without any training data for the target rig, and our model reaches 12.11 MPJPE and 8.74 PA-MPJPE, outperforming all query-design variants (14.62–15.32 MPJPE). Since all variants share the same retrieval procedure, this gain is attributable to the combined semantic and structural query design.

Method	Seen rigs	Seen rigs	Unseen rigs	Unseen rigs
	MPJPE↓	PA-MPJPE↓	MPJPE↓	PA-MPJPE↓
SMPLest-X [2026]	10.57	6.53	—	—
OSX [2023]	11.20	7.12	—	—
Learned embedding	10.24	6.58	14.62	9.32
CLIP only	10.51	7.20	15.32	9.59
Topo only	10.84	6.67	14.82	9.63
Ours	9.46	6.01	12.11	8.74

Empirical validation of query design. We verify that combining CLIP and topological features satisfies both properties in section 3.3 through a cross-rig bone retrieval experiment: given a bone from one dragon, we retrieve its semantically equivalent bone in another dragon by the cosine similarity between their queries. The experiment covers 8 dragon rigs, with 1,252 queries in total and an average of 47.9 candidate bones. Queries built from CLIP features alone show strong cross-rig similarity for corresponding bones, but often confuse bones sharing the same prefix at different depths, failing the intra-rig discriminability property. Topological features alone resolve such structural ambiguities but lack the semantic information needed to distinguish different parts. Their combination yields the most reliable retrieval, confirming that CLIP captures what a bone is, while topology captures where it is located.

Effect of query design on motion capture. The learned-embedding, CLIP-only, and topology-only variants in Table 1 share the same decoder, training objective, and residual retrieval, so their differences isolate the effect of query construction. Neither CLIP-only nor topology-only queries outperform learned embeddings, while the full model, which combines both descriptors, achieves the lowest MPJPE and PA-MPJPE in both settings. On unseen rigs, it reduces MPJPE from 14.62 for learned embeddings to 12.11, supporting the cross-rig generalization of the combined design.

To qualitatively analyze these effects, we provide t-SNE visualizations of the bone embeddings in fig. 5. As illustrated in fig. 5, the representations in (A), (B), and (C) are significantly less discriminative. Specifically, using only CLIP embeddings (A) leads to fragmented and sparse clusters, while relying solely on topological embeddings (B) fails to distinguish between anatomically similar parts, such as the “Forelimb” (blue) and “Hindlimb” (purple), which appear heavily overlapped. Baseline methods (C) like SMPLest-X (learned embedding only) exhibit a highly chaotic distribution where different semantic categories are intermingled. In contrast, our full design (D) achieves the most cohesive and distinct clustering, effectively separating even functionally similar limbs into well-defined groups. This visual evidence confirms that the synergy of CLIP-based semantic priors and structural topology is essential for learning a robust, rig-agnostic latent space.

Qualitative Results

Figure 3 compares DragonMoCap with the competing baselines on two seen-rig cases, a winged quadruped with an articulated tail and a winged biped (wyvern) with wing fans, and additionally shows our motion capture results on an unseen rig with a distinct joint layout, to which the baselines cannot be applied. Figure 4 illustrates a capability that follows directly from rig-conditioned decoding: simultaneously driving several dragon rigs of different topologies from a single input video, with each output respecting its own rig’s structure and without training data for the target rigs.

User Study

Since AI-generated dragon videos have no ground-truth motion, we conduct a user study. Forty participants compared DragonMoCap with the retrained SMPLest-X on 30 AI-generated dragon clips. The two outputs were shown side by side in random order without method labels and rated for motion fidelity, anatomical plausibility, and overall visual quality on a 5-point Likert scale; paired differences are assessed with the Wilcoxon signed-rank test. As shown in Table 2, participants consistently preferred DragonMoCap, especially for anatomical plausibility (+1.3 Likert, 83.6% win rate, $p < 0.001$), indicating more natural dragon motion even when both methods are trained on the same data.

Criterion	SMPLest-X	Ours	$p$	Ours vs. SMPLest-X
Motion fidelity	3.7	4.4	$<0.001$	72.5%
Anatomical plausibility	3.2	4.5	$<0.001$	83.6%
Overall visual quality	4.1	4.6	$<0.001$	74.8%

Residual Space Analysis

Residual retrieval assumes that residuals learned for functionally equivalent bones in different training rigs are similar. To verify this, we group all learned residuals by semantic bone category (spine, limb, wing, tail, head, extremity) and compute pairwise cosine similarity across rigs. Within a category, residuals from different rigs reach $>0.85$ similarity on average for spine, limb, and tail bones, whereas similarity across categories drops below 0.2. Although each residual is learned for a specific joint of a specific character, residuals thus converge to motion priors organized by functional role rather than by character identity: the shared query specifies which joint is predicted, and the residual adds the motion characteristics of its role. This shared structure is what makes retrieval-based transfer possible.

Retrieval quality. In a leave-one-rig-out experiment, where each training rig is removed from the bone bank and its joints are retrieved from the remaining rigs, retrieval achieves 78.3% top-1 and 94.6% top-5 semantic-role match, supporting residual transfer between functionally related bones.

Explainability

To examine whether the model attends to semantically meaningful regions, we visualize the cross-attention maps between bone queries and input frames (Figure 6). Queries for anatomical parts such as “Head” consistently activate the dragon’s head across different poses and rigs (rows A and B). The model also distinguishes symmetric components, such as “Wrist_L” and “Wrist_R”, attending precisely to the respective wing joints. This indicates that CLIP-based semantic grounding and topological priors allow the decoder to “look” at the correct locations when estimating the motion of each bone, rather than relying on purely statistical correlations.

Runtime Efficiency

On a single NVIDIA L20 GPU, the full per-frame pipeline (DINOv2 encoder and rig-conditioned decoder) runs at $\sim30$ FPS, sufficient for real-time preview and efficient offline processing of long clips. Rig-level quantities are computed once per rig and cached, adding no per-frame cost. When driving multiple rigs from one video, the image encoder is shared and only the lightweight decoder is re-run per rig, so the marginal cost of each additional rig is minor. This efficiency enables integration into mainstream animation workflows such as UE5 (section 6).

Application

Synthesizing motion from key-frame sketches. Beyond video-based capture, our framework enables a streamlined sketch-to-motion workflow. As illustrated in Figure 7, an artist can provide a sparse set of expressive key-frame sketches (A). DragonMoCap treats each sketch as an input to recover its corresponding 3D pose, which then serves as a key-frame constraint for a motion interpolation algorithm (e.g., GenMM [16]. By synthesizing intermediate frames between these recovered poses (shown as solid vs. semi-transparent rigs in B), the system automatically generates a dense and temporally coherent animation sequence on the target rig. This interface allows artists to produce complex dragon animations by specifying only a few critical poses.

Motion reassembly from one video. Our model generalizes well to unseen reassembled skeletons. Specifically, we construct hybrid rigs by combining parts from different characters, e.g., attaching the wings of dragon A to the body of dragon B. Although such combinations are not seen during training, our method still produces plausible and coherent motion results, as shown in Figure 8. This indicates that the model captures transferable motion priors and can handle novel bone compositions.

Animation User Interface

To facilitate the seamless integration of our model into professional production pipelines, we developed a dedicated Unreal Engine 5 (UE-5) plugin leveraging UE-5 editor framework and reflection system. As illustrated in Figure 9, the streamlined GUI panel serves as the primary interface for motion capture configuration. Users first select the target video (A) via an integrated file picker, which is automatically rendered by an in-scene viewer (F). After assigning the dragon mesh (B), the user specifies a subset of bones (C) for capture, effectively filtering out auxiliary gameplay bones that do not require animation. Once the output directory (D) is defined, clicking the DragonMoCap button (E) initiates the backend processing. The system tracks the progress in real-time and subsequently compiles the returned motion data into a native UE-5 animation asset. To support efficient evaluation, the plugin features synchronized playback, allowing users to perform a rigorous side-by-side comparison between the original footage (F) and the resulting character motion (G) within the engine 3D viewport.

Conclusion

We presented DragonMoCap, to our knowledge the first video-based motion-capture system for dragon-type characters. By treating the target rig as an input rather than an architectural constant, and specifying it through language-grounded query tokens that fuse semantic, topological, and kinematic features through a shared skeleton encoder, a single decoder operates on heterogeneous dragon rigs; a non-parametric query-token retrieval further extends this to rigs never seen during training. This design makes the system practical for production settings, where character rigs often differ substantially across assets and are rarely standardized. On our dataset of 8 training and 7 held-out dragon rigs, DragonMoCap outperforms HMR-style and learned-embedding baselines in every setting and remains usable on AI-generated inputs. These results suggest that language- and structure-conditioned motion capture is a promising direction for animating non-human characters with diverse skeletal layouts. The main remaining bottleneck is the image encoder’s appearance coverage, which we mitigate with generative appearance augmentation and a first/last-frame-conditioned production pipeline; scaling this further, together with extending beyond dragons, is our primary future direction.

References