betaFeedback

WorksComputer Science

RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection

Abstract

Facial landmark detection has advanced substantially under regular imaging conditions, yet occlusion, poor illumination, and large pose changes remain major sources of error because local facial evidence can become incomplete or misleading. Context-guided latent patch recovery offers a promising way to address this issue by inferring unreliable patch representations from surrounding facial context before heatmap prediction. However, the recovered context can itself be unreliable when it is weakly adapted to the input face, influenced by corrupted neighboring patches, or blended without preserving local spatial coherence. We propose RCFormer, a Reliability-aware Context Transformer for occlusion-robust facial landmark detection. RCFormer strengthens latent patch recovery with three reliability-oriented components: visible-context conditioning (VCC), which adapts the context representation using a visibility-weighted global facial summary; visibility-guided context aggregation (VGCA), which reduces the influence of unreliable source patches during attention; and spatial residual correction (SRC), which refines blended latent features using neighboring patch information. The recovered heatmaps are then used as reliability-aware structural cues for existing landmark detection networks. Experiments show that RCFormer improves heatmap recovery and achieves strong landmark localization results on the WFLW, COFW, and 300W datasets.

bmvc_review_ash.pdf

Introduction

Facial landmark detection (FLD) aims to identify semantically meaningful facial keypoints, such as those around the eyes, nose, mouth, and facial contour. It serves as a fundamental component in many face-related applications, including face recognition [24], facial expression analysis [1], head-pose estimation [6], and augmented reality [9]. With advances in deep neural networks, FLD methods have achieved strong performance under regular imaging

© 2026. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. conditions. However, their performance often degrades in challenging scenarios involving occlusion, extreme illumination, and large head-pose variation, where local facial evidence may be incomplete or misleading. In such cases, features extracted from affected regions can become corrupted, making accurate landmark localization difficult. Therefore, improving FLD robustness requires not only detecting reliable facial evidence but also recovering useful structural cues when local observations are degraded or unavailable.

A promising direction in occlusion-robust FLD is context-guided latent patch recovery, which addresses unreliable facial regions in the latent feature space before landmark prediction. Rather than relying only on features extracted from a potentially occluded patch, this approach estimates its representation from surrounding facial context. This is motivated by the strong structural dependency among facial regions: visible areas around the eyes, nose, mouth, and contour can provide cues about the expected representation of a missing or corrupted region. The recovered patch feature therefore serves as a context-derived alternative when the target patch is unreliable or unavailable. By recovering degraded latent features before heatmap decoding, context-guided recovery can generate more reliable heatmaps and improve landmark localization under partial visibility. ORFormer [3] is a representative method in this direction, introducing messenger tokens that aggregate information from non-target patches and use it for occlusion-aware feature recovery.

Although context-guided latent patch recovery offers an effective way to handle corrupted facial regions, it introduces a second reliability problem: the recovered context is useful only when the context itself is trustworthy. A context query that is mainly shaped by learned tokens may not fully adapt to the visible structure of the current face, where pose, expression, identity, and occlusion patterns can vary significantly. Moreover, the surrounding context may also contain unreliable patches, so aggregating features without explicitly suppressing corrupted source regions can propagate degraded information into the recovered representation. Finally, recovering each latent patch through patch-wise blending can overlook the local continuity of facial contours and neighboring structures, leading to features that are context-derived but not spatially coherent. These limitations motivate a recovery mechanism that is image-adaptive, visibility-aware, and spatially refined before heatmap decoding.

Building on context-guided latent patch recovery, we introduce RCFormer, a Reliability-aware Context Transformer for occlusion-robust FLD. RCFormer addresses the reliability limitations above by making context recovery image-adaptive, visibility-aware, and spatially coherent. First, visible-context conditioning (VCC) adapts the context representation using a visibility-weighted global facial summary, allowing the recovery branch to reflect the visible structure of the current face. Second, visibility-guided context aggregation (VGCA) suppresses unreliable source patches during attention, so corrupted regions contribute less to the recovered representation. Third, spatial residual correction (SRC) refines the blended latent features with neighboring patch information, improving local continuity especially in regions predicted to be unreliable.

We incorporate the recovered heatmaps generated by RCFormer as reliability-aware structural cues for existing landmark detection networks. These heatmaps transfer recovered facial structure to the landmark network and improve prediction under partial visibility. Experiments on WFLW [28], COFW [7], and 300W [25] show that RCFormer improves heatmap quality and achieves strong landmark localization performance. Our main contributions are summarized as follows:

  • We identify a reliability gap in context-guided latent patch recovery, where the recov-

ered context may be insufficiently adaptive to the input face, affected by unreliable source patches, or weakly constrained by local spatial structure.

  • We propose RCFormer, a reliability-aware context transformer with visible-context conditioning (VCC), visibility-guided context aggregation (VGCA), and spatial residual correction (SRC) to improve context recovery.

  • We validate RCFormer on WFLW, COFW, and 300W, showing improved heatmap recovery and strong landmark localization under challenging facial conditions.

Related Work

Facial Landmark Detection. Existing FLD methods mainly follow coordinate-regression or heatmap-regression formulations. Coordinate-regression methods directly predict landmark locations from CNN features using regression heads [4] [5]. Feng et al. [4] introduce wing loss for improved landmark supervision, Wu et al. [28] use facial boundary information to impose structural constraints, and Miao et al. [19] model complex facial shape variations through Fourier feature pooling. Transformer-based methods further use attention and structured representation learning for relational modeling [12]: Xia et al. [30] propose a coarse-to-fine decoder over sparse local patches, while Li et al. [15] learn landmark queries from pyramid features. Heatmap-regression methods predict spatial response maps for landmarks, with Newell et al. [21] using stacked hourglass networks and later works improving heatmap supervision and localization accuracy [7, 27, 33]. To reduce reliance on non-differentiable heatmap decoding, Jin et al. [8] reformulate prediction as confidence and offset estimation, Huang et al. [7] and Zhou et al. [33] design losses using both heatmap and landmark supervision, and Micaelli et al. [20] use a deep equilibrium formulation for cascaded landmark refinement. This ability to preserve facial structure while supporting both heatmap and landmark supervision has made heatmap-based methods highly effective in recent FLD, although their predictions can still degrade under occlusion.

Occlusion-aware Facial Landmark Detection and Feature Recovery. Several FLD methods improve robustness by modeling occlusion, visibility, or uncertainty during landmark prediction. Kumar et al. [11] estimate landmark location with uncertainty and visibility likelihood, while Li et al. [16] jointly predict landmarks, occlusion probability, and head pose. Wu et al. [29] also address landmark localization under large pose and occlusion. While effective, these methods often require visibility or occlusion-related supervision. Other works use local-predictor agreement or facial context to handle missing evidence: Burgos-Artizzu et al. [2] identify reliable regions through consensus among patch-based regressors, Merget et al. [18] incorporate global context in a fully convolutional framework, Zhu et al. [35] model geometric relationships between facial components, and Zhu et al. [34] capture hierarchical facial-component relations. Although these approaches improve robustness under partial visibility, they mainly down-weight, bypass, or compensate for unreliable regions.

A more direct alternative is to recover the latent representation of corrupted regions from visible facial context before heatmap decoding. Transformers provide a natural tool for such recovery because they model long-range token interactions, but their attention can become unreliable when some tokens are corrupted or occluded. Several methods recover degraded visual representations in related tasks, including occluded person re-identification [31], and mesh estimation [23], blind face restoration [32], and shadow-robust image generation [13]. However, these methods are designed for different settings, such as multi-frame recovery or generic face restoration, and are not directly tailored to single-image FLD heatmap recovery. For single-image FLD, this recovery problem is more specifically addressed by context-guided latent patch recovery, where corrupted facial patch representations are estimated from surrounding facial context before heatmap decoding. ORFormer [3] follows this idea by introducing messenger tokens that exclude the target patch during context aggregation and provide context-derived features for occlusion-aware blending with regular latent features. However, its recovery quality still depends on how reliably the context is formed: the context representation is weakly adapted to the current face, unreliable surrounding patches may still affect aggregation, and patch-wise blending may not fully preserve local spatial continuity. RCFormer addresses these limitations through visible-context conditioning (VCC), visibility-guided context aggregation (VGCA), and spatial residual correction (SRC).

Method

This section presents RCFormer for reliability-aware context-guided latent recovery. Following codebook-based recovery works [3] [32], we first adopt vector quantization [22] to learn a quantized heatmap prior (§ 4.1; Figure 1). After pre-training, the learned codebook and decoder are fixed and used as the prior for latent recovery. We then introduce RCFormer, which recovers reliable latent features from partially visible facial inputs using visible-context conditioning, visibility-guided context aggregation, and spatial residual correction (§ 4.2; Figure 2). Finally, the recovered heatmaps are used as structural cues for downstream FLD networks (§ 4.6).

Overview of the quantized heatmap prior learning stage

Figure 1. Overview of the quantized heatmap prior learning stage. A clean face image is encoded into latent features, quantized through a learned codebook, and decoded into an edge heatmap representation.

Architecture of RCFormer for reliable context-guided latent recovery

Figure 2. Architecture of RCFormer for reliable context-guided latent recovery. Given latent patches from an occluded input, RCFormer maintains regular and context token streams within an LL-layer reliability-aware recovery transformer. Visible-context conditioning (VCC) conditions context queries using reliable global evidence, while visibility-guided context aggregation (VGCA) suppresses unreliable source patches during context aggregation. The final regular and context tokens predict code sequences SIS_I and SCS_C, which are mapped through the fixed codebook to obtain ZIZ_I and ZCZ_C. The final unreliability map α\alpha guides patch-wise blending and gates spatial residual correction (SRC) to produce ZrecZ_{\mathrm{rec}}. The fixed decoder maps ZrecZ_{\mathrm{rec}} to the recovered heatmap HrecH_{\mathrm{rec}}. For downstream FLD, HrecH_{\mathrm{rec}} is concatenated with an early-stage detector feature map and fused using a lightweight 1×11 \times1 convolutional block before landmark prediction.

Learning the Quantized Heatmap Prior

Following codebook-based recovery works [3] [32], we first learn a quantized heatmap prior before training RCFormer. This prior is trained on clean facial structures so that the learned codebook and decoder capture heatmap patterns under reliable facial evidence. Such a clean heatmap-aligned prior reduces the uncertainty of later recovery, where corrupted latent patches must be mapped back to plausible facial structures. Specifically, we train a VQVAE-style [22] heatmap generator that quantizes encoder features through a learned codebook and decodes the quantized representation into facial edge heatmaps. We follow the edge heatmap definition used in FLD works [3] [28], where facial contours are represented as structural heatmap channels. Heatmap generation details and dataset-specific landmark-toedge mappings are provided in the supplementary material. After training, the codebook and decoder are fixed and used as the prior for the subsequent RCFormer recovery stage. Figure 1 illustrates this prior learning stage.

A clean facial image I∈Rh×w×3I \in\mathbb{R}^{h \times w \times3} is first mapped by an encoder EE to a latent representation Z=E(I)Z = E(I), where Z∈Rm×n×dZ \in\mathbb{R}^{m \times n \times d}. We denote the K=mnK = mn latent tokens in ZZ as {zk}k=1K\{z_k\}_{k=1}^{K}, with each token zk∈Rdz_k \in\mathbb{R}^{d}. Given a learnable codebook C={cr}r=1NC = \{c_r\}_{r=1}^{N}, where each code vector cr∈Rdc_r \in\mathbb{R}^{d}, each latent token is quantized by assigning it to its nearest codebook entry:

ak=arg⁡min⁡r∈{1,…,N} ∥zk−cr∥22,zkq=cak.(1)a_k = \underset{r \in\{1,\ldots,N\}}{\arg\min}\ \left\|z_k-c_r\right\|_2^2,\qquad z_k^q = c_{a_k}. \tag*{(1)}

The resulting index sequence is a={ak}k=1K∈{1,…,N}K\mathbf{a} = \{a_k\}_{k=1}^{K} \in\{1,\ldots,N\}^{K}, and the quantized latent representation is Zq={zkq}k=1K∈Rm×n×dZ^q = \{z_k^q\}_{k=1}^{K} \in\mathbb{R}^{m \times n \times d}. The decoder DD maps this quantized representation to predicted edge heatmaps H=D(Zq)\mathbf{H} = D(Z^q), where H∈Rh×w×NE\mathbf{H} \in\mathbb{R}^{h \times w \times N_E} and NEN_E denotes the number of facial edge heatmap channels.

The generator is trained with heatmap supervision and vector-quantization regularization. The heatmap term penalizes the difference between the predicted edge heatmaps H\mathbf{H} and the target heatmaps H^\hat{\mathbf{H}}, while the VQ term aligns the encoder features with their selected codebook vectors:

Lhm=∥H−H^∥22,Lvq=∥sg⁡(Z)−Zq∥22+β∥Z−sg⁡(Zq)∥22.(2)\mathcal{L}_{hm} = \left\|\mathbf{H}-\hat{\mathbf{H}}\right\|_2^2,\qquad\mathcal{L}_{vq} = \left\|\operatorname{sg}(\mathbf{Z})-\mathbf{Z}^q\right\|_2^2+\beta\left\|\mathbf{Z}-\operatorname{sg}(\mathbf{Z}^q)\right\|_2^2. \tag*{(2)}

Here, sg⁡(⋅)\operatorname{sg}(\cdot) denotes the stop-gradient operation, and β\beta controls the commitment term. The overall objective for training the quantized heatmap generator is

LQHP=Lhm+λvqLvq,(3)\mathcal{L}_{QHP} = \mathcal{L}_{hm}+\lambda_{vq}\mathcal{L}_{vq}, \tag*{(3)}

where λvq\lambda_{vq} balances heatmap reconstruction and vector-quantization regularization. After training, the codebook and decoder are kept fixed for the subsequent RCFormer recovery stage.

RCFormer for Reliable Context-Guided Latent Recovery

For faces affected by occlusion or other visibility degradations, encoder latent patches can become inconsistent with the clean facial structures captured by the learned heatmap prior. Building on context-guided latent patch recovery, RCFormer predicts reliable latent codes before heatmap decoding. However, context-guided recovery can still fail when the context branch copies from the target patch, attends to corrupted surrounding patches, or produces features that are not locally coherent.

To address these issues, RCFormer combines an LL-layer reliability-aware token recovery module with code prediction, latent blending, and spatial residual correction. It introduces three reliability-oriented components: visible-context conditioning (VCC), visibility-guided context aggregation (VGCA), and spatial residual correction (SRC). Given latent patches from the encoder, RCFormer predicts a patch-wise unreliability map α\alpha, a regular code sequence SIS_I, and a context code sequence SCS_C. The predicted codes are mapped to quantized latent features using the fixed codebook, combined according to α\alpha, and refined to obtain ZrecZ_{rec}. The fixed decoder then maps ZrecZ_{rec} to the recovered heatmaps HrecH_{rec}. Figure 2 illustrates the overall RCFormer architecture, including the regular and context token streams, VCC, VGCA, code prediction, latent blending, and SRC.

Regular and Context Token Encoding. RCFormer maintains two token streams over the encoder latent patches. At each layer, the regular stream updates the observed patch tokens XlX^l using standard self-attention followed by an FFN, with projections (QIl,KIl,VIl)(Q_I^l,K_I^l,V_I^l) computed from XlX^l. These tokens retain the available image evidence for code prediction. In parallel, the context stream maintains one context token for each patch, denoted by ClC^l. Each context token attends to the regular patch tokens, using its own query while taking KIlK_I^l and VIlV_I^l from the regular stream as keys and values to form an alternative representation of the corresponding patch from surrounding facial regions. This provides the base context stream, which is further conditioned by VCC and visibility-reweighted by VGCA in the following steps.

To prevent the context representation from directly copying its own patch, RCFormer applies a hard self-exclusion mask in the base context attention. For the ii-th context token and the jj-th regular token, the attention logit is defined as

Bijl=qC,il(kI,jl)Tdfor i≠j,Biil=−∞(4)B^l_{ij}=\frac{q^l_{C,i}(k^l_{I,j})^{\mathsf{T}}}{\sqrt{d}}\quad\text{for }i\ne j,\qquad B^l_{ii}=-\infty \tag*{(4)}

where qC,ilq^l_{C,i} is the query of the ii-th context token and kI,jlk^l_{I,j} is the key of the jj-th regular token. The context tokens are then updated by multiplying softmax(Bl)(B^l) with the regular-token value matrix VIlV_I^l. This gives each patch a context-derived counterpart inferred from other facial regions, while keeping direct self-patch information blocked.

Unreliability Estimation

RCFormer estimates patch unreliability from the discrepancy between each regular token and its context-derived counterpart. For the kk-th patch after layer ll, we compute

αkl+1=σ(Wαl+1dist⁡(xkl+1,ckl+1)),(5)\alpha_k^{l+1} = \sigma\left(\mathbf{W}_{\alpha}^{l+1}\operatorname{dist}(x_k^{l+1}, c_k^{l+1})\right), \tag*{(5)}

where dist⁡(⋅,⋅)\operatorname{dist}(\cdot,\cdot) is the element-wise squared difference, Wαl+1\mathbf{W}_{\alpha}^{l+1} maps the discrepancy vector to a scalar, and σ(⋅)\sigma(\cdot) maps αkl+1\alpha_k^{l+1} to (0,1)(0,1). A larger value indicates greater unreliability. We use this discrepancy because it directly measures disagreement between the observed patch representation and its context-derived estimate. In our analysis, attention-entropy-based unreliability and a 3×33 \times3 convolutional head applied to the regular–context discrepancy produced higher WFLW heatmap L2 loss. Moreover, α\alpha acts as a soft weight rather than a hard mask, so inaccurate estimates modulate rather than discard patch information. We set α1=0\alpha^1=0 and denote the final-layer estimate by α\alpha.

Visible-Context Conditioning (VCC)

A context token initialized only from a learned query is weakly image-adaptive, since its context formation is not explicitly conditioned on the visible structure of the current face. To make the context branch input-adaptive before aggregation, RCFormer conditions each context query on a visibility-weighted global summary of reliable facial evidence. Given the unreliability map from the previous layer, we first compute visibility weights:

wjl=1−sg⁡(αjl)∑t=1K(1−sg⁡(αtl))+ε.(6)w_j^l = \frac{1-\operatorname{sg}(\alpha_j^l)}{\sum_{t=1}^{K}(1-\operatorname{sg}(\alpha_t^l))+\varepsilon}. \tag*{(6)}

Using these weights, the global visible context is obtained as

gl=∑j=1Kwjlxjl.(7)g^l = \sum_{j=1}^{K} w_j^l x_j^l. \tag*{(7)}

The context query is then conditioned by adding a projected version of this global visible context:

q~C,il=qC,il+Wglgl.(8)\widetilde{q}_{C,i}^l = q_{C,i}^l + \mathbf{W}_g^l g^l. \tag*{(8)}

Here, Wgl\mathbf{W}_g^l is a learnable projection that maps glg^l to the context-query dimension. The operator sg⁡(⋅)\operatorname{sg}(\cdot) stops gradients through the unreliability weights, so 1−αl1-\alpha^l is used as a visibility estimate rather than a shortcut for optimizing query conditioning. At the first layer, where α1=0\alpha^1=0, the summary reduces to an average facial context; in later layers, it emphasizes patches predicted to be reliable. The conditioned query q~C,il\widetilde{q}_{C,i}^l replaces the base query qC,ilq_{C,i}^l in Eq. (4) when computing the context-attention score matrix.

Visibility-Guided Context Aggregation (VGCA)

Using the VCC-conditioned queries, RCFormer computes self-excluded context attention and then modulates the resulting attention probabilities according to source reliability. Let Al=softmax⁡(Bl)A^l=\operatorname{softmax}(B^l) denote the context-attention matrix obtained after VCC-based query conditioning and hard self-exclusion. Since some source patches may themselves be corrupted, we use 1−αl1-\alpha^l as the source visibility estimate:

A~ijl=Aijl(1−αjl)∑t=1KAitl(1−αtl)+ε.(9)\widetilde{A}_{ij}^l = \frac{A_{ij}^l(1-\alpha_j^l)}{\sum_{t=1}^{K} A_{it}^l(1-\alpha_t^l)+\varepsilon}. \tag*{(9)}

We apply visibility guidance after softmax so that attention first captures query–key compatibility, after which (1−αjl)(1-\alpha_j^l) reweights the normalized probabilities according to source reliability. This separates semantic matching from reliability modulation without altering the raw attention-logit scale. The subsequent renormalization suppresses unreliable source patches while increasing the relative contribution of reliable ones. The context tokens are then updated as Cl+1=FFN⁡(A~lVl)C^{l+1}=\operatorname{FFN}(\widetilde{A}_lV_l).

Code Prediction and Spatial Residual Correction (SRC). After the final layer of the reliability-aware token recovery module, the regular tokens XL+1X^{L+1} and the context tokens CL+1C^{L+1} are used to predict two code sequences, denoted by SIS_I and SCS_C, respectively. The predicted codes are mapped through the fixed codebook C\mathcal{C} to obtain the quantized latent features ZIZ_I and ZCZ_C. Here, ZIZ_I preserves information from the regular patch stream, while ZCZ_C provides a context-derived alternative inferred from surrounding facial regions.

Using the final unreliability map α\alpha, RCFormer first forms a patch-wise blended latent representation:

Zblend=(1−α)⊗ZI+α⊗ZC,(10)Z_{\mathrm{blend}}=(1-\alpha)\otimes Z_I+\alpha\otimes Z_C, \tag*{(10)}

where ⊗\otimes denotes element-wise multiplication between a patch-wise weight map and latent features along the channel dimension. This blend trusts the regular feature on reliable patches and shifts toward the context-derived feature on unreliable patches.

Patch-wise blending treats each latent position independently and may not fully preserve the local continuity of facial structures. RCFormer therefore applies spatial residual correction to refine the blended latent feature with neighborhood-aware information. A lightweight depthwise spatial operator extracts local context from ZblendZ_{\mathrm{blend}}, and a linear projection layer predicts the residual correction from both the blended feature and its local response:

Zrec=Zblend+α⊗ϕ([Zblend,ψsp(Zblend)]),(11)Z_{\mathrm{rec}}=Z_{\mathrm{blend}}+\alpha\otimes\phi([Z_{\mathrm{blend}},\psi_{\mathrm{sp}}(Z_{\mathrm{blend}})]), \tag*{(11)}

where ψsp(⋅)\psi_{\mathrm{sp}}(\cdot) denotes the depthwise spatial operator, ϕ(⋅)\phi(\cdot) is a linear projection layer that predicts the residual correction, and [⋅,⋅][\cdot,\cdot] denotes channel-wise concatenation. Since the correction is weighted by α\alpha, it is emphasized on unreliable regions while reliable patches remain largely preserved. Finally, the fixed decoder maps ZrecZ_{\mathrm{rec}} to the recovered heatmaps HrecH_{\mathrm{rec}}.

Training Objective. After learning the quantized heatmap prior, we train RCFormer while keeping the codebook C\mathcal{C} and decoder DD fixed, and fine-tune the encoder together with RCFormer. Although encoder adaptation may shift its feature distribution, the fixed codebook and decoder, together with code-prediction and heatmap-reconstruction supervision, constrain the recovered representation toward the learned clean heatmap prior. For supervision, each training image is passed through the learned quantized heatmap prior to obtain the target code sequence a\mathbf{a} by the nearest-code assignment in Eq. 1. This target sequence supervises both predicted code sequences SIS_I and SCS_C using cross-entropy over code indices. We also apply a heatmap reconstruction loss, Lhm=∥Hrec−H^∥22\mathcal{L}_{hm}=\lVert H_{\mathrm{rec}}-\widehat{H}\rVert_2^2, between the recovered heatmaps HrecH_{\mathrm{rec}} and the target heatmaps H^\widehat{H}. The complete objective is

LRCFormer=Lcode(SI,a)+Lcode(SC,a)+λhmLhm,(12)\mathcal{L}_{\mathrm{RCFormer}}=\mathcal{L}_{\mathrm{code}}(S_I,\mathbf{a})+\mathcal{L}_{\mathrm{code}}(S_C,\mathbf{a})+\lambda_{hm}\mathcal{L}_{hm}, \tag*{(12)}

where Lcode\mathcal{L}_{\mathrm{code}} denotes cross-entropy loss for code prediction and λhm\lambda_{hm} balances the heatmap reconstruction term.

Using Recovered Heatmaps for FLD

Following the integration protocol of ORFormer [3], we use the recovered heatmaps Hrec\mathbf{H}_{rec} as additional structural guidance for existing FLD methods [7, 33]. Specifically, Hrec\mathbf{H}_{rec} is concatenated with an early-stage feature map of the FLD backbone and fused using the same lightweight convolutional block as ORFormer. The fused representation is then forwarded to the landmark prediction head. We intentionally keep this detector-side fusion unchanged so that the performance gains can be attributed to RCFormer’s heatmap recovery mechanism rather than to a redesigned fusion module.

Experiments

Experimental Setup

Datasets and Metrics. We evaluate RCFormer on three standard facial landmark detection benchmarks: WFLW [28], COFW [2], and 300W [25]. WFLW contains 7,500 training images and 2,500 test images annotated with 98 landmarks, and provides subset labels for challenging conditions including pose, expression, illumination, makeup, occlusion, and blur. COFW contains 1,345 training images and 507 test images with 29 landmarks, and is mainly used to evaluate robustness under heavy facial occlusion; although landmark-level occlusion labels are available, we do not use them for training or evaluation. 300W contains 3,148 training images and 689 test images with 68 landmarks, and we follow the standard split of the test set into 554 common and 135 challenging images. Following prior FLD works [7, 33], we report normalized mean error (NME), failure rate (FR), and area under the curve (AUC). Inter-ocular normalization is used for WFLW and 300W, and inter-pupil normalization is used for COFW. The threshold for both FR and AUC is set to 10%.

Implementation Details. The quantized heatmap prior takes a 64×64×364 \times64 \times3 face image as input and predicts 64×64×NE64 \times64 \times N_E edge heatmaps, where NEN_E is the number of facial edges. Its latent feature has size m×n×dm \times n \times d, and the codebook contains NN entries of dimension dd. Following ORFormer [3], we set m=n=16m = n = 16, d=256d = 256, and N=2048N = 2048, and keep the remaining heatmap-prior settings unchanged for fair comparison. RCFormer uses an L=3L = 3 reliability-aware recovery transformer in this latent space, with token size 1×1×d1 \times1 \times d and m×nm \times n regular/context tokens. We set ε=10−6\varepsilon= 10^{-6} in VCC and VGCA. The SRC spatial operator is implemented as a 3×33 \times3 depthwise convolution to capture local neighborhood information with minimal overhead.

For landmark detection, we follow ADNet [7] and STAR [33], using a four-stacked hourglass network [21] as the backbone. The recovered heatmaps are concatenated with the backbone feature maps and fused by a lightweight 1×11 \times1 convolutional block before the first hourglass module. All models are trained from scratch on the target dataset without external data or pretrained weights.

Following ORFormer [3], we use Adam [10] with a cosine annealing warm-restart scheduler [17] for all stages, using the same restart period of 5 and period multiplier of 2. Training is performed on a single NVIDIA RTX 4090 GPU with 24GB memory, and all stages are trained for 300 epochs. For the quantized heatmap prior, the batch size is 128 for all datasets. The learning rate is 7×10−47 \times10^{-4} for WFLW and COFW, and 8×10−48 \times10^{-4} for 300W. We set β=0.25\beta= 0.25 for all datasets, with λvq=100\lambda_{vq} = 100 for WFLW and COFW and λvq=1000\lambda_{vq} = 1000 for 300W. For RCFormer, the batch size is 64 and the learning rate is 1×10−41 \times10^{-4} for all datasets. We set λhm=50\lambda_{hm} = 50 for WFLW and COFW, and λhm=100\lambda_{hm} = 100 for 300W. For the downstream FLD detector, the batch size is 16 and the learning rate is 1×10−31 \times10^{-3} for all datasets.

Data Augmentation. Following common training protocols in facial landmark detection and face recovery [7, 32, 33], we crop the face region and resize it to 64×6464 \times64 for the heatmap-prior and RCFormer stages. We apply random horizontal flipping with probability 0.5, random grayscale conversion with probability 0.5, rotation within ±30°, translation within ±4%, and scaling within ±5%. Following [3], an additional random occlusion is inserted into each image during RCFormer training to simulate unreliable facial regions; these synthetic occlusions are used only as augmentation. For the downstream landmark detector, images are resized to 256×256256 \times256, and random occlusion is applied with probability 0.4 while keeping the remaining augmentations unchanged. Improvements on the WFLW occlusion subset and COFW, both containing real occlusions, indicate that RCFormer does not merely learn synthetic occlusion patterns.

Comparison with Existing Methods

Table 1 reports the comparison on WFLW, COFW, and 300W. For WFLW, we separate methods according to the bounding-box protocol and restrict direct comparisons to methods evaluated using HRNet-based boxes. Under this protocol, RCFormer achieves the lowest NME and FR while matching the best AUC. RCFormer also obtains the lowest NME on COFW and all 300W splits. These results demonstrate consistent improvements across the evaluated FLD benchmarks.

MethodBackboneWFLW-FullCOFW300W (NME↓\downarrow)
NME↓\downarrowFR↓\downarrowAUC↑\uparrowNME↓\downarrowFullComm.Chal.
Original WFLW bounding boxes
LAB [28]Hourglass5.277.560.532-3.492.985.19
Wing [4]ResNet-504.996.000.5505.44---
AWing [27]Hourglass4.362.840.5724.943.072.724.52
PIPNet [8]ResNet-1014.31---3.192.784.89
RePFormer [15]ResNet-1014.11---3.01--
HRNet-based WFLW bounding boxes
HRNet [26]HRNet-W184.604.64--3.322.875.15
LUVLi [11]DU-Net4.373.120.577-3.232.765.16
ADNet [7]Hourglass4.142.720.6024.682.932.534.58
HIH [14]Hourglass4.082.600.6054.633.092.654.89
SLPT [30]HRNet-W184.142.760.5954.793.172.754.90
†STAR [33]Hourglass4.032.320.6114.622.902.524.46
LDEQ [20]Hourglass3.922.480.624----
ORFormer [3]Hourglass3.861.760.6224.462.902.534.43
RCFormerHourglass3.731.650.6244.292.812.404.32

Table 1. Comparison with baseline FLD methods. NME is reported for all datasets; FR and AUC at the 10% threshold are reported for WFLW. WFLW methods are grouped by bounding-box protocol, with direct comparisons and highlighting restricted to the HRNet-based protocol. Best results are shown in bold and second-best results are underlined. † denotes baseline results sourced from ORFormer [3].

For qualitative evaluation, we visualize landmark predictions on challenging WFLW samples in Figure 3. We compare RCFormer with STAR [33], which uses the same hourglass backbone. In the highlighted regions, RCFormer generally shows shorter landmark-error lines than STAR, particularly around occluded or weakly visible facial boundaries. This suggests that the recovered heatmaps provide useful structural guidance for landmark prediction in challenging cases.

Figure 3. Qualitative landmark comparison on challenging WFLW samples. Blue points denote ground-truth landmarks, red points denote predicted landmarks, and green line segments show the displacement between matched landmark pairs. Yellow ellipses highlight difficult regions where RCFormer produces shorter landmark-error lines than STAR.

Furthermore, we evaluated RCFormer on the challenging subsets of WFLW in Table 2. RCFormer improves over ORFormer on the full test set and achieves lower NME on all six subsets, showing that the proposed recovery design is effective across different challenging conditions. On the expression subset, RCFormer still improves over ORFormer but ranks second overall. This may indicate that expression variation is less directly addressed by visibility-oriented recovery than occlusion, pose, illumination, makeup, or blur. We also visualize the predicted unreliability maps in Figure 4. High-response regions tend to appear around occlusions, low-visibility areas, and ambiguous facial boundaries, while more reliable facial regions show lower responses. This suggests that the learned unreliability maps correlate with visually degraded regions and provide spatial guidance for feature recovery.

MethodBackboneFullPoseExp.Ill.Mak.Occ.Blur
NME↓
Original WFLW bounding boxes
LAB [28]Hourglass5.2710.245.515.235.156.796.12
Wing [4]ResNet-504.998.435.214.885.266.215.81
AWing [27]Hourglass4.367.384.584.324.275.194.96
PIPNet [8]ResNet-1014.317.514.444.194.025.365.02
RePFormer [15]ResNet-1014.117.254.224.043.915.114.76
HRNet-based WFLW bounding boxes
HRNet [26]HRNet-W184.607.944.854.554.295.445.42
ADNet [7]Hourglass4.146.964.384.094.055.064.79
HIH [14]Hourglass4.086.874.064.343.854.854.66
SLPT [30]HRNet-W184.146.964.454.054.005.064.79
†STAR [33]Hourglass4.036.784.263.973.854.824.59
LDEQ [20]Hourglass3.926.863.944.173.754.774.59
ORFormer [3]Hourglass3.866.634.133.833.614.574.50
RCFormer (Ours)Hourglass3.736.483.963.713.494.384.36

Table 2. WFLW subset evaluation. NME is reported on the full test set and six challenging subsets. Methods are grouped by bounding-box protocol, with direct comparisons and highlighting restricted to the HRNet-based protocol. Best results are shown in bold and second-best results are underlined. † denotes baseline results sourced from ORFormer [3].

Predicted unreliability maps from RCFormer

Figure 4. Predicted unreliability maps from RCFormer. Red regions indicate higher unreliability, highlighting occlusions, low-visibility areas, and ambiguous facial boundaries.

Ablation Studies

Recovered Heatmap Quality. To evaluate heatmap generation quality, we report the heatmap L2 loss in Table 3. The comparison includes VQVAE as the quantized heatmap generator baseline and ORFormer as the context-guided latent recovery baseline. ORFormer improves over VQVAE, confirming the benefit of context-guided latent patch recovery. RCFormer further reduces the L2 loss from 22.29 to 15.22 on WFLW-Full and achieves relative gains from

ArchitectureWFLWCOFW300W
FullPoseExp.Ill.Mak.Occ.BlurFullFull
Heatmap L2 Loss ↓
VQVAE29.3859.3732.6428.4533.2540.1234.2316.5725.97
ORFormer22.2943.2526.4922.0723.3128.4827.0512.8016.58
RCFormer15.2225.4118.4616.5616.7519.1217.208.3912.59
Rel. gain vs. ORFormer31.7%41.2%30.3%25.0%28.1%32.9%36.4%34.5%24.1%

Table 3. Heatmap generation comparison using L2 loss. All architectures are implemented under the same setting. Results are reported on WFLW and its six subsets, with full-set results on COFW and 300W. Bold percentages indicate relative L2 reduction of RCFormer over ORFormer.

24.1% to 41.2% over ORFormer across the evaluated settings. These reductions show that reliability-aware context recovery substantially improves recovered heatmap quality. The gains are larger for heatmap recovery than for NME because RCFormer directly optimizes the recovered heatmaps, while the downstream detector uses them only as structural guidance and remains constrained by detector capacity and annotation ambiguity. Nevertheless, the consistent NME improvements in Table 1 and Table 2 confirm their downstream benefit. Figure 5 directly compares the recovered heatmaps from VQVAE, ORFormer, and RCFormer with ground truth. RCFormer produces cleaner facial structures and lower per-sample L2 errors than the baselines.

Edge heatmap outputs comparing Input Face, Ground Truth, VQVAE, ORFormer, and RCFormer on two WFLW samples

Figure 5. Edge heatmap outputs on WFLW samples. Ground truth (GT) heatmaps are derived from landmark-based facial edge annotations, with brighter regions indicating stronger responses. Values denote per-sample L2 errors against GT.

Component Analysis of RCFormer. We evaluate the contribution of each RCFormer component on WFLW using heatmap L2 loss and landmark NME. The base model keeps the regular/context token streams, hard self-exclusion context attention in Eq. (4), unreliability estimation, dual code prediction, and patch-wise latent blending, but removes VCC, VGCA, and SRC. As shown in Table 4, adding VCC improves both metrics, showing that conditioning the context query with a visibility-weighted global context makes the recovered context more adaptive to the input face. Adding VGCA on top of VCC further improves both metrics, suggesting that visibility-guided context aggregation complements visible-context conditioning. SRC gives a strong standalone reduction in heatmap L2 loss by applying spatial correction under the guidance of the unreliability map, although its NME gain is limited when using the base reliability estimates. Combining SRC with VCC and VGCA gives the best L2 loss and NME, suggesting that reliability-aware context formation and spatial correction work together in RCFormer.

MethodVCCVGCASRCL2 Loss↓NME↓
Base–––22.053.96
Base + VCC✓––21.083.90
Base + VCC + VGCA✓✓–20.143.86
Base + SRC––✓18.463.89
RCFormer✓✓✓15.223.73

Table 4. Component ablation on WFLW. VCC, VGCA, and SRC denote visible-context conditioning, visibility-guided context aggregation, and spatial residual correction, respectively.

Computational Efficiency. Table 5 reports the complete inference cost of STAR, ORFormer, and RCFormer on WFLW. Compared with ORFormer, RCFormer adds only 0.004 G FLOPs and 0.898 ms latency while reducing NME from 3.86 to 3.73; peak GPU memory remains similar at 145.25 MB versus 143.48 MB. Compared with STAR, RCFormer adds 1.468 G FLOPs and 3.777 ms latency. At the recovery-network level, RCFormer has 5.31M parameters compared with 4.78M for ORFormer, an increase of approximately 0.53M (11.0%), with VCC and SRC accounting for the additional parameters while VGCA introduces no learnable parameters. Overall, these results show that the improved heatmap recovery and landmark localization are obtained with modest additional computational cost.

MethodParams (M)FLOPs (G)Latency (ms)NME↓\downarrow
STAR17.17734.79634.1544.03
ORFormer22.03836.26037.0333.86
RCFormer22.69936.26437.9313.73

Table 5. Inference efficiency on WFLW measured on an NVIDIA RTX 4090 with batch size 1 and FP32 precision.

Conclusion, Limitations and Future Work

We introduced RCFormer, a reliability-aware extension of context-guided latent patch recovery for occlusion-robust FLD. By combining visible-context conditioning, visibility-guided context aggregation, and spatial residual correction, RCFormer improves both recovered heatmap quality and landmark localization. Experiments on WFLW, COFW, and 300W demonstrate consistent improvements over strong baselines across challenging facial conditions, supporting the effectiveness of reliability-aware context recovery. The recovery mechanism may become less effective when occlusions are extremely large, reliable contextual evidence is scarce, or pose and expression variations dominate the localization error. While the results across three benchmarks support the robustness of RCFormer, extending the evaluation to cross-dataset settings would provide further insight into its generalization under unseen data distributions and represents a natural direction for future work.

Supplementary Material for “RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection”

Ashish Kumar; Prashant Kumar; Mallika Garg

supplementary_ash.pdf

Supplementary Materials

Heatmap Definition

RCFormer aims to identify unreliable facial regions and recover missing structural information from degraded face images, enabling the generation of high-fidelity heatmaps under challenging conditions such as occlusion, extreme pose, blur, and lighting variation. These recovered heatmaps provide facial contour guidance that helps facial landmark detection (FLD) methods maintain robustness in challenging scenarios.

Following ORFormer [2], which adopts the edge-based heatmap representation of Wu et al. [4], we use the same heatmap construction protocol and dataset-specific landmark-to-edge mappings. Using edge heatmaps reduces computational cost while preserving sufficient structural information for downstream FLD methods.

Heatmap Generation

For a given face image I∈Rh×w×3I \in\mathbb{R}^{h \times w \times3} and its ground-truth landmark annotations L={li}i=0NL−1L = \{l_i\}_{i=0}^{N_L-1}, we divide LL into NEN_E subsets Lj⊂LL_j \subset L, where j=0,…,NE−1j = 0, \ldots, N_E - 1, to represent facial edges such as the cheek, eyebrow, eye contour, nose, and mouth contour. Here, NLN_L denotes the number of landmarks per face, and NEN_E denotes the number of facial edges.

Each facial edge LjL_j is used to interpolate an edge line, forming a binary edge map BjB_j with the same spatial size as the face image. We then apply a distance transform to BjB_j, which computes the nearest distance from each pixel to the corresponding edge line and produces the distance map MjM_j. The ground-truth edge heatmap H^j\hat{H}_j used to supervise the quantized heatmap generator and RCFormer is defined as

H^j(x,y)={exp⁡(−Mj(x,y)22σ2),if Mj(x,y)<3σ,0,otherwise.(13)\widehat{H}_j(x,y) = \begin{cases} \exp\left(-\frac{M_j(x,y)^2}{2\sigma^2}\right), & \text{if } M_j(x,y) < 3\sigma, \\ 0, & \text{otherwise.} \end{cases} \tag*{(13)}

where σ\sigma denotes the standard deviation of the values in the distance map MjM_j.

Index Mapping

Our experiments are conducted on three datasets: WFLW [4], COFW [1], and 300W [3]. Since the number of landmarks varies across these datasets, the facial edge heatmap definitions are dataset-specific. Therefore, we provide the landmark-to-edge index mappings below.

For the WFLW dataset, with NL=98N_L = 98 and NE=15N_E = 15, the index mapping is given as follows:

Edge 0 : [0-32]

Edge 1 : [33-37]

Edge 2 : [38-41, 33]

Edge 3 : [42-46]

Edge 4 : [46-49, 50]

Edge 5 : [51-54]

Edge 6 : [55-59]

Edge 7 : [60-64]

Edge 8 : [64-67, 60]

Edge 9 : [68-72]

Edge 10 : [72-75, 68]

Edge 11 : [76-82]

Edge 12 : [82-87, 76]

Edge 13 : [88-92]

Edge 14 : [92-95, 88].

For the 300W dataset, with NL=68N_L = 68 and NE=13N_E = 13, the index mapping is

Edge 0 : [?, 1, 2, 3, 4, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?] Edge 1 : [?, ?, ?, ?, ?] Edge 2 : [?, ?, ?, ?, ?] Edge 3 : [?, ?, ?, ?] Edge 4 : [?, ?, ?, ?, ?] Edge 5 : [?, ?, ?, ?] Edge 6 : [?, ?, ?, ?] Edge 7 : [?, ?, ?, ?] Edge 8 : [?, ?, ?, ?] Edge 9 : [?, ?, ?, ?, ?, ?, ?] Edge 10 : [?, ?, ?, ?, ?, ?, ?] Edge 11 : [?, ?, ?, ?, ?] Edge 12 : [?, ?, ?, ?, ?].

For the COFW dataset, with NL=29N_L = 29 and NE=14N_E = 14, the index mapping is

Edge 0 : [?, 4, 2] Edge 1 : [2, ?, ?] Edge 2 : [1, ?, 3] Edge 3 : [3, ?, 1] Edge 4 : [?, ?, ?] Edge 5 : [?, ?, ?] Edge 6 : [?, ?, ?] Edge 7 : [?, ?, ?] Edge 8 : [?, ?, ?] Edge 9 : [?, ?] Edge 10 : [?, ?, ?] Edge 11 : [?, ?, ?] Edge 12 : [?, ?, ?] Edge 13 : [?, ?, ?].

References

  1. [1]Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollár. Robust face landmark estimation under occlusion. In IEEE/CVF International Conference on Computer Vision (ICCV), 2013.
  2. [2]Jui-Che Chiang, Hou-Ning Hu, Bo-Syuan Hou, Chia-Yu Tseng, Yu-Lun Liu, Min-Hung Chen, and Yen-Yu Lin. ORFFormer: Occlusion-robust transformer for accurate facial landmark detection. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.arxiv.org/abs/2412.13174
  3. [3]Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2013.
  4. [4]Wenyan Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look at boundary: A boundary-aware face alignment algorithm. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2129–2138, 2018. doi: 10.1109/CVPR.2018.00227.DOI

Paper details

vlnc:W_grvywnee
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents