betaFeedback

WorksComputer Science

Asking Less, Adapting Better: Feedback-Efficient Test-Time Adaptation for Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) requires agents to follow instructions and navigate to target locations. However, current VLN agents often suffer severe performance degradation in unseen environments due to distribution shifts. Test-Time Adaptation (TTA) offers a promising paradigm to bridge this gap, yet existing methods that rely on unsupervised entropy minimization can propagate errors through the trajectory, causing cascading failures in navigation. Recent methods mitigate this by incorporating human feedback, but they apply trajectory-level supervision uniformly across all steps without localizing the actions is truly informative, leading to inefficient feedback utilization and mismatched optimization granularity. To address these limitations, we propose FETA, a Feedback-Efficient Test-Time Adaptation method that improves both feedback efficiency and optimization reliability. To reduce unnecessary human interventions, we introduce a hierarchical sample selection strategy to localize informative samples. Specifically, we first filter trajectories whose normalized policy entropy exhibits large fluctuations. Within each selected trajectory, we further select the critical action via an entropy-gradient saliency criterion and query a single binary feedback on its correctness. A discrepancy-aware entropy optimization objective then updates the policy only at the queried step, with adaptive scaling that assigns larger relative weights to samples with greater prediction-feedback discrepancy. Extensive experiments across multiple benchmarks demonstrate that FETA achieves state-of-the-art performance with fewer human interventions, while remaining robust under noisy feedback and continual adaptation settings.

Introduction

Vision-and-Language Navigation (VLN) [1, 2, 3, 4] is a fundamental embodied AI task in which an agent interprets natural-language instructions and navigate through a 3D environment to reach a target location. In practical applications, VLN agents are often deployed under conditions that differ from those seen during training. Such deployment shifts may arise from new room layouts, varying illumination or different instruction granularity, leading to substantial discrepancies between the training and test distributions [5, 6, 7, 8, 9]. As a result, even strong VLN models often suffer marked performance drop when transferred to new environments, making online adaptation crucial for robust embodied navigation [10, 11, 12, 13, 14].

Test-Time Adaptation (TTA) [15, 16, 17, 18] offers a natural way to mitigate such distribution shifts by updating model parameters on incoming test data. Motivated by its success in static vision tasks such as image classification and semantic segmentation, recent studies have begun to extend TTA to VLN. However, existing methods [6, 19] typically rely on unsupervised objectives, such as entropy minimization, and perform policy updates at every step along the navigation trajectory. This strategy is particularly brittle in VLN, where navigation unfolds as a sequential decision-making process: an erroneous update at one step can change the subsequent state distribution, distort later optimization signals, and eventually cause cascading failures. Therefore, the key challenge in VLN test-time adaptation is not simply how to adapt online, but how to derive optimization signals that remain reliable throughout sequential decision-making.

To obtain more reliable optimization signals, recent VLN-TTA methods have begun to incorporate human feedback at test time. FeedTTA [20] queries human whether each trajectory is successful or not, and uses the feedback as a reward signal to update the policy via reinforcement learning. ATENA [7] requests feedback only for trajectories with high entropy, calibrating policy uncertainty by boosting decision confidence on successful trajectories and suppressing it on failed ones. Despite their promise, these methods still suffer from two key limitations. 1) Inefficient Feedback Utilization: Once a trajectory-level label (i.e., success or failure) is obtained, existing methods generally use every obtained label to update the policy on the corresponding trajectory, even when the resulting adaptation gains are limited. 2) Mismatched Optimization Granularity: Existing methods use the same trajectory-level feedback signal across all steps, even though successful trajectories may still contain suboptimal actions and failed ones may still include locally correct decisions. This creates a mismatch between trajectory-level supervision and local action decisions being optimized.

To address these challenges, we propose a Feedback-Efficient Test-Time Adaptation method (FETA) that improves both feedback efficiency and optimization reliability. Our key observation is that the choice of trajectories and steps for feedback acquisition affects adaptation performance (as shown in Figure 2 and Table IV). Based on this observation, we design a hierarchical feedback selection strategy that improves both feedback efficiency and optimization granularity. At the trajectory level, we prioritize trajectories with large fluctuations in normalized policy entropy, as such trajectories indicate unstable decision confidence and may benefit from external correction. At the step level, for each selected trajectory, we select the critical decision state using an entropy-gradient saliency criterion, which measures the sensitivity of normalized policy entropy to the action logits. We then query a single binary human feedback signal on the correctness of the selected action. This hierarchical design significantly reduces unnecessary human intervention while refining supervision from the trajectory level to the action level.

Image not extracted

Figure 2. Trajectory-level selection criteria comparison. Solid/dashed lines select trajectories above/below threshold. δσ\delta_{\sigma} and δμ\delta_{\mu} are thresholds for σH^(τ)\sigma_{\hat{H}}(\tau) and μH^(τ)\mu_{\hat{H}}(\tau), respectively.

ModuleVal Unseen
TLSSLSDASSR ↑OSR ↑SPL ↑RGSPL ↑RGS ↑
Baseline46.9851.0733.7323.0332.15
✓--55.5464.2136.1024.1237.06
-✓-61.4169.0143.3229.1441.47
--✓56.8362.5237.6325.1137.91
-✓✓63.1967.5246.8631.3242.34
✓-✓57.6662.6941.2027.7738.90
✓✓-65.4672.0548.4232.6244.08
✓✓✓68.3373.2750.5533.1545.08

Table IV. Ablation study of Trajectory-Level Selection (TLS), Step-Level Selection (SLS), and Discrepancy-Aware Scaling (DAS) on the REVERIE Val Unseen split. Baseline denotes the unadapted model.

Based on the queried feedback, we design a discrepancy-aware objective for policy optimization. Instead of updating the entire trajectory from a single final outcome, our method corrects the policy only at the queried step, where the feedback determines both the update magnitude and direction of entropy optimization. Specifically, we introduce an adaptive scaling strategy that assigns larger relative weights to samples with greater mismatch between the current policy prediction and the feedback. We then apply directional entropy optimization: for actions verified as correct, the objective sharpens the policy distribution to reinforce the current decision. Otherwise, it relaxes overconfident predictions and encourages alternative actions. Our contributions can be summarized as follows:

  • We propose a Feedback-Efficient Test-Time Adaptation method for VLN and substantially reduce the number of human interventions through a hierarchical sample selection strategy, achieving a favorable trade-off between feedback cost and performance gain.

  • We design a discrepancy-aware entropy optimization loss that dynamically adjusts the direction of entropy optimization and the relative sample weights according to the human feedback, enabling efficient utilization of sparse feedback signals.

  • Extensive experiments show that FETA achieves the state-of-the-art performance across multiple benchmarks with fewer human interventions, including an 18.30% relative improvement in SPL over the strongest existing navigation TTA baseline, using DUET as the backbone on the REVERIE val unseen split, while remaining robust under noisy feedback and continual adaptation.

Related Work

Vision-and-language navigation

Vision-and-language navigation (VLN) [1, 4, 21, 22] is a fundamental task in embodied AI, requiring an agent to translate natural-language instructions into grounded perception and sequential actions [23]. Early VLN methods were largely based on recurrent seq2seq architectures, augmented with mechanisms such as speaker-based data augmentation, progress estimation, and search to improve instruction following and long-horizon decision making [24, 25, 26, 27]. Subsequent work shifted toward pretraining and Transformer-based models [5, 11, 28, 29, 30, 31, 32] which substantially improved multimodal fusion, history modeling, and benchmark performance. Despite this progress, most VLN models are still trained offline and tend to suffer significant performance degradation when deployed in unseen environments [33]. Recent efforts explore zero-shot navigation with Large Language Models (LLMs) [34, 35, 36, 37, 38, 39] to improve generalization without task-specific training. However, their reliance on large models incurs high inference latency, limiting practicality for real-time navigation. Another line of work studies test-time adaptation through online updates at deployment. Existing methods typically optimize self-supervised objectives such as entropy minimization [6, 15, 16, 40], which are weakly aligned with navigation objectives and can lead to brittle adaptation under distribution shifts. In contrast, our work leverages binary human feedback to directly align adaptation with the navigation goal, enabling more stable updates in unseen environments.

Test-Time Adaptation

Test-Time Adaptation (TTA) aims to update the model parameters directly using target domain data during inference, without access to source domain labels [41]. Most methods [15, 16, 42, 43, 44] adopt entropy minimization as the optimization objective, improving the confidence of policy predictions to enable online adaptation to unseen domains, and have demonstrated strong performance on tasks such as image classification [45, 46] and semantic segmentation [47, 48]. Recent work has begun to explore the application of TTA to Vision-and-Language Navigation. FSTTA [6] achieves a balance between fast adaptation and long-term stability through gradient and parameter decomposition. FeedTTA [20] incorporates binary trajectory-level human feedback as a reinforcement learning reward signal to update the navigation policy online. ATENA [7] selectively requests feedback only for high-entropy trajectories and calibrates policy confidence accordingly. However, existing methods that rely on human feedback use the same trajectory-level feedback signal across all steps, potentially reinforcing suboptimal actions or suppressing locally correct ones. By contrast, our method uses hierarchical feedback selection and improving adaptation performance while requiring fewer feedback queries.

Problem Statement and Motivation

Vision-and-Language Navigation (VLN) is a partially observable sequential decision-making task, where an agent follows the instruction to navigate in a 3D environment. Given an instruction II, at each time step tt the agent receives a visual observation oto_t and maintains a history representation hth_t, forming the agent state st=(ot,ht,I)s_t = (o_t, h_t, I). The agent then selects the highest-probability action ata_t from the candidate set At\mathcal{A}_t according to its policy πθ\pi_\theta. The navigation continues until the agent predicts STOP or reaches the maximum step limit, yielding a trajectory τ={(ot,at)}t=1T\tau= \{(o_t, a_t)\}_{t=1}^{T}.

In real deployment, VLN agents often encounter environments that differ from those seen during training, which can degrade navigation performance. This motivates test-time adaptation (TTA), which updates model parameters online

Algorithm 1 The pipeline of Feedback-Efficient Test-Time Adaptation method for VLN

Require: Pretrained policy πθ\pi_\theta, threshold δσ\delta_\sigma, scaling exponent γ\gamma, learning rate η\eta

  1. for each episode do

  2.     Collect trajectory τ={(ot,at)}t=1T\tau= \{(o_t, a_t)\}_{t=1}^{T} using πθ\pi_\theta

  3.     Compute σ^H(τ)\hat{\sigma}_H(\tau) via Eq. (3)

  4.     if σ^H(τ)>δσ\hat{\sigma}_H(\tau) > \delta_\sigma then

  5.         Compute GtG_t via Eq. (5)

  6.         Select step t∗=arg⁡max⁡tGtt^* = \arg\max_t G_t

  7.         Query binary feedback rt∗r_{t^*} for action at∗a_{t^*}

  8.         αt∗←sg⁡(∣rt∗−πθ(at∗∣st∗)∣γ)\alpha_{t^*} \leftarrow\operatorname{sg}(\lvert r_{t^*} - \pi_\theta(a_{t^*} \mid s_{t^*})\rvert^\gamma)

  9.         Compute L(τ)=αt∗⋅Lent(t∗)\mathcal{L}(\tau) = \alpha_{t^*} \cdot\mathcal{L}_{\mathrm{ent}}(t^*) via Eq. (8)

  10.         Update θ←θ−η∇θL(τ)\theta\leftarrow\theta- \eta\nabla_\theta\mathcal{L}(\tau) return Adapted policy πθ∗\pi_{\theta^*}

using incoming test samples. Existing TTA methods typically rely on unsupervised surrogate objectives, e.g., entropy minimization. However, unlike static perception tasks, VLN is inherently sequential: each action affects future states, observations, and subsequent optimization signals. As a result, once the adaptation signal is unreliable, an erroneous update can propagate through the trajectory and cause cascading failures. The challenge is how to obtain reliable optimization signals under sequential decision-making and environment shift.

To improve adaptation reliability, recent VLN test-time adaptation methods have begun to introduce reliable human feedback during deployment. However, existing human-feedback-driven methods still remain limited in two important aspects. First, the feedback they acquire may not always translate into effective adaptation, as policy updates on some labeled trajectories yield only limited gains. Second, they typically use a single trajectory-level feedback signal to optimize all actions within that trajectory in the same direction. Such coarse supervision is mismatched to the fine-grained nature of sequential decision-making: successful trajectories may still contain suboptimal actions, while failed trajectories may still include locally correct decisions. Applying the same feedback to all steps can therefore provide inappropriate optimization signals for individual actions, reducing the effectiveness of human guidance for policy adaptation. These limitations suggest that human-feedback-driven TTA for VLN requires a more structured use of sparse supervision, with both feedback-efficient querying and fine-grained action-level correction.

Feedback-Efficient Test-Time Adaptation

We propose FETA, a Feedback-Efficient Test-Time Adaptation method for VLN. At test time, after the agent executes a navigation trajectory in the target environment, FETA first performs Hierarchical Sample Selection (c.f. § 4-B) to determine where human supervision should be used, including trajectory-level selection to identify informative trajectories and step-level selection to locate a promising decision point for feedback within each selected trajectory. FETA then queries a binary feedback signal on the selected action (c.f. § 4-A). Based on the acquired feedback, FETA finally performs Feedback-Driven Policy Optimization (c.f. § 4-C) to convert sparse supervision into discrepancy-aware step-level policy updates. In this way, FETA concentrates limited human supervision on the informative decisions and enables efficient and reliable adaptation under distribution shift. Figure 1 provides an overview of our method, and Algorithm 1 summarizes the complete procedure.

Image not extracted

Figure 1. Overview of FETA. For the trajectory executed by the agent, we hierarchically select informative samples by filtering trajectories with higher normalized-entropy standard deviation σH^(τ)\sigma_{\hat{H}}(\tau) and querying the step with the largest entropy-gradient saliency GtG_t. The queried feedback is then used to update the policy via directional entropy optimization modulated by discrepancy-aware scaling, enabling efficient adaptation in novel environments.

Preliminary

Progress-Based Binary Feedback. FETA performs adaptation after each completed navigation episode, rather than updating the policy at every decision step. Accordingly, human feedback is queried only after the full trajectory is available. For each queried action, the human provides a binary signal indicating whether the action is beneficial for reaching the navigation goal. In the simulator, we instantiate this supervision with a progress-based oracle. Let dtd_t denote the geodesic distance from the agent’s position to the goal at time step tt. For a non-stop action, the feedback is positive if the action reduces the agent-to-goal geodesic distance; for the stop action, the feedback is positive if the agent has successfully reached the goal. The binary reward is defined as:

rt={I[dt>dt+1],if at≠STOP,I[dT≤dth],if at=STOP,(1)r_t = \begin{cases} \mathbb{I}[d_t > d_{t+1}], & \text{if } a_t \ne\mathrm{STOP},\\ \mathbb{I}[d_T \le d_{\mathrm{th}}], & \text{if } a_t = \mathrm{STOP}, \end{cases} \tag*{(1)}

where dTd_T is the final agent-to-goal geodesic distance and dthd_{\mathrm{th}} is the success threshold. rt=1r_t = 1 and rt=0r_t = 0 denote positive and negative feedback, respectively. This oracle enables reproducible evaluation, while local progress serves as an approximation to human judgment on action usefulness.

Hierarchical Sample Selection

Requesting human feedback on every action is impractical at test time. Under a limited feedback budget, we must answer two questions: which trajectories are worth querying, and where to query within each selected trajectory. We address these questions through a hierarchical sample selection strategy, whose design is motivated by two validation studies. As shown in Figure 2, under comparable numbers of feedback, selecting trajectories with larger fluctuations in normalized policy entropy yields better adaptation gains, motivating our trajectory-level criterion. As shown in Figure 3, although querying additional steps can further improve performance, the gains quickly diminish while the query cost continues to increase, indicating that one carefully selected step per trajectory provides the favorable trade-off. Guided by these observations, FETA first performs Trajectory-Level Selection to identify informative trajectories and then performs Step-Level Selection to select a critical decision step for feedback querying.

Image not extracted

Figure 3. Effect of maximum number of feedback per trajectory (KK) on success rate and total feedback count. Our method uses K=1K=1 to balance feedback cost and performance.

Trajectory-Level Selection. Our goal is to identify the trajectories that are most likely to benefit from external feedback under a limited query budget. Since uncertainty is a natural indicator of adaptation potential, we measure the policy uncertainty along each trajectory using normalized entropy. Let πθ(at∣st)\pi_\theta(a_t \mid s_t) denote the softmax probability assigned to action ata_t at time step tt, and let ∣At∣\lvert\mathcal{A}_t \rvert denote the number of feasible actions at step tt. The policy entropy and its normalized form are defined as:

Ht=−∑a∈Atπθ(a∣st)log⁡πθ(a∣st),H^t=Htlog⁡∣At∣(2)H_t=-\sum_{a\in\mathcal{A}_t}\pi_\theta(a\mid s_t)\log\pi_\theta(a\mid s_t),\qquad\hat{H}_t=\frac{H_t}{\log\lvert\mathcal{A}_t\rvert} \tag*{(2)}

where H^t∈[0,1]\hat{H}_t\in[0,1] captures the relative uncertainty of the policy over the current action space. Based on the per-step normalized entropy within a trajectory τ\tau of length TT, we further define the mean and standard deviation of normalized entropy as:

μH^(τ)=1T∑t=1TH^t,σH^(τ)=1T∑t=1T(H^t−μH^(τ))2.(3)\mu_{\hat{H}}(\tau)=\frac{1}{T}\sum_{t=1}^{T}\hat{H}_t,\qquad\sigma_{\hat{H}}(\tau)=\sqrt{\frac{1}{T}\sum_{t=1}^{T}\left(\hat{H}_t-\mu_{\hat{H}}(\tau)\right)^2}. \tag*{(3)}

To compare the two criteria with closely matched feedback counts, we set their respective dynamic thresholds δσ\delta_\sigma and δμ\delta_\mu to the same pp-th percentile of separate sliding windows, each containing the 150 most recent values of the corresponding criterion. As shown in Figure 2, using the standard deviation σH^(τ)\sigma_{\hat{H}}(\tau) is more effective than using the mean entropy μH^(τ)\mu_{\hat{H}}(\tau). Intuitively, a large entropy standard deviation indicates that the agent alternates between confident and uncertain phases along the trajectory, suggesting the presence of localized unreliable decisions that can benefit more from sparse external correction. In contrast, high mean entropy indicates globally diffuse uncertainty, which is harder to correct with limited feedback. Based on this observation, we use σH^(τ)\sigma_{\hat{H}}(\tau) as the trajectory-level selection criterion:

Sτ=I(σH^(τ)>δσ),(4)S_\tau=\mathbb{I}\left(\sigma_{\hat{H}}(\tau)>\delta_\sigma\right), \tag*{(4)}

where δσ\delta_\sigma is a dynamic threshold derived from recent trajectories (see Section VII) and I(⋅)\mathbb{I}(\cdot) denotes the indicator function. Only trajectories with Sτ=1S_\tau=1 are forwarded to the next stage.

Step-Level Selection. After selecting informative trajectories, the next question is where to query within each trajectory. Figure 3 shows that the first queried step yields the largest performance gain. While additional queried steps can further improve the success rate, their marginal benefits diminish quickly, whereas the feedback count increases nearly linearly. This makes a single carefully selected step the most cost-effective choice. Thus, FETA queries only one step for each selected trajectory.

Given this one-query design, we further need to determine which step to query for correction. Intuitively, a useful query target should allow the feedback signal to guide policy optimization effectively. A natural alternative is to query the step with the highest entropy. However, high entropy only indicates that the policy is uncertain, and may simply correspond to a near-uniform distribution, where the entropy gradient with respect to the logits approaches zero. Instead, we select the step whose normalized entropy is most sensitive to the logits, using this sensitivity as a heuristic for identifying steps that may benefit from the correction. We capture this property using the gradient norm of the normalized entropy with respect to the action logits, which we term entropy-gradient saliency. Let ztz_t denote the action logits at step tt. The saliency score is defined as:

Gt=∥∇ztH^t∥=1log⁡∣At∣∑aj∈Atπθ(aj∣st)2(log⁡πθ(aj∣st)+Ht)2.(5)G_t = \left\lVert\nabla_{z_t}\hat{H}_t \right\rVert = \frac{1}{\log\lvert A_t\rvert}\sqrt{\sum_{a_j\in A_t}\pi_\theta(a_j\mid s_t)^2\left(\log\pi_\theta(a_j\mid s_t)+H_t\right)^2}. \tag*{(5)}

For each selected trajectory, we rank all decision steps by GtG_t and query feedback only for the executed action at the critical step t∗=arg⁡max⁡t∈τGtt^*=\arg\max_{t\in\tau}G_t.

Feedback-Driven Policy Optimization

To effectively exploit sparse human feedback, we decompose each queried signal into two roles in policy adaptation. The prediction–feedback discrepancy modulates the update magnitude by reweighting how strongly a queried sample contributes to optimization, while the feedback label determines the direction of entropy optimization to encourage reinforcement or relaxation of the queried decision. In this way, feedback modulates the strength and determines the direction of entropy optimization.

Discrepancy-Aware Scaling. Applying uniform-weight updates to all queried samples overlooks the fact that different feedback samples are not equally informative for policy adaptation. To address this, we introduce a discrepancy-aware scaling factor according to the discrepancy between the current policy prediction and the queried feedback:

∣rt−πθ(at∣st)∣γ,γ>0,(6)\lvert r_t-\pi_\theta(a_t\mid s_t)\rvert^\gamma,\qquad\gamma>0, \tag*{(6)}

where πθ(at∣st)\pi_\theta(a_t\mid s_t) denotes the probability assigned by the policy to the executed action ata_t. When πθ(at∣st)\pi_\theta(a_t\mid s_t) is close to rtr_t, the current policy already agrees with the feedback, and the corresponding sample should contribute less to optimization. In contrast, when the discrepancy is large, the queried sample receives a larger relative weight, allowing the update to focus more on decision points where the current policy disagrees with the feedback signal.

Directional Entropy Optimization. Different feedback signals imply different local adjustment needs. For correct actions, the current decision should be reinforced; for incorrect ones, the current preference should be relaxed so that alternative actions can be explored. To realize this intuition, we use the policy entropy HtH_t as the optimization objective and introduce a directional weight wtw_t:

Lent(t)=wt⋅Ht,where wt={+1,rt=1,−1,rt=0.(7)\mathcal{L}_{\mathrm{ent}}(t)=w_t\cdot H_t,\qquad \text{where }w_t= \begin{cases} +1,&r_t=1,\\ -1,&r_t=0. \end{cases} \tag*{(7)}

When the feedback is positive, wt=+1w_t=+1 encourages the local policy distribution to become more concentrated around the current action, thereby reinforcing the verified decision. When the feedback is negative, wt=−1w_t=-1 encourages the current preference to be relaxed, allowing probability mass to shift toward alternative actions.

Full Optimization Objective. Combining the hierarchical sample filtering mechanism described in § 4-B, the overall loss is defined as:

L(τ)=I[σH^(τ)>δσ]⋅sg⁡[∣rt∗−πθ(at∗∣st∗)∣γ]⋅Lent(t∗),(8)\mathcal{L}(\tau)=\mathbb{I}\left[\sigma_{\hat{H}}(\tau)>\delta_\sigma\right]\cdot\operatorname{sg}\left[\lvert r_{t^*}-\pi_\theta(a_{t^*}\mid s_{t^*})\rvert^\gamma\right]\cdot\mathcal{L}_{\mathrm{ent}}(t^*), \tag*{(8)}

where t∗=arg⁡max⁡t∈τGtt^*=\arg\max_{t\in\tau}G_t and sg⁡(⋅)\operatorname{sg}(\cdot) denotes the stop-gradient operator, which treats the discrepancy term as a fixed weighting factor during optimization. This objective is computed only at the single queried step within trajectories that pass the trajectory-level filter. Here, the discrepancy term re-weights each queried sample according to its consistency with the received feedback, while wt∗w_{t^*} determines whether the local decision is reinforced or relaxed. Together, these components enable FETA to convert sparse binary feedback into effective local policy adaptation.

Experiments

Benchmark and Metrics

We evaluate our method on four benchmarks: REVERIE [49], R2R [1], GSA-R2R [19], and R2R-CE [50]. REVERIE requires the agent to navigate to a target region and identify the referred object based on a high-level instruction. R2R focuses on fine-grained instruction following in discrete indoor environments. GSA-R2R extends R2R with more diverse environments and instruction styles to evaluate agent adaptability in both in-distribution and out-of-distribution settings. R2R-CE transfers the R2R task to continuous environments, requiring low-level action execution. We report SR (Success Rate), OSR (Oracle Success Rate), SPL (Success rate weighted by Path Length), NDTW (Normalized Dynamic Time Warping), TL (trajectory length) and NE (Navigation Error). For REVERIE, we additionally report RGS and RGSPL, which measure remote object grounding success without and with path efficiency weighting. We also introduce FC (Feedback Count) to quantify the total number of human feedback queries during evaluation, directly reflecting the annotation cost of test-time adaptation process. R2R results are provided in the Appendix A.

MethodsVal SeenVal UnseenTime(ms)
OSR ↑\uparrowSR ↑\uparrowSPL ↑\uparrowRGSPL ↑\uparrowFC ↓\downarrowOSR ↑\uparrowSR ↑\uparrowSPL ↑\uparrowRGSPL ↑\uparrowFC ↓\downarrow
DUET [3]73.8671.7563.9451.14–51.0746.9833.7323.03–156.5
• FSTTA [6]73.7971.6864.0051.21–50.9346.8633.7123.00–457.7
• FeedTTA [20]82.8377.2865.3954.28142462.6053.8528.6619.023522433.1
• ATENA [7]82.6281.4072.8558.9057365.7861.9542.7328.392595321.3
• FETA84.3182.2973.7559.0151973.2768.3350.5533.152475243.5
GOAT [13]82.3680.7473.4458.82–57.9753.8237.5227.00–144.2
• FSTTA [6]82.8580.6772.6558.11–57.5453.4836.7625.80–461.1
• FeedTTA [20]84.9282.0869.5459.12142465.1256.7533.6826.113522390.0
• ATENA [7]84.6883.0276.4661.4680363.6660.5242.9630.433012342.3
• FETA90.0287.6577.9862.1453372.7568.6554.2238.182477307.9
ScaleVLN [11]73.3070.9860.8644.30–61.4354.2541.2825.00–187.7
• FSTTA [6]73.2370.9160.8244.25–61.4554.1541.2425.00–544.7
• FeedTTA [20]78.7172.1055.5345.92142471.7156.1834.8523.833522437.5
• ATENA [7]77.9376.0466.0148.0696568.4963.3249.0829.202870344.4
• FETA83.0280.8670.7651.6756275.5069.8456.6133.522465300.0

Table 1. Experimental results on REVERIE. The last column reports the average runtime per episode.

MethodsTest Residential BasicTest Non-residential Scene
SR ↑\uparrowSPL ↑\uparrowNDTW ↑\uparrowNE ↓\downarrowSR ↑\uparrowSPL ↑\uparrowNDTW ↑\uparrowNE ↓\downarrow
GR-DUET [19]6964713.094742545.55
• FSTTA [6]6962703.144842535.46
• FeedTTA [20]6964723.074843555.43
• ATENA [7]6764713.344840505.45
• FETA7167742.935044554.94
BEVBert [12]5845544.113927366.69
• FSTTA [6]5844534.224028376.50
• FeedTTA [20]5944534.093926356.70
• ATENA [7]5844534.204028376.49
• FETA5945543.914228375.75

Table 2. Experimental results on GSA-R2R.

MethodsVal SeenVal Unseen
OSR ↑\uparrowSR ↑\uparrowSPL ↑\uparrowNE ↓\downarrowOSR ↑\uparrowSR ↑\uparrowSPL ↑\uparrowNE ↓\downarrow
ETPNav [10]7164574.126457494.72
• FSTTA [6]7063564.196355474.86
• FeedTTA [20]7265584.066457494.73
• ATENA [7]7065594.066357504.69
• FETA7266594.006559514.52
BEVBert [12]7871633.226658494.53
• FSTTA [6]7568603.526759494.59
• FeedTTA [20]7772643.216758494.47
• ATENA [7]7771643.246760484.34
• FETA7973643.086961504.20

Table 3. Experimental results on R2R-CE.

Results on REVERIE. Table I reports the comparative results on the REVERIE [49] benchmark. We conduct adaptation using DUET [3], GOAT [13], and ScaleVLN [11] as base models. Across all base models and data splits, FETA consistently outperforms ATENA [7], the current state-of-the-art test-time adaptation method for navigation. In particular, on the more challenging val unseen split, FETA achieves relative SPL improvements over ATENA of 18.30%, 26.21%, and 15.34% for DUET, GOAT, and ScaleVLN, respectively. Meanwhile, FETA also achieves lower feedback count (FC) across all settings, as the hierarchical selection strategy filters informative trajectories and queries critical navigation step within each selected trajectory. Furthermore, FETA requires consistently less execution time per episode than ATENA, demonstrating that the proposed method improves both effectiveness and efficiency.

Results on GSA-R2R. We further evaluate FETA on the scene-diverse GSA-R2R [19] benchmark. As shown in Table II, ATENA brings only limited gains and even degrades the performance of GR-DUET on the residential split. In contrast, FETA delivers consistent improvements across all settings and achieves up to a 10.00% relative improvement in SPL over ATENA on the more challenging non-residential split. This suggests that FETA remains effective even when deployment scenarios differ substantially from the training distribution.

Results on R2R-CE. Unlike the discrete navigation settings above, R2R-CE [50] requires the agent to execute low-level continuous control commands. As shown in Table III, FETA achieves the best results with both ETPNav [10] and BEVBert [12]. On the val unseen split, it achieves relative SR improvements over ATENA of 3.51% and 1.67% with ETPNav and BEVBert, respectively, demonstrating consistent gains in continuous environments.

Ablation Study

To verify the effectiveness of each component in our method, we conduct ablation studies on the REVERIE Val Unseen split, with results reported in Table IV. To ensure a fair comparison under an equal feedback budget, the ablations of Trajectory-Level Selection (TLS) and Step-Level Selection (SLS) replace the respective selection method with uniform random sampling while keeping the total number of feedback queries identical. Disabling Discrepancy-Aware Scaling (DAS) assigns unit weight to each queried sample. Relative to the full method, replacing TLS with random trajectory sampling leads to a performance drop across all reported metrics, supporting the effectiveness of trajectory selection based on the standard deviation of normalized entropy. Replacing SLS with random step sampling also decreases performance, indicating that entropy-gradient saliency helps make effective use of sparse feedback. Removing DAS while retaining TLS and SLS also decreases performance, supporting the benefit of weighting each queried sample according to the prediction–feedback discrepancy. The variants with only TLS, SLS, or DAS enabled also outperform the unadapted baseline, showing that adaptation remains effective when only one of the three components is enabled. These results collectively demonstrate that our method efficiently exploits sparse human feedback to enable effective agent adaptation in unseen environments.

More Discussions

Comparison of Step-Level Selection Methods. Table VII compares entropy-gradient saliency against querying feedback

Selection MethodVal SeenVal Unseen
OSR↑SR↑SPL↑RGSPL↑OSR↑SR↑SPL↑RGSPL↑
Entropy81.1979.0371.3157.4169.6363.9746.3931.13
Entropy-Gradient Saliency84.3182.2973.7559.0173.2768.3350.5533.15

Table VII. Comparison of step-level selection methods on the REVERIE dataset.

at the step with highest normalized entropy. Under an equal budget, entropy gradient saliency outperforms the entropy-based selection method across all metrics, yielding an 8.97% relative improvement in SPL on the REVERIE Val Unseen split. A possible reason is that time steps with high entropy may correspond to policy outputs that approach a uniform distribution. Negative feedback at such steps provides a weak entropy-maximization signal, as the entropy gradient with respect to the logits approaches zero near a uniform distribution. By contrast, selecting the step with the largest entropy-gradient saliency GtG_t directs feedback toward steps where normalized entropy is more sensitive to logit changes, which may facilitate feedback-driven policy adjustment.

Robustness of the Method. In practical applications, binary feedback provided by humans is rarely perfectly accurate. To evaluate the robustness of our method under noisy feedback, we simulate noise by randomly flipping feedback labels with varying probabilities, gradually reducing the feedback accuracy from 100% to 50%. As shown in Figure 4, the navigation success rate decreases as the noise level increases. Even at 60% feedback accuracy, our method still outperforms the unadapted baseline. Among the tested noise levels, performance falls below the baseline only when feedback becomes random (50% accuracy). These results demonstrate that FETA can tolerate substantial simulated label noise while retaining an advantage over the unadapted baseline.

Image not extracted

Figure 4. Effect of noisy binary feedback on navigation performance in the REVERIE Val Unseen split.

Continual Adaptation. We evaluate continual adaptation by progressively adapting the model over multiple rounds and measuring performance after each round, starting from the unadapted model (Round 0). As shown in Figure 5, ATENA improves over the baseline after the first round but degrades in subsequent rounds, eventually falling below the unadapted performance. In contrast, FETA improves in each of the three adaptation rounds. This contrast may be related to the granularity of supervision: ATENA uses the same trajectorylevel feedback signal across all steps, potentially reinforcing suboptimal actions or suppressing locally correct ones. By contrast, FETA computes the adaptation loss at a selected step using feedback on the corresponding action, aligning the supervision granularity with the decision being optimized.

Image not extracted

Figure 5. Continual adaptation results on REVERIE Unseen split, averaged over 3 random seeds.

Cross-Dataset Adaptation. We evaluate cross-dataset adaptation by training the model on one dataset and then adapting it to the other. Table V reports cross-dataset generalization under two transfer directions: R2R→REVERIE and REVERIE→R2R. Compared with the competing methods, FETA shows strong transferability across datasets and tasks, achieving better adaptation performance. These results confirm that our adaptation strategy is not dataset-specific but generalizes robustly across heterogeneous task distributions.

Data (Train → Test)MethodsVal Unseen
SR ↑SPL ↑NE ↓TL ↓
R2R → REVERIEDUET [3]24.9120.007.2214.67
• ATENA29.4219.436.7524.45
• FETA40.0223.535.2039.09
REVERIE → R2RDUET [3]29.849.037.2844.20
• ATENA31.258.416.9848.40
• FETA36.9119.066.2334.65

Table V. Experimental results of cross-dataset adaptation. Models are trained on one dataset and adapted on the other.

Real-Robot Navigation Results. We evaluate FETA on a physical robot in an office and a laboratory, with 20 navigation episodes per environment. We use BEVBert [12] as the baseline and retain the same settings as in simulation. As shown in Table VI, FETA improves success rate by 15 and 10 percentage points in the office and laboratory, respectively, while also reducing navigation error in both environments. These results provide initial evidence that FETA can improve navigation performance in physical environments beyond simulation.

MethodOfficeLaboratory
SR ↑NE ↓SR ↑NE ↓
BEVBert [12]404.30304.20
• FETA553.87404.03

Table VI. Experimental results in real-world environments.

Human–Simulator Feedback Agreement

To assess how closely the progress-based simulator feedback reflects human judgments of action usefulness, we conduct a human annotation study on 90 feedback queries from the REVERIE Val Unseen split. Three annotators each label a disjoint set of 30 queries, so each query receives one human annotation. For each query, the annotator is shown the navigation instruction, the complete observation sequence

along the trajectory, the observations immediately before and after the queried action, and the action itself. The annotator then assigns a binary label indicating whether the action is helpful for completing the task. Human annotations agree with the simulator feedback on 78.9% of queries, with agreement rates of 81.8% for simulator-positive queries and 76.1% for simulator-negative queries. The label distributions are approximately balanced: 48.9% of simulator labels are positive, while annotators judge 52.2% of actions as helpful. These results provide empirical support for using the simulator feedback as a proxy for human judgments in this setting. The annotation interface is shown in Figure 8 in the appendix.

Image not extracted

Figure 8. Human annotation interface for binary action feedback. Annotators assess action usefulness using the task instruction, trajectory video, and panoramic observations before and after the queried action.

Implementation Details

To simulate real-world deployment conditions, we adopt an online test-time adaptation paradigm with a batch size of 1. All adaptation methods are evaluated three times with distinct random orderings of test samples, and we report the mean performance across runs. We evaluate our method against several representative VLN architectures, including DUET [3], GOAT [13], ScaleVLN [11], ETPNav [10], and BEVBert [12]. All baseline results are reproduced using the official implementations. During adaptation, we perform full-parameter updates with learning rates chosen from {6×10−6,2×10−6,6×10−7}\{6 \times10^{-6}, 2 \times10^{-6}, 6 \times10^{-7}\}. A gradient update is triggered at the end of an episode only if the standard deviation of the normalized entropy σH^(τ)\sigma_{\hat{H}(\tau)} across the trajectory exceeds a dynamic threshold δσ\delta_\sigma. Specifically, we maintain a sliding window of the W=150W = 150 most recent standard deviations. Before the window is filled, all episodes pass the filter since feedback is valuable when the agent first encounters a novel environment. On REVERIE, R2R and R2R-CE, δσ\delta_\sigma is set to the 70th percentile of this window for Val Seen and the 30th percentile for Val Unseen. On GSA-R2R, δσ\delta_{\sigma} is set to the 30th percentile for all evaluations. These settings balance feedback cost and navigation performance. For episodes that pass this filter, the update is applied at the navigation step exhibiting the highest entropy-gradient saliency GtG_t. The discrepancy-aware scaling factor γ\gamma in Eq. (6) is set to 0.6. The results of FSTTA [6], FeedTTA [20], and ATENA [7] are reproduced using its officially released codebase. All experiments are conducted on a single NVIDIA Tesla A800 GPU.

Qualitative Analysis

Figure 6 visualizes the navigation trajectories of the base policy (DUET [3]) and the policy adapted by FETA on two representative episodes from the REVERIE validation unseen split. In the left panel of Figure 6, the instruction asks the agent to go to the dining room and identify the table support closest to the tray of food. The unadapted model navigates toward a plausible dining area containing a round table, but ultimately stops at an incorrect location. After adaptation, FETA steers the agent toward the correct dining room, yielding a trajectory that better matches the instructed destination. In the right panel of Figure 6, the instruction requires the agent to navigate to the balcony and locate the book closest to the lantern on the table. The unadapted model terminates in the living room rather than reaching the requested balcony. After adaptation, FETA leads the agent to the balcony specified by the instruction. Across both examples, a consistent pattern emerges: the unadapted policy makes a plausible but erroneous decision at a branching point and subsequently fails to reach the target, whereas the adapted policy follows a route toward the instructed destination.

Image not extracted

Figure 6. Qualitative comparison of navigation trajectories between the base model and our test-time adaptation method (FETA). Red waypoints indicate the trajectory predicted by the base model, while green waypoints denote the trajectory predicted after applying FETA at test time. Blue waypoints are the shared prefix visited by both methods before their trajectories diverge.

The improvement in reaching the instructed destination is also reflected in the real-robot examples in Figure 7, where BEVBert serves as the base policy. In example (a), the adapted policy proceeds past the sofas and stops near the blue display board, which the baseline fails to reach. In example (b), the adapted policy reaches the specified glass door, while the baseline remains near the starting location. Together, the simulation and real-robot examples provide qualitative evidence that FETA can help navigation policies follow routes toward instruction-specified destinations in both simulated and physical environments.

Image not extracted

Figure 7. Real-robot navigation examples. (a) BEVBert fails to reach the blue display board, while FETA proceeds past the sofas and stops near the board. (b) BEVBert remains near the starting location, while FETA successfully stops in front of the glass door.

Conclusion

We propose FETA, a Feedback-Efficient Test-Time Adaptation method for Vision-and-Language Navigation. To address the inefficient feedback utilization and mismatched optimization granularity of prior methods, FETA performs hierarchical sample selection that filters informative trajectories via the standard deviation of normalized entropy and selects a critical step for targeted feedback via entropy-gradient saliency. The queried feedback is then converted into a fine-grained step-level update through a discrepancy-aware entropy optimization objective. Extensive experiments show that FETA achieves state-of-the-art performance with fewer human queries and lower per-episode latency.

Biographies

Zixu Wang

Zixu Wang is currently pursuing a Ph.D. degree at South China University of Technology, Guangzhou, China. He received the B.E. degree in Automation from Central South University, China, in 2022. His research interests include vision-and-language navigation, test-time adaptation and continual learning.

Huaquan Zhang

Huaquan Zhang is currently pursuing the M.S. degree at South China University of Technology, Guangzhou, China. He received his B.Sc. degree from South China University of Technology, Guangzhou, China, in 2025. His research interests include vision-and-language navigation and test-time adaptation.

Shoukai Xu

Shoukai Xu is Postdoctoral Researcher with School of Software Engineering at South China University of Technology. He received his Ph.D. degree in the School of Software Engineering in 2025 from South China University of Technology in Guangzhou, China. His research interests include computer vision and lightweight neural model optimization. He has published papers in top AI venues, including ICML, ICLR, ECCV and IEEE TCSVT.

Jinwu Hu

Jinwu Hu is currently pursuing a Ph.D. degree at South China University of Technology, Guangzhou, China. He received the M.S. degree from Chongqing University of Posts and Telecommunications, Chongqing, China, in 2023. His research interests include LLMs, computer vision, and reinforcement learning. He has published papers in top venues, including ICML, NeurIPS, CVPR, IJCAI, ACM MM, ECCV, IEEE TKDE, IEEE TIP, IEEE TIFS, IEEE TMM, IEEE TCYB, IEEE TMI, etc. He has served as a reviewer for many journals/conferences, including TPAMI, ICML, NeurIPS, CVPR, etc.

Zhiquan Wen

Zhiquan Wen is recently a Post-doctoral Researcher with Shen-Ming Wu School of Intelligent Engineering at South China University of Technology. He received his Ph.D. degree in the School of Software Engineering in 2024 from South China University of Technology in Guangzhou, China. His research interests include vision-and-language and test-time adaptation. He has published papers in top venues, including NeurIPS, ICLR, ECCV, AAAI, ACL, IEEE TMM, IEEE TCSVT and Neural Networks. He has been invited as a reviewer for top-tier conferences and journals including IEEE TPAMI, IEEE TMM, ICLR, ICML, NeurIPS, CVPR, ICCV, AAAI, etc.

Mingkui Tan

Mingkui Tan is currently a Professor with the School of Software Engineering at South China University of Technology. He received his B.Sc. degree in Environmental Science and Engineering and M.Eng. degree in Control Science and Engineering from Hunan University, Changsha, China, in 2006 and 2009, respectively. He obtained his Ph.D. degree in Computer Science from Nanyang Technological University, Singapore, in 2014. From 2014 to 2016, he worked as a Senior Research Associate in the School of Computer Science at the University of Adelaide, Australia, where his research focused on computer vision. He currently serves as an Associate Editor for both IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) and Machine Intelligence Research (MIR). His research interests include machine learning, deep learning, computer vision, sparse analysis, and large-scale optimization.

Appendix A: Adaptation Results on R2R

We further evaluate FETA on the classic R2R benchmark using DUET, GOAT, and ScaleVLN as navigation backbones. As shown in Table VIII, FETA improves overall navigation performance across all three backbones. On the Val Unseen split, it achieves higher navigation success than ATENA while maintaining or improving path efficiency. These results demonstrate the effectiveness of FETA on R2R across different navigation backbones.

MethodsVal SeenVal Unseen
OSR ↑SR ↑SPL ↑NE ↓OSR ↑SR ↑SPL ↑NE ↓
DUET [3]8679732.288172603.31
• FSTTA [6]8376692.517868573.47
• FeedTTA [20]8578732.298072613.26
• ATENA [7]8579742.287972623.23
• FETA8680742.208274623.01
GOAT [13]8985801.678678682.31
• FSTTA [6]8883781.828577662.45
• FeedTTA [20]8985801.668678682.30
• ATENA [7]8985801.678678692.27
• FETA9085801.658679692.21
ScaleVLN [11]8781752.128881702.09
• FSTTA [6]8478722.328678672.30
• FeedTTA [20]8782762.108780712.12
• ATENA [7]8682772.048679712.18
• FETA8882771.968881722.08

Table VIII. Experimental results on R2R dataset.

Appendix B: Derivation of Entropy-Gradient Saliency

We derive the closed-form expression for the entropy-gradient saliency Gt=∥∇ztH^t∥G_t = \lVert\nabla_{z_t}\hat{H}_t \rVert used in the step-level selection criterion (§ 4-B). Let zt=(z1,z2,…,zN)Tz_t = (z_1,z_2,\ldots,z_N)^\mathsf{T} denote the action logits at step tt, where N=∣At∣N = |\mathcal{A}_t| is the number of feasible actions. The policy is defined via the softmax function:

πj≜πθ(aj∣st)=exp⁡(zj)∑k=1Nexp⁡(zk),j=1,…,N.(9)\pi_j \triangleq\pi_\theta(a_j \mid s_t) = \frac{\exp(z_j)}{\sum_{k=1}^{N}\exp(z_k)}, \qquad j=1,\ldots,N. \tag*{(9)}

The policy entropy and its normalized form are:

Ht=−∑j=1Nπjlog⁡πj,H^t=Htlog⁡N.(10)H_t = -\sum_{j=1}^{N}\pi_j\log\pi_j,\qquad\hat{H}_t = \frac{H_t}{\log N}. \tag*{(10)}

a) Step 1: Softmax Jacobian. The partial derivative of the softmax output πi\pi_i with respect to the logit zjz_j is:

∂πi∂zj=πi(δij−πj),(11)\frac{\partial\pi_i}{\partial z_j} = \pi_i(\delta_{ij}-\pi_j), \tag*{(11)}

where δij\delta_{ij} is the Kronecker delta.

b) Step 2: Gradient of the entropy with respect to logits.:

Taking the partial derivative of HtH_t with respect to zjz_j:

∂Ht∂zj=−∑i=1N∂∂zj[πilog⁡πi]=−∑i=1N[∂πi∂zjlog⁡πi+πi⋅1πi∂πi∂zj]=−∑i=1N∂πi∂zj(log⁡πi+1).(12)\frac{\partial H_t}{\partial z_j} = -\sum_{i=1}^{N}\frac{\partial}{\partial z_j}[\pi_i\log\pi_i] = -\sum_{i=1}^{N}\left[\frac{\partial\pi_i}{\partial z_j}\log\pi_i+\pi_i\cdot\frac{1}{\pi_i}\frac{\partial\pi_i}{\partial z_j}\right] = -\sum_{i=1}^{N}\frac{\partial\pi_i}{\partial z_j}(\log\pi_i+1). \tag*{(12)}

Substituting the softmax Jacobian from Eq. (11):

∂Ht∂zj=−∑i=1Nπi(δij−πj)(log⁡πi+1)=−πj(log⁡πj+1)+πj∑i=1Nπi(log⁡πi+1).(13)\frac{\partial H_t}{\partial z_j} = -\sum_{i=1}^{N}\pi_i(\delta_{ij}-\pi_j)(\log\pi_i+1) = -\pi_j(\log\pi_j+1)+\pi_j\sum_{i=1}^{N}\pi_i(\log\pi_i+1). \tag*{(13)}

Note that ∑i=1Nπilog⁡πi=−Ht\sum_{i=1}^{N}\pi_i\log\pi_i=-H_t and ∑i=1Nπi=1\sum_{i=1}^{N}\pi_i=1. Therefore:

∂Ht∂zj=−πj(log⁡πj+1)+πj(−Ht+1)=−πjlog⁡πj−πj+πj−πjHt=−πj(log⁡πj+Ht).(14)\frac{\partial H_t}{\partial z_j} = -\pi_j(\log\pi_j+1)+\pi_j(-H_t+1) = -\pi_j\log\pi_j-\pi_j+\pi_j-\pi_jH_t = -\pi_j(\log\pi_j+H_t). \tag*{(14)}

c) Step 3: Gradient of the normalized entropy.: Since H^t=Ht/log⁡∣At∣\hat{H}_t=H_t/\log|\mathcal{A}_t| and ∣At∣|\mathcal{A}_t| is treated as a constant at step tt:

∂H^t∂zj=1log⁡∣At∣∂Ht∂zj=−πθ(aj∣st)(log⁡πθ(aj∣st)+Ht)log⁡∣At∣.(15)\frac{\partial\hat{H}_t}{\partial z_j} = \frac{1}{\log|\mathcal{A}_t|}\frac{\partial H_t}{\partial z_j} = -\frac{\pi_\theta(a_j\mid s_t)(\log\pi_\theta(a_j\mid s_t)+H_t)}{\log|\mathcal{A}_t|}. \tag*{(15)}

d) Step 4: Entropy-gradient saliency.: The saliency score is defined as the ℓ2\ell_2-norm of the gradient vector. Substituting Eq. (15):

Gt=∥∇ztH^t∥=∑aj∈At(∂H^t∂zj)2=1log⁡∣At∣∑aj∈Atπθ(aj∣st)2(log⁡πθ(aj∣st)+Ht)2.(16)G_t = \lVert\nabla_{z_t}\hat{H}_t\rVert = \sqrt{\sum_{a_j\in\mathcal{A}_t}\left(\frac{\partial\hat{H}_t}{\partial z_j}\right)^2} = \frac{1}{\log|\mathcal{A}_t|}\sqrt{\sum_{a_j\in\mathcal{A}_t}\pi_\theta(a_j\mid s_t)^2\left(\log\pi_\theta(a_j\mid s_t)+H_t\right)^2}. \tag*{(16)}

This expression can be computed in closed form from the softmax probabilities and the entropy, requiring no backpropagation through the model.

References

  1. [1]P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sunderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683.arxiv.org/abs/1711.07280
  2. [2]Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu, “Hop: History-and-order aware pre-training for vision-and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15418–15427.
  3. [3]S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16537–16547.arxiv.org/abs/2202.11742
  4. [4]Y. Hong, Z. Wang, Q. Wu, and S. Gould, “Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
  5. [5]P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, “Ainbert: In-domain Pretraining for Vision-and-Language Navigation,” 2021.
  6. [6]J. Gao, X. Yao, and C. Xu, “Fast-flow test-time adaptation for online vision-and-language navigation,” in ICML, 2024.DOI
  7. [7]H. Ko, S. Kim, G. Oh, J. Yoon, H. Lee, S. Jang, S. Kim, and S. Kim, “Active test-time vision-language navigation,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.06630
  8. [8]B. Lin, Y. Zhu, Y. Long, X. Liang, Q. Ye, and L. Lin, “Adversarial reinforced instruction attacker for robust vision-language navigation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 7175–7189, 2021.DOI
  9. [9]X. Wang, W. Wang, J. Shao, and Y. Yang, “Learning to follow and generate instructions for language-capable navigation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, pp. 3334–3350, 2023.DOI
  10. [10]D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. He, and L. Wang, “Etpnav: Evolving topological planning for vision-language navigation in continuous environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.DOI
  11. [11]Z. Wang, J. Li, Y. Hong, Y. Wang, Q. Wu, M. Bansal, S. Gould, H. Tan, and Y. Qiao, “Scaling data generation in vision-and-language navigation,” in ICCV, 2023, pp. 12 009–12 020.arxiv.org/abs/2307.15644
  12. [12]D. An, Y. Qi, Y. Li, Y. Huang, L. Wang, T. Tan, and J. Shao, “Bevbert: Multimodal map pre-training for language-guided navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2212.04385
  13. [13]L. Wang, Z. He, R. Dang, M. Shen, C. Liu, and Q. Chen, “Vision-and-language navigation via causal learning,” in The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2404.10241
  14. [14]X. Wang, J. Dong, B. Liu, Q. Lyu, L. Liu, and Z. Han, “Lifelong embodied navigation learning,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=PaYos9rij
  15. [15]D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, “Tent: Fully test-time adaptation by entropy minimization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=uXI3bZLkr3carxiv.org/abs/2006.10726
  16. [16]S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan, “Efficient test-time model adaptation without forgetting,” in The International Conference on Machine Learning, 2022.arxiv.org/abs/2204.02610
  17. [17]S. Niu, C. Miao, G. Chen, P. Wu, and P. Zhao, “Test-time model adaptation with only forward passes,” in Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2404.01650
  18. [18]Y. Su, X. Xu, T. Li, and K. Jia, “Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering regularized self-training,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5524–5540, 2024.DOI
  19. [19]H. Hong, Y. Qiao, S. Wang, J. Liu, and Q. Wu, “General scene adaptation for vision-and-language navigation,” in The Thirteenth International Conference on Learning Representations, 2025.
  20. [20]S. Kim, G. Oh, H. Ko, D. Ji, D. Lee, B.-J. Lee, S. Jang, and S. Kim, “Test-time adaptation for online vision-language navigation with feedback-based reinforcement learning,” in Forty-second International Conference on Machine Learning, 2025.
  21. [21]X. Shi, Z. Li, W. Lyu, J. Xia, F. Dayoub, Y. Qiao, and Q. Wu, “Smartway: Enhanced waypoint prediction and backtracking for zero-shot vision-and-language navigation,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 16 923–16 930.arxiv.org/abs/2503.10069
  22. [22]B. Lin, Y. Nie, K. L. Zai, Z. Wei, M. Han, R. Xu, M. Niu, J. Han, H. Zhang, L. Lin et al., “Evolvenav: Empowering llm-based vision-language navigation via self-improving embodied reasoning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.DOI
  23. [23]K. He, Y. Huang, Y. Jing, Q. Wu, and L. Wang, “Fine-grained alignment supervision matters in vision-and-language navigation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.DOI
  24. [24]D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker-follower models for vision-and-language navigation,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2018/file/6a81681a7af700c6385d36577ebec359-Paper.pdf
  25. [25]C.-Y. Ma, J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong, “Self-monitoring navigation agent via auxiliary progress estimation,” in Proceedings of the International Conference on Learning Representations (ICLR), 2019. [Online]. Available: https://arxiv.org/abs/1901.03035
  26. [26]H. Wang, W. Wang, T. Shu, W. Liang, and J. Shen, “Active visual information gathering for vision-language navigation,” in ECCV, 2020.arxiv.org/abs/2007.08037
  27. [27]X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W. Yang, and L. Zhang, “Vision-language navigation policy learning and adaptation,” IEEE transactions on pattern analysis and machine intelligence, no. 12, pp. 4205–4216, 2020.DOI
  28. [28]Y. Hong, Q. Wu, Y. Qi, C. Rodriguez-Opazo, and S. Gould, “A recurrent vision-and-language bert for navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 1643–1653.arxiv.org/abs/2011.13922
  29. [29]S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev, “History aware multimodal transformer for vision-and-language navigation,” in Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2110.13309
  30. [30]J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang, “Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks,” in Robotics: Science and Systems, 2025.arxiv.org/abs/2412.06224
  31. [31]M. Wei, C. Wan, X. Yu, T. Wang, Y. Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y. Chen et al., “Streamvln: Streaming vision-and-language navigation via slowfast context modeling,” in 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.
  32. [32]Y. Qiao, Y. Qi, Y. Hong, Z. Yu, P. Wang, and Q. Wu, “Hop+: History-enhanced and order-aware pre-training for vision-and-language navigation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8524–8537, 2023.DOI
  33. [33]B. Lin, Y. Nie, Z. Wei, Y. Zhu, H. Xu, S. Ma, J. Liu, and X. Liang, “Correctable landmark discovery via large models for vision-language navigation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 8534–8548, 2024.DOI
  34. [34]Y. Qiao, W. Lyu, H. Wang, Z. Wang, Z. Li, Y. Zhang, M. Tan, and Q. Wu, “Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms,” in ICRA, 2025.arxiv.org/abs/2409.18794
  35. [35]G. Zhou, Y. Hong, and Q. Wu, “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 7641–7649.arxiv.org/abs/2305.16986
  36. [36]P. Chen, X. Sun, H. Zhi, R. Zeng, T. H. Li, G. Liu, M. Tan, and C. Gan, “a²nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models,” 2023. [Online]. Available: https://arxiv.org/abs/2308.07997
  37. [37]Y. Long, W. Cai, H. Wang, G. Zhan, and H. Dong, “Instructnav: Zero-shot system for generic instruction navigation in unexplored environment,” in 8th Annual Conference on Robot Learning, 2024. [Online]. Available: https://openreview.net/forum?id=fCDOfpTCzZarxiv.org/abs/2406.04882
  38. [38]Y. Qiao, H. Hong, W. Lyu, D. An, S. Zhang, Y. Xie, X. Wang, and Q. Wu, “Navbench: Probing multimodal large language models for embodied navigation,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.01031
  39. [39]K. Chen, D. An, Y. Huang, R. Xu, Y. Su, Y. Ling, I. Reid, and L. Wang, “Constraint-aware zero-shot vision-language navigation in continuous environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.DOI
  40. [40]Q. Zhang, Y. Bian, X. Kong, P. Zhao, and C. Zhang, “Come: Test-time adaptation by conservatively minimizing entropy,” in International Conference on Learning Representations, 2025.arxiv.org/abs/2410.10894
  41. [41]H. Dong, M. Liu, K. Zhou, E. Chatzi, J. Kannala, C. Stachniss, and O. Fink, “Advances in multimodal adaptation and generalization: From traditional approaches to foundation models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.DOI
  42. [42]Q. Wang, O. Fink, L. Van Gool, and D. Dai, “Continual test-time domain adaptation,” in Proceedings of Conference on Computer Vision and Pattern Recognition, 2022.arxiv.org/abs/2203.13591
  43. [43]S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan, “Towards stable test-time adaptation in dynamic wild world,” in International Conference on Learning Representations, 2023.arxiv.org/abs/2302.12400
  44. [44]M. Tan, G. Chen, J. Wu, Y. Zhang, Y. Chen, P. Zhao, and S. Niu, “Uncertainty-calibrated test-time model adaptation without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 8, pp. 6274–6289, 2025.DOI
  45. [45]T. Lee, S. Chottananurak, J. Kim, J. Shin, T. Gong, and S.-J. Lee, “Test-time adaptation with binary feedback,” in Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2505.18514
  46. [46]A. Karmanov, D. Guan, S. Lu, A. El Saddik, and E. Xing, “Efficient test-time adaptation of vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 162–14 171.arxiv.org/abs/2403.18293
  47. [47]H. Cao, Y. Xu, J. Yang, P. Yin, S. Yuan, and L. Xie, “Multi-modal continual test-time adaptation for 3d semantic segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 18 809–18 819.arxiv.org/abs/2303.10457
  48. [48]J. Lee, D. Das, J. Choo, and S. Choi, “Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 16 380–16 389.arxiv.org/abs/2308.06879
  49. [49]Y. Qi, Q. Wu, P. Anderson, X. Wang, W. Y. Wang, C. Shen, and A. van den Hengel, “Reverie: Remote embodied visual referring expression in real indoor environments,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.arxiv.org/abs/1904.10151
  50. [50]J. Krantz, E. Wijnans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision and language navigation in continuous environments,” in European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/2004.02857

Paper details

vlnc:W_y6vtx9sz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents