betaFeedback

WorksComputer Science

What Actually Matters? A Paired Evaluation of Inference Settings for π0.5π_{0.5} on BEHAVIOR-1K

Abstract

Vision-Language-Action (VLA) models offer a promising paradigm for general-purpose household robotics. However, inference and data-collection strategies that have been deployed in smaller-scale settings do not necessarily generalize to large-scale, diverse task suites. To address this gap, we conduct a systematic, paired evaluation of previously proposed inference and data-collection strategies across all 100 tasks. Our evaluation uses a fine-tuned π0.5π_{0.5} model based on the 2025 BEHAVIOR Challenge winner, which achieves a Q-score of 0.39 on the public test set. These results show that proposed strategies do not transfer uniformly to new environments and tasks. Project page, videos, and detailed analysis: https://rubenhammele.github.io/behavior2026

Introduction

VLA performance depends not only on learned parameters but also on inference and data-collection strategies. Inference strategies are often adopted from prior fine-tuning pipelines, yet their effectiveness beyond the original evaluation settings remains unclear. The inference strategies of the winner of the BEHAVIOR Challenge 2025 [1], Larchenko et al. [2], remain understudied. Other approaches investigate advantage guidance and rollout optimization in the context of garment folding [3], while industrial reports evaluate π0.5\pi_{0.5} fine-tuning and inference settings on a single real-world manufacturing task [4]. The extent to which these strategies generalize across diverse household tasks therefore remains an open question.

We systematically evaluate proposed inference and data-collection strategies using a fine-tuned π0.5\pi_{0.5} model across 4 instances of all 100 tasks of BEHAVIOR-1K [1]. Halving the number of actions executed per prediction decreases Q-score by 0.16, potentially causing consecutive chunks to pursue different strategies. The inherited 1.3×1.3\times playback speed-up remains beneficial, likely because it fits more of each task into the time budget. Reducing denoising from twenty steps to one causes no detectable loss, suggesting that a single step already lands close to its mode. For data collection, snapshot-based restarts reduce simulation-step consumption by 29–43%, since retries resume near the failure point instead of re-simulating already completed stages from the initial state.

Experimental Setup

Training Our policy is initialized from the pre-trained π0.5\pi_{0.5} checkpoint introduced by Larchenko et al. [2]. The policy is fully fine-tuned on the BEHAVIOR-1K dataset. Training then continues in a loop in which the current checkpoint collects rollouts on training instances, and the (partial) successful rollouts are mixed with the demonstrations for further training. Each training frame carries a binary advantage label for RECAP-style conditioning [5].

Benchmark and Statistical Analysis. We evaluate all 100 BEHAVIOR-1K tasks on public test instances under official constraints [6]. For each cell, the control is the mean of two independent baseline runs. We assess paired score differences (ΔQ\Delta Q) using task-clustered bootstrap 95% confidence intervals (CIs), classifying effects as helpful or harmful when the CI excludes zero.

Experiments & Results

Inference Ablations

Action-Chunk Execution Horizons: Prior work adopts different action-chunk execution horizons [7, 8], while findings on the benefits of longer execution horizons remain task- and policy-dependent [2, 9]. Consistent with this trade-off, halving the executed prefix from 26 to 13 actions substantially reduced Q-score and success rate while increasing episode length (Table 1). Playback Speed-Up: The 1.3×\times playback speed-up used by Larchenko et al. [2] also outperformed standard-speed execution, whereas increasing the speed-up to 2×\times leaned worse without reaching significance. Denoising Steps: Prior work uses different numbers of flow-matching inference steps, from 4 in GR00T N1 [10] to 10 in π0\pi_0 [7]. Starting from the 20-step configuration of Larchenko et al. [2], reducing denoising to 10 or one step produced no detectable performance change. Overlap Inpainting: Real-Time Chunking (RTC) uses inpainting to maintain chunk continuity during asynchronous inference [11]. Larchenko et al. [2] also uses overlap inpainting but does not ablate it. Disabling it had no detectable effect on Q-score. Advantage-Guidance Strength: Increasing advantage-guidance strength to β=2\beta= 2 was worse, while β=4\beta= 4 matched β=1\beta= 1. Rendering: Motivated by the gains from high-resolution rendering reported by Bai et al. [9], we compare downsampling with direct rendering at 224×224224 \times224.

SettingBase.TestedΔQ\Delta QΔ\DeltaSRΔ\DeltaStepsSig.
Actions/chunk2613−0.16±0.03-0.16 \pm0.03−11-11+752+752Yes
30−0.01±0.02-0.01 \pm0.020−53-53No
Speed-up1.3×1.3\times1×1\times−0.04±0.02-0.04 \pm0.02−4-4+433+433Yes
2×2\times−0.03±0.02-0.03 \pm0.02−2-2−135-135No
Denoising2010−0.02±0.02-0.02 \pm0.02−1-1+7+7No
1+0.01±0.02+0.01 \pm0.021−23-23No
InpaintingOnOff−0.02±0.02-0.02 \pm0.02−2-2−7-7No
Guidance β\beta12−0.03±0.02-0.03 \pm0.02−4-4+333+333Yes
4−0.00±0.02-0.00 \pm0.020+42+42No
Gripper Snaponoff−0.03±0.02-0.03 \pm0.02−5-5+326+326No
RenderingDownsampledDirect−0.03±0.02-0.03 \pm0.02−3-3+125+125No
SamplingSingleBest-Of-4−0.01±0.02-0.01 \pm0.02−2-2−1-1No

Table 1. Inference ablations relative to baseline.

Data Collection

To maximize the number of successful rollouts collected for training, we evaluate strategies to reduce data-collection costs and improve recovery from failed episodes (Table 2). Restart Points: Prior work replays failure states [3], revisits promising states [12], and expands start states outward from the goal [13]. Among the tested restart points, restarting before the first drop in predicted task progress earned more progress than restarting at the last progress point. Simulation Cost: Restarts reduced simulation cost but did not significantly improve successes per simulation step. Stage-Belief Restoration: Since the policy feeds back its predicted stage [2], omitting this state during restoration reduced Q-score and task progress, supporting the preservation of policy-side state.

ComparisonΔQ\Delta QΔ\DeltaSRSteps
Restart point (Baseline: fresh start)
Before first drop+0.076±0.028+0.076 \pm0.028+0.3±2.5+0.3 \pm2.5−29%-29\%
Before last progress+0.059±0.023+0.059 \pm0.023−3.1±2.1-3.1 \pm2.1−38%-38\%
At last progress+0.056±0.021+0.056 \pm0.021−3.4±2.3-3.4 \pm2.3−43%-43\%
Modified restore vs. full restore
Stage belief not restored−0.05±0.02-0.05 \pm0.02−5.1±2.3-5.1 \pm2.3–

Table 2. Data collection ablations.

Conclusion

We have evaluated inference and data-collection strategies across all 100 BEHAVIOR-1K tasks. Shorter action chunks significantly reduced Q-score and a playback speed of 1.3 improved performance. Fewer denoising steps and disabling overlap inpainting maintained performance, whereas stronger advantage guidance offered no benefit. Snapshot restarts reduced simulation cost but not success, and required restoring the policy’s stage belief. These results suggest that inference and data-collection strategies do not necessarily scale uniformly across task complexity.

Acknowledgments

Thank you to the Stanford BEHAVIOR Challenge organizers, who created this benchmark. This work has taken place in the VCAI Lab at Kiel University and in the RobotiXX Lab at George Mason University. The research was conducted under the RobOdin project funded by the European Union through its Interreg Germany-Denmark funding program. RobotiXX research is supported by National Science Foundation (NSF, 2350352), Army Research Office (ARO, W911NF2320004, W911NF2520011), Army Ground Vehicle Systems Center (GVSC), Google DeepMind (GDM), Microsoft Research (MSR), Clearpath Robotics, FrodoBots Lab, Raytheon Technologies (RTX), Tangenta, 4-VA, Mason Innovation Exchange (MIX), and Walmart. We acknowledge technical support by the Embodied AI Center at Kiel University.

References

  1. [1]C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, W. Ai, B. Martinez, H. Yin, M. Lingelbach, M. Hwang, A. Hiranaka, S. Garlanka, A. Aydin, S. Lee, J. Sun, M. Anvari, M. Sharma, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, Y. Li, S. Savarese, H. Gweon, C. K. Liu, J. Wu, and L. Fei-Fei. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXiv:2403.09227, 2024.arxiv.org/abs/2403.09227
  2. [2]I. Larchenko, G. Zarin, and A. Karnatak. Task adaptation of vision-language-action model: 1st place solution for the 2025 behavior challenge, 2025. URL https://arxiv.org/abs/2512.06951.
  3. [3]I. Larchenko. Learning to fold: prizewinning solution at lehome challenge 2026 (1st place online, 2nd offline), 2026. URL https://arxiv.org/abs/2606.27163.
  4. [4]Dream Machines. Empirical results fine-tuning π₀.₅ on a real manufacturing task. https://dream-machines.eu/blog/pi05-fine-tuning, Sept. 2026. Blog post.
  5. [5]P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levine, A. Li-Bell, Y. Lu, V. Mano, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, C. Sharma, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, A. Swerdlow, J. Tanner, M. T. Thorne, Q. Vuong, A. Walling, H. Wang, B. Williams, S. Yoo, L. Yu, U. Zhilinsky, and Z. Zhou. π₀.₆: a vla that learns from experience, 2025. URL https://arxiv.org/abs/2511.14759.
  6. [6]Stanford Vision and Learning Lab. Behavior challenge 2026: Evaluation and rules. https://behavior.stanford.edu/challenge/evaluation.html, 2026. Accessed: 2026-10-24.
  7. [7]K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. π₀: A vision-language-action flow model for general robot control, 2026. URL https://arxiv.org/abs/2410.24164.
  8. [8]C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2024.
  9. [9]J. Bai, Y.-W. Chao, Q. Chen, J. Gu, M. J. Kim, Z. Li, X. Li, T.-Y. Lin, M.-Y. Liu, N. Ma, K. Mo, D. Qu, S. Sun, H. Xia, F. Wei, and X. Zeng. Openpi comet: Competition solution for 2025 behavior challenge. arXiv preprint arXiv:2512.10071, 2025. URL https://arxiv.org/abs/2512.10071.
  10. [10]NVIDIA, :. J. Bjork, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu. Gr00t n1: An open foundation model for generalist humanoid robots, 2025. URL https://arxiv.org/abs/2503.14734.
  11. [11]K. Black, M. Y. Galliker, and S. Levine. Real-time execution of action chunking flow policies, 2025. URL https://arxiv.org/abs/2506.07339.
  12. [12]A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return then explore. CoRR, abs/2004.12919, 2020. doi:10.48550/ARXIV.2004.12919. URL https://doi.org/10.48550/arXiv.2004.12919.
  13. [13]C. Florensa, D. Held, M. Wulfmeier, and P. Abbeel. Reverse curriculum generation for reinforcement learning. CoRR, abs/1707.05300, 2017. doi:10.48550/ARXIV.1707.05300. URL https://doi.org/10.48550/arXiv.1707.05300.

Paper details

vlnc:W_n7nx9sfg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents