betaFeedback

WorksComputer Science

Task-Specialized Language-Conditioned Models for Fine-Grained Robotic Manipulation

Abstract

Fine-grained robotic manipulation requires instruction grounding, robust perception, and precise contact control. We describe our submission to the MetaFine Open Competition at CoRL 2026. Five independently trained models use a common architecture with frozen DINOv2 and CLIP encoders, trainable language-conditioned feature modulation, and task-specific action heads. We improve demonstrations through contact-aware grasp filtering, instruction-matched data, and expert correction of insertion failures. At inference, task-dependent strategies include photometric test-time augmentation, stochastic action averaging, and uncertainty-triggered replanning. Our submission achieves an official overall score of 0.914 and ranks third among 15 teams. We analyze the models and their task-specific behavior.

Introduction

Fine-grained manipulation requires robots to identify instructed object parts and execute precise movements despite viewpoint and lighting changes. MetaFine [1] tests these abilities through part grasping (T1), mug transport (T2), switch manipulation (T3), block placement (T4), and letter insertion (T5), evaluating understanding, perception, and controlled behavior.

We use a shared vision-language architecture with separate task-specific models. To address different failure modes, we combine instruction-conditioned visual features with curated demonstrations and task-dependent execution strategies. Figure 1 summarizes the common architecture, while Table 1 reports the distinct training and inference configurations.

Figure 1. Overall architecture of our approach. Frozen DINOv2 and CLIP encoders provide language-conditioned visual features through trainable FiLM modules. A multimodal Transformer drives action heads that produce end-effector targets for T1–T4 or joint-space actions for T5.

TaskHead / actionEpisodesTraining emphasisInference
T1ℓ1\ell_1 / EE5,024Contact audit, rolloutsTTA, residual
T2FM / EE1,200Directional demosFM mean, replan
T3ℓ1\ell_1 / EE4,800Paired target groundingTTA
T4FM / EE4,644Color-conditioned demosFM mean
T5FM / joint6,723DAgger-style correctionFM mean, replan

Table 1. Task-specific training and deployment settings. Episode counts include rendered camera views before applying training masks.

Model Architecture and Inference

Language-Conditioned Representation

Each model receives base and wrist RGB observations, a nine-dimensional robot state containing seven arm joint positions and two gripper-finger positions, and a language instruction. The 512×512512 \times512 images are resized to 224×224224 \times224 and encoded by frozen DINOv2-Base [2], producing 257 visual tokens per view. A frozen Contrastive Language–Image Pre-training (CLIP) text encoder [3] supplies the language representation. Trainable Feature-wise Linear Modulation (FiLM) [4] adjusts 12 visual blocks through instruction-dependent scale and bias, hℓ=(1+γℓ(c))⊙hℓ+βℓ(c)h_{\ell} = (1 + \gamma_{\ell}(c)) \odot h_{\ell} + \beta_{\ell}(c), with language condition cc and identity initialization. The visual features are projected to 512 dimensions and concatenated with a robot-state token and 32 learnable action queries. A four-layer, eight-head Transformer encoder processes these tokens. T1 and T3 use target-aware features, while T5 uses phase conditioning in its decoder.

Action Prediction and Control

All models predict 32-step action chunks [5]. T1 and T3 use direct regression with a masked ℓ1\ell_1 loss for target-directed actions. T2, T4, and T5 use conditional flow matching (FM) [6], allowing stochastic generation of action sequences. For demonstrated actions aa, noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), time t∼Beta⁡(1.5,1)t \sim\operatorname{Beta}(1.5,1), and Transformer context coc_o, the velocity field is trained as

xt=(1−t)a+tϵ,LFM=E[∥vθ(xt,t,co)−(ϵ−a)∥22].(1)\begin{aligned} x_t &= (1-t)a+t\epsilon,\\ \mathcal{L}_{\mathrm{FM}} &= \mathbb{E}\left[\left\|v_\theta(x_t,t,c_o)-(\epsilon-a)\right\|_2^2\right]. \tag*{(1)} \end{aligned}

Here, t=1t=1 denotes noise and t=0t=0 denotes the demonstrated action. Generation integrates backward from t=1t=1 to t=0t=0 using ten Euler steps. For T1–T4, the heads output ten-dimensional end-effector (EE) targets comprising three position coordinates, a six-dimensional rotation representation, and a gripper command. Damped-least-squares inverse kinematics (IK) converts the target pose into arm commands, while the gripper command is passed separately. T5 directly predicts seven arm joint deltas and one gripper command.

Task-Dependent Inference

For T1 and T3, photometric test-time augmentation (TTA) [7] produces eight action predictions from the original observations and seven brightness- and contrast-perturbed variants. We select the ℓ1\ell_1 medoid, which minimizes the total distance to the other predictions, without updating model parameters. For FM-based models, we instead average eight action chunks generated from different Gaussian noise samples under the same observation to reduce dependence on a single sample.

For T2 and T5, the execution horizon adapts to disagreement among the eight sampled chunks. The action disagreement at future step jj is calculated as follows:

uj=1da∑k=1daStd⁡m=18[aj,k(m)].(2)u_j=\frac{1}{d_a}\sum_{k=1}^{d_a}\operatorname{Std}_{m=1}^{8}\left[a_{j,k}^{(m)}\right]. \tag*{(2)}

Here, dad_a is the action dimension and aj,k(m)a_{j,k}^{(m)} is the normalized kk-th action component at step jj in sample mm. The model executes at least four steps and then truncates the chunk before the first step with uj>0.11u_j>0.11 in normalized action units. It then generates a new chunk from a fresh observation. This is related to adaptive action chunking (AAC) [8], but uses a fixed sample-disagreement threshold rather than entropy-based horizon selection. T1 additionally uses a wrist-conditioned joint residual around gripper transitions to refine contact-sensitive actions.

Task-Specific Learning

We train task-specific models with AdamW, batch size 16, and a learning rate of 10−410^{-4} with 1,000 warm-up steps followed by cosine decay. Camera-view augmentation broadens the demonstrations across viewpoint changes. Table 1 summarizes the deployed configurations.

T1. Contact-aware part grasping. We filter scripted demonstrations to exclude premature contact, arm collisions, and unintended object displacement. After initial training, successful model rollouts replace 30% of expert body-grasp demonstrations to incorporate states encountered during execution. Grasp-point features condition the action Transformer, while a 3D grasp-vector head provides auxiliary supervision.

T2–T4. Instruction-specific demonstrations. T2 uses 1,200 camera-rendered episodes spanning left, right, and forward mug transport. T3 pairs red and blue switch instructions within the same scenes, yielding 4,800 rendered episodes. An instruction-conditioned grounding module converts the correspondence between observed switch attributes and the requested target into a soft action-conditioning token. T4 uses 4,644 rendered episodes for color-conditioned block placement. These datasets emphasize directional, target-selection, and sequential placement requirements, respectively.

T5. Corrective learning for insertion. The initial model could grasp and carry letters but often stalled near the narrow slots. We therefore adopt a Dataset Aggregation (DAgger)-style strategy [9]. Following failed rollouts, an expert resumes from recorded near-slot states to complete insertion. Collection emphasizes letters with higher failure rates and retains 458 successful corrections. Rendering these corrections under three camera conditions and combining them with the original multi-view demonstrations yields 6,723 episodes. The FM decoder also employs phase- and language-conditioned Low-Rank Adaptation (LoRA) [10] with auxiliary slot-alignment supervision.

Experimental Results

Official Evaluation

The competition reports understanding (UU), perception (PP), and controlled behavior (BB). Perception uses area under the success curve (AUSC) across visual perturbations. Each task score is St=(Ut+Pt+Bt)/3S_t = (U_t + P_t + B_t)/3, and the overall score is Sall=15∑t=15StS_{\mathrm{all}} = \frac{1}{5}\sum_{t=1}^{5} S_t. Our final submission ranks third among 15 teams with the official overall score 0.914. Table 2 reports the task scores.

TaskPPUUBBScore
T1 Grasp part0.7350.7750.7750.762
T2 Move mug0.9671.0001.0000.989
T3 Toggle switch0.9881.0001.0000.996
T4 Place blocks1.0001.0001.0001.000
T5 Insert letter0.8160.8250.8250.822
Overall–––0.914

Table 2. Official MetaFine evaluation results. Scores are rounded for display.

Task-wise Results and Analysis

T1. Grasp part. T1 achieves an official score of 0.762, with perception AUSC of 0.735 and understanding of 0.775. Part-level grasping remains less reliable than mug transport, switch manipulation, and block placement under the official evaluation.

T2. Mug transport. T2 achieves 0.989, with perfect understanding and perception AUSC of 0.967. The multi-view-trained model reliably performs left, right, and forward transport under the evaluated visual perturbations.

T3. Switch manipulation. T3 achieves 0.996, with perfect understanding and perception AUSC of 0.988. The model trained with paired instructions and a grounding token reliably selects the requested switch across the evaluated viewpoint and lighting changes.

T4. Block placement. T4 achieves 1.000 across all official dimensions. As shown in Figure 2(a), fixed-horizon execution outperformed replanning with identical weights and eight-sample averaging, achieving 96.4% versus 93.6% over 140 perception episodes and 98.3% versus 96.7% over 60 understanding episodes. Block placement may require fewer alignment corrections than narrow-slot insertion in T5, while repeated chunk regeneration could disrupt motion continuity. We therefore retained fixed-horizon execution for T4.

Figure 2. Task-dependent effects of replanning. Episode success rates with and without uncertainty-triggered replanning for (a) T4 block placement and (b) T5 letter insertion.

T5. Letter insertion. T5 achieves an official score of 0.822 and 86.9% understanding success in a separate 320-episode evaluation. Figure 2(b) compares execution strategies using identical weights, eight-sample action averaging, and 10 episodes per letter. Replanning increased insertion success from 55% to 95%, with R improving from 30% to 90%, supporting its use for T5. Remaining failures involved incomplete seating near the target slot.

Conclusion

We presented a shared language-conditioned model architecture with task-specific data refinement and execution strategies to address five distinct manipulation tasks. Our final submission achieves 0.914, ranking third among 15 teams. The execution ablations support task-dependent use of replanning rather than applying it uniformly.

References

  1. [1]H.-Y. Xu, P. Zhang, Z. Ge, X. Hao, S. Belongie, X. Geng, Y. Peng, and X.-S. Wei. Beyond binary success: A diagnostic meta-evaluation framework for fine-grained manipulation. arXiv preprint arXiv:2605.19986, 2026.arxiv.org/abs/2605.19986
  2. [2]M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, W. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Trans. Mach. Learn. Res., 2024.arxiv.org/abs/2304.07193
  3. [3]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., volume 139, pages 8748–8763, 2021.arxiv.org/abs/2103.00020
  4. [4]E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville. FiLM: Visual reasoning with a general conditioning layer. In Proc. AAAI Conf. Artif. Intell., pages 3942–3951, 2018.arxiv.org/abs/1709.07871
  5. [5]T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proc. Robot. Sci. Syst., 2023.arxiv.org/abs/2304.13705
  6. [6]Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In Proc. Int. Conf. Learn. Represent., 2023.arxiv.org/abs/2210.02747
  7. [7]D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag. Better aggregation in test-time augmentation. In Proc. IEEE/CVF Int. Conf. Comput. Vis., pages 1214–1223, 2021.arxiv.org/abs/2011.11156
  8. [8]Y. Liang, X. Wang, K. Wang, S. Wang, X. Peng, H. Chen, D. K. H. Chua, and P. Vadakkepat. Adaptive action chunking at inference-time for vision-language-action models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pages 20802–20811, 2026.arxiv.org/abs/2604.04161
  9. [9]S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proc. Int. Conf. Artif. Intell. Statist., volume 15, pages 627–635, 2011.arxiv.org/abs/1011.0686
  10. [10]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Proc. Int. Conf. Learn. Represent., 2022.arxiv.org/abs/2106.09685

Paper details

vlnc:W_gsrtchg5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents