BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

Abstract

Vision-Language-Action (VLA) models have emerged as a promising paradigm for grounding visual-language understanding into real-world robotic manipulation. However, dexterous manipulation remains challenging for VLA policies due to high-dimensional hand control and compounding execution errors, which makes real-world RL post-training essential for bridging the gap between visually grounded action generation and physically reliable dexterous execution. However, high-dimensional dexterous exploration often triggers temporal inconsistency, sample inefficiency and hardware risks in the real world. To address these challenges, we propose BORA, an offline-to-online RL post-training framework designed for real-world dexterous VLA models. In the offline phase, BORA constructs a critic that takes both the VLM's cognition tokens and action chunks as inputs. This design enables action-conditioned value guidance, allowing the critic to evaluate dexterous hand motions beyond visual context alone. During the subsequent online phase, BORA freezes the VLA base and introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism to mitigate real-world execution errors and further correct the offline-learned intents within the actual physical environment. By inheriting the offline critic and employing intervention-driven rewards, BORA effectively corrects execution discrepancies and adapts to real-world physical variances while preserving the pretrained policy as a stable prior. Extensive evaluations across five complex real-world dexterous tasks demonstrate that BORA significantly outperforms pure imitation learning and traditional decoupled RL baselines, achieving a 33% absolute increase in average success rate under standard settings and up to a 43% improvement in unseen object generalization.

Contributions

• Token-Action Integrated Architecture: a critic that fuses continuous action chunks with VLM semantic features to mitigate visual value hacking and improve intent evaluation in offline RL.
• Lightweight Residual Online Adaptation: a Human-in-the-Loop residual chunk mechanism that freezes the VLA base while inheriting the offline critic for safe physical adaptation.
• The BORA Unified Framework: a complete Offline-to-Online post-training paradigm that improves success rates and real-world robustness for dexterous manipulation.

Illustration of the BORA framework. BORA bridges offline token-action reinforcement learning and online residual adaptation for real-world dexterous VLA policies. In the offline stage, the VLM encoder and action expert produce shared VLM tokens and action chunks, which are jointly evaluated by an integrated critic with semantic anchoring, temporal credit disentanglement, and IQL-based offline policy optimization. In the online stage, the offline-aligned base VLA is frozen, while a lightweight residual chunk actor is trained with inherited critic feedback, sparse task rewards, and human-in-the-loop intervention signals. The final action chunk is obtained by residual composition, A_final = A_base + lambda A_res, enabling low-cost physical adaptation and improved dexterous execution.

Citation

If you use this work, please cite:

@article{chen2026bora,
  title={BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models},
  author={Zhongxi Chen and Yifan Han and Yanming Shao and Huanming Liu and Congsheng Xu and Xiaoyu Chen and Yao Mu and Wenzhao Lian},
  journal={arXiv preprint arXiv:2605.30226},
  year={2026},
  url={https://arxiv.org/abs/2605.30226}
}