COROMER: A Multimodal Transformer for Collaborative Robots’ Skill Learning
Аннотация
Achieving intuitive and effective collaboration between humans and robots requires robotic systems capable of understanding and acting on rich, multimodal cues that reflect human intent and environmental context. Designing robots to integrate vision, language, and proprioception remains challenging due to the complexity of aligning diverse modalities. This paper introduces Collaborative Robot Transformer (COROMER), a novel multimodal transformer designed to advance skill learning in collaborative robots by fusing visual data, natural language instructions, and robot state information. Trained on the Mobile ALOHA dataset, COROMER leverages multi-scale Vision Transformers (ViT), a Bidirectional Encoder Representations from Transformers (BERT) based language encoder, and a temporal transformer decoder to generate coherent and context-aware control commands for mobile manipulation tasks. Empirical evaluations demonstrate that COROMER achieves a success rate of 87.14% with a validation loss Mean Squared Error (MSE) of 0.1729, outperforming larger baseline models in task-specific generalization while maintaining a compact architecture of ~ 195 M parameters. Qualitative analysis shows robust spatial reasoning, effective grounding of natural language, and smooth base–arm coordination. These findings highlight COROMER’s potential to set a new benchmark for scalable, interpretable, and resource-efficient multimodal policy learning, paving the way for next-generation collaborative robots that can adapt to dynamic environments and diverse human inputs.