Transformer-based vision-language alignment for robot navigation and question answering

Abstract

The task of robot navigation and question answering, which is also known as Embodied Question Answering (EQA), places its emphasis on empowering agents to actively explore their environments and deliver answers to user inquiries. Considering the extensive range of potential applications, particularly in the realms of home robots and personal assistants, the Embodied Question Answering task has attracted growing attention from researchers. Owing to the difficulties in bridging the semantic divide between inputs from different modalities and capturing extended connections between widely separated pixels, most existing methods face challenges in attaining adequate performance concerning accuracy in navigation and responses. To address these challenges, we present a transformer-based framework that aligns vision and language information for the task of robot navigation and question answering. Firstly, an information fusion model is designed to utilize object tags as reference points for aligning the vision and linguistic modalities into a coherent semantic space. Secondly, a dedicated transformer block is employed to capture extensive dependencies within visual scenes, enabling the generation of more contextually appropriate actions. Lastly, the two transformer-based components are seamlessly integrated into a cohesive framework, effectively handling the complete Embodied Question Answering task. The results of our experiments clearly indicate that our approach substantially boosts the performance of every module in the system, resulting in a notable 4.1% enhancement in overall accuracy.