PILL: Plug into LLM with adapter expert and attention gate

Abstract

Due to the remarkable capabilities of powerful Large Language Models (LLMs) in effectively following instructions, significant progress has recently been made in the development of Vision Language Models (VLMs), expanding the capabilities of LLMs in multi-modal learning and enabling them to be fine-tuned in a parameter-efficient manner. Promising as the existing pre-trained vision-language models might be, there are two intractable issues including modal entanglement and insufficient capability of LLMs in handling visual information. To tackle these two issues, we introduce an architecture called PILL which Plugs Into LLM with adapter expert and attention gate. To maintain the capability of the original LLMs and the more progressive injection of visual modality, one Modality-Attention-Gating (MAG) module is introduced, enabling adaptive control of the contribution of modality tokens to the overall representation. Specifically, one Mixture-of-Modality-Adapter-Expert (MoMAE) module is proposed to handle different modalities with the dedicated adapters. In addition, further improvement is made to the adapter to enhance its learning and expressive capabilities. We adopt a two-stage training paradigm to optimize different modules of our model. Experimental results demonstrate that our approach exhibits competitive performance compared to other mainstream methods for modality fusion with much lower resources. We provide free access to the code and models.22https://github.com/DsaltYfish/PILL.