Visual Question Answering Performance Research Articles

VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. Although the “pre-training & finetuning” learning paradigm significantly improves the VQA performance, the adversarial robustness of such a learning paradigm has not been explored. In this paper, we delve into a new problem: using a pre-trained multimodal source model to create adversarial image-text pairs and then transferring them to attack the target VQA models. Correspondingly, we propose a novel VQATTACK model, which can iteratively generate both im- age and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module. At each iteration, the LLM-enhanced image attack module first optimizes the latent representation-based loss to generate feature-level image perturbations. Then it incorporates an LLM to further enhance the image perturbations by optimizing the designed masked answer anti-recovery loss. The cross-modal joint attack module will be triggered at a specific iteration, which updates the image and text perturbations sequentially. Notably, the text perturbation updates are based on both the learned gradients in the word embedding space and word synonym-based substitution. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQATTACK in the transferable attack setting, compared with state-of-the-art baselines. This work reveals a significant blind spot in the “pre-training & fine-tuning” paradigm on VQA tasks. The source code can be found in the link https://github.com/ericyinyzy/VQAttack.

ZVQAF: Zero-shot visual question answering with feedback from large language models

Due to the prominent zero-shot generalization in new language tasks shown by large language models (LLMs), applying LLMs for zero-shot visual question answering (VQA) has been a new trend. However, most prior approaches directly use off-the-shelf captioning models to generate captions that compose in-context examples for LLMs, and such generated captions may be uninformative, thus leading the LLMs to give false predictions. To address this, we propose zero-shot VQA with feedback from LLMs (ZVQAF), a framework that applies LLMs to discriminate the quality of generated captions and leverages this feedback to train the captioning model. ZVQAF consists of two stages: the first stage is the training with feedback, which enables the captioning model to recognize the task objective and information requirements from the LLM, and the second stage is utilizing the optimized captioning model and LLM for inference. Extensive experiments show that ZVQAF achieves zero-shot VQA performance that is comparable or even superior to those previous zero-shot, few-shot, and end-to-end training approaches. For example, on VQAv2 test dataset, ZVQAF outperforms Flamingo (Alayrac et al., 2022) which employs end-to-end training by a large margin of 8.0%. In addition, on A-OKVQA dataset, ZVQAF outperforms zero-shot method Img2LLM (Guo et al., 2023) by 3.8% when employing LLMs with similar scales.

Improving visual question answering for bridge inspection by pre‐training with external data of image–text pairs

AbstractThis paper explores the application of visual question answering (VQA) in bridge inspection using recent advancements in multimodal artificial intelligence (AI) systems. VQA involves an AI model providing natural language answers to questions about the content of an input image. However, applying VQA to bridge inspection poses challenges due to the high cost of creating training data that requires expert knowledge. To address this, we propose leveraging existing bridge inspection reports, which already include image–text pairs, as external knowledge to enhance VQA performance. Our approach involves training the model on a large collection of image–text pairs, followed by fine‐tuning it on a limited amount of training data specifically designed for the VQA task. The results demonstrate a significant improvement in VQA accuracy using this approach. These findings highlight the potential of AI models for VQA as valuable tools for assessing the condition of bridges.

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3’s power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.

Accuracy vs. complexity: A trade-off in visual question answering models

Visual Question Answering (VQA) has emerged as a Visual Turing Test to validate the reasoning ability of AI agents. The pivot to existing VQA models is the joint embedding that is learned by combining the visual features from an image and the semantic features from a given question. Consequently, a large body of literature has focused on developing complex joint embedding strategies coupled with visual attention mechanisms to effectively capture the interplay between these two modalities. However, modelling the visual and semantic features in a high dimensional (joint embedding) space is computationally expensive, and more complex models often result in trivial improvements in the VQA accuracy. In this work, we systematically study the trade-off between the model complexity and the performance on the VQA task. VQA models have a diverse architecture comprising of pre-processing, feature extraction, multimodal fusion, attention and final classification stages. We specifically focus on the effect of “multi-modal fusion” in VQA models that is typically the most expensive step in a VQA pipeline. Our thorough experimental evaluation leads us to three proposals, one optimized for minimal complexity, one for balanced complexity-accuracy and the last one for state-of-the-art VQA performance.

Local relation network with multilevel attention for visual question answering

With the tremendous success of the visual question answering (VQA) tasks, visual attention mechanisms have become an indispensable part of VQA models. However, these attention-based methods do not consider any relationship among regions, which is crucial for the thorough understanding of the image by the model. We propose local relation networks for generating context-aware image features for each image region, which contain information on the relationship among the other image regions. Furthermore, we propose a multilevel attention mechanism to combine semantic information from the LRNs and the original image regions, rendering the decision of the model more reasonable. With these two measures, we improve the region representation and achieve better attentive effect and VQA performance. We conduct numerous experiments on the COCO-QA dataset and the largest VQA v2.0 benchmark dataset. Our model achieves competitive results, proving the effectiveness of our proposed LRNs and multilevel attention mechanism through visual demonstrations.

Fair-VQA: Fairness-Aware Visual Question Answering Through Sensitive Attribute Prediction

Visual Question Answering (VQA) is a task that answers questions on given images. Although previous works achieve a great improvement in VQA performance, they do not consider the fairness of answers in terms of ethically sensitive attributes, such as gender. Therefore, we propose a Fair-VQA model that contains two modules: VQA module and SAP (Sensitive Attribute Prediction) module. On top of VQA module, which predicts various kinds of answers, SAP module predicts only sensitive attributes using the same inputs. The predictions of SAP module are utilized to rectify answers from VQA module to be fairer in terms of the sensitive attributes with graceful performance degradation. To validate the proposed method, we conduct extensive experiments on VQA, GQA, and our proposing VQA-Gender datasets. In all the experiments, our method shows the fairest results in various metrics for fairness. Moreover, we demonstrate that our method works interpretably through the analysis of visualized attention maps.

Multimodal Encoder-Decoder Attention Networks for Visual Question Answering

Visual Question Answering (VQA) is a multimodal task involving Computer Vision (CV) and Natural Language Processing (NLP), the goal is to establish a high-efficiency VQA model. Learning a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions is the heart of VQA. In this paper, a novel Multimodal Encoder-Decoder Attention Networks (MEDAN) is proposed. The MEDAN consists of Multimodal Encoder-Decoder Attention (MEDA) layers cascaded in depth, and can capture rich and reasonable question features and image features by associating keywords in question with important object regions in image. Each MEDA layer contains an Encoder module modeling the self-attention of questions, as well as a Decoder module modeling the question-guided-attention and self-attention of images. Experimental evaluation results on the benchmark VQA-v2 dataset demonstrate that MEDAN achieves state-of-the-art VQA performance. With the Adam solver, our best single model delivers 71.01% overall accuracy on the test-std set, and with the AdamW solver, we achieve an overall accuracy of 70.76% on the test-dev set. Additionally, extensive ablation studies are conducted to explore the reasons for MEDAN’s effectiveness.

Affective question answering on video

Visual Question Answering (VQA) is an increasingly popular research area in machine learning. Most of the existing VQA tasks only focus on static images, and only a few models are based on videos. The primary purpose of this project is to develop an innovative model that performs Affective Question Answering on Video (AQAV), a multi-tasking architecture that implements a Video QA route and an Affective route. A pre-trained CNN emotion detector recognizes emotions on the frames of a video, and a string of the emotion labels is relayed to the Token-based, Frame-based and Integrated attention mechanisms. The attention model uses the visual features, the question and the emotion labels to focus on relevant frames of the video and relevant regions of the frames. The string of emotion labels is used to generate an emotion caption that will be used by the Text QA module to prepare an affective answer. A conventional answer is generated from processes that take place along the Video QA route, while the affective answer is a product of both the Video QA and the Affective routes. Our model does not only make VQA more analytic by generating an explanatory answer, but also registers quantitative improvement in performance, when compared with previous baselines. We managed to prove that the injection of emotions in the attention mechanism boosts VQA performance. The AQAV model contributes towards efforts in making machines understand sequential and dynamic visual scenes in the real world.

Question-Led object attention for visual question answering

Question plays a leading role for Visual Question Answering (VQA) because it specifies the particular visual objects or conjures vivid visual that the machine should attend. However, existing approaches predominantly predict the answer by utilizing the question and the whole image without considering the leading role of the question. Also, recent object spatial inference is usually conducted on pixel level instead of object level. Therefore, we propose a novel but simple framework, namely Question-Led Object Attention (QLOB), to improve the VQA performance by exploring question semantics, fine-grained object information, and the relationship between those two modalities. First, we extract sentence semantics by a question model, and utilize the efficient object detection network to obtain a global visual feature and local features from top r object region proposals. Second, our QLOB attention mechanism selects those question-related object regions. Third, we optimize question model and QLOB attention by a softmax classifier to predict the final answer. Extensive experimental results on three public VQA datasets demonstrate that our QLOB outperforms the state-of-the-arts.

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

Visual question answering (VQA) is challenging, because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multimodal feature fusion that is able to capture the complex interactions between multimodal features; and 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a "coattention" mechanism is developed using a deep neural network (DNN) architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multimodal feature fusion, a generalized multimodal factorized high-order pooling approach (MFH) is developed to achieve more effective fusion of multimodal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the Kullback-Leibler divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A DNN architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA data sets and win the runner-up in VQA Challenge 2017.

Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?

We conduct large-scale studies on ‘human attention’ in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a question. Thus, we introduce the VQA-HAT (Human ATtention) dataset. We evaluate attention maps generated by state-of-the-art VQA models against human attention both qualitatively (via visualizations) and quantitatively (via rank-order correlation). Our experiments show that current attention models in VQA do not seem to be looking at the same regions as humans. Finally, we train VQA models with explicit attention supervision, and find that it improves VQA performance.