Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

Visual question answering (VQA) demands simultaneous comprehension of both the image visual content and natural language questions. In some cases, the reasoning needs the help of common sense or general knowledge which usually appear in the form of text. Current methods jointly embed both the visual information and the textual feature into the same space. However, how to model the complex interactions between the two different modalities is not an easy task. In contrast to struggling on multimodal feature fusion, in this paper, we propose to unify all the input information by natural language so as to convert VQA into a machine reading comprehension problem. With this transformation, our method not only can tackle VQA datasets that focus on observation based questions, but can also be naturally extended to handle knowledge-based VQA which requires to explore large-scale external knowledge base. It is a step towards being able to exploit large volumes of text and natural language processing techniques to address VQA problem. Two types of models are proposed to deal with open-ended VQA and multiple-choice VQA respectively. We evaluate our models on three VQA benchmarks. The comparable performance with the state-of-the-art demonstrates the effectiveness of the proposed method.

Multimodal feature fusion by relational reasoning and attention for visual question answering

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multimodal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multimodal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a `co-attention' mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-theart performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb.

RGB-T crowd counting methods aim to enhance the counting accuracy of network models under conditions of uneven lighting and low visibility by fusing features from the RGB and thermal modalities. Previous approaches primarily utilized attention mechanisms to extract and fuse complementary RGB and thermal features. However, these methods lack guidance and constraints during the extraction and fusion of multi-modal features and do not fully leverage the complementary advantages between global and local features, leading to suboptimal performance. This paper argues that, by transitioning from global attention to local attention, extracting and fusing the complementary information between global and local multi-modal features can significantly improve the model’s counting performance. To achieve this, we propose an RGB-T crowd counting network based on global-local multimodal feature fusion (GLFNet). Specifically, we first use a multi-head attention mechanism to fuse global multi-modal features and guide the global multi-modal fusion using learnable block-counting guided tokens (BCT). Next, we employ composite spatial attention mechanisms (CSAM) to focus on the local detail information of multi-modal crowd features and facilitate the fusion of local multimodal features. Finally, we utilize a detail contrast loss function (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L_d</i>) to capture the complementary advantages between global and local multi-modal features and to guide and constrain the fusion process of multi-modal features. Experimental results on the RGBT-CC and DroneRGBT datasets demonstrate the superior performance of our method.

Visual Question Answering (VQA) is a multi-modal AI-complete task of answering natural language questions about images. Literature solved VQA with a three-phase pipeline: image and question featurisation, multi-modal feature fusion and answer generation or prediction. Most of the works have given attention to the second phase, where multi-modal features get combined ignoring the effect of individual input features. This work investigates VQA’s natural language question embedding phase by proposing a new question featurisation framework based on Supervised Term Weighting (STW) schemes. In addition, two new STW schemes integrating text semantics, qf.cos and tf.rf.sim, have been introduced to boost the framework’s performance. A series of tests on the DAQUAR VQA dataset is used to compare the new system to conventional pre-trained word embedding. Over the past few years, STW schemes have been commonly used in text classification research. In light of this, tests are carried out to verify the effectiveness of the two newly proposed STW schemes in the general text classification task.

Visual Question Answering (VQA) tasks must provide correct answers to the questions posed by given images. Such requirement has been a wide concern since this task was presented. VQA consists of four steps: image feature extraction, question text feature extraction, multi-modal feature fusion and answer reasoning. During multi-modal feature fusion, outer product calculation is used in existing models, which leads to excessive model parameters, high training overhead, and slow convergence. To avoid these problems, we applied the Variational Autoencoder (VAE) method to calculate the probability distribution of the hidden variables of image and question text. Furthermore, we designed a question feature hierarchy method based on the traditional attention mechanism model and VAE. The objective is to investigate deep questions and image correlation features to improve the accuracy of VQA tasks.

MFUR-Net: Multimodal feature fusion and unimodal feature refinement for RGB-D salient object detection

Traffic sign recognition system is an important subsystem in advanced driver assistance systems (ADAS) that assisting a driver to detect a critical driving scenario and subsequently making an immediate decision. Recently, deep architecture neural network is popular because it adapts well in various kind of scenarios, even those which were not used during training. Therefore, a deep architecture neural network is implemented to perform traffic sign classification in order to improve the traffic sign recognition rate. A comparative study for a deep and shallow architecture neural network is presented in this paper. Deep and shallow architecture neural network refer to convolutional neural network (CNN) and radial basis function neural network (RBFNN) respectively. In the simulation result, two types of training modes had been compared i.e. incremental training and batch training. Experimental results show that incremental training mode trains faster than batch training mode. The performance of the convolutional neural network is evaluated with the Malaysian traffic sign database and achieves 99% of the recognition rate.

As an interdisciplinary field of natural language processing and computer vision, Visual Question Answering (VQA) has emerged as a prominent research focus in artificial intelligence. The core of the VQA task is to combine natural language understanding and image analysis to infer answers by extracting meaningful features from textual and visual inputs. However, most current models struggle to fully capture the deep semantic relationships between images and text owing to their limited capacity to comprehend feature interactions, which constrains their performance. To address these challenges, this paper proposes an innovative Trilinear Multigranularity and Multimodal Adaptive Fusion algorithm (TriMMF) that is designed to improve the efficiency of multimodal feature extraction and fusion in VQA tasks. Specifically, the TriMMF consists of three key modules: (1) an Answer Generation Module, which generates candidate answers by extracting fused features and leveraging question features to focus on critical regions within the image; (2) a Fine-grained and Coarse-grained Interaction Module, which achieves multimodal interaction between question and image features at different granularities and incorporates implicit answer information to capture complex multimodal correlations; and (3) an Adaptive Weight Fusion Module, which selectively integrates coarse-grained and fine-grained interaction features based on task requirements, thereby enhancing the model’s robustness and generalization capability. Experimental results demonstrate that the proposed TriMMF significantly outperforms existing methods on the VQA v1.0 and VQA v2.0 datasets, achieving state-of-the-art performance in question–answer accuracy. These findings indicate that the TriMMF effectively captures the deep semantic associations between images and text. The proposed approach provides new insights into multimodal interaction and fusion research, combining domain adaptation techniques to address a broader range of cross-domain visual question answering tasks.

Multimodal feature fusion and exploitation with dual learning and reinforcement learning for recipe generation

The development towards autonomous driving directly relates to the system's robustness and reliability in environmental perception. Multimodal feature fusion, particularly fusing data from high-resolution cameras and highprecision LiDAR, is now a new way to have all the information needed to drive safely in complex urban scenarios. While this type of fusion greatly improves perception precision, and is more conducive to coping with single sensor failure or poor weather, it imposes an excessively heavy computational load and consumes excessive memory. Deep neural networks needed for feature-level fusion are often highly complex, sometimes using transformerbased attention mechanisms, which means they require a lot of resources. This makes it difficult to process things fast enough (more than 10-20 frames per second) in autonomous vehicles, where rapid perception and response are critical. This gap between accuracy and computational feasibility is currently a major roadblock to deployment in the real world. To address this issue, this paper proposes a comprehensive optimization scheme for speeding up multimodal fusion networks without making it too tough on our ability to recognize things. We integrate three key strategies: (1) the lightweight and efficient fusion architecture Fast-Fuse Net, which removes the heavy backbone and employs a Sparse Attention-based modal interaction; (2) structured pruning, which removes redundant network parameters while maintaining hardware compatibility. (3) quantization-aware training (QAT) with low-precision (INT8) inference. We validate our method on the large-scale nuScenes dataset, and demonstrate that our fully optimized model achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.8 \times$</tex> faster inference speed and a 65% reduction in model size, while retaining 98.5% of the baseline model's mAP for 3D object detection. This research provides a practical solution for deploying high-performance multimodal perception systems on the on-board computing devices of autonomous vehicles.

Addressing the limitations in current visual question answering (VQA) models face limitations in multimodal feature fusion capabilities and often lack adequate consideration of local information, this study proposes a multimodal Transformer VQA network based on local and global information integration (LGMTNet). LGMTNet employs attention on local features within the context of global features, enabling it to capture both broad and detailed image information simultaneously, constructing a deep encoder-decoder module that directs image feature attention based on the question context, thereby enhancing visual-language feature fusion. A multimodal representation module is then designed to focus on essential question terms, reducing linguistic noise and extracting multimodal features. Finally, a feature aggregation module concatenates multimodal and question features to deepen question comprehension. Experimental results demonstrate that LGMTNet effectively focuses on local image features, integrates multimodal knowledge, and enhances feature fusion capabilities.

[ENG] This thesis presents the design and implementation of end-to-end deep neural network architectures for predicting vehicle control variables in autonomous driving. Given the complexity of driving tasks, direct sensor-to-control mapping using end-to-end architectures presents a promising approach. This research aims to develop effective and efficient deep learning architectures capable of accurately predicting steering angles and vehicle speed using raw sensor data. For this purpose, a thorough review of existing end-to-end neural network architectures and data fusion techniques was conducted, as well as a study of the existing driving datasets available. A real-world dataset was collected using the UPCT-CICar, a specially designed autonomous vehicle platform equipped with multiple sensors. This dataset served as the foundation for training and testing the proposed architectures. Various deep convolutional neural network architectures were designed, optimising network hyperparameters such as kernel sizes, network depths, and convolutional layer configurations. Experimental results demonstrated that the integration of image data and angular velocity as a fused input modality significantly improved the accuracy of the predictions, proving the effectiveness of multimodal data fusion in enhancing the performance of end-to-end architectures. Additionally, a synthetic dataset was generated using the CARLA driving simulator to obtain data from a variety of driving conditions, including a wide range of weather and traffic scenarios. The architectures were trained and tested with the synthetic dataset and the results were compared to those obtained with the real-world data. To address the challenges presented by the domain gap between synthetic and real-world datasets, transfer learning and domain adaptation techniques were explored and end-to-end architectures were designed and implemented using an EfficientNet backbone. It was found that architectures pretrained on the synthetic dataset and subsequently fine-tuned with the real-world dataset achieved faster convergence and improved training efficiency. However, despite these benefits, architectures trained exclusively with real-world data outperformed those trained with synthetic data. The results obtained were compared with similar studies and show that end-to-end deep neural network architectures are capable of effectively predicting vehicle control commands from raw sensor data. The use of real and synthetic datasets, combined with sensor fusion and transfer learning strategies, has led to significant improvements in architecture performance and computational efficiency. Overall, this thesis demonstrates the advantages of using end-to-end deep neural networks to perform autonomous driving tasks, highlighting important design considerations and future research directions for enhancing architecture generalisability and efficiency.

Survey on Deep Neural Networks in Speech and Vision Systems

Causality guided co-attention network for visual question answering.