Multimodal Objective Research Articles

Informative Data Selection With Uncertainty for Multimodal Object Detection.

Noise has always been nonnegligible trouble in object detection by creating confusion in model reasoning, thereby reducing the informativeness of the data. It can lead to inaccurate recognition due to the shift in the observed pattern, that requires a robust generalization of the models. To implement a general vision model, we need to develop deep learning models that can adaptively select valid information from multimodal data. This is mainly based on two reasons. Multimodal learning can break through the inherent defects of single-modal data, and adaptive information selection can reduce chaos in multimodal data. To tackle this problem, we propose a universal uncertainty-aware multimodal fusion model. It adopts a multipipeline loosely coupled architecture to combine the features and results from point clouds and images. To quantify the correlation in multimodal information, we model the uncertainty, as the inverse of data information, in different modalities and embed it in the bounding box generation. In this way, our model reduces the randomness in fusion and generates reliable output. Moreover, we conducted a completed investigation on the KITTI 2-D object detection dataset and its derived dirty data. Our fusion model is proven to resist severe noise interference like Gaussian, motion blur, and frost, with only slight degradation. The experiment results demonstrate the benefits of our adaptive fusion. Our analysis on the robustness of multimodal fusion will provide further insights for future research.

The Russian Language Textbook as a Multimodal Object

The school textbook has always been the most important tool for organizing the educational process. Many studies by Kazakhstani scholars have been devoted to the issue of the quality of Kazakhstani textbooks. However, most of the research primarily focuses on the verbal component of textbooks, while their visual design is often underestimated. This suggests that the school textbook is not considered a holistic multimodal object, each element of which (both verbal and non-verbal) is intended to contribute to developing students' multimodal literacy. Since the development of educational literature today is carried out not only by scholars but also by practicing educators, there is a need to study teachers' professional attitudes toward the multimodality of textbooks. The article employs a survey method to investigate teachers' evaluative perception of the visual design of Russian language textbooks for the 5th grade from the publishers “Mektep” and “AEO NIS”. The authors found out that teachers transfer their negative attitude towards the verbal component of the textbook to its non-verbal aspect, poorly understanding the didactic potential of the multimodality of the textbook, which hinders their successful work in shaping and developing students' multimodal literacy. In light of the insufficiently high ranking of Kazakhstani students in international studies such as PIRLS and PISA, the results obtained in the article acquire special relevance. Based on the conducted analysis, the authors propose recommendations for solving the identified problem.

MambaReID: Exploiting Vision Mamba for Multi-Modal Object Re-Identification.

Multi-modal object re-identification (ReID) is a challenging task that seeks to identify objects across different image modalities by leveraging their complementary information. Traditional CNN-based methods are constrained by limited receptive fields, whereas Transformer-based approaches are hindered by high computational demands and a lack of convolutional biases. To overcome these limitations, we propose a novel fusion framework named MambaReID, integrating the strengths of both architectures with the effective VMamba. Specifically, our MambaReID consists of three components: Three-Stage VMamba (TSV), Dense Mamba (DM), and Consistent VMamba Fusion (CVF). TSV efficiently captures global context information and local details with low computational complexity. DM enhances feature discriminability by fully integrating inter-modality information with shallow and deep features through dense connections. Additionally, with well-aligned multi-modal images, CVF provides more granular modal aggregation, thereby improving feature robustness. The MambaReID framework, with its innovative components, not only achieves superior performance in multi-modal object ReID tasks, but also does so with fewer parameters and lower computational costs. Our proposed MambaReID's effectiveness is validated by extensive experiments conducted on three multi-modal object ReID benchmarks.

MODIFICATION OF A QUANTUM-INSPIRED GENETIC ALGORITHM FOR NUMERICAL OPTIMIZATION USING QUDIT UNDER CONDITIONS OF SIMULATING QUANTUM DECOHERENCE

The genetic algorithm for numerical optimization (GA) of the metaheuristic class is a method for finding optimal solutions based on the biological principles of natural selection and variability. GA is characterized by high operating speed, resistance to noise in the data, low probability of hitting the local extremum of the multimodal objective function, as well as the simultaneous application of probabilistic and deterministic rules for generating search space points. An alternative to the classical GA is the quantum-inspired genetic algorithm for numerical optimization (QIGA), which has advantages that are unattainable for GA by using the concepts and principles of quantum computing. The article proposes a new approach to the implementation of a quantum-inspired genetic numerical optimization algorithm for searching for the global maximum of the objective function, based on modeling the functioning of the GA by simulating the execution of quantum calculations based on qudit in the conditions of the existence of quantum decoherence in the era of noisy medium-scale quantum algorithms. For this purpose, to carry out quantum operations of rotating the states of multilevel quantum systems, the paper presents a density matrix based on Heisenberg-Weyl operators as an analogue of the Bloch sphere for qudits. The simulation of quantum decoherence is interpreted from the point of view of the influence of extraneous noise emanating from the environment on the qudit and is presented as the use of a normal random variable with zero mathematical expectation and unit variance in quantum gates. At the same time, the work presents detailed pseudocodes of the functioning of both the most modified quantum-inspired genetic algorithm for numerical optimization and its individual operations. Testing is carried out by conducting computational experiments with the implementation of a modified algorithm on two-dimensional and multidimensional functions of test optimization problems, as well as when solving an applied optimization problem of planning hybrid flow production in the manufacturing industry based on financial costs and solving the problem of increasing forecasting accuracy based on compact extreme learning machines. The experimental results demonstrate the superiority of the new algorithm over QIGA and classical optimization algorithms in the accuracy of the solution, the speed of convergence with the target value of the global maximum and the execution time of the algorithm.

Subpopulations of neurons in the perirhinal cortex enable both modality-specific and modality-invariant recognition of objects.

The perirhinal cortex (PER) supports multimodal object recognition, but how multimodal information of objects is integrated within the PER remains unknown. Here, we recorded single units within the PER while rats performed a PER-dependent multimodal object-recognition task. In this task, audiovisual cues were presented simultaneously (multimodally) or separately (unimodally). We identified 2 types of object-selective neurons in the PER: crossmodal cells, showing constant firing patterns for an object irrespective of its modality, and unimodal cells, showing a preference for a specific modality. Unimodal cells further dissociated unimodal and multimodal versions of the object by modulating their firing rates according to the modality condition. A population-decoding analysis confirmed that the PER could perform both modality-invariant and modality-specific object decoding-the former for recognizing an object as the same in various conditions and the latter for remembering modality-specific experiences of the same object.

PRORETA 5 – building blocks for automated urban driving enhancing city road safety

Abstract In the joint research project PRORETA 5, building blocks for automated driving in urban areas have been developed, implemented, and tested. The developed blocks involve an object tracking for cars, bicycles, and pedestrians that feeds a multimodal object prediction which is able to predict the traffic participants’ most likely trajectories. Then, an anytime tree-based planning algorithm calculates the vehicle’s desired path. Finally, logic-based safety functions ensure a collision-free trajectory for the ego vehicle. The mentioned building blocks were integrated and tested in a prototype vehicle in urban scenarios. Furthermore, a novel general framework for specifying and testing traffic rule compliance has been developed. In this paper, the automated driving concept of PRORETA 5 is introduced and the developed methods are briefly explained.

Multimodal Features Alignment for Vision–Language Object Tracking

Vision–language tracking presents a crucial challenge in multimodal object tracking. Integrating language features and visual features can enhance target localization and improve the stability and accuracy of the tracking process. However, most existing fusion models in vision–language trackers simply concatenate visual and linguistic features without considering their semantic relationships. Such methods fail to distinguish the target’s appearance features from the background, particularly when the target changes dramatically. To address these limitations, we introduce an innovative technique known as multimodal features alignment (MFA) for vision–language tracking. In contrast to basic concatenation methods, our approach employs a factorized bilinear pooling method that conducts squeezing and expanding operations to create a unified feature representation from visual and linguistic features. Moreover, we integrate the co-attention mechanism twice to derive varied weights for the search region, ensuring that higher weights are placed on the aligned visual and linguistic features. Subsequently, the fused feature map with diverse distributed weights serves as the search region during the tracking phase, facilitating anchor-free grounding to predict the target’s location. Extensive experiments are conducted on multiple public datasets, and our proposed tracker obtains a success score of 0.654/0.553/0.447 and a precision score of 0.872/0.556/0.513 on OTB-LANG/LaSOT/TNL2K. These results are satisfying compared with those of recent state-of-the-art vision–language trackers.

Multimodal multi-objective hierarchical distributed consensus method for multimodal multi-objective economic dispatch of hierarchical distributed power systems

As the scale of power systems increases, the multi-layer distributed multi-objective consensus algorithm (MLDMOCA) can speed up computation and enhance privacy. However, the MLDMOCA needs to face: (1) the problems of error enlargement of hierarchy bringing deteriorating results; (2) over-dependence of certain weights on the choice of power balance adjustment factors. Moreover, multimodal multi-objective economic dispatch has not been performed for either small or large systems, resulting in poor diversity of economic scheduling solutions. Therefore, this study proposes a multimodal multi-objective hierarchical distributed consensus approach. The multimodal multi-objective hierarchical distributed consensus approach contains multimodal increments, and the proposed multimodal search method can obtain multimodal results. By changing the multimodal factor and multimodal critical threshold, the multimodal results and multimodal decision solutions of the proposed method are adjusted. In this study, the multimodal multi-objective hierarchical distributed consensus approach is simulated in IEEE 118-bus and IEEE 3059-bus cases. The simulation results show that the multimodal multi-objective hierarchical distributed consensus approach can achieve multiple multimodal solutions quickly and can adapt the power balance adjustment factor to almost all weights. Moreover, multiple linear-weighted multimodal objective values of the proposed approach are smaller than the objective value of the existing hierarchical distributed method.

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

AbstractPerception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

Deep-feature-based asymmetrical background-aware correlation filter for object tracking

Correlation filter-based video object-tracking algorithms have gained widespread attention due to their efficiency and excellent tracking performance. However, traditional correlation filtering tracking algorithms possess several limitations. (1) They extract image features only by using rule sampling, which ignores the shape information of the target, resulting in insufficient discriminative power of the features. (2) They focus only on the difference between the background and the object, ignoring the effects of more challenging intra-class interference. (3) They do not further evaluate the reliability of the optimal candidate samples, resulting in easy failure in occlusion scenarios. This paper proposes an asymmetric background-aware correlation filter method for object tracking with deep features to solve the above limitations. First, an asymmetrical background-aware sampling method based on the shape information of the object is proposed. This sampling method significantly differs from the symmetrical sampling method in the traditional correlation filter framework. By exploring the shape information of the object, improving the otherness between the background and object samples is easy, thus suppressing the intra-class interferences. Second, deep neural networks are introduced in the correlation filter framework to extract the deep object features and a spatio-temporal regularization factor is adaptively assigned to suppress the boundary effects, intra-class distractors and aberrance between frames. Finally, a multi-modal object pool is constructed to evaluate the optimal candidate sample in each frame. This template pool fully exploits object diversity and solves the tracking drift and failure caused by invalid appearance changes in scenes such as occlusion and vigorous motion scenarios. To validate the effectiveness of the proposed method, it was compared with the state-of-the-art methods on public datasets. The experimental results show that the tracking performance of the proposed method is competitive.

CrossFormer: Cross-guided attention for multi-modal object detection

Object detection is one of the essential tasks in a variety of real-world applications such as autonomous driving and robotics. In a real-world scenario, unfortunately, there are numerous challenges such as illumination changes, adverse weather conditions, and geographical changes, to name a few. To tackle the problem, we propose a novel multi-modal object detection model that is built upon a hierarchical transformer and cross-guidance between different modalities. The proposed hierarchical transformer consists of domain-specific feature extraction networks where intermediate features are connected by the proposed Cross-Guided Attention Module (CGAM) to enrich their representational power. Specifically, in the CGAM, one domain is regarded as a guide and the other is assigned to a base. After that, the cross-modal attention from the guide to the base is applied to the base feature. The CGAM works bidirectionally in parallel by exchanging roles between modalities to refine multi-modal features simultaneously. Experimental results on FLIR-aligned, LLVIP, and KAIST multispectral pedestrian datasets demonstrate that the proposed method is superior to previous multi-modal detection algorithms quantitatively and qualitatively.

Slime mold algorithm for topology optimization: metagratings inverse design

In this paper we discuss the use of a metaheuristic (MH) gradient-free optimization method, specifically the slime mold algorithm (SMA), combined with the topology optimization (TO) method to design metasurfaces using a spectral modal method. The motivation behind using a MH approach comes from the drawbacks associated with traditional gradient-based methods. Normally, gradient-based methods require the calculation of the electromagnetic (EM) field at certain nodes within the computation domain. However, in spectral modal methods, this is unnecessary since these methods can compute the EM response without the need for field component values. Second, optimizing metagratings often involves a multimodal objective function with multiple local minimums and gradient-based methods might struggle with finding the global optimum. So to overcome these drawbacks, we propose using a MH approach, specifically the slime mold algorithm (SMA). We apply SMA to a metasurface design, especially in the context of TO and spectral methods, which is relatively unexplored. By coupling both TO with SMA, we successfully design metagratings capable of deflecting incident waves into a desired transmission angle.

MTUW-GAN: A Multi-Teacher Knowledge Distillation Generative Adversarial Network for Underwater Image Enhancement

Underwater imagery is plagued by issues such as image blurring and color distortion, which significantly impede the detection and operational capabilities of underwater robots, specifically Autonomous Underwater Vehicles (AUVs). Previous approaches to image fusion or multi-scale feature fusion based on deep learning necessitated multi-branch image preprocessing prior to merging through fusion modules. However, these methods have intricate network structures and a high demand for computational resources, rendering them unsuitable for deployment on AUVs, which have limited resources at their disposal. To tackle these challenges, we propose a multi-teacher knowledge distillation GAN for underwater image enhancement (MTUW-GAN). Our approach entails multiple teacher networks instructing student networks simultaneously, enabling them to enhance color and detail in degraded images from various perspectives, thus achieving an image-fusion-level performance. Additionally, we employ middle layer channel distillation in conjunction with the attention mechanism to extract and transfer rich middle layer feature information from the teacher model to the student model. By eliminating multiplexed branching and fusion modules, our lightweight student model can directly generate enhanced underwater images through model compression. Furthermore, we introduce a multimodal objective enhancement function to refine the overall framework training, striking a balance between a low computational effort and high-quality image enhancement. Experimental results, obtained by comparing our method with existing approaches, demonstrate the clear advantages of our proposed method in terms of visual quality, model parameters, and real-time performance. Consequently, our method serves as an effective solution for real-time underwater image enhancement, specifically tailored for deployment on AUVs.

Multimodal Object Detection of UAV Remote Sensing Based on Joint Representation Optimization and Specific Information Enhancement

Multimodal Object Detection of UAV Remote Sensing Based on Joint Representation Optimization and Specific Information Enhancement

Weakly Aligned Feature Fusion for Multimodal Object Detection.

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.

An improved YOLOv7 network using RGB-D multi-modal feature fusion for tea shoots detection

Due to the increasing scarcity of tea pickers, the implementation of intelligent harvesting for premium tea is a crucial prerequisite for the sustainable development of the premium tea industry. The initial step towards achieving intelligent and precise harvesting is the accurate detection of tender shoots, which consist of one bud and one leaf. However, accurately identifying tea shoots poses a challenging visual task due to their small size, variable shapes, as well as similar colors and backgrounds. The existing model, based on RGB images, can only detect partial targets. To address this issue and further enhance the detection of tea buds, this study proposes the utilization of multi-modal features encompassing red, green, blue, and depth (RGB-D) for identification. In addition, a unidirectional complementary multi-modal fusion method is introduced to minimize the adverse effects caused by low-quality depth information. Firstly, an RGB-D dataset comprising high-quality tea leaves is constructed, and the samples are carefully calibrated. Subsequently, an enhanced end-to-end RGB-D multi-modal object detection network, referred to as YOLO-RGBDtea, is developed based on You Only Look Once version 7 (YOLOv7). This model incorporates a parallel lightweight depth image feature extraction backbone network and incorporates a self-attention mechanism to prioritize contextual information. Lastly, a cross-modal spatial attention fusion module (CSFM) is devised to collaboratively integrate depth features with RGB features in a unidirectional manner. The experimental results reveal that YOLO-RGBDtea achieves an AP50 of 91.12% when confronting complex outdoor tea shoots, exhibiting significant performance improvements compared to YOLOv7, especially in scenarios involving small targets, overlapping target groups, and highly overexposed images. Notably, the parameter increment in YOLO-RGBDtea compared to the original YOLOv7 model is merely 17.8%, and the additional components can be seamlessly transferred to other models. Overall, this study introduces a straightforward yet effective multi-modal fusion method that bears theoretical and practical significance in advancing the detection of high-quality tea shoots in complex outdoor environments.

Towards audiovisual scene analysis and object formation

The neural mechanisms underpinning auditory scene analysis and object formation have been of intense research interest in the past two decades. Fundamentally, however, we live in a multisensory environment. Even Cherry in his original paper posited that “lip reading” as a way for us to solve the cocktail party problem. Yet, how different aspects of visual cues (e.g., timing, linguistic information) help listeners follow conversation in a complex acoustic scene is still not well understood. In this talk, we present a theoretical framework to study audiovisual scene analysis that has been extrapolated from the unisensory object-based attention literature and posit the following questions: How do we define a multi-modal object? What are the predictions from unisensory object-based attention theory when we apply to the audiovisual domain? What are the conceptual models to test the different neural mechanisms that underpin audiovisual scene analysis? Answering these questions would move us closer to addressing the cocktail party problem in the real-world setting as well as help us create, de novo, audiovisual scenes that are more engaging in the augmented/virtual reality world.

Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection

RGB-Infrared multi-modal object detection utilizes diverse and complementary information, showing some advantages in intelligent transportation field. The main challenge of RGB-Infrared object detection is how to fuse the two modalities. The difficulty of fusion is reflected in two aspects: 1) large visual differences between modalities make it difficult to learn effective complementary features, 2) some misaligned RGB-Infrared images increase the difficulty of fusion. To this end, based on feature pyramid commonly used in object detection, we propose Multi-modal Feature Pyramid Transformer (MFPT) to fuse the two modalities. The proposed MFPT learns semantic and modal complementary information to enhance each modal features via intra-modal feature pyramid transformer and inter-modal feature pyramid transformer. The intra-modal feature pyramid transformer enables features to interact across space and scales, improving the semantic representations of features in each modality. The inter-modal feature pyramid transformer conducts feature interaction between modalities, enabling each modality to learn complementary features from other modalities. Meanwhile, the inter-modal feature pyramid transformer can also learn distance independent dependencies between modalities, which are not sensitive to misaligned images. Furthermore, a local attention mechanism is introduced within different windows into MFPT to achieve efficient correlation between regions of different scales or different modalities. Experimental results on two RGB-Infrared detection datasets demonstrate the proposed method is superior to state-of-the-art methods.

Parameter estimation in non-linear chemical processes: an opposite point-based differential evolution (OPDE) approach

Abstract In recent years, evolutionary algorithms have been gaining popularity for finding optimal solutions to non-linear multimodal problems encountered in many engineering disciplines. Differential evolution (DE), an evolutionary algorithm, is a novel optimization method capable of handling nondifferentiable, non-linear, and multimodal objective functions. DE is an efficient, effective, and robust evolutionary optimization method. Still, DE takes large computational time to optimize the computationally expensive objective functions. Therefore, an attempt to speed up DE is considered necessary. This paper introduces a modification to the original DE that enhances the convergence rate without compromising solution quality. The proposed opposite point-based differential evolution (OPDE) algorithm utilizes opposite point-based population initialization, in addition to random initialization. Such an improvement reduces computational effort. The OPDE has been applied to benchmark test functions and high-dimensional non-linear chemical engineering problems. The proposed method of population initialization accelerates the convergence speed of DE, as indicated by the results obtained using benchmark test functions and non-linear chemical engineering problems.

Modeling Suicidality with Multimodal Impulsivity Characterization in Participants with Mental Health Disorder.

Suicide is one of the leading causes of death across different age groups. The persistence of suicidal ideation and the progression of suicidal ideations to action could be related to impulsivity, the tendency to act on urges with low temporal latency, and little forethought. Quantifying impulsivity could thus help suicidality estimation and risk assessments in ideation-to-action suicidality frameworks. To model suicidality with impulsivity quantification, we obtained questionnaires, behavioral tests, heart rate variability (HRV), and resting state functional magnetic resonance imaging measurements from 34 participants with mood disorders. The participants were categorized into three suicidality groups based on their Mini-International Neuropsychiatric Interview: none, low, and moderate to severe. Questionnaire and HRV-based impulsivity measures were significantly different between the suicidality groups with higher subscales of impulsivity associated with higher suicidality. A multimodal system to characterize impulsivity objectively resulted in a classification accuracy of 96.77% in the three-class suicidality group prediction task. This study elucidates the relative sensitivity of various impulsivity measures in differentiating participants with suicidality and demonstrates suicidality prediction with high accuracy using a multimodal objective impulsivity characterization in participants with mood disorders.