Multimodal Objective Research Articles

Cross-Modal Object Detection Via UAV

UAV-based object detection aims at locating and recognizing targets in aerial images, which is widely applied to traffic surveillance, disaster rescue and anomaly monitoring. However, due to expensive sensors and complicated architectures, it is unrealistic to deploy precise but heavy multi-modal object detectors into UAV nodes. To get over the dilemma, inspired by model compression and cross-modal signal processing techniques, this paper proposes a cross-modal knowledge distillation (CKD) enabled object detection paradigm, which achieves comparable detection performance with multi-modal techniques, yet requires less computational resource. On the one hand, in order to avoid transferring redundant feature knowledge, we design a Selective Feature Imitation (SFI) to selectively shorten the distance between cross-modal features according to their types. On the other hand, in order to transfer the most valuable prediction knowledge, we design an Adaptive Prediction Imitation (API). It evaluates the quality of prediction knowledge, and then adaptively adjusts the distillation intensity for cross-modal prediction. Extensive experiments on the DroneVehicle dataset have shown the performance improvement of the proposed scheme.

Learning from the global view: Supervised contrastive learning of multimodal representation

The development of technology enables the availability of abundant multimodal data, which can be utilized in many representation learning tasks. However, most methods ignore the rich modality correlation information stored in each multimodal object and fail to fully exploit the potential of multimodal data. To address the aforementioned issue, cross-modal contrastive learning methods are proposed to learn the similarity score of each modality pair in a self-/weakly-supervised manner and improve the model robustness. Though effective, contrastive learning based on unimodal representations might be, in some cases, inaccurate as unimodal representations fail to reveal the global information of multimodal objects. To this end, we propose a contrastive learning pipeline based on multimodal representations to learn from the global view, and devise multiple techniques to generate negative and positive samples for each anchor. To generate positive samples, we apply the mix-up operation to mix two multimodal representations of different objects that have the maximal label similarity. Moreover, we devise a permutation-invariant fusion mechanism to define the positive samples by permuting the input order of modalities for fusion and sampling various contrastive fusion networks. In this way, we force the multimodal representation to be invariant regarding the order of modalities and the structures of fusion networks, so that the model can capture high-level semantic information of multimodal objects. To define negative samples, for each modality, we randomly replace the unimodal representation with that from another dissimilar object when synthesizing the multimodal representation. By this means, the model is led to capture the high-level concurrence information and correspondence relationship between modalities within each object. We also directly define the multimodal representation from another object as a negative sample, where the chosen object shares the minimal label similarity with the anchor. The label information is leveraged in the proposed framework to learn a more discriminative multimodal embedding space for downstream tasks. Extensive experiments demonstrate that our method outperforms previous state-of-the-art baselines on the tasks of multimodal sentiment analysis and humor detection.

Lightweight Transformer for Multi-Modal Object Detection (Student Abstract)

It has become a common practice for many perceptual systems to integrate information from multiple sensors to improve the accuracy of object detection. For example, autonomous vehicles use visible light, and infrared (IR) information to ensure that the car can cope with complex weather conditions. However, the accuracy of the algorithm is usually a trade-off between the computational complexity and memory consumption. In this study, we evaluate the performance and complexity of different fusion operators in multi-modal object detection tasks. On top of that, a Poolformer-based fusion operator (PoolFuser) is proposed to enhance the accuracy of detecting targets without compromising the efficiency of the detection framework.

Low saliency crack detection based on improved multimodal object detection network: an example of wind turbine blade inner surface

Low saliency crack detection based on improved multimodal object detection network: an example of wind turbine blade inner surface

Exploring fusion strategies for accurate RGBT visual object tracking

We address the problem of multi-modal object tracking in video and explore various options available for fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities, including pixel-level, feature-level and decision-level fusion. Specifically, in contrast to the existing approaches, we propose and develop the paradigm for combining multi-modal information for image fusion at pixel level. At the feature level, two different kinds of fusion strategies are investigated for completeness, i.e., the attention-based online fusion strategy and the offline-trained fusion block. At the decision level, a novel fusion strategy is put forward, inspired by the success of the simple averaging configuration which has shown so much promise. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of the proposed decision fusion method produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). A comprehensive comparison of the innovative pixel and feature-level fusion strategies with the proposed decision-level fusion method highlights the advantages fusing multimodal information at the decision score level. Extensive experimental results on five challenging datasets, i.e., GTOT, VOT-RGBT2019, RGBT234, LasHeR and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. The Code is available at https://github.com/Zhangyong-Tang/DFAT.

멀티모달 오브젝트를 통한 협업 서비스 목적 기반 효율적인 XR 콘텐트 저작 방법

확장현실(Extended Reality, XR) 기술은 가상 환경과 디지털 정보를 현실 세계에 접목시켜 물리적 한계를 뛰어넘는 경험의 확장을 가능하게 한다. Covid-19 팬데믹으로 인한 사회적 거리두기는 비대면 협업의 요구를 급증하게 했고 제한적 의사소통과 시공간의 제약을 가지는 기존의 2D 온라인 협업 서비스의 한계를 XR 기술을 통해 극복할 수 있다. 이에 따라 시각적 사실감 증대를 위한 기술 개발을 중심으로 XR 분야 연구는 렌더링 속도, 정밀도 등 성능향상에 집중되어 있었다. 하지만, 사용자에게 향상된 효율적 협업 경험 제공을 위해서는 고품질 서비스를 위한 효율적인 콘텐트 저작이 중요하다. 특히 사용자의 감각 경험을 다중화하여 개별 XR 기술의 제한된 성능을 넘어서는 실감도를 제공할 수 있다. 본 연구에서는 XR 콘텐트 저작을 위한 가상의 협업 공간, 아바타를 위한 공존감 구현, 그리고 멀티모달 오브젝트의 구성을 통해 다자간 협업 서비스를 위한 콘텐트 저작 방법을 제시한다. 협업 서비스의 목적을 기반으로 기능 요소와 대상 요소를 구분하여 저작하는 모델을 통해 서비스의 사용자가 손쉽게 콘텐트의 생산자가 될 수 있는 효율적인 XR 콘텐트 저작 가능성을 확인할 수 있다.

A Fast Frequency Sweep approach for performance-based optimization of earthquake-resistant irregular large-scale buildings

Performance-Based Design (PBD) optimization of buildings subjected to seismic excitations has increasingly been applied to regular small-scale structural systems to minimize initial or life-cycle costs. Nonetheless, a comprehensive bibliographic survey reveals that studies dealing with irregular buildings are scarce, although they can present higher seismic vulnerability than regular buildings. Specifically, the PBD optimization of irregular buildings represented by large-scale finite element (FE) models, and considering dozens of discrete design variables, cannot be found to the best of the authors’ knowledge. Hence, this paper presents an original framework for the PBD optimization of irregular buildings subjected to seismic excitations, handling thousands of DOFs and dozens of discrete design variables. To relieve the computational burden, enabling the problem to be solved, a Model Order Reduction (MOR) strategy based on the Well-Conditioned Asymptotic Waveform Evaluation (WCAWE) approach is combined with a hybrid optimization algorithm, which integrates a local optimizer with a restart strategy. The former is employed to speed up the frequency-domain response, while the latter allows for tackling a non-convex and multimodal objective function with many discrete variables. The procedure is adaptatively devised to operate in a robust and fully black-box approach. The application case consists of a stiffness in-plane irregular H-shape building with 16,452 active DOFs and 34 discrete design variables. The proposed scheme largely reduces the computational burden, leading to speed-up factors (SUF) up to 19 times compared to direct frequency-domain solutions.

An Accurate UAV 3-D Path Planning Method for Disaster Emergency Response Based on an Improved Multiobjective Swarm Intelligence Algorithm.

Planning a practical three-dimensional (3-D) flight path for unmanned aerial vehicles (UAVs) is a key challenge for the follow-up management and decision making in disaster emergency response. The ideal flight path is expected to balance the total flight path length and the terrain threat, to shorten the flight time and reduce the possibility of collision. However, in the traditional methods, the tradeoff between these concerns is difficult to achieve, and practical constraints are lacking in the optimized objective functions, which leads to inaccurate modeling. In addition, the traditional methods based on gradient optimization lack an accurate optimization capability in the complex multimodal objective space, resulting in a nonoptimal path. Thus, in this article, an accurate UAV 3-D path planning approach in accordance with an enhanced multiobjective swarm intelligence algorithm is proposed (APPMS). In the APPMS method, the path planning mission is converted into a multiobjective optimization task with multiple constraints, and the objectives based on the total flight path length and degree of terrain threat are simultaneously optimized. In addition, to obtain the optimal UAV 3-D flight path, an accurate swarm intelligence search approach based on improved ant colony optimization is introduced, which can improve the global and local search capabilities by using the preferred search direction and random neighborhood search mechanism. The effectiveness of the proposed APPMS method was demonstrated in three groups of simulated experiments with different degrees of terrain threat, and a real-data experiment with 3-D terrain data from an actual emergency situation.

Revisiting Analogical Reasoning in Computing Education: Use of Similarities Between Robot Programming Tasks in Debugging

Analogical reasoning is considered to be a critical cognitive skill in programming. However, it has been rarely studied in a block-based programming context, especially involving both virtual and physical objects. In this multi-case study, we examined how novice programming learners majoring in early childhood education used analogical reasoning while debugging block code to make a robot perform properly. Screen recordings, scaffolding entries, reflections, and block code were analyzed. The cross-case analysis suggested multimodal objects enabled the novice programming learners to identify and use structural relations. The use of a robot eased the verification process by enabling them to test their analogies immediately after the analogy application. Noticing similar functional analogies led to noticing similarities in the relation between block code as well as between block code and the robot, guiding to locate bugs. Implications and directions for future educational computing research are discussed.

Visuo-haptic object perception for robots: an overview

The object perception capabilities of humans are impressive, and this becomes even more evident when trying to develop solutions with a similar proficiency in autonomous robots. While there have been notable advancements in the technologies for artificial vision and touch, the effective integration of these two sensory modalities in robotic applications still needs to be improved, and several open challenges exist. Taking inspiration from how humans combine visual and haptic perception to perceive object properties and drive the execution of manual tasks, this article summarises the current state of the art of visuo-haptic object perception in robots. Firstly, the biological basis of human multimodal object perception is outlined. Then, the latest advances in sensing technologies and data collection strategies for robots are discussed. Next, an overview of the main computational techniques is presented, highlighting the main challenges of multimodal machine learning and presenting a few representative articles in the areas of robotic object recognition, peripersonal space representation and manipulation. Finally, informed by the latest advancements and open challenges, this article outlines promising new research directions.

Empirical analysis of multiple modalities for emotion recognition using convolutional neural network

Emotion recognition is a dynamic process that focuses on a person's emotional state, which implies that the emotions associated with each individual's activities differ. Humans, in general, express their emotions in a variety of ways. It is critical to accurately evaluate these emotions in order to promote effective communication. However, emotional detection in our daily lives is vital for social engagement because emotions influence human behavior. In this research, we use a variety of machine learning methods for sensing and analysing EEG and audio signals in an effort to establish emotions as a multimodal object. The analysis depicts a multi-sensory emotion identification system based on the integration of multi-channel information from speech and EEG signals. The suggested system's purpose is to increase the accuracy of emotion identification during human-computer interaction. The combination of EEG and the speech-based emotion recognition method can be utilised to distinguish and perceive various conditions of feelings more precisely. A challenging issue in today's world is how to give a machine emotional intelligence. However, when studying human emotions, researchers do not combine many methods into a single framework. As emotion impacts nearly every modality, humans can use this information to predict emotional states in human behavior. A brain-computer interface (BCI) can be used to implement a model of the brain in conjunction with a model of voice signals to normalise emotional responses. To improve the accuracy of emotion detection, this processing method has to be investigated as a fusion model. In this paper, we accept this challenge and analyse multiple modalities for emotion recognition using a convolutional neural network by taking input signals from EEG and audio. Furthermore, both signal features were extracted and fused using the principal component analysis method and the Grey Wolf optimization algorithm, which were applied to select combined features. Moreover, the convolutional neural network is to be trained, and evaluation is done in terms of accuracy. The result shows that the proposed technique provides better accuracy (94.44%) in comparison to the existing one.

The registration of visible and thermal images through multi-objective optimization

Multimodal imaging with visible and thermal sensors attracts much attention due to its robustness under challenging illumination conditions. Due to spectral differences, the visible and thermal images are typically misaligned, where image registration is necessary before high-level vision tasks such as information fusion, multimodal object detection, etc. Contemporary registration methods measure either an angular distance or a linear distance as an objective function to solve affine parameters between visible and thermal images. It is noticeable that both distances have unique advantages in image alignment, which suggests that integrating these measurements may enhance the registration quality. Thus, this paper proposes a new algorithm, namely multi-objective optimization-based image registration (MOIR), to align visible and thermal images. The MOIR comprises a normalized gradient measurement (NGM) that quantifies and normalizes differences of image gradient in both angular and linear aspects, and a regularized stochastic gradient descent (RSGD) that smoothly solves affine parameters. Both quantitative and qualitative results validate the effectiveness and robustness of MOIR for the registration of visible and thermal images in both controlled and real-world scenarios. Some animated examples can be found in https://github.com/jb2849/MOIR.

Perception of multimodal objects in NLP through computer vision

This project is based on voice interaction and object detecting properties. It will allow the users to do voice interaction with the artificial intelligence and it will reply with the system voice. That is how users will use their voice to command as a trigger to find out the category of any object by showing it using the camera module. At first, the user will show an object with the help of a camera and ask for identifying it in the system. The object detection system then captures a frame from the camera and predicts through the structure to identify which class the object belongs to by extracting the feature from there. The process of this application is to search the database to match the structural data to find out the exact category of the object. When this system approximately matches with the information of a category then the application will suggest the category for the object by mentioning the category name through voice. This application can also give some basic information by asking for it. Our general-purpose approach can be effective in interpreting the structure and properties of objects in different networks through natural language processing.

Auditory cortical connectivity in humans.

To understand auditory cortical processing, the effective connectivity between 15 auditory cortical regions and 360 cortical regions was measured in 171 Human Connectome Project participants, and complemented with functional connectivity and diffusion tractography. 1. A hierarchy of auditory cortical processing was identified from Core regions (including A1) to Belt regions LBelt, MBelt, and 52; then to PBelt; and then to HCP A4. 2. A4 has connectivity to anterior temporal lobe TA2, and to HCP A5, which connects to dorsal-bank superior temporal sulcus (STS) regions STGa, STSda, and STSdp. These STS regions also receive visual inputs about moving faces and objects, which are combined with auditory information to help implement multimodal object identification, such as who is speaking, and what is being said. Consistent with this being a "what" ventral auditory stream, these STS regions then have effective connectivity to TPOJ1, STV, PSL, TGv, TGd, and PGi, which are language-related semantic regions connecting to Broca's area, especially BA45. 3. A4 and A5 also have effective connectivity to MT and MST, which connect to superior parietal regions forming a dorsal auditory "where" stream involved in actions in space. Connections of PBelt, A4, and A5 with BA44 may form a language-related dorsal stream.

The goat search algorithms.

This article develops an evolutionary nature inspired algorithm based on the social behavior of the goat, a pet of a farmer in a village life. In village life, we generally see the shepherds keep their goats free/untie from collar thread for grazing in the early morning and receives them at the end of the day when they come back into the home with their own efforts. But some day the goats did not come back in due time because of overfeeding of grass causing unable to move any more after meeting their grasp and began to get rest there. The shepherd feels more tempted and began to search for his/her goat. After untie, the goat began to graze herself through the walk on the path of the cultivated land and bank of the village ponds. The search process is going on through that path until it is not finally got. To characterize this problem some definitions like false walk, uniform and non-uniform steps, goat's jump, periodic walk and goodness of fit for various walk functions have been discussed here rigorously. Inspiring from this fact novel metaheuristic algorithms along with pseudocode and hardware specification have been discussed to optimize a benchmark multi-modal objective function having some singularity zones explicitly. Numerical results have been compared with some of the existing state- of -arts under 95% confidence intervals. Also, graphical illustrations are performed to validate the proposed approach. Finally, a conclusion is made followed by scope of future work.

Forming 3-dimensional multimodal object representations relies on integrative coding

Forming 3-dimensional multimodal object representations relies on integrative coding

Fusion of tactile and visual information in deep learning models for object recognition

Humans use multimodal sensory information to understand the physical properties of their environment. Intelligent decision-making systems such as the ones used in robotic applications could also utilize the fusion of multimodal information to improve their performance and reliability. In recent years, machine learning and deep learning methods are used at the heart of such intelligent systems. Developing visuo-tactile models is a challenging task due to various problems such as performance, limited datasets, reliability, and computational efficiency. In this research, we propose four efficient models based on dynamic neural network architectures for unimodal and multimodal object recognition. For unimodal object recognition, TactileNet and VisionNet are proposed. For multimodal object recognition, the FusionNet-A and the FusionNet-B are designed to implement early and late fusion strategies, respectively. The proposed models have a flexible structure and are able to change at the train or test phase to accommodate the amount of available information. Model confidence calibration is employed to enhance the reliability and generalization of the models. The proposed models are evaluated on MIT CSAIL large-scale multimodal dataset. Our results demonstrate accurate performance in both unimodal and multimodal scenarios. It has been illustrated that by using different fusion strategies and augmenting the tactile-based models with visual information, the top-1 error rate of the single-frame tactile model was reduced by 78% and the mean average precision was increased by 2.19 times. Although the focus has been on the fusion of tactile and visual modalities, the proposed design methodology can be generalized to include more modalities.

Image classification and identification for rice leaf diseases based on improved WOACW_SimpleNet.

In view of the problem that manual selection of hyperparameters may lead to low performance and large consumption of manpower cost of the convolutional neural network (CNN), this paper proposes a nonlinear convergence factor and weight cooperative self-mapping chaos optimization algorithm (WOACW) to optimize the hyperparameters in the identification and classification model of rice leaf disease images, such as learning rate, training batch size, convolution kernel size and convolution kernel number. Firstly, the opposition-based learning is added to the whale population initialization with improving the diversity of population initialization. Then the algorithm improves the convergence factor, increases the weight coefficient, and calculates the self-mapping chaos. It makes the algorithm have a strong ability to find optimization in the early stage of iteration and fast convergence rate. And disturbance is carried out to avoid falling into local optimal solution in the late stage of iteration. Next, a polynomial mutation operator is introduced to correct the current optimal solution with a small probability, so that a better solution can be obtained in each iteration, thereby enhancing the optimization performance of the multimodal objective function. Finally, eight optimized performance benchmark functions are selected to evaluate the performance of the algorithm, the experiment results show that the proposed WOACW outperforms than 5 other common improved whale optimization algorithms. The WOACW_SimpleNet is used to identify rice leaf diseases (rice blast, bacterial leaf blight, brown spot disease, sheath blight and tungro disease), and the experiment results show that the identification average recognition accuracy rate reaches 99.35%, and the F1-score reaches 99.36%.

Multimodal object tracking by exploiting appearance and class information

In this work, we study the method exploiting natural language network to improve tracking performance. We propose a novel architecture which can combine class and visual information presented in tracking. To this end, we introduce a multimodal feature association network, allowing us to correlate the target class with its appearance during training and aid the localization of the target during inference. Specifically, we first utilize an appearance model to extract the target visual features, from which we obtain appearance cues, for instance shape and color. In order to employ target class information, we design a learned lightweight embedding network to embed the target class into a feature representation. The association network of our architecture contains a multimodal fusion module and a predictor module. The fusion module is used to combine features from class and appearance, yielding multimodal features with more expressive representations for the subsequent module. The predictor module is used to determine the target location in the current frame, from which we associate the class to the appearance. The class embedding module thus can learn appearance cues by exploiting the back-propagation functionality. To verify the abilities of our method, we select the official training and test splits of the LaSOT with annotated images and classes to perform experiments. In particular, we analyze the imbalance in the samples and employ a class validator discriminator to alleviate this problem. Extensive experimental results on LaSOT, UAV20L and UAV123@10fps demonstrate our method achieves competitive results while maintaining a considerable real-time speed.

Modifications of Flower Pollination, Teacher-Learner and Firefly Algorithms for Solving Multiextremal Optimization Problems

The article offers a possible treatment for the numerical research of tasks which require searching for an absolute optimum. This approach is established by employing both globalized nature-inspired methods as well as local descent methods for exploration and exploitation. Three hybrid nonconvex minimization algorithms are developed and implemented. Modifications of flower pollination, teacher-learner, and firefly algorithms are used as nature-inspired methods for global searching. The modified trust region method based on the main diagonal approximation of the Hessian matrix is applied for local refinement. We have performed the numerical comparison of variants of the realized approach employing a representative collection of multimodal objective functions. The implemented nonconvex optimization methods have been used to solve the applied problems. These tasks utilize an optimization of the low-energy metal Sutton-Chen clusters potentials with a very large number of atoms and the parametric identification of the nonlinear dynamic model. The results of this research confirms the performance of the suggested algorithms.