Multimodal Deep Neural Network Research Articles

Automatic pavement crack detection using multimodal features fusion deep neural network

ABSTRACT The existing pavement crack detection algorithms are all dealing with detection and segmentation separately, ignoring the feature correlation between the bounding box coordinates and mask information, and there are still problems such as low crack detection accuracy, incomplete detection, and segmentation fracture in practical application. In view of the above situation, this paper took the detection bounding box coordinates and mask information as multimodal features of the same pavement crack area, and proposed a One-Stage MFFNet (Multimodal Feature Fusion Network), which improved the pavement crack detection accuracy and segmentation integrity obviously. And experimental results of different models were compared on self-collected datasets and two public datasets (CFD and CRACK500) respectively. Compared with Mask R-CNN, the average detection accuracy and average segmentation accuracy were improved by 2.6% and 4.7%, respectively. Compared with the optimised model of RDSNet, the detection accuracy and processing speed were improved by 1.8% and 2FPS, respectively. In addition, MFFNet had significantly improved the integrity of pavement crack segmentation results. The results showed that the proposed MFFNet model achieved the best detection and segmentation accuracy, and was an effective and high-precision pavement crack detection model.

Behaviour Detection and Recognition of College Basketball Players Based on Multimodal Sequence Matching and Deep Neural Networks.

This study fuses multimodal sequence matching with a deep neural network algorithm for college basketball player behavior detection and recognition to conduct in-depth research and analysis, analyzing the basic components of basketball technical action videos by studying the practical application of technical actions in professional games and teaching videos from self-published authors of short video platforms. The characteristics of the dataset are also analyzed through literature research related to the basketball action dataset. On the established basketball technical action dataset, combined with the SSD target detection algorithm, the video images with cropping of human motion regions to reduce the size of image frames, generating a basketball technical action dataset based on cropped frames, reduces the amount of network training and improves the efficiency of subsequent action recognition training. In this study, by analyzing the characteristics of basic camera motion, a univariate global motion model is proposed to introduce a quadratic term to accurately express the shaking transformation, while the horizontal and vertical motion are independently represented to reduce the model complexity. Comparative experimental results show that the proposed model achieves a good balance between complexity and accuracy of global motion representation. It is suitable for global motion modeling in behavior recognition applications, laying the foundation for global and local motion estimation. On this basis, the visual feature change pattern of the key area of the scene (basketball area) is combined with the group behavior recognition based on motion patterns and the success-failure classification based on key visual information to achieve basketball semantic event recognition. The experimental results at NCAA show that the fusion of global and local motion patterns can effectively improve group behavior recognition performance. The semantic event recognition algorithm combining motion patterns and video key visual information achieves the best performance.

An effective structure of multi-modal deep convolutional neural network with adaptive group teaching optimization

An effective structure of multi-modal deep convolutional neural network with adaptive group teaching optimization

Ad Creative Discontinuation Prediction with Multi-Modal Multi-Task Neural Survival Networks

Discontinuing ad creatives at an appropriate time is one of the most important ad operations that can have a significant impact on sales. Such operational support for ineffective ads has been less explored than that for effective ads. After pre-analyzing 1,000,000 real-world ad creatives, we found that there are two types of discontinuation: short-term (i.e., cut-out) and long-term (i.e., wear-out). In this paper, we propose a practical prediction framework for the discontinuation of ad creatives with a hazard function-based loss function inspired by survival analysis. Our framework predicts the discontinuations with a multi-modal deep neural network that takes as input the ad creative (e.g., text, categorical, image, numerical features). To improve the prediction performance for the two different types of discontinuations and for the ad creatives that contribute to sales, we introduce two new techniques: (1) a two-term estimation technique with multi-task learning and (2) a click-through rate-weighting technique for the loss function. We evaluated our framework using the large-scale ad creative dataset, including 10 billion scale impressions. In terms of the concordance index (short: 0.896, long: 0.939, and overall: 0.792), our framework achieved significantly better performance than the conventional method (0.531). Additionally, we confirmed that our framework (i) demonstrated the same degree of discontinuation effect as manual operations for short-term cases, and (ii) accurately predicted the ad discontinuation order, which is important for long-running ad creatives for long-term cases.

Utilization of Image/Force/Tactile Sensor Data for Object-Shape-Oriented Manipulation: Wiping Objects With Turning Back Motions and Occlusion

There has been an increasing demand for housework robots to handle various objects. It is, however, difficult to achieve object-shape-oriented tasks in conventional research owing to the requirement for dealing with multiple surfaces, invisible area, and occlusion; moreover, robots must perceive shapes and adjust movements even if they cannot be seen directly. Humans usually tackle questions by integrating several sensory information; inspired by this perception mechanism of humans, in this study, we considered the effective utilization of image/force/tactile data in constructing a multimodal deep neural networks (DNN) model for the shape of an object perception and motion generation. As an example, we constructed a robot to wipe around the outside of objects that are imitating light shades. The wiping motions include the moment when the hands of the robot must be away from the surface as well as the turning directions required to wipe the next surface, even though some parts of the surfaces, such as the backside or parts occluded by the arm of the robot, may not be seen directly. If DNN model uses continuous visual information, it is badly influenced by the occluded images. Hence, the best-performing DNN model is the one that uses an image of the initial time-step to approximately perceive the shape and size and then generate motions by integrating the perception and sense of tactile and force. We conclude that the effective approach to object-shape-oriented manipulation is to initially utilize image to outline the target shape and, thereafter, to use force and tactile to understand concrete features while performing tasks.

A zero-shot deep metric learning approach to Brain–Computer Interfaces for image retrieval

In this paper we propose a deep learning based approach for image retrieval using EEG. Our approach makes use of a multi-modal deep neural network based on metric learning, where the EEG signal from a user observing an image is mapped together with visual information extracted from the image. The inspiration behind this work is the vision of a system which allows the user to navigate their image catalogue just by thinking about the image they want to see. Thanks to our metric learning approach, the system is scalable in that it can operate with new images that have never been used in training, resulting in a zero-shot image retrieval system. This framework is tested in two different standard EEG image-viewing datasets, where we demonstrate state-of-the-art results in this complex scenario.

A Quantile Regression Random Forest-Based Short-Term Load Probabilistic Forecasting Method

In this paper, a novel short-term load forecasting method amalgamated with quantile regression random forest is proposed. Comprised with point forecasting, it is capable of quantifying the uncertainty of power load. Firstly, a bespoke 2D data preprocessing taking advantage of empirical mode decomposition (EMD) is presented. It can effectively assist subsequent point forecasting models to extract spatial features hidden in the 2D load matrix. Secondly, by exploiting multimodal deep neural networks (DNN), three short-term load point forecasting models are conceived. Furthermore, a tailor-made multimodal spatial–temporal feature extraction is proposed, which integrates spatial features, time information, load, and electricity price to obtain more covert features. Thirdly, relying on quantile regression random forest, the probabilistic forecasting method is proposed, which exploits the results from the above three short-term load point forecasting models. Lastly, the experimental results demonstrate that the proposed method outperforms its conventional counterparts.

CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets

During the past decade, social media platforms have been extensively used during a disaster for information dissemination by the affected community and humanitarian agencies. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build the deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed cross-attention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting the more relevant cross-domain features of text and image tweets. This study affirms that social media platforms become a rich source of multimodal data during a disaster. This data can be utilized to build automated tools for quick filtration of informative messages to assess the post-disaster needs of the affected community and provide timely help.

Sensor-Fusion for Smartphone Location Tracking Using Hybrid Multimodal Deep Neural Networks.

Many engineered approaches have been proposed over the years for solving the hard problem of performing indoor localization using smartphone sensors. However, specialising these solutions for difficult edge cases remains challenging. Here we propose an end-to-end hybrid multimodal deep neural network localization system, MM-Loc, relying on zero hand-engineered features, but learning automatically from data instead. This is achieved by using modality-specific neural networks to extract preliminary features from each sensing modality, which are then combined by cross-modality neural structures. We show that our choice of modality-specific neural architectures can estimate the location independently. But for better accuracy, a multimodal neural network that fuses the features of early modality-specific representations is a better proposition. Our proposed MM-Loc system is tested on cross-modality samples characterised by different sampling rate and data representation (inertial sensors, magnetic and WiFi signals), outperforming traditional approaches for location estimation. MM-Loc elegantly trains directly from data unlike conventional indoor positioning systems, which rely on human intuition.

Tool-Use Model to Reproduce the Goal Situations Considering Relationship Among Tools, Objects, Actions and Effects Using Multimodal Deep Neural Networks.

We propose a tool-use model that enables a robot to act toward a provided goal. It is important to consider features of the four factors; tools, objects actions, and effects at the same time because they are related to each other and one factor can influence the others. The tool-use model is constructed with deep neural networks (DNNs) using multimodal sensorimotor data; image, force, and joint angle information. To allow the robot to learn tool-use, we collect training data by controlling the robot to perform various object operations using several tools with multiple actions that leads different effects. Then the tool-use model is thereby trained and learns sensorimotor coordination and acquires relationships among tools, objects, actions and effects in its latent space. We can give the robot a task goal by providing an image showing the target placement and orientation of the object. Using the goal image with the tool-use model, the robot detects the features of tools and objects, and determines how to act to reproduce the target effects automatically. Then the robot generates actions adjusting to the real time situations even though the tools and objects are unknown and more complicated than trained ones.

On the Effect of Observed Subject Biases in Apparent Personality Analysis From Audio-Visual Signals

Personality perception is implicitly biased due to many subjective factors, such as cultural, social, contextual, gender, and appearance. Approaches developed for automatic personality perception are not expected to predict the real personality of the target but the personality external observers attributed to it. Hence, they have to deal with human bias, inherently transferred to the training data. However, bias analysis in personality computing is an almost unexplored area. In this article, we study different possible sources of bias affecting personality perception, including emotions from facial expressions, attractiveness, age, gender, and ethnicity, as well as their influence on prediction ability for apparent personality estimation. To this end, we propose a multimodal deep neural network that combines raw audio and visual information alongside predictions of attribute-specific models to regress apparent personality. We also analyze spatio-temporal aggregation schemes and the effect of different time intervals on first impressions. We base our study on the ChaLearn first impressions dataset, consisting of one-person conversational videos. Our model shows state-of-the-art results regressing apparent personality based on the Big-Five model. Furthermore, given the interpretability nature of our network design, we provide an incremental analysis on the impact of each possible source of bias on final network predictions.

Inferring Emotion from Large-scale Internet Voice Data: A Semi-supervised Curriculum Augmentation based Deep Learning Approach

Effective emotion inference from user queries helps to give a more personified response for Voice Dialogue Applications(VDAs). The tremendous amounts of VDA users bring in diverse emotion expressions. How to achieve a high emotion inferring performance from large-scale Internet Voice Data in VDAs? Traditionally, researches on speech emotion recognition are based on acted voice datasets, which have limited speakers but strong and clear emotion expressions. Inspired by this, in this paper, we propose a novel approach to leverage acted voice data with strong emotion expressions to enhance large-scale unlabeled internet voice data with diverse emotion expressions for emotion inferring. Specifically, we propose a novel semi-supervised multi-modal curriculum augmentation deep learning framework. First, to learn more general emotion cues, we adopt a curriculum learning based epoch-wise training strategy, which trains our model guided by strong and balanced emotion samples from acted voice data and sub-sequently leverages weak and unbalanced emotion samples from internet voice data.Second, to employ more diverse emotion expressions, we design a Multi-path Mix-match Multimodal Deep Neural Network(MMMD), which effectively learns feature representations for multiple modalities and trains labeled and unlabeled data in hybrid semi-supervised methods for superior generalization and robustness. Experiments on an internet voice dataset with 500,000 utterances show our method outperforms (+10.09% in terms of F1) several alternative baselines, while an acted corpus with 2,397 utterances contributes 4.35%. To further compare our method with state-of-the-art techniques in traditionally acted voice datasets, we also conduct experiments on public dataset IEMOCAP. The results reveal the effectiveness of the proposed approach.

Multimodal deep neural networks for attribute prediction and applications to e-commerce catalogs enhancement

Compiling and managing huge e-commerce catalogs is a hard and time-consuming task for a retailer. In particular, deriving standardized and structured descriptions from unstructured data modalities, such as texts and images, is crucial to the performance of search engines and the general organization of virtual store databases. In this paper, we propose methodologies and strategies based on Deep Learning classifiers to structure, update, and inspect large e-commerce catalogs. To this purpose, we exploit multimodal representations combining data from images and unstructured textual descriptions to identify relevant labels for e-commerce applications. Such modalities of data are employed to train deep neural network architectures, which are then able to automatically recognize attributes. Three classes of architecture were investigated: variations of the VGG architecture for recognition from images; architectures combining embedding, convolutional and recurrent layers for text recognition; and hybrid architectures that combine elements from each of the previous architectures. We also propose tools that allow the detection of insufficiently descriptive visual and textual data, which can be later manually improved; and automatic enhancement of attribute annotations through neural network predictions. Using a database that we collected through a Web Crawler from a large e-commerce site, we show in our experiments that hybrid architectures achieve a better result in the classification task by combining both types of data. Finally, we show results of a case study performed to demonstrate the potential of our strategy for insufficiently descriptive data detection. We conclude that the proposed tools are effective to rectify, enhance, and efficiently update e-commerce catalogs.

Multimodal GANs: Toward Crossmodal Hyperspectral–Multispectral Image Segmentation

This article addresses the problem of semantic segmentation with limited cross-modality data in large-scale urban scenes. Most prior works have attempted to address this issue by using multimodal deep neural networks (DNNs). However, their ability to effectively blending different properties across multimodalities and robustly learning representations from complex scenes remains limited, particularly in the absence of sufficient and well-annotated training images. This leads to a challenge related to cross-modality learning with multimodal DNNs. To this end, we introduce two novel plug-and-play units in the network: self-generative adversarial networks (GANs) module and mutual-GANs module, to learn perturbation-insensitive feature representations and to eliminate the gap between multimodalities, respectively, yielding more effective and robust information transfer. Furthermore, a patchwise progressive training strategy is devised to enable effective network learning with limited samples. We evaluate the proposed network on two multimodal (hyperspectral and multispectral) overhead image data sets and achieve a significant improvement in comparison with several state-of-the-art methods.

DeepLPI: a multimodal deep learning method for predicting the interactions between lncRNAs and protein isoforms

BackgroundLong non-coding RNAs (lncRNAs) regulate diverse biological processes via interactions with proteins. Since the experimental methods to identify these interactions are expensive and time-consuming, many computational methods have been proposed. Although these computational methods have achieved promising prediction performance, they neglect the fact that a gene may encode multiple protein isoforms and different isoforms of the same gene may interact differently with the same lncRNA.ResultsIn this study, we propose a novel method, DeepLPI, for predicting the interactions between lncRNAs and protein isoforms. Our method uses sequence and structure data to extract intrinsic features and expression data to extract topological features. To combine these different data, we adopt a hybrid framework by integrating a multimodal deep learning neural network and a conditional random field. To overcome the lack of known interactions between lncRNAs and protein isoforms, we apply a multiple instance learning (MIL) approach. In our experiment concerning the human lncRNA-protein interactions in the NPInter v3.0 database, DeepLPI improved the prediction performance by 4.7% in term of AUC and 5.9% in term of AUPRC over the state-of-the-art methods. Our further correlation analyses between interactive lncRNAs and protein isoforms also illustrated that their co-expression information helped predict the interactions. Finally, we give some examples where DeepLPI was able to outperform the other methods in predicting mouse lncRNA-protein interactions and novel human lncRNA-protein interactions.ConclusionOur results demonstrated that the use of isoforms and MIL contributed significantly to the improvement of performance in predicting lncRNA and protein interactions. We believe that such an approach would find more applications in predicting other functional roles of RNAs and proteins.

Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning.

A lot of research has been done on the detection of mental workload (MWL) using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of measurement modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify MWL. The goal of this research was to classify perceived mental workload (PMWL) using a deep neural network (DNN) that flexibly makes use of multiple modalities, in order to allow for feature sharing between modalities. To achieve this goal, an experiment was conducted in which MWL was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had 1 h to solve as many puzzles as they could. Between puzzles, they gave a difficulty rating between 1 and 7, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms and eye movements were collected simultaneously using LabStreamingLayer (LSL). Marker information from the puzzles was also streamed on LSL. We designed and evaluated a novel intermediate fusion multimodal DNN for the classification of PMWL using the aforementioned four modalities. Two main criteria that guided the design and implementation of our DNN are modularity and generalisability. We were able to classify PMWL within-level accurate (0.985 levels) on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available.

LSTM and Filter Based Comparison Analysis for Indoor Global Localization in UAVs

Deep learning (DL) based localization and Simultaneous Localization and Mapping (SLAM) has recently gained considerable attention demonstrating remarkable results. Instead of constructing hand-crafted algorithms through geometric theories, DL based solutions provide a data-driven solution to the problem. Taking advantage of large amounts of training data and computing capacity, these approaches are increasingly developing into a new field that offers accurate and robust localization systems. In this work, the problem of global localization for unmanned aerial vehicles (UAVs) is analyzed by proposing a sequential, end-to-end, and multimodal deep neural network based monocular visual-inertial localization framework. More specifically, the proposed neural network architecture is three-fold; a visual feature extractor convNet network, a small IMU integrator bi-directional long short-term memory (LSTM), and a global pose regressor bi-directional LSTM network for pose estimation. In addition, by fusing the traditional IMU filtering methods instead of LSTM with the convNet, a more time-efficient deep pose estimation framework is presented. It is worth pointing out that the focus in this study is to evaluate the precision and efficiency of visual-inertial (VI) based localization approaches concerning indoor scenarios. The proposed deep global localization is compared with the various state-of-the-art algorithms on indoor UAV datasets, simulation environments and real-world drone experiments in terms of accuracy and time-efficiency. In addition, the comparison of IMU-LSTM and IMU-Filter based pose estimators is also provided by a detailed analysis. Experimental results show that the proposed filter-based approach combined with a DL approach has promising performance in terms of accuracy and time efficiency in indoor localization of UAVs.

보증데이터를 이용한 텍스트 마이닝 기반 고장 현상 예측 연구

Purpose: This paper uses unstructured warranty data, such as customer complaints and repair information, extracted through text mining and proposes a methodology for identifying and predicting automotive failure.BRMethods: The Word2Vec and CountVectorizer algorithms were used to determine the similarity between and frequency of the vectorized meanings of the words. A word embedding model was constructed on the bases of these distinct features of the words, and multimodal-deep neural network (DNN) modeling was performed to derive the prediction results of the failure phenomenon. A comparative analysis of the performance metrics of different combinations of the models was performed.BRResults: The model using both the CountVectorizer algorithm for extracting features within similar text clusters and the multimodal-DNN for training the data exhibited the best performance.BRConclusion: This study shows the effects of different data structures and feature extraction algorithms on failure prediction performance. The developed model improves the prediction accuracy by applying the relevant feature extraction algorithm and text classification learning model.

Distributed Training and Inference of Deep Learning Models for Multi-Modal Land Cover Classification

Deep Neural Networks (DNNs) have established themselves as a fundamental tool in numerous computational modeling applications, overcoming the challenge of defining use-case-specific feature extraction processing by incorporating this stage into unified end-to-end trainable models. Despite their capabilities in modeling, training large-scale DNN models is a very computation-intensive task that most single machines are often incapable of accomplishing. To address this issue, different parallelization schemes were proposed. Nevertheless, network overheads as well as optimal resource allocation pose as major challenges, since network communication is generally slower than intra-machine communication while some layers are more computationally expensive than others. In this work, we consider a novel multimodal DNN based on the Convolutional Neural Network architecture and explore several different ways to optimize its performance when training is executed on an Apache Spark Cluster. We evaluate the performance of different architectures via the metrics of network traffic and processing power, considering the case of land cover classification from remote sensing observations. Furthermore, we compare our architectures with an identical DNN architecture modeled after a data parallelization approach by using the metrics of classification accuracy and inference execution time. The experiments show that the way a model is parallelized has tremendous effect on resource allocation and hyperparameter tuning can reduce network overheads. Experimental results also demonstrate that proposed model parallelization schemes achieve more efficient resource use and more accurate predictions compared to data parallelization approaches.

MULTI-MODAL DEEP LEARNING WITH SENTINEL-3 OBSERVATIONS FOR THE DETECTION OF OCEANIC INTERNAL WAVES

Abstract. The observation of waves that propagate along density interfaces inside the ocean poses a significant challenge, as their visible surface signatures are much lower compared to their internal amplitudes. However, monitoring internal waves is important as they redistribute large amounts of energy, play a role in mixing and vertical heat transfer, and modify water and nutrient transports. Although satellite observations would allow global monitoring of internal waves at constant time intervals, their automatic detection is challenging: In optical images, internal waves are hardly visible and can be obscured by clouds, whereas radar data have limitations in coastal regions and their spatial coverage is not perfect. Furthermore, the occurrence of internal waves can be confused with other ocean phenomena. In this work, we present an automated detection framework for internal waves based on multiple data sources in order to compensate for the shortcoming of single data sources. In our application, we use Ocean and Land Color Imager and Synthetic Aperture Radar Altimeter data. Our contributions are (1) we develop a multi-modal deep neural network SONet with multi-streams and late fusion, which performs a classification on the basis of training with both modalities, and (2) we establish a method to deal with missing modalities. Experiments in the Amazon Shelf region show SONet achieves adequate results when both modalities are available, but also when only a single modality is available. By exploiting correlations between the modalities, SONet classifies OLCI images off the SRAL ground track better than uni-modal network ONet, which describes a great advantage of our multi-modal network.