Early Fusion Schemes Research Articles

Robust multi-view graph clustering in latent energy-preserving embedding space

Due to the powerful capacity of exploring complementary and consistent information by generating a consensus affinity graph from multi-view data, multi-view graph clustering (MVGC) methods have attracted intensive attention. However, multi-view data is usually existed in high-dimensional space, where redundant and irrelevant features may result in the curse of dimensionality . Moreover, original data often mix with noise and outliers that will destroy the underlying clustering structure , such that unreliable and inaccurate affinity graphs will be generated. To alleviate the aforementioned problems, we propose a novel multi-view latent energy-preserving embedding (MLEE) method, which seamlessly integrates the clean embedding space learning and consensus affinity graph learning into a unified objective function. Concretely, for each view, we first learn the low-dimensional yet clean data by proposing a full-energy projection and recovering method. This can well reduce the redundancy and interference in the data. Furthermore, by leveraging adaptive neighbors graph learning (ANGL), the local manifold structure of the clean embedding data can be implicitly preserved. To integrate the complementary and consistent information of different views, an early-fusion scheme is proposed to directly yield a consensus graph for clustering purpose. Experiments on six benchmark datasets demonstrate that our method achieves state-of-the-art clustering performance.

A Multimodal Deep Learning-Based Fault Detection Model for a Plastic Injection Molding Process

The authors of this work propose a deep learning-based fault detection model that can be implemented in the field of plastic injection molding. Compared to conventional approaches to fault detection in this domain, recent deep learning approaches prove useful for on-site problems involving complex underlying dynamics with a large number of variables. In addition, the advent of advanced sensors that generate data types in multiple modalities prompts the need for multimodal learning with deep neural networks to detect faults. This process is able to facilitate information from various modalities in an end-to-end learning fashion. The proposed deep learning-based approach opts for an early fusion scheme, in which the low-level feature representations of modalities are combined. A case study involving real-world data, obtained from a car parts company and related to a car window side molding process, validates that the proposed model outperforms late fusion methods and conventional models in solving the problem.

Development of a deep multimodal hedonic recognition database for oral stimuli

The emotional reactions that certain products evoke (like or dislike) play a key role in the user acceptance or preference toward these products and influence the choice behaviors. Facial expressions constitute a primary channel to express and communicate emotions. In particular, the facial reactions evoked by certain stimuli are generated subconsciously, and they can be easily captured by video recording. However, it is challenging to quantify and objectively describe facial expressions. Until now, most implicit measurements based on facial expression analyses have been implemented considering “basic emotions”, which cannot directly represent like/dislike. How can we identify the affective state of consumers directly by using facial expression analysis technology? In this study, hedonic rating categories are defined based on the hedonic scale, and automatic facial expression analysis technology is applied to evaluate the suitability of the defined categories. Moreover, a visual multimodal facial database (RGB, depth, and infrared images) for the hedonic recognition of Asian individuals is constructed. A hybrid deep learning framework is employed to exploit the spatiotemporal information of the collected data to realize hedonic recognition. Early and late fusion strategies are used to investigate the suitability of the individual modalities and their complementarity. The experimental results indicate that the late fusion of the three modalities yielded the highest recognition accuracy compared to those obtained using individual modalities and the early fusion scheme. These results demonstrate the effectiveness of fusing different visual facial data and indicate that the use of multimodality-based facial expression analyses could enhance the accuracy of hedonic recognition.

NeuralIO: Indoor?Outdoor Detection via Multimodal Sensor Data Fusion on Smartphones

The indoor–outdoor (IO) status of mobile devices is fundamental information for various smart city applications. In this paper, we present NeuralIO, a neural-network-based method for dealing with the IO detection problem for smartphones. Multimodal data from various sensors on a smartphone are fused through neural network models to determine the IO status. A data set containing more than one million labeled samples is then constructed. We test the performance of an early fusion scheme in various settings. NeuralIO achieves an accuracy above 98% in 10-fold cross-validation and an accuracy above 90% in a real-world test.

Cooperative Perception for 3D Object Detection in Driving Scenarios Using Infrastructure Sensors

3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the environment as a way to mitigate these limitations. This article proposes two schemes for cooperative 3D object detection using single modality sensors. The early fusion scheme combines point clouds from multiple spatially diverse sensing points of view before detection. In contrast, the late fusion scheme fuses the independently detected bounding boxes from multiple spatially diverse sensors. We evaluate the performance of both schemes, and their hybrid combination, using a synthetic cooperative dataset created in two complex driving scenarios, a T-junction and a roundabout. The evaluation shows that the early fusion approach outperforms late fusion by a significant margin at the cost of higher communication bandwidth. The results demonstrate that cooperative perception can recall more than 95% of the objects as opposed to 30% for single-point sensing in the most challenging scenario. To provide practical insights into the deployment of such system, we report how the number of sensors and their configuration impact the detection performance of the system.

Ensemble learning on visual and textual data for social image emotion classification

Texts, images and other information are posted everyday on the social network and provides a large amount of multimodal data. The aim of this work is to investigate if combining and integrating both visual and textual data permits to identify emotions elicited by an image. We focus on image emotion classification within eight emotion categories: amusement, awe, contentment, excitement, anger, disgust, fear and sadness. Within this classification task we here propose to adopt ensemble learning approaches based on the Bayesian model averaging method, that combine five state-of-the-art classifiers. The proposed ensemble approaches consider predictions given by several classification models, based on visual and textual data, through respectively a late and an early fusion schemes. Our investigations show that an ensemble method based on a late fusion of unimodal classifiers permits to achieve high classification performance within all of the eight emotion classes. The improvement is higher when deep image representations are adopted as visual features, compared with hand-crafted ones.

Human action recognition using fusion of features for unconstrained video sequences

Effective modeling of the human action using different features is a critical task for human action recognition; hence, the fusion of features concept has been used in our proposed work. By fusing several modalities, features, or classifier decision scores, we present six different fusion models inspired by the early fusion schemes, late fusion schemes, and intermediate fusion schemes. In the first two models, we have utilized early fusion technique. The third and fourth models exploit intermediate fusion techniques. In the fourth model, we confront a kernel-based fusion scheme, which takes advantage of kernel basis of classifiers i.e. Support Vector Machine (SVM). In the fifth and sixth models, we have demonstrated late fusion techniques. The performance of all models is evaluated with ASLAN and UCF11 benchmark dataset of action videos. We obtained significant improvements with the proposed fusion schemes relative to the usual fusion schemes relative state-of-the-art methods.