Multimodal fusion for sensor data using stacked autoencoders

Abstract

As sensor networks collect different kinds of data to make better-informed decisions, we need multimodal fusion for basic analytical tasks such as event prediction, error reduction and data compression. Inspired by unsupervised feature discovery methods in deep learning, we propose an approach based on the stacked autoencoder: a multi-layer feed-forward neural network. After extracting key features from multimodal sensor data, the algorithm computes compact representations which can also be used directly in analytical tasks. Using simulated and real-world environmental data, we evaluate the performance of our approach for data fusion and regression. We demonstrate improvements over situations where only single modality data is available and where multimodal data are fused together without learning intermodality correlations. For regression, we attained more than 45% improvement in root-mean-square errors (RMSE) over linear approaches, and up to 10% improvement over shallow neural network methods.