Multi-head Convolutional Neural Network Research Articles

Attention induced multi-head convolutional neural network for human activity recognition

Deep neural networks, including convolutional neural networks (CNNs), have been widely adopted for human activity recognition in recent years. They have attained significant performance improvement over traditional techniques due to their strong feature representation capabilities. Some of the challenges faced by the HAR community is the non-availability of a substantial amount of labeled training samples, and the higher computational cost and system resources requirements of deep learning architectures as opposed to shallow learning algorithms. To address these challenges, we propose an attention-based multi-head model for human activity recognition (HAR). This framework contains three lightweight convolutional heads, with each head designed using one-dimensional CNN to extract features from sensory data. The lightweight multi-head model is induced with attention to strengthen the representation ability of CNN, allowing for automatic selection of salient features and suppress unimportant ones. We conducted ablation studies and experiments on two publicly available benchmark datasets: WISDM and UCI HAR, to evaluate our model. The experimental outcome demonstrates the effectiveness of the proposed framework in activity recognition and achieves better accuracy while ensuring computational efficiency.

Evolutionary neural architecture search for remaining useful life prediction

With the advent of Industry 4.0, making accurate predictions of the remaining useful life (RUL) of industrial components has become a crucial aspect in predictive maintenance (PdM). To this aim, various Deep Neural Network (DNN) models have been proposed in the recent literature. However, while the architectures of these models have a large impact on their performance, they are usually determined empirically. To exclude the time-consuming process and the unnecessary computational cost of manually engineering these models, we present a Neural Architecture Search (NAS) technique based on an Evolutionary Algorithm (EA) applied to optimize the architecture of a DNN used to predict the RUL. The EA explores the combinatorial parameter space of a multi-head Convolutional Neural Network with Long Short Term Memory (CNN-LSTM) to search for the best architecture. In particular, our method requires minimum computational resources by making use of an early stopping policy and a history of the evaluated architectures. We dub the proposed method ENAS-PdM. To our knowledge, this is the first work where an EA-based NAS is used to optimize a CNN-LSTM architecture in the field of PdM. In our experiments, we use the well-established Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset from NASA. Compared to the current state-of-the-art, our method obtains better results in terms of two different metrics, RMSE and Score, when aggregating across all the C-MAPSS sub-datasets. Without aggregation, we achieve lower RMSE in 3 out of 4 sub-datasets. Our experimental results verify that the proposed method is a reliable tool for obtaining state-of-the-art RUL predictions and as such it can have a strong impact in several industrial applications, especially those with limited available computing power.

Automatic Image-Based Event Detection for Large-N Seismic Arrays Using a Convolutional Neural Network

Passive seismic experiments have been proposed as a cost-effective and non-invasive alternative to controlled-source seismology, allowing body–wave reflections based on seismic interferometry principles to be retrieved. However, from the huge volume of the recorded ambient noise, only selected time periods (noise panels) are contributing constructively to the retrieval of reflections. We address the issue of automatic scanning of ambient noise data recorded by a large-N array in search of body–wave energy (body–wave events) utilizing a convolutional neural network (CNN). It consists of computing first both amplitude and frequency attribute values at each receiver station for all divided portions of the recorded signal (noise panels). The created 2-D attribute maps are then converted to images and used to extract spatial and temporal patterns associated with the body–wave energy present in the data to build binary CNN-based classifiers. The ensemble of two multi-headed CNN models trained separately on the frequency and amplitude attribute maps demonstrates better generalization ability than each of its participating networks. We also compare the prediction performance of our deep learning (DL) framework with a conventional machine learning (ML) algorithm called XGBoost. The DL-based solution applied to 240 h of ambient seismic noise data recorded by the Kylylahti array in Finland demonstrates high detection accuracy and the superiority over the ML-based one. The ensemble of CNN-based models managed to find almost three times more verified body–wave events in the full unlabelled dataset than it was provided at the training stage. Moreover, the high-level abstraction features extracted at the deeper convolution layers can be used to perform unsupervised clustering of the classified panels with respect to their visual characteristics.

EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications

Human Activity Recognition (HAR) can be defined as the automatic prediction of the regular human activities performed in our day-to-day life, such as walking, running, cooking, performing office work, etc. It is truly beneficial in the field of medical care services, for example, personal health care assistants, old-age care services, maintaining patient records for future help, etc. Input data to a HAR system can be (a) videos or still images capturing human activities, or (b) time-series data of human body movements while performing the activities taken from sensors in the smart devices like accelerometer, gyroscope, etc. In this work, we mainly focus on the second category of the input data. Here, we propose an ensemble of three classification models, namely CNN-Net, Encoded-Net, and CNN-LSTM, which is named as EnsemConvNet. Each of these classification models is built upon simple 1D Convolutional Neural Network (CNN) but differs in terms of the number of dense layers, kernel size used along with other key differences in the architecture. Each model accepts the time series data as a 2D matrix by taking a window of data at a time in order to infer information, which ultimately predicts the type of human activity. Classification outcome of the EnsemConvNet model is decided using various classifier combination methods that include majority voting, sum rule, product rule, and a score fusion approach called adaptive weighted approach. Three benchmark datasets, namely WISDM activity prediction, UniMiB SHAR, MobiAct, are used for evaluating our proposed model. We have compared our EnsemConvNet model with some existing deep learning models such as Multi Headed CNN, hybrid of CNN, and Long Short Term Memory (LSTM) models. The results obtained here establish the supremacy of the EnsemConvNet model over the other mentioned models.

MH-MetroNet-A Multi-Head CNN for Passenger-Crowd Attendance Estimation.

Knowing an accurate passengers attendance estimation on each metro car contributes to the safely coordination and sorting the crowd-passenger in each metro station. In this work we propose a multi-head Convolutional Neural Network (CNN) architecture trained to infer an estimation of passenger attendance in a metro car. The proposed network architecture consists of two main parts: a convolutional backbone, which extracts features over the whole input image, and a multi-head layers able to estimate a density map, needed to predict the number of people within the crowd image. The network performance is first evaluated on publicly available crowd counting datasets, including the ShanghaiTech part_A, ShanghaiTech part_B and UCF_CC_50, and then trained and tested on our dataset acquired in subway cars in Italy. In both cases a comparison is made against the most relevant and latest state of the art crowd counting architectures, showing that our proposed MH-MetroNet architecture outperforms in terms of Mean Absolute Error (MAE) and Mean Square Error (MSE) and passenger-crowd people number prediction.

An End-to-End Grasping Stability Prediction Network for Multiple Sensors

As we all know, the output of the tactile sensing array on the gripper can be used to predict grasping stability. Some methods utilize traditional tactile features to make the decision and some advanced methods use machine learning or deep learning ways to build a prediction model. While these methods are all limited to the specific sensing array and have two common disadvantages. On the one hand, these models cannot perform well on different sensors. On the other hand, they do not have the ability of inferencing on multiple sensors in an end-to-end manner. Thus, we aim to find the internal relationships among different sensors and inference the grasping stability of multiple sensors in an end-to-end way. In this paper, we propose the MM-CNN (mask multi-head convolutional neural network), which can be utilized to predict the grasping stability on the output of multiple sensors with the weight sharing mechanism. We train this model and evaluate it on our own collected datasets. This model achieves 99.49% and 94.25% prediction accuracy on two different sensing arrays, separately. In addition, we show that our proposed structure is also available for other CNN backbones and can be easily integrated.

Forecasting Solar PV Output Using Convolutional Neural Networks with a Sliding Window Algorithm

The stochastic nature of renewable energy sources, especially solar PV output, has created uncertainties for the power sector. It threatens the stability of the power system and results in an inability to match power consumption and production. This paper presents a Convolutional Neural Network (CNN) approach consisting of different architectures, such as the regular CNN, multi-headed CNN, and CNN-LSTM (CNN-Long Short-Term Memory), which utilizes a sliding window data-level approach and other data pre-processing techniques to make accurate forecasts. The output of the solar panels is linked to input parameters such as irradiation, module temperature, ambient temperature, and windspeed. The benchmarking and accuracy metrics are calculated for 1 h, 1 day, and 1 week for the CNN based methods which are then compared with the results from the autoregressive moving average and multiple linear regression models in order to demonstrate its efficacy in making short-term and medium-term forecasts.

Energy forecasting using multiheaded convolutional neural networks in efficient renewable energy resources equipped with energy storage system

AbstractRenewable energy resources (RERs) motivate electricity users to reduce their energy bills by taking benefit of self‐generated green energy. Different studies have already pointed out that, because of the absence of proper technical support and awareness, the energy users were not able to sufficiently take paybacks from the RERs. However, with the commencement of smart grids, the potential benefits of RERs and dynamic pricing schemes can be exploited. Nonetheless, the big issue is the accurate prediction of energy produced by intermittent RERs. In this work, we have proposed an efficient framework by integrating energy storage system (ESS) and RERs with smart homes. This framework has shown significant results, which make it helpful and suitable for energy management at a community level. We applied a multiheaded convolutional neural network model for precise and accurate prediction of produced energy by RERs. Moreover, we have considered a smart community consisting of 80 homes. Simulation results prove that the proposed framework helps to decrease the energy bill of consumers by 58.32% and 63.02% through integration of RERs without and with ESS, respectively.

Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study

Detecting anomalies in time series data is becoming mainstream in a wide variety of industrial applications in which sensors monitor expensive machinery. The complexity of this task increases when multiple heterogeneous sensors provide information of different nature, scales and frequencies from the same machine. Traditionally, machine learning techniques require a separate data pre-processing before training, which tends to be very time-consuming and often requires domain knowledge. Recent deep learning approaches have shown to perform well on raw time series data, eliminating the need for pre-processing. In this work, we propose a deep learning based approach for supervised multi-time series anomaly detection that combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) in different ways. Unlike other approaches, we use independent CNNs, so-called convolutional heads, to deal with anomaly detection in multi-sensor systems. We address each sensor individually avoiding the need for data pre-processing and allowing for a more tailored architecture for each type of sensor. We refer to this architecture as Multi-head CNN–RNN. The proposed architecture is assessed against a real industrial case study, provided by an industrial partner, where a service elevator is monitored. Within this case study, three type of anomalies are considered: point, context-specific, and collective.The experimental results show that the proposed architecture is suitable for multi-time series anomaly detection as it obtained promising results on the real industrial scenario.

Microgrid-Level Energy Management Approach Based on Short-Term Forecasting of Wind Speed and Solar Irradiance

Background: The Distributed Energy Resources (DERs) are beneficial in reducing the electricity bills of the end customers in a smart community by enabling them to generate electricity for their own use. In the past, various studies have shown that owing to a lack of awareness and connectivity, end customers cannot fully exploit the benefits of DERs. However, with the tremendous progress in communication technologies, the Internet of Things (IoT), Big Data (BD), machine learning, and deep learning, the potential benefits of DERs can be fully achieved, although a significant issue in forecasting the generated renewable energy is the intermittent nature of these energy resources. The machine learning and deep learning models can be trained using BD gathered over a long period of time to solve this problem. The trained models can be used to predict the generated energy through green energy resources by accurately forecasting the wind speed and solar irradiance. Methods: We propose an efficient approach for microgrid-level energy management in a smart community based on the integration of DERs and the forecasting wind speed and solar irradiance using a deep learning model. A smart community that consists of several smart homes and a microgrid is considered. In addition to the possibility of obtaining energy from the main grid, the microgrid is equipped with DERs in the form of wind turbines and photovoltaic (PV) cells. In this work, we consider several machine learning models as well as persistence and smart persistence models for forecasting of the short-term wind speed and solar irradiance. We then choose the best model as a baseline and compare its performance with our proposed multiheaded convolutional neural network model. Results: Using the data of San Francisco, New York, and Los Vegas from the National Solar Radiation Database (NSRDB) of the National Renewable Energy Laboratory (NREL) as a case study, the results show that our proposed model performed significantly better than the baseline model in forecasting the wind speed and solar irradiance. The results show that for the wind speed prediction, we obtained 44.94%, 46.12%, and 2.25% error reductions in root mean square error (RMSE), mean absolute error (MAE), and symmetric mean absolute percentage error (sMAPE), respectively. In the case of solar irradiance prediction, we obtained 7.68%, 54.29%, and 0.14% error reductions in RMSE, mean bias error (MBE), and sMAPE, respectively. We evaluate the effectiveness of the proposed model on different time horizons and different climates. The results indicate that for wind speed forecast, different climates do not have a significant impact on the performance of the proposed model. However, for solar irradiance forecast, we obtained different error reductions for different climates. This discrepancy is certainly due to the cloud formation processes, which are very different for different sites with different climates. Moreover, a detailed analysis of the generation estimation and electricity bill reduction indicates that the proposed framework will help the smart community to achieve an annual reduction of up to 38% in electricity bills by integrating DERs into the microgrid. Conclusions: The simulation results indicate that our proposed framework is appropriate for approximating the energy generated through DERs and for reducing the electricity bills of a smart community. The proposed framework is not only suitable for different time horizons (up to 4 h ahead) but for different climates.

Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks

We propose the multi-head convolutional neural network (MCNN) architecture for waveform synthesis from spectrograms. Nonlinear interpolation in MCNN is employed with transposed convolution layers in parallel heads. MCNN achieves more than an order of magnitude higher compute intensity than commonly-used iterative algorithms like Griffin-Lim, yielding efficient utilization for modern multi-core processors, and very fast (more than 300x real-time) waveform synthesis. For training of MCNN, we use a large-scale speech recognition dataset and losses defined on waveforms that are related to perceptual audio quality. We demonstrate that MCNN constitutes a very promising approach for high-quality speech synthesis, without any iterative algorithms or autoregression in computations.