An Intelligent Implementation of Multi-Sensing Data Fusion With Neuromorphic Computing for Human Activity Recognition

Abstract

The increasing demand for considering multisensor data fusion technology has drawn attention for precise human activity recognition (HAR) over standalone technology due to its reliability and robustness. This article presents a framework that fuses data from multiple sensing systems and applies neuromorphic computing to sense and classify human activities. The data is collected by utilizing inertial measurement unit (IMU) sensors, software-defined radios, and radars, and feature extraction and selection are performed on the data. For each of the actions, such as sitting and standing, an activity matrix is generated, which is then fed into a discrete Hopfield neural network as a binary feature pattern for one-shot learning. Following the Hopfield network neurons’ feedback output, the conformity to the standard activity feature pattern is also determined. Following the Hopfield network neurons’ feedback output, the training of neurons is completed after two steps under the Hebbian learning law, and the conformity to the standard activity feature pattern is also determined. According to the probabilistic statistics on inference predictions, the proposed method, that is the neuromorphic computing of the three data fused framework, achieved the box plot for the highest lower quartile output of 95.34%, while the confusion matrix classification accuracy of the two activities was 98.98%. The results have shown that neuromorphic computing is most capable of multisensor data-fusion-based HAR. Furthermore, the proposed method can be enhanced by incorporating additional hardware signal processing in the system to enable the flexible integration of human activity data.