A Novel Hybrid Deep Learning Model for Activity Detection Using Wide-Angle Low-Resolution Infrared Array Sensor

K A Muthukumar,Tomoaki Ohtsuki,Mondher Bouazizi

doi:10.1109/access.2021.3084926

K A Muthukumar, Tomoaki Ohtsuki + Show 1 more

Open Access

https://doi.org/10.1109/access.2021.3084926

Copy DOI

Abstract

In this paper, we propose a deep learning-based technique for activity detection that uses wide-angle low-resolution infrared (IR) array sensors. Alongside with the main challenge which is how to further improve the performance of IR array sensor-based methods for activity detection, throughout this work, we address the following challenges: we employ a wide-angle infrared array sensor with peripheral vision in comparison to a standard IR array sensor. This makes activities at different positions have different patterns of temperature distribution, making it challenging to learn these different patterns. In addition, unlike previous works, our goal is to perform the activity detection using the least possible amount of information. While the conventional works use a time window equal to 10 seconds where a single event occurs, we aim to identify the activity using a time window of less than 1 second. Nevertheless, we aim to improve over the accuracy obtained in previous work by employing deep learning, while keeping the approach light for it to run on devices with low computational power. Therefore, we use a hybrid deep learning model well suited for the classification of distorted images because the neural network learns the features automatically. In our work, we use two IR sensors ( 32×24 pixels), one placed on the wall and one on the ceiling. We collect data simultaneously from both the IR sensors and apply hybrid deep learning classification techniques to classify various activities including “walking”, “standing”, “sitting”, “lying”, “falling”, and the transition between the activities which is referred to as “action change”. This is done in two steps. In the first step, we classify ceiling data and wall data separately as well as the combination of both (ceiling and wall) using a Convolutional Neural Network (CNN). In the second step, the output of the CNN is fed to a Long Short Term Memory (LSTM) with a window size equal to 5 frames to classify the sequence of activities. Through experiments, we show that the classification accuracy of the ceiling data, wall data, and combined data with the LSTM reach 0.96, 0.95, and 0.97, respectively.

Highlights

Population ageing is a societal issue facing many countries nowadays that affects social life and the economy
We propose a hybrid deep learning technique to detect the activities using a wide-angle low-resolution infrared array sensors
The True Positive (TP), False Positive (FP), True Negative (TN), and, False Negative (FN) values are reported in the confusion matrix

Summary

INTRODUCTION

Population ageing is a societal issue facing many countries nowadays that affects social life and the economy. B. CNN AND LSTM ARCHITECTURE FOR SENSOR DATA CLASSIFICATION In the first step, we use data collected by each sensor individually to perform the activity detection. The architecture that we propose, as it stands is novel and has been designed taking in mind 3 factors: 1) the type of input data (i.e., sequences of 32 × 24 images) which are very low resolution, 2) the requirement in terms of performance: more complex neural networks might increase the accuracy slightly but not much, and less complex ones have a remarkable performance degradation, and 3) the complexity itself: we expect our model to run on low computation devices such as the Raspberry Pi (which we used to collect the data). The LSTM has a higher potential in detecting such activities

EXPERIMENTAL RESULTS

CNN CLASSIFICATION RESULTS

DISCUSSION

VIII. CONCLUSION