Continuous Learning Enabled Dual Attention-based Fairness-aware eXplainable Convolutional Network for bias mitigation in a critical decision system for human action recognition

Modern human activity recognition (HAR) systems are designed using large amounts of experimental data. So far, real-data-driven or experimental-based HAR systems using Wi-Fi or radar systems have shown considerable results. However, the acquisition of large, clean, and labeled training datasets remains a crucial impediment to the progress of experimental-based HAR systems. Therefore, in this paper, a paradigm shift from the experimental to a fully simulation-based design of HAR systems is proposed in the context of radar sensing. An end-to-end simulation framework is proposed as a proof-of-concept that can simulate realistic millimeter-wave radar signatures for synthesized human motion. We designed a human motion synthesis tool that emulates different types of human activities and generates the spatial trajectories accordingly. These trajectories are processed by a geometric model with respect to user-defined antenna configurations. Considering the long-and short-time stationarity of wireless channels, we synthesize the raw in-phase and quadrature data and process the data to simulate the radar signatures for emulated human activities. Finally, a simulated and a real HAR dataset were used to train and test a simulation-based HAR system, respectively, which gave an average (maximum) classification accuracy of 94% (98.4%). The main advantage of the proposed simulation framework is that the training effort for radar-based classifiers, e.g., gesture recognition systems, can be minimized drastically.

Modern monostatic radar-based human activity recognition (HAR) systems perform very well as long as the direction of human activities is either towards or away from the radar. The monostatic single-input single-output (SISO) and monostatic multiple-input multiple-output (MIMO) radar systems cannot detect motion of an object that moves perpendicularly to the radar’s boresight axis. Due to this physical layer limitation, today’s radar-based HAR systems fail to classify multi-directional human activities. In this paper, we resolve this typical but critical physical layer problem of contemporary HAR systems. We propose a HAR system underlying a distributed MIMO radar configuration, where multiple antennas of a millimeter wave MIMO radar system (Ancortek SDR-KIT 2400T2R4) are distributed in an indoor environment. In our proposed HAR system, we have two independent and identical monostatic radar subsystems that irradiate and capture the multi-directional human movement from two perspectives, which allows to compute two distinct time-variant radial velocity distributions. A feature extraction network extracts numerous features from the measured time-variant radial velocity distributions, which are then fused by a multiclass classifier to detect five types of human activities. The proposed multi-perspective MIMO-radar-based HAR system achieves a classification accuracy of 98.52%, which surpasses the accuracy of SISO radar-based HAR system by more than 9%. Our approach resolves the physical layer limitations of modern HAR systems that are based on either monostatic SISO or monostatic MIMO radar systems.

The availability of cheap wearable motion and biometric sensors has favoured the research on wearable human activity recognition (HAR) systems. However, a HAR system comprehends many complex signal processing stages that usually require some computationally demanding operations which can hardly be directly performed in an embedded system. Modern FPGA technologies and the system-on-chip (SoC) approach open the door to the implementation of complex single-chip signal processing systems to produce tiny, wearable and autonomous embedded HAR systems. However, compared to a pure embedded software approach, the potentially higher performance-to-power ratio of FPGAs can only be exploited in very demanding applications and by a careful design of the implemented system. In this work we describe a first step in the consecution of an FPGA-based completely autonomous singlechip HAR system which can be adapted and optimized to the user with no need of external computing means neither of human intervention. The system includes all stages in a HAR process, i.e., signal segmentation, signal processing for feature extraction, input space dimensionality reduction (feature selection), and activity estimation by means of a neural classifier. A physical activity recognition example is used as a reference design to evaluate the performance of the system and to draw conclusions on the potential benefits of using FPGAs in future wearable HAR applications.

Elderly and individuals with disabilities can greatly benefit from human activity recognition (HAR) systems, which have recently advanced significantly due to the integration of the Internet of Things (IoT) and artificial intelligence (AI). The blending of IoT and AI methodologies into HAR systems has the potential to enable these populations to lead more autonomous and comfortable lives. HAR systems are equipped with various sensors, including motion capture sensors, microcontrollers, and transceivers, which supply data to assorted AI and machine learning (ML) algorithms for subsequent analyses. Despite the substantial advantages of this integration, current frameworks encounter significant challenges related to computational overhead, which arises from the complexity of AI and ML algorithms. This article introduces a novel ensemble of gated recurrent networks (GRN) and deep extreme feedforward neural networks (DEFNN), with hyperparameters optimized through the artificial water drop optimization (AWDO) algorithm. This framework leverages GRN for effective feature extraction, subsequently utilized by DEFNN for accurately classifying HAR data. Additionally, AWDO is employed within DEFNN to adjust hyperparameters, thereby mitigating computational overhead and enhancing detection efficiency. Extensive experiments were conducted to verify the proposed methodology using real-time datasets gathered from IoT testbeds, which employ NodeMCU units interfaced with Wi-Fi transceivers. The framework's efficiency was assessed using several metrics: accuracy at 99.5%, precision at 98%, recall at 97%, specificity at 98%, and F1-score of 98.2%. These results then were benchmarked against other contemporary deep learning (DL)-based HAR systems. The experimental outcomes indicate that our model achieves near-perfect accuracy, surpassing alternative learning-based HAR systems. Moreover, our model demonstrates reduced computational demands compared to preceding algorithms, suggesting that the proposed framework may offer superior efficacy and compatibility for deployment in HAR systems designed for elderly or individuals with disabilities.

Robust and accurate human activity recognition (HAR) systems are essential to many human-centric services within active assisted living and healthcare facilities. Traditional HAR systems mostly leverage a single sensing modality (e.g., either wearable, vision, or radio frequency sensing) combined with machine learning techniques to recognize human activities. Such unimodal HAR systems do not cope well with real-time changes in the environment. To overcome this limitation, new HAR systems that incorporate multiple sensing modalities are needed. Multiple diverse sensors can provide more accurate and complete information resulting in better recognition of the performed activities. This article presents WiWeHAR-a multimodal HAR system that uses combined Wi-Fi and wearable sensing modalities to simultaneously sense the performed activities. WiWeHAR makes use of standard Wi-Fi network interface cards to collect the channel state information (CSI) and a wearable inertial measurement unit (IMU) consisting of accelerometer, gyroscope, magnetometer sensors to collect the user's local body movements. We compute the time-variant mean Doppler shift (MDS) from the processed CSI data and magnitude from the inertial data for each sensor of the IMU. Thereafter, we separately extract various time- and frequency-domain features from the magnitude data and the MDS. We apply feature-level fusion to combine the extracted features, and finally supervised learning techniques are used to recognize the performed activities. We evaluate the performance of WiWeHAR by using a multimodal human activity data set, which was obtained from 9 participants. Each participant carried out four activities, such as walking, falling, sitting, and picking up an object from the floor. Our results indicate that the proposed multimodal WiWeHAR system outperforms the unimodal CSI, accelerometer, gyroscope, and magnetometer HAR systems and achieves an overall recognition accuracy of 99.6%-100%.

This paper presents a methodology based on multimodal sensors to configure a simple, comfortable and fast fall detection and human activity recognition system that can be easily implemented and adopted. The methodology is based on the configuration of specific types of sensors, machine-learning methods and procedures. The protocol is divided into four phases: (1) database creation (2) data analysis (3) system simplification and (4) evaluation. Using this methodology, we created a multimodal database for fall detection and human activity recognition, namely UP-Fall Detection. It comprises data samples from 17 subjects that perform 5 types of falls and 6 different simple activities, during 3 trials. All information was gathered using 5 wearable sensors (tri-axis accelerometer, gyroscope and light intensity), 1 electroencephalograph helmet, 6 infrared sensors as ambient sensors, and 2 cameras in lateral and front viewpoints. The proposed novel methodology adds some important stages to perform a deep analysis of the following design issues in order to simplify a fall detection system: a) select which sensors or combination of sensors are to be used in a simple fall detection system, b) determine the best placement of the sources of information, and c) select the most suitable machine learning classification method for fall and human activity detection and recognition. Even though some multimodal approaches reported in literature only focus on one or two of the above-mentioned issues, our methodology allows simultaneously solving these three design problems related to a human fall and activity detection and recognition system.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

A human activity recognition (HAR) system acts as the backbone of many human-centric applications, such as active assisted living and in-home monitoring for elderly and physically impaired people. Although existing Wi-Fi-based human activity recognition methods report good results, their performance is affected by the changes in the ambient environment. In this work, we present Wi-Sense—a human activity recognition system that uses a convolutional neural network (CNN) to recognize human activities based on the environment-independent fingerprints extracted from the Wi-Fi channel state information (CSI). First, Wi-Sense captures the CSI by using a standard Wi-Fi network interface card. Wi-Sense applies the CSI ratio method to reduce the noise and the impact of the phase offset. In addition, it applies the principal component analysis to remove redundant information. This step not only reduces the data dimension but also removes the environmental impact. Thereafter, we compute the processed data spectrogram which reveals environment-independent time-variant micro-Doppler fingerprints of the performed activity. We use these spectrogram images to train a CNN. We evaluate our approach by using a human activity data set collected from nine volunteers in an indoor environment. Our results show that Wi-Sense can recognize these activities with an overall accuracy of 97.78%. To stress on the applicability of the proposed Wi-Sense system, we provide an overview of the standards involved in the health information systems and systematically describe how Wi-Sense HAR system can be integrated into the eHealth infrastructure.

Many Human Activity Recognition (HAR) systems are able to detect sequential executed Activity of Daily Living (ADL). However, a person is capable of doing two things in parallel or pausing one ADL and finishing it later. Thus, a HAR system must be capable of remembering and deciding which ADL is completed and which might be continued after the current ADL. We address this case by combining a stochastic Markov model and a psychological memory function to detect parallel ADL. For the evaluation, we use an input dataset and a publicly available benchmark. Our approach outperforms the leading HAR systems for the used benchmark by 5%, while using a more cost-effective installation environment. Furthermore, we address an unsupervised learning method to train the HAR system and explain the algorithm of parallel ADL detection in detail.

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

The population of aged people has been increasing globally due to various factors like life expectancy and declining birth rates. So, there is a reduction in physical or cognitive decline, which affects the quality of life of the people where people compromise comfort. It is essential to bring the technology to assist the elderly, which helps them to serve effectively in terms of cost and reliability. As the technologies are more accessible in terms of price, size and speed, these can be adapted to assist the people. Human activity recognition (HAR) system plays a vital role in understanding human actions to assist such people. In this work, we carry out an extensive survey on research in the field of HAR. We propose a review, based on the various sensory modalities used in the HAR system. Along with this, the issues and challenges faced by the various sensor-based HAR are listed.

Video sensor based human activity recognition systems have potential applications in life care and health care areas. The paper presents a system for elderly care by recognizing six abnormal activities; forward fall, backward fall, chest pain, faint, vomit, and headache, selected from the daily life activities of elderly people. Privacy of elderly people is ensured by automatically extracting the binary silhouettes from video activities. Two problems are addressed in this research, which decrease recognition accuracy during the process of abnormal human activity recognition (HAR) system development. First, the problem of continuous changing distance of a moving person from two viewpoints is resolved by using the R-transform. R-transform extracts periodic, scale and translation invariant features from the sequences of activities. Second, the high similarities in postures of different activities is significantly improved by using the kernel discriminant analysis (KDA). KDA increases discrimination between different classes of activities by using non-linear technique. Hidden markov model (HMM) is used for training and recognition of activities. The system is evaluated against linear discriminant analysis (LDA) on the original silhouette features and LDA on the R-transform features. Average recognition rate of 95.8% proves the feasibility of the system for elderly care at home.

Human activity recognition (HAR) systems aim to provide low-cost, low-power, unobtrusive and non-invasive solutions to monitor and collect data accurately for human-centric applications, such as health monitoring, assisted living and rehabilitation. Although wearable sensor_based HAR systems have been demonstrated to be effective in the literature, they raise various concerns such as energy consumption and hardware cost. In this work, we examine the pattern of radio signal strength variations in different activity classes in absence of sensor hardware. We present a performance comparison analysis by setting up two testbeds to compare a sensor_based with a radio_based HAR system over a range of variable metrics such as the number of sensor nodes, and the nodes and the sink node placement with respect to the accuracy and the energy efficiency. Wearable HAR datasets are constructed based on our reported testbeds. The main contributions of this work are in two folds: (1) when eliminating the use of accelerometers in the radio_based system, beside the reduced hardware cost, prolonged lifetime of the HAR system by nearly 30% can be achieved while maintaining the accuracy. The impact of the selected overlapping window size (WS) is also investigated with respect to the accuracy level in both systems over a range of activity classes. (2) The impact of the node placement on the accuracy indicates a higher dependency to the number of nodes, the nodes and the sink node placements in the radio_based system due to the dependency of the results to the distance.

Smart home technologies are getting considerable attentions nowadays for better care of the residents, especially the elderly. One of the key technologies is the human activity recognition (HAR) system which automatically recognizes various indoor activities of a resident and reacts upon the needs of the resident, known as a proactive system. In this work, we propose a novel HAR system that utilizes depth imaging. Our HAR system utilizes local binary patterns (LBP) as local activity features from depth silhouettes and recognizes human activities via Hidden Markov Model (HMM). In our methodology, first LBP features were extracted from depth human body silhouettes from each frame of a video containing human activity. Then, principal component analysis (PCA) and linear discriminant analysis (LDA) were performed over the LBP features to obtain condensed features. Applying these features, each activity HMM was trained. Finally, HAR was performed with the trained HMMs. Our approach shows superior recognition performance over the traditional silhouette feature-based approaches. The system should be practical to be used for smart homes.

Abstract. A human activity recognition (HAR) system carried by masseurs for controlling a therapy table via different movements of legs or hip is studied. This work starts with a survey on HAR systems using the sensor position named “trouser pockets”. Afterwards, in the experiments, the impacts of different hardware systems, numbers of subjects, data generation processes (online streams/offline data snippets), sensor positions, sampling rates, sliding window sizes and shifts, feature sets, feature elimination processes, operating legs, tag orientations, classification processes (concerning method, parameters and an additional smoothing process), numbers of activities, training databases, and the use of a preceding teaching process on the classification accuracy are examined to get a thorough understanding of the variables influencing the classification quality. Besides the impacts of different adjustable parameters, this study also serves as an advisor for the implementation of classification tasks. The proposed system has three operating classes: do nothing, pump therapy table up or pump therapy table down. The first operating class consists of three activity classes (go, run, massage) such that the whole classification process exists with five classes. Finally, using online data streams, a classification accuracy of 98 % could be achieved for one skilled subject and about 90 % for one randomly chosen subject (mean of 1 skilled and 11 unskilled subjects). With the LOSO (leave-one-subject-out) technique for 12 subjects, up to 86 % can be attained. With our offline data approach, we get accuracies of 98 % for 12 subjects and up to 100 % for 1 skilled subject.