Human Motion Recognition Research Articles

High-performance fibrous strain sensor with synergistic sensing layer for human motion recognition and robot control

High-performance fibrous strain sensor with synergistic sensing layer for human motion recognition and robot control

Subspace and second-order statistical distribution alignment for cross-domain recognition of human hand motions

Subspace and second-order statistical distribution alignment for cross-domain recognition of human hand motions

CubeLearn: End-to-End Learning for Human Motion Recognition From Raw mmWave Radar Signals

mmWave FMCW radar has attracted huge amount of research interest for human-centered applications in recent years, such as human gesture and activity recognition. Most existing pipelines are built upon conventional Discrete Fourier Transform (DFT) pre-processing and deep neural network classifier hybrid methods, with a majority of previous works focusing on designing the downstream classifier to improve overall accuracy. In this work, we take a step back and look at the pre-processing module. To avoid the drawbacks of conventional DFT pre-processing, we propose a complex-weighted learnable pre-processing module, named CubeLearn, to directly extract features from raw radar signal and build an end-to-end deep neural network for mmWave FMCW radar motion recognition applications. Extensive experiments show that our CubeLearn module consistently improves the classification accuracies of different pipelines, especially benefiting those simpler models, which are more likely to be used on edge devices due to their computational efficiency. We provide ablation studies on initialization methods and structure of the proposed module, as well as an evaluation of the running time on PC and edge devices. This work also serves as a comparison of different approaches towards data cube slicing. Through our task agnostic design, we propose a first step towards a generic end-to-end solution for radar recognition problems.

Self-Repairing and Energy-Harvesting Triboelectric Sensor for Tracking Limb Motion and Identifying Breathing Patterns.

The increasing prevalence of health problems stemming from sedentary lifestyles and evolving workplace cultures has placed a substantial burden on healthcare systems. Consequently, remote health wearable monitoring systems have emerged as essential tools to track individuals' health and well-being. Self-powered triboelectric nanogenerators (TENGs) have exhibited significant potential for use as emerging detection devices capable of recognizing body movements and monitoring breathing patterns. However, several challenges remain to be addressed in order to fulfill the requirements for self-healing ability, air permeability, energy harvesting, and suitable sensing materials. These materials must possess high flexibility, be lightweight, and have excellent triboelectric charging effects in both electropositive and electronegative layers. In this work, we investigated self-healable electrospun polybutadiene-based urethane (PBU) as a positive triboelectric layer and titanium carbide (Ti3C2Tx) MXene as a negative triboelectric layer for the fabrication of an energy-harvesting TENG device. PBU consists of maleimide and furfuryl components as well as hydrogen bonds that trigger the Diels-Alder reaction, contributing to its self-healing properties. Moreover, this urethane incorporates a multitude of carbonyl and amine groups, which create dipole moments in both the stiff and the flexible segments of the polymer. This characteristic positively influences the triboelectric qualities of PBU by facilitating electron transfer between contacting materials, ultimately resulting in high output performance. We employed this device for sensing applications to monitor human motion and breathing pattern recognition. The soft and fibrous-structured TENG generates a high and stable open-circuit voltage of up to 30 V and a short-circuit current of 4 μA at an operation frequency of 4.0 Hz, demonstrating remarkable cyclic stability. A significant feature of our TENG is its self-healing ability, which allows for the restoration of its functionality and performance after sustaining damage. This characteristic has been achieved through the utilization of the self-healable PBU fibers, which can be repaired via a simple vapor solvent method. This innovative approach enables the TENG device to maintain optimal performance and continue functioning effectively even after multiple uses. After integration with a rectifier, the TENG can charge various capacitors and power 120 LEDs. Moreover, we employed the TENG as a self-powered active motion sensor, attaching it to the human body to monitor various body movements for energy-harvesting and sensing purposes. Additionally, the device demonstrates the capability to recognize breathing patterns in real time, offering valuable insights into an individual's respiratory health.

Radar-Based Human Motion Recognition by Using Vital Signs with ECA-CNN

Radar-Based Human Motion Recognition by Using Vital Signs with ECA-CNN

An assumption of UWB enhancement of human motion recognition

Today's VR games on the market use computers to connect a head-mounted monitor and two grips to build a virtual game world for players, providing players with a fresh sensory experience. However, the current VR game only uses the player's hand movement as a way of human-computer communication, the way is relatively single, and the player can not express their desire to play through only two hands, which cannot fully express the player's will. There are currently two types of motion capture used in consumer VR devices: optical capture, which is based on inertial sensor motion capture. These two are faced with the problem of extremely high cost and burden. The ultra-broadband system has the advantages of high positioning accuracy, low latency, cheap price, being light and portable, etc., which is very suitable for this kind of application scenario. This paper considers using two main technologies RTT ranging motion recognition and keyframe to achieve the aim.

Human Motion Recognition in Dance Video Images Based on Attitude Estimation

With the deep integration of science and technology and culture, the estimation of human movements in dance video images will become an important application field of computer vision technology, which can be used not only for professional dancers’ movement correction, dance self-help teaching, and other application scenarios but also for athletes’ movement analysis. Therefore, it will greatly promote the implementation of teaching students in accordance with their aptitude by applying information technology to estimate dancers’ movements and postures in real time and obtaining information of classroom dance teaching status in time. In this paper, human motion recognition in dance video images is studied based on an attitude estimation algorithm. When the number of experiments reaches 20, the average value of deep learning algorithm and particle swarm optimization algorithm is 76.23 and 75.23, respectively, while the average value of attitude estimation algorithm in this paper is 77.95. Therefore, the average results of attitude estimation algorithm in this paper are slightly higher than those of other algorithms.

A Self-Powered Piezoelectric Nanofibrous Membrane as Wearable Tactile Sensor for Human Body Motion Monitoring and Recognition.

The online version contains supplementary material available at 10.1007/s42765-023-00282-8.

3D Dynamic Image Modeling Based on Machine Learning in Film and Television Animation

With the deep integration and rapid development of computer technology and film and television animation in recent years, computer animation technology has gradually created objective practical value. 3D animation technology also plays a key role in film and television special effects and advertising special effects. As a new type of multimedia data, human motion capture data can be used for 3D human model modeling and human motion simulation because of its high fidelity. A human motion pattern recognition method based on long short-term memory network (LSTM) is proposed. The model uses a deep learning neural network composed of a 2-layer LSTM network to automatically extract features from the collected human body feature information. Then, the multi-class motion patterns are modeled in time series to quickly identify different motion patterns in real time. To evaluate the performance of this model, the effectiveness of this method in identifying six different motion modes is validated using an open dataset. At the same time, this method is compared with four methods based on in-depth learning model. Experimental results verify the effectiveness of the method. It provides a feasible method for human motion recognition and modeling based on capture data in video animation.

Flexible multi-modal sensors based on CNT hollow spheres/PDMS composites for human motion recognition and colloid concentration detection

Because flexible sensors with multi-modal responses are essential for applications in the complex environment of wearable products, a unique carbon nanotube (CNT) hollow-sphere scaffold is fabricated using self-assembling of microspheres and microwave irradiation to achieve flexible strain, pressure, and laser sensors. The CNT hollow-sphere scaffold comprising intertwined CNTs is featured with a sophisticated conductive network, large tensile and compressive deformability, intrinsic photothermoelectric nature, and good accommodation for polydimethylsiloxane (PDMS) matrix. Thus, the fabricated CNT hollow spheres/PDMS composite sensors showed good responses to the applied strain (a high gauge factor exceeding 55000 in the strain region of 95–107%, 27 ms response time, and over 20000 cycles durability at 40% strain), pressure (a gauge factor of 0.07–3.15 kPa−1 with a wide sensing range (up to 248 kPa pressure), and laser irradiation (0.066 s−1 at 5 s of 532 nm laser irradiation). Moreover, their uses in body movement detection, healthcare, tactile perception, and colloid concentration detection are demonstrated. It is believed that the CNT hollow spheres/PDMS composite sensors will pave the way for practical applications for future wearable electronic devices, which could recognize human motion as well as sense air quality and disease diagnosis.

Autonomous gesture recognition using multi-layer LSTM networks and laban movement analysis

In recent years, due to the reasonable price of RGB-D devices, the use of skeletal-based data in the field of human-computer interaction has attracted a lot of attention. Being free from problems such as complex backgrounds as well as changes in light is another reason for the popularity of this type of data. In the existing methods, the use of joint and bone information has had significant results in improving the recognition of human movements and even emotions. However, how to combine these two types of information in the best possible way to define the relationship between joints and bones is a problem that has not yet been solved. In this article, we used the Laban Movement Analysis (LMA) to build a robust descriptor and present a precise description of the connection of the different parts of the body to itself and its surrounding environment while performing a gesture. To do this, in addition to the distances between the hip center and other joints of the body and the changes of the quaternion angles in time, we define the triangles formed by the different parts of the body and calculate their area. We also calculate the area of the single conforming 3-D boundary around all the joints of the body. We use a long short-term memory (LSTM) network to evaluate this descriptor. The proposed algorithm is implemented on five public datasets: NTU RGB+D 120, SYSU 3D HOI, FLORENCE 3D ACTIONS, MSR Action3D and UTKinect-Action3D datasets, and the results are compared with those available in the literature.

High-Sensitivity Composite Dual-Network Hydrogel Strain Sensor and Its Application in Intelligent Recognition and Motion Monitoring

Flexible electronic sensors are attracting more and more attention because of their outstanding flexibility and foldability. Based on this background, a composite hydrogel with excellent flexibility, good stability, and high sensitivity is prepared in this work. This high-performance hydrogel is composed of polyacrylamide, sodium alginate, MXene, and PEDOT:PSS through dynamic supramolecular cross-linking. The electrical conductivity is significantly enhanced by the addition of the conductive polymer PEDOT:PSS and MXene. Hydrogel conductivity increased by 150% with the addition of conductive materials at the same tensile strain. It exhibits high sensitivity (GF = 1.99) and excellent linearity over a wide operating range of 1000% of strain, which ensures the detection performance of the hydrogel sensor under large strain conditions. In addition, it has excellent repeatability (600 cycles) and fast response/recovery times (0.624 s/0.912 s). Due to this excellent performance, it was made into a smart sensor to explore its application in smart electronic skin. The hydrogel sensor and data acquisition module were integrated for real-time monitoring of pressure. We combine it with deep learning algorithms to provide a new solution for human motion recognition, such as joint flexion, facial expression change, health monitoring, and handwriting recognition.

Human Motion Capture Based on MEMS Sensor

In order to realize the monitoring of human joint rehabilitation, a human motion capture and recognition system is constructed by using micro electro mechanical system (MEMS) sensor nodes. A two-stage extended Kalman filter algorithm is proposed for multi-sensor data fusion. The error matrix between the coordinate system of sensor node and the coordinate system of body was calculated by using the stationary posture calibration. The root mean squared error (RMSE) of the computed joint angle time series is less than 0.5°. The feature of joint angle time series was extracted and the support vector machine (SVM) classification model based on particle swarm optimization (PSO) was established. The experimental results show that the SVM algorithm optimized by PSO has better recognition effect than BP neural network. The average recognition rate can reach more than 97%. The human motion capture system designed in this paper can effectively realize human motion capture, recognition and joint rehabilitation monitoring.

SEMG Signal-Based Lower Limb Movements Recognition Using Tunable Q-Factor Wavelet Transform and Kraskov Entropy

BackgroundThe recognition of lower limb movement has a wide range of applications in rehabilitation training, wearable exoskeleton control, and human activity monitoring. Surface electromyography (sEMG) signals can directly reflect the intention of human movement and can be used as the source of lower limb movement recognition. Literature reports have shown that extracting features from sEMG signals is the core of human movement recognition based on sEMG signals. However, how to effectively extract features from the sEMG signal of the lower limbs affected by body gravity is a difficult problem for the recognition of lower limb movement based on the sEMG signal. ObjectivesThe main objective of this paper is to propose an efficient lower limb movement recognition model based on sEMG signals to accurately recognize the four lower limb movements. Methods and resultsWe proposed a novel method of lower limbs movements recognition based on tunable Q-factor wavelet transform (TQWT) and Kraskov entropy (KrEn). Firstly, the sEMG signals of four different lower limb movements from twenty subjects were recorded by seven wearable sEMG signal sensors, and the recorded sEMG signals were denoised by multi-scale principal component analysis (MSPCA). Then, the denoised sEMG signal is decomposed into multiple sub-band signals by TQWT and the KrEn feature is extracted from each sub-band signal. Next, the representative features are selected from the extracted KrEn features by the minimum redundancy maximum relevance (mRMR) feature selection method. Finally, the four lower limb movements are recognized by three machine learning classifiers. Besides, to improve the recognition performance, a majority voting (MV) technology is proposed for the post-processing of decision flow. Experimental results show that the combination of TQWT, KrEn, and MV technology achieved the average recognition accuracy of 98.42% using the linear discriminant analysis (LDA) classifier. ConclusionThe method proposed in this paper can recognize lower limb movements with high accuracy. Compared with existing methods, this method is more advanced and accurate, indicating that it has great application potential in rehabilitation training, wearable exoskeleton control, and daily activity monitoring.

Flexible strain sensor with a hat-shaped structure for in situ measurement of 3D deformation

Flexible strain sensors that are currently available are mainly used in human motion recognition and medical health detection applications, and there is still an urgent need for sensors to realize real-time monitoring of the 3D deformation of industrial and agricultural products. In this work, a flexible strain sensor with a hat-shaped structure was fabricated using a molding technique to perform in situ measurement of 3D deformation. An algorithm for resistance change detection and linear calibration equations were proposed to enable analysis of the deformation data and calculation of local shape changes. The sensor was applied to monitor the growth deformation of a kumquat fruit, and the results were highly consistent with the algorithm. The proposed technique has great potential for application to 3D deformation detection of flexible objects.

Human motion recognition by a shoes-floor triboelectric nanogenerator and its application in fall detection

With the advent of the Internet of Things (IoT), various IoT devices have been widely used, and in recent years, the rapid development of smart sensors as wearable electronic devices has increasingly promoted human-computer integration. Traditional methods using solid-state power supplies suffer from limited battery life, high maintenance costs and environmental pollution. In this work, we propose a simple yet powerful battery-free human motion sensing system that can recognize human motion through a shoes-ground comprised natural triboelectric nanogenerator (TENG), with a conductive PVA-PEDOT:PSS hydrogel as the signal collection component. In addition, we develop an artificial intelligence (AI)-based fall detection system based on the TENG. To this end, we extend TENG with a custom Bluetooth module to transmit the collected signals to the cloud and develop an anomaly detection AI algorithm to detect fall accidents during walking in real time and send instant messages for notification. We experimentally evaluate the proposed TENG fall detection system and conclude that the human sensing-based TENG system has a wide range of potential in wearable electronic devices, and also provides a viable reference for other applications related to human motion detection.

An attention-based deep learning approach for inertial motion recognition and estimation in human-robot collaboration

In line with a human-centric smart manufacturing vision, human-robot collaboration is striving to combine robots’ high efficiency and quality with humans’ rapid adaptability and high flexibility. In particular, perception, recognition and estimation of human motion determine when and what robot to collaborate with humans. This work presents an attention-based deep learning approach for inertial motion recognition and estimation in order to infer when robotic assistance will be requested by the human and to allow the robot to perform partial human tasks. First, in the stage of motion perception and recognition, quaternion-based calibration and forward kinematic analysis methods enable the reconstruction of human motion based on data streaming from an inertial motion capture device. Then, in the stage of motion estimation, residual module and Bidirectional Long Short-Term Memory module are integrated with proposed attention mechanism for estimating arm motion trajectories further. Experimental results show the effectiveness of the proposed approach in achieving better recognition and estimation in comparison with traditional approaches and existing deep learning approaches. It is experimentally verified in a laboratory environment involving a collaborative robot employed in a small part assembly task.

Human motion recognition based on Nano-CMOS Image sensor.

Human motion recognition is of great value in the fields of intelligent monitoring systems, driver assistance system, advanced human-computer interaction, human motion analysis, image and video processing. However, the current human motion recognition methods have the problem of poor recognition effect. Therefore, we propose a human motion recognition method based on Nano complementary metal oxide semiconductor (CMOS) image sensor. First, using the Nano-CMOS image sensor to transform and process the human motion image, and combines the background mixed model of pixels in the human motion image to extract the human motion features, and feature selection is conducted. Second, according to the three-dimensional scanning features of Nano-CMOS image sensor, the human joint coordinate information data is collected, the state variables of human motion are sensed by the sensor, and the human motion model is constructed according to the measurement matrix of human motions. Finally, the foreground features of human motion images are obtained by calculating the feature parameters of each motion gesture. According to the posterior conditional probability of human motion images, the recognition objective function of human motion is obtained to realize human motion recognition. The results show that the human motion recognition effect of the proposed method is good, the extraction accuracy is high, the average human motion recognition rate is 92%, the classification accuracy is high, and the recognition speed is up to 186 frames/s.

Integration of capillaric strain sensors toward recognition of human movements

Skin-strain-field is used as a fingerprint of the human movement to guide the design of an integrated strain sensor for digital human motion recognition.

Toward Ambient Intelligence: Federated Edge Learning With Task-Oriented Sensing, Computation, and Communication Integration

With the breakthroughs in deep learning and contactless sensors, the recent years have witnessed a rise of ambient intelligence applications and services, spanning from healthcare delivery to intelligent home. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Federated edge learning</i> (FEEL), as a privacy-enhancing paradigm of collaborative learning at the network edge, is expected to be the core engine to achieve ambient intelligence. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Sensing, computation, and communication</i> (SC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> ) are highly coupled processes in FEEL and need to be jointly designed in a task-oriented manner to achieve the best FEEL performance under stringent resource constraints at edge devices. However, this remains an open problem as there is a lack of theoretical understanding on how the SC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> resources jointly affect the FEEL performance. In this paper, we address the problem of joint SC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> resource allocation for FEEL via a concrete case study of human motion recognition based on wireless sensing in ambient intelligence. First, by analyzing the wireless sensing process in human motion recognition, we find that there exists a thresholding value for the sensing transmit power, exceeding which yields sensing data samples with approximately the same satisfactory quality. Then, the joint SC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> resource allocation problem is cast to maximize the convergence speed of FEEL, under the constraints on training time, energy supply, and sensing quality of each edge device. Solving this problem entails solving two subproblems in order: the first one reduces to determine the joint sensing and communication resource allocation that maximizes the total number of samples that can be sensed during the entire training process; the second one concerns the partition of the attained total number of sensed samples over all the communication rounds to determine the batch size at each round for convergence speed maximization. The first subproblem on joint sensing and communication resource allocation is converted to a single-variable optimization problem by exploiting the derived relation between different control variables (resources), which thus allows an efficient solution via one-dimensional grid search. For the second subproblem, it is found that the number of samples to be sensed (or batch size) at each round is a decreasing function of the loss function value attained at the round. Based on this relationship, the approximate optimal batch size at each communication round is derived in closed-form as a function of the round index. Finally, extensive simulation results are provided to validate the superiority of the proposed joint SC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> resource allocation scheme over baseline schemes in terms of FEEL performance.