Video Magnification Research Articles

Transfer-learning enabled micro-expression recognition using dense connections and mixed attention

Micro-expression recognition (MER) is a challenging computer vision problem, where the limited amount of available training data and insufficient intensity of the facial expressions are among the main issues adversely affecting the performance of existing recognition models. To address these challenges, this paper explores a transfer–learning enabled MER model using a densely connected feature extraction module with mixed attention. Unlike previous works that utilize transfer learning to facilitate MER and extract local facial-expression information, our model relies on pretraining with three diverse macro-expression datasets and, as a result, can: (i) overcome the problem of insufficient sample size and limited training data availability, (ii) leverage (related) domain-specific information from multiple datasets with diverse characteristics, and (iii) improve the model adaptability to complex scenes. Furthermore, to enhance the intensity of the micro-expressions and improve the discriminability of the extracted features, the Euler video magnification (EVM) method is adopted in the preprocessing stage and then used jointly with a densely connected feature extraction module and a mixed attention mechanism to derive expressive feature representations for the classification procedure. The proposed feature extraction mechanism not only guarantees the integrity of the extracted features but also efficiently captures local texture cues by aggregating the most salient information from the generated feature maps, which is key for the MER task. The experimental results on multiple datasets demonstrate the robustness and effectiveness of our model compared to the state-of-the-art.

Micro-expression recognition based on euler video magnification and 3D residual network under imbalanced sample

A student’s verbal behavior plays a crucial role in education, while nonverbal behavior, such as micro-expressions, significantly improves teaching quality. To address the problem of small facial expression movements, imbalanced data categories, and lack of temporal information in static expressions, a micro-expression recognition method is proposed based on Eulerian Video Magnification (EVM) and a 3D Residual Network (3D ResNet) under imbalanced samples. Firstly, face detection in the Dlib library is used to locate the face in the micro-expression video sample and crop it. Secondly, the EVM is used to magnify the motion features in micro-expressions. Then, the 3D ResNet is used to extract spatio-temporal information from micro-expression video samples, and the Cyclical Focal Loss (CFL) function is introduced in the network training process to solve the class imbalance problem in micro-expression datasets. Finally, the roles of the EVM and the CFL function in recognizing micro-expressions by the 3D ResNet are analyzed. The experimental results on the Spontaneous Micro-expression Database (SMIC) and Chinese Academy of Sciences Micro-expression Database II (CASME II) demonstrate the effectiveness and superiority of this method. The proposed method can assist in teaching evaluation and promote the development of smart classrooms, and further research is needed on the storage and computing of the proposed method on devices.

Efficient Vibration Measurement and Modal Shape Visualization Based on Dynamic Deviations of Structural Edge Profiles.

As a non-contact method, vision-based measurement for vibration extraction and modal parameter identification has attracted much attention. In most cases, artificial textures are crucial elements for visual tracking, and this feature limits the application of vision-based vibration measurement on textureless targets. As a computation technique for visualizing subtle variations in videos, the video magnification technique can analyze modal responses and visualize modal shapes, but the efficiency is low, and the processing results contain clipping artifacts. This paper proposes a novel method for the application of a modal test. In contrast to the deviation magnification that exaggerates subtle geometric deviations from only a single image, the proposed method extracts vibration signals with sub-pixel accuracy on edge positions by changing the perspective of deviations from space to timeline. Then, modal shapes are visualized by decoupling all spatial vibrations following the vibration theory of continuous linear systems. Without relying on artificial textures and motion magnification, the proposed method achieves high operating efficiency and avoids clipping artifacts. Finally, the effectiveness and practical value of the proposed method are validated by two laboratory experiments on a cantilever beam and an arch dam model.

Phase-based video magnification on a single board computer

Phase based video magnification is currently one of the best motion magnifying algorithms available. Unlike the linear video magnification which preceded it, however, it is significantly more computationally demanding. In this work we present a modified version of this algorithm which we developed in C++ to be usable on low powered platforms such as the Raspberry Pi.

A Novel Contactless Blood Pressure Measurement System and Algorithm Based on Vision Intelligence

The measurement of vital signs such as blood pressure plays a key role in human health. Usually, we encounter some problems when we check them in the traditional way; for example, it is impossible to check continuously, and measuring vital signs requires direct contact with the patient, which can be uncomfortable for certain individuals. In this research, we present a vision-based system for estimating blood pressure using pulse transit time (PTT) and the Eulerian video magnification (EVM) technique to amplify tiny color variations caused by blood flow to calculate arterial pulse waves traveling between two arterial sites. Calculating the PTT by processing the video signal for each subject, an oscillometer BP device was used to evaluate the performance between measurements in different conditions, including rest, exercise, and during recovery. Mean systolic BP was 115 mmHg at rest, 137 mmHg during high-intensity exercise, and 114 mmHg during recovery, respectively. The average value of diastolic blood pressure did not change significantly before, during, and after exercise. When we compared the systolic and diastolic blood pressure with ground-truth results, our system showed an accuracy of 91% for systolic blood pressure and 90% for diastolic blood pressure.

Estimation of vital signs from facial videos via video magnification and deep learning

The continuous monitoring of vital signs is one of the hottest topics in healthcare. Recent technological advances in sensors, signal processing, and image processing spawned the development of no-contact techniques such as remote photoplethysmography (rPPG). To solve the common problems of rPPG including weak extracted signals, body movements, and generalization with limited data resources, we proposed a dual-path estimation method based on video magnification and deep learning. First, image processes are applied to detect, track, and magnificate facial ROIs automatically. Then, the steady part of the wave of each processed ROI is used for the extraction of features including heart rate, PTT, and features of pulse wave waveform. The blood pressures are estimated from the features via a small CNN. Results comply with the current standard and promise potential clinical applications in the future.

Predicting Depression Risk from Facial Video-Derived Heart Rate Estimates

Common mental disorder is caused due to depression. Medical study shows that heart rate is linked to depression. Heart rate can give early warning of potential depression. Heart rate could predict the risk of depression. This link helps diagnosis and treatment of mental health issues like depression. Heart rate of healthy person is 60-85 beats per minute. When a person is depressed, his heart rate is not in normal range. Heart rate of depressed human being is increased beyond 85 beats per minute. Depression can be predicted with a 90% accuracy by analyzing a person’s heart rate. Using Eulerian Video Magnification algorithm, it is possible to calculate heart rate of person from facial video. Which gives benefit that no need of physical contact with the person. In the proposed research Heart rate is calculated by inputting face videos. Questionnaire is formed that contains 32 questions useful for depression assessment. Real time video dataset is collected while asking depression questionnaire to the people of all age groups. Using Eulerian Video Magnification algorithm, facial video is amplified and heart rate is estimated. Based on range, dataset is labeled as depressed or not depressed. By applying machine learning algorithms like Decision Tree (DT), Support Vector Machine (SVM) and Random forest (RF), dataset is classified with accuracy ranging from 96% to 100%. The performance of this research when compared with related work carried out for same research purpose, it is observed that accuracy obtained for this research work is 51% to 85.7% till date. This research gives more accurate model with new approach to predict risk of depression using heart rate estimated from facial videos.

A novel video magnification method for revealing subtle signals based on adaptive Laplacian pyramid

Numerous difficult to perceive minor signals, such as minute color variations and modest movements, carry crucial information. The ability to see small signals in video has been demonstrated to be greatly aided by video magnification techniques. The Eulerian video magnification (EVM) method produces excellent results when magnifying minute colors and motions; but, it is susceptible to artifacts and outcomes that are blurry, especially when bigger magnifications are used. In this research, we propose a novel video magnification method based on adaptive Laplacian pyramid to achieve non-contact optical measurements. Experiments were conducted on real video sequences containing biological or mechanical motion signals, the results demonstrated that our approach achieves video amplification and extraction of subtle signals, supports larger amplification factors, and obtains sharper video quality with reduced artifacts and blurring.

Heart Rate Estimation using Hermite Transform Video Magnification and Deep Learning

Abstract: Monitoring of heart rate can be used in many medical and sports applications. Lack of portability and connection problems make traditional monitoring methods difficult to use outside of clinical environments. The computer vision techniques have shown that some physiological variables as heart rate can be measured without contact. Video magnification is one of these approaches used for the detection of the pulse signal. In this paper we propose a new strategy to magnify motion in a video sequence using the Hermite transform. In addition, a deep learning technique is implemented to estimate the beat-by-beat pulse signal. We trained the system and validated our results using an electronic pulse monitoring device. Our approach is compared with the classical video magnification using a Gaussian pyramid. The results show a better enhancement of spectral information from the color changes allowing an accurate estimation of the instantaneous beat by beat pulse than the Gaussian approach.

A Study of Machine Learning Regression Techniques for Non-Contact SpO2 Estimation from Infrared Motion-Magnified Facial Video

This work explores the use of infrared low-cost cameras for monitoring peripheral oxygen saturation (SpO2), a vital sign that is particularly important for individuals with fragile health, such as the elderly. The development of contactless SpO2 monitoring utilizing RGB cameras has already proven successful. This study utilizes the Eulerian Video Magnification (EVM) technique to enhance minor variations in skin pixel intensity in particular facial regions. More specifically, the emphasis in this study is in the utilization of infrared cameras, in order to explore the possibility of contactless SpO2 monitoring under low-light or night-time conditions. Many different methods were employed for regression. A study of machine learning regression methods was performed, including a Generalized Additive Model (GAM) and an Extra Trees Regressor, based on 12 novel features extracted from the extracted amplified photoplethysmography (PPG) signal. Deep learning methods were also explored, including a 3D Convolution Neural Network (CNN) and a Video Vision Transformer (ViViT) architecture on the amplified forehead/cheeks video. The estimated SpO2 values of the best performing method reach a low root mean squared error of 1.331 and an R2 score of 0.465 that fall within the acceptable range for these applications.

Contactless Camera-Based Heart Rate and Respiratory Rate Monitoring Using AI on Hardware.

Detecting vital signs by using a contactless camera-based approach can provide several advantages over traditional clinical methods, such as lower financial costs, reduced visit times, increased comfort, and enhanced safety for healthcare professionals. Specifically, Eulerian Video Magnification (EVM) or Remote Photoplethysmography (rPPG) methods can be utilised to remotely estimate heart rate and respiratory rate biomarkers. In this paper two contactless camera-based health monitoring architectures are developed using EVM and rPPG, respectively; to this end, two different CNNs, (Mediapipe's BlazeFace and FaceMesh) are used to extract suitable regions of interest from incoming video frames. These two methods are implemented and deployed on four off-the-shelf edge devices as well as on a PC and evaluated in terms of latency (in each stage of the application's pipeline), throughput (FPS), power consumption (Watt), efficiency (throughput/Watt), and value (throughput/cost). This work provides important insights about the computational costs and bottlenecks of each method on each hardware platform, as well as which platform to use depending on the target metric. One of our insights shows that the Jetson Xavier NX platform is the best platform in terms of throughput and efficiency, while Raspberry Pi 4 8 GB is the best platform in terms of value.

Vision-based estimation of MDS-UPDRS scores for quantifying Parkinson's disease tremor severity.

Parkinson's disease (PD) is a common neurodegenerative movement disorder among older individuals. As one of the typical symptoms of PD, tremor is a critical reference in the PD assessment. A widely accepted clinical approach to assessing tremors in PD is based on part III of the Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). However, expert assessment of tremor is a time-consuming and laborious process that poses considerable challenges to the medical evaluation of PD. In this paper, we proposed a novel model, Global Temporal-difference Shift Network (GTSN), to estimate the MDS-UPDRS score of PD tremors based on video. The PD tremor videos were scored according to the majority vote of multiple raters. We used Eulerian Video Magnification (EVM) pre-processing to enhance the representations of subtle PD tremors in the videos. To make the model better focus on the tremors in the video, we proposed a special temporal difference module, which stacks the current optical flow to the result of inter-frame difference. The prediction scores were obtained from the Residual Networks (ResNet) embedded with a novel module, the Global Shift Module (GSM), which allowed the features of the current segment to include the global segment features. We carried out independent experiments using PD tremor videos of different body parts based on the scoring content of the MDS-UPDRS. On a fairly large dataset, our method achieved an accuracy of 90.6% for hands with rest tremors, 85.9% for tremors in the leg, and 89.0% for the jaw. An accuracy of 84.9% was obtained for postural tremors. Our study demonstrated the effectiveness of computer-assisted assessment for PD tremors based on video analysis. The latest version of the code is available at https://github.com/199507284711/PD-GTSN.

Non-invasive measurements of respiration and heart rate across wildlife species using Eulerian Video Magnification of infrared thermal imagery

BackgroundAn animal’s metabolic rate, or energetic expenditure, both impacts and is impacted by interactions with its environment. However, techniques for obtaining measurements of metabolic rate are invasive, logistically difficult, and costly. Red–green–blue (RGB) imaging tools have been used in humans and select domestic mammals to accurately measure heart and respiration rate, as proxies of metabolic rate. The purpose of this study was to investigate if infrared thermography (IRT) coupled with Eulerian video magnification (EVM) would extend the applicability of imaging tools towards measuring vital rates in exotic wildlife species with different physical attributes.ResultsWe collected IRT and RGB video of 52 total species (39 mammalian, 7 avian, 6 reptilian) from 36 taxonomic families at zoological institutions and used EVM to amplify subtle changes in temperature associated with blood flow for respiration and heart rate measurements. IRT-derived respiration and heart rates were compared to ‘true’ measurements determined simultaneously by expansion of the ribcage/nostrils and stethoscope readings, respectively. Sufficient temporal signals were extracted for measures of respiration rate in 36 species (85% success in mammals; 50% success in birds; 100% success in reptiles) and heart rate in 24 species (67% success in mammals; 33% success in birds; 0% success in reptiles) using IRT-EVM. Infrared-derived measurements were obtained with high accuracy (respiration rate, mean absolute error: 1.9 breaths per minute, average percent error: 4.4%; heart rate, mean absolute error: 2.6 beats per minute, average percent error: 1.3%). Thick integument and animal movement most significantly hindered successful validation.ConclusionThe combination of IRT with EVM analysis provides a non-invasive method to assess individual animal health in zoos, with great potential to monitor wildlife metabolic indices in situ.

Non-Contact Breathing Rate Estimation Using Machine Learning with an Optimized Architecture

The breathing rate monitoring is an important measure in medical applications and daily physical activities. The contact sensors have shown their effectiveness for breathing monitoring and have been mostly used as a standard reference, but with some disadvantages for example in burns patients with vulnerable skins. Contactless monitoring systems are then gaining attention for respiratory frequency detection. We propose a new non-contact technique to estimate the breathing rate based on the motion video magnification method by means of the Hermite transform and an Artificial Hydrocarbon Network (AHN). The chest movements are tracked by the system without the use of an ROI in the image video. The machine learning system classifies the frames as inhalation or exhalation using a Bayesian-optimized AHN. The method was compared using an optimized Convolutional Neural Network (CNN). This proposal has been tested on a Data-Set containing ten healthy subjects in four positions. The percentage error and the Bland–Altman analysis is used to compare the performance of the strategies estimating the breathing rate. Besides, the Bland–Altman analysis is used to search for the agreement of the estimation to the reference.The percentage error for the AHN method is 2.19±2.1 with and agreement with respect of the reference of ≈99%.

The estimation of broiler respiration rate based on the semantic segmentation and video amplification

Respiratory rate is an indicator of a broilers’ stress and health status, thus, it is essential to detect respiratory rate contactless and stress-freely. This study proposed an estimation method of broiler respiratory rate by deep learning and machine vision. Experiments were performed at New Hope (Shandong Province, P. R. China) and Wen’s group (Guangdong Province, P. R. China), and a total of 300 min of video data were collected. By separating video frames, a data set of 3,000 images was made, and two semantic segmentation models were trained. The single-channel Euler video magnification algorithm was used to amplify the belly fluctuation of the broiler, which saved 55% operation time compared with the traditional Eulerian video magnification algorithm. The contour features significantly related to respiration were used to obtain the signals that could estimate broilers’ respiratory rate. Detrending and band-pass filtering eliminated the influence of broiler posture conversion and motion on the signal. The mean absolute error, root mean square error, average accuracy of the proposed respiratory rate estimation technique for broilers were 3.72%, 16.92%, and 92.19%, respectively.

Remote Heart Rate Prediction in Virtual Reality Head-Mounted Displays Using Machine Learning Techniques

Head-mounted displays are virtual reality devices that may be equipped with sensors and cameras to measure a patient's heart rate through facial regions. Heart rate is an essential body signal that can be used to remotely monitor users in a variety of situations. There is currently no study that predicts heart rate using only highlighted facial regions; thus, an adaptation is required for beats per minute predictions. Likewise, there are no datasets containing only the eye and lower face regions, necessitating the development of a simulation mechanism. This work aims to remotely estimate heart rate from facial regions that can be captured by the cameras of a head-mounted display using state-of-the-art EVM-CNN and Meta-rPPG techniques. We developed a region of interest extractor to simulate a dataset from a head-mounted display device using stabilizer and video magnification techniques. Then, we combined support vector machine and FaceMash to determine the regions of interest and adapted photoplethysmography and beats per minute signal predictions to work with the other techniques. We observed an improvement of 188.88% for the EVM and 55.93% for the Meta-rPPG. In addition, both models were able to predict heart rate using only facial regions as input. Moreover, the adapted technique Meta-rPPG outperformed the original work, whereas the EVM adaptation produced comparable results for the photoplethysmography signal.

Vision-Based Structural Modal Identification Using Hybrid Motion Magnification

As a promising alternative to conventional contact sensors, vision-based technologies for a structural dynamic response measurement and health monitoring have attracted much attention from the research community. Among these technologies, Eulerian video magnification has a unique capability of analyzing modal responses and visualizing modal shapes. To reduce the noise interference and improve the quality and stability of the modal shape visualization, this study proposes a hybrid motion magnification framework that combines linear and phase-based motion processing. Based on the assumption that temporal variations can represent spatial motions, the linear motion processing extracts and manipulates the temporal intensity variations related to modal responses through matrix decomposition and underdetermined blind source separation (BSS) techniques. Meanwhile, the theory of Fourier transform profilometry (FTP) is utilized to reduce spatial high-frequency noise. As all spatial motions in a video are linearly controllable, the subsequent phase-based motion processing highlights the motions and visualizes the modal shapes with a higher quality. The proposed method is validated by two laboratory experiments and a field test on a large-scale truss bridge. The quantitative evaluation results with high-speed cameras demonstrate that the hybrid method performs better than the single-step phase-based motion magnification method in visualizing sound-induced subtle motions. In the field test, the vibration characteristics of the truss bridge when a train is driving across the bridge are studied with a commercial camera over 400 m away from the bridge. Moreover, four full-field modal shapes of the bridge are successfully observed.

EXPLORATORY IN-CLINIC STUDY EVALUATING FEASIBILITY OF JVP HEIGHT ASSESSMENT BY EARLY PROTOTYPE OF MACHINE VISION ALGORITHM

EXPLORATORY IN-CLINIC STUDY EVALUATING FEASIBILITY OF JVP HEIGHT ASSESSMENT BY EARLY PROTOTYPE OF MACHINE VISION ALGORITHM

Assessing Hand Perfusion With Eulerian Video Magnification and Waveform Extraction

Timely and accurate triage of upper extremity injuries is critical, but current perfusion monitoring technologies have shortcomings. These limitations are especially pronounced in patients with darker skin tones. This pilot study evaluates a Eulerian Video Magnification (EVM) algorithm combined with color channel waveform extraction to enable video-based measurement of hand and finger perfusion. Videos of 10 volunteer study participants with Fitzpatrick skin types III-VI were taken in a controlled environment during normal perfusion and tourniquet-induced ischemia. Videos were EVM processed, and red/green/blue color channel characteristics were extracted to produce waveforms. These videos were assessed by surgeons with a range of expertise in hand injuries. The videos were randomized and presented in 1 of 3 ways: unprocessed, EVM processed, and EVM with waveform output (EVM+waveform). Survey respondents indicated whether the video showed an ischemic or perfused hand or if they were unable to tell. We used group comparisons to evaluate response accuracy across video types, skin tones, and respondent groups. Of the 51 providers to whom the surveys were sent, 25 (49%) completed them. Using the Pearson χ2 test, the frequencies of correct responses were significantly higher in the EVM+waveform category than in the unprocessed or EVM videos. Additionally, the agreement was higher among responses to the EVM+waveform questions than among responses to the unprocessed or EVM processed. The accuracy and agreement from the EVM+waveform group were consistent across all skin pigmentations evaluated. Video-based EVM processing combined with waveform extraction from color channels improved the surgeon's ability to identify tourniquet-induced finger ischemia via video across all skin types tested. Eulerian Video Magnification with waveform extraction improved the assessment of perfusion in the distal upper extremity and has potential future applications, including triage, postsurgery vascular assessment, and telemedicine.

Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature

Estimating heart rate is important for monitoring users in various situations. Estimates based on facial videos are increasingly being researched because they allow the monitoring of cardiac information in a non-invasive way and because the devices are simpler, as they require only cameras that capture the user’s face. From these videos of the user’s face, machine learning can estimate heart rate. This study investigates the benefits and challenges of using machine learning models to estimate heart rate from facial videos through patents, datasets, and article review. We have searched the Derwent Innovation, IEEE Xplore, Scopus, and Web of Science knowledge bases and identified seven patent filings, eleven datasets, and twenty articles on heart rate, photoplethysmography, or electrocardiogram data. In terms of patents, we note the advantages of inventions related to heart rate estimation, as described by the authors. In terms of datasets, we have discovered that most of them are for academic purposes and with different signs and annotations that allow coverage for subjects other than heartbeat estimation. In terms of articles, we have discovered techniques, such as extracting regions of interest for heart rate reading and using video magnification for small motion extraction, and models, such as EVM-CNN and VGG-16, that extract the observed individual’s heart rate, the best regions of interest for signal extraction, and ways to process them.