Phase-based Video Magnification Research Articles

Phase-based video magnification on a single board computer

Phase based video magnification is currently one of the best motion magnifying algorithms available. Unlike the linear video magnification which preceded it, however, it is significantly more computationally demanding. In this work we present a modified version of this algorithm which we developed in C++ to be usable on low powered platforms such as the Raspberry Pi.

Video Magnification Techniques: Medical Applications and Comparison of Methods

The unassisted visual system cannot note minute temporal variations in video and image sequences. In many applications, these differences and small signals are highly informative. A new technique used to expose video variations by measuring and amplifying video variations over time in a fixed position (pixel) was used to Eulerian video magnification (EVM). The objective of the study is to investigate and evaluate different processes for the creation and testing of EVM techniques and video quality parameters for each one of those methods. This research employed four new methods; EVM, Riesz pyramid for fast phase-based video magnification (FPBM), phase-based video magnification (PBM), and Enhanced Eulerian video magnification (E2VM). The experimental findings compared with their output for certain enlargement methods; time and quality parameters of image. A new magnification method is required based on the study of the exiting methods, which takes account of noise elimination, video quality and time reduction.

Manifestation of Flexural Vibration Modes of Rails by the Phase-Based Magnification Method

With the recent development of high performance photography devices, vibration measurements using digital image processing can be utilized for the condition monitoring and the analysis of generation mechanisms. High-quality image recording provides very dense spatial information unlike physical sensors with a limited number of point-oriented measurements and output channels such as accelerometers. Moreover, image acquisition is a non-contact technique and ensures that the installed sensors do not affect the mass or stiffness. In this study, the vibration mode of rails is investigated using the phase-based video magnification method. An extensive amount of information that cannot be detected by human eyes, can be obtained with this imaging motion magnification technique by amplifying and visualising the minute vibrational movements. Flexural vibration modes in rails generate rolling noise, especially at high frequencies where the displacement amplitudes are small. The motion magnification method is applied using a high-speed camera to measure the rail mode shapes. The measured mode shapes are compared to the numerical results obtained from the waveguide finite element method. This method helps in understanding the vibration generation and the transfer mechanisms in the waveguide structures with a uniform cross-section in the lengthwise direction such as a railway track.

Local Riesz Pyramid for Faster Phase-Based Video Magnification

Phase-based video magnification methods can magnify and reveal subtle motion changes invisible to the naked eye. In these methods, each image frame in a video is decomposed into an image pyramid, and subtle motion changes are then detected as local phase changes with arbitrary orientations at each pixel and each pyramid level. One problem with this process is a long computational time to calculate the local phase changes, which makes high-speed processing of video magnification difficult. Recently, a decomposition technique called the Riesz pyramid has been proposed that detects only local phase changes in the dominant orientation. This technique can remove the arbitrariness of orientations and lower the over-completeness, thus achieving high-speed processing. However, as the resolution of input video increases, a large amount of data must be processed, requiring a long computational time. In this paper, we focus on the correlation of local phase changes between adjacent pyramid levels and present a novel decomposition technique called the local Riesz pyramid that enables faster phase-based video magnification by automatically processing the minimum number of sufficient local image areas at several pyramid levels. Through this minimum pyramid processing, our proposed phase-based video magnification method using the local Riesz pyramid achieves good magnification results within a short computational time.

Detection of the respiratory rate of standing cows by combining the Deeplab V3+ semantic segmentation model with the phase-based video magnification algorithm

The respiratory rate (RR) of dairy cows is closely related to heat stress and reproductive diseases. In this study, a method was proposed to detect the RR of standing resting dairy cows by using deep learning and video analysis. First, a Deeplab V3+ semantic segmentation model was developed using the framework of ResNet-101, and the model was trained and tested using 3,000 manually labelled images to realise the segmentation of the cows. Next, a phase-based video magnification (PBVM) algorithm was applied to amplify the weak breathing movements of the cow. Finally, the Lucas-Kanade (LK) optical flow algorithm was used to detect breathing direction, and an RR detection model was used to detect the RR. The effectiveness of the method was verified by testing a total of 67,812 frames of data collected from 70 video segments of different cows. A comparison with the original video LK optical flow method (Method 1), target segmentation video LK optical flow method (Method 2), and original video motion amplification LK optical flow method (Method 3) showed that the accuracy of the proposed method was between 80.00% and 100.00%, and the mean accuracy was 93.04%. These values were 45.82%, 17.72%, and 22.94% higher than those of the other three methods, respectively. In addition, the method was tested in 105 videos of cows under heat stressed, and the RR detection accuracy was 98.69%. The results showed that the method could provide a reference for the automatic detection of RR and remote diagnosis of respiratory-related diseases.

Analysis of Soot Propensity in Combustion Processes Using Optical Sensors and Video Magnification.

Industrial combustion processes are an important source of particulate matter, causing significant pollution problems that affect human health, and are a major contributor to global warming. The most common method for analyzing the soot emission propensity in flames is the Smoke Point Height (SPH) analysis, which relates the fuel flow rate to a critical flame height at which soot particles begin to leave the reactive zone through the tip of the flame. The SPH and is marked by morphological changes on the flame tip. SPH analysis is normally done through flame observations with the naked eye, leading to high bias. Other techniques are more accurate, but are not practical to implement in industrial settings, such as the Line Of Sight Attenuation (LOSA), which obtains soot volume fractions within the flame from the attenuation of a laser beam. We propose the use of Video Magnification techniques to detect the flame morphological changes and thus determine the SPH minimizing observation bias. We have applied for the first time Eulerian Video Magnification (EVM) and Phase-based Video Magnification (PVM) on an ethylene laminar diffusion flame. The results were compared with LOSA measurements, and indicate that EVM is the most accurate method for SPH determination.

An efficient motion magnification system for real-time applications

The human eye cannot see subtle motion signals that fall outside human visual limits, due to either limited resolution of intensity variations or lack of sensitivity to lower spatial and temporal frequencies. Yet, these invisible signals can be highly informative when amplified to be observable by a human operator or an automatic machine vision system. Many video magnification techniques have recently been proposed to magnify and reveal these signals in videos and image sequences. Limitations, including noise level, video quality and long execution time, are associated with the existing video magnification techniques. Therefore, there is value in developing a new magnification method where these issues are the main consideration. This study presents a new magnification method that outperforms other magnification techniques in terms of noise removal, video quality at large magnification factor and execution time. The proposed method is compared with four methods, including Eulerian video magnification, phase-based video magnification, Riesz pyramid for fast phase-based video magnification and enhanced Eulerian video magnification. The experimental results demonstrate the superior performance of the proposed magnification method regarding all video quality metrics used. Our method is also 60–70% faster than Eulerian video magnification, whereas other competing methods take longer to execute than Eulerian video magnification.

Feasibility of extracting operating shapes using phase-based motion magnification technique and stereo-photogrammetry

Optically-based vibration measurements can be used to determine the full-field dynamic response of structures without the need of mounted transducers (e.g. accelerometers, strain gages, and LVDTs) that can add cost, have wiring, power signal transmission issues, and may affect the dynamic characteristics by adding mass or stiffness. The three-dimensional digital image correlation (3D DIC) and three-dimensional point tracking (3DPT) methods based on a stereo-vision system can provide the full-field dynamic displacements of a structure with sub-pixel accuracy. However, stereo-photogrammetry systems are limited by camera resolution and intrinsic noise of the acquired images. Moreover, exciting a structure with only a small amount of energy may result in a subtle mechanical response that is not perceptible by observing the raw data of the conventional optical measurement systems (i.e. 3D DIC and 3DPT). Thus, in order to use the optical-sensing techniques to identify dynamic characteristics of a structure at high frequencies, the signal-to-noise ratio (SNR) in the sequence of images captured with a stereo-vision system needs to be improved. Within this paper, phase-based video magnification in conjunction with the stereo-photogrammetry techniques are used to measure the higher-frequency operating shapes of a cantilever beam and a 2.3-m long Skystream 4.7 wind turbine blade. The phase-based motion estimation method uses complex steerable pyramids to decompose the original sequence of images to amplitude and phase at different spatial resolutions (subsampled images). Motion information is conserved within the phase values which can be filtered in the time domain and amplified to detect the subtle motions. The magnified sequence of images using the motion magnification technique are post-processed using 3D DIC or 3DPT to quantify infinitesimal deformations that are not recognizable using only the traditional stereo-photogrammetry methods. The results obtained within this paper reveal the great potential of motion magnification in conjunction with stereo-photogrammetry techniques in detecting small motions of structures and extracting 3D operating shapes from the optically measured data with low SNR. Compared to traditional stereo-photogrammetry, additional modes of the test structures are obtainable by means of combining the phase based motion magnification and conventional image-sensing techniques (i.e. 3D DIC and 3DPT). The proposed methodology could be used to enhance the established procedure to extract more information about the dynamic characteristics of structures (only once effectively calibrated), compared to existing image-sensing techniques and contact measurement sensors.

Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video

Hilar dissection is an important and delicate stage in partial nephrectomy, during which surgeons remove connective tissue surrounding renal vasculature. Serious complications arise when the occluded blood vessels, concealed by fat, are missed in the endoscopic view and as a result are not appropriately clamped. Such complications may include catastrophic blood loss from internal bleeding and associated occlusion of the surgical view during the excision of the cancerous mass (due to heavy bleeding), both of which may compromise the visibility of surgical margins or even result in a conversion from a minimally invasive to an open intervention. To aid in vessel discovery, we propose a novel automatic method to segment occluded vasculature from labeling minute pulsatile motion that is otherwise imperceptible with the naked eye. Our segmentation technique extracts subtle tissue motions using a technique adapted from phase-based video magnification, in which we measure motion from periodic changes in local phase information albeit for labeling rather than magnification. Based on measuring local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs, our approach assigns segmentation labels by detecting regions exhibiting temporal local phase changes matching the heart rate. We demonstrate how our technique is a practical solution for time-critical surgical applications by presenting quantitative and qualitative performance evaluations of our vessel detection algorithms with a retrospective study of fifteen clinical robot-assisted partial nephrectomies.

Joint view expansion and filtering for automultiscopic 3D displays

Multi-view autostereoscopic displays provide an immersive, glasses-free 3D viewing experience, but they require correctly filtered content from multiple viewpoints. This, however, cannot be easily obtained with current stereoscopic production pipelines. We provide a practical solution that takes a stereoscopic video as an input and converts it to multi-view and filtered video streams that can be used to drive multi-view autostereoscopic displays. The method combines a phase-based video magnification and an interperspective antialiasing into a single filtering process. The whole algorithm is simple and can be efficiently implemented on current GPUs to yield a near real-time performance. Furthermore, the ability to retarget disparity is naturally supported. Our method is robust and works well for challenging video scenes with defocus blur, motion blur, transparent materials, and specularities. We show that our results are superior when compared to the state-of-the-art depth-based rendering methods. Finally, we showcase the method in the context of a real-time 3D videoconferencing system that requires only two cameras.