Sequence Of Image Frames Research Articles

A Real‐Time and Long‐Term Face Tracking Method Using Convolutional Neural Network and Optical Flow in IoT‐Based Multimedia Communication Systems

The development of the Internet of Things (IoT) stimulates many research works related to Multimedia Communication Systems (MCS), such as human face detection and tracking. This trend drives numerous progressive methods. Among these methods, the deep learning‐based methods can spot face patch in an image effectively and accurately. Many people consider face tracking as face detection, but they are two different techniques. Face detection focuses on a single image, whose shortcoming is obvious, such as unstable and unsmooth face position when adopted on a sequence of continuous images; computing is expensive due to its heavy reliance on Convolutional Neural Networks (CNNs) and limited detection performance on the edge device. To overcome these defects, this paper proposes a novel face tracking strategy by combining CNN and optical flow, namely, C‐OF, which achieves an extremely fast, stable, and long‐term face tracking system. Two key things for commercial applications are the stability and smoothness of face positions in a sequence of image frames, which can provide more probability for face biological signal extraction, silent face antispoofing, and facial expression analysis in the fields of IoT‐based MCS. Our method captures face patterns in every two consequent frames via optical flow to get rid of the unstable and unsmooth problems. Moreover, an innovative metric for measuring the stability and smoothness of face motion is designed and adopted in our experiments. The experimental results illustrate that our proposed C‐OF outperforms both face detection and object tracking methods.

Crop Yield Prediction Using Multitemporal UAV Data and Spatio-Temporal Deep Learning Models

Unmanned aerial vehicle (UAV) based remote sensing is gaining momentum worldwide in a variety of agricultural and environmental monitoring and modelling applications. At the same time, the increasing availability of yield monitoring devices in harvesters enables input-target mapping of in-season RGB and crop yield data in a resolution otherwise unattainable by openly availabe satellite sensor systems. Using time series UAV RGB and weather data collected from nine crop fields in Pori, Finland, we evaluated the feasibility of spatio-temporal deep learning architectures in crop yield time series modelling and prediction with RGB time series data. Using Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM) networks as spatial and temporal base architectures, we developed and trained CNN-LSTM, convolutional LSTM and 3D-CNN architectures with full 15 week image frame sequences from the whole growing season of 2018. The best performing architecture, the 3D-CNN, was then evaluated with several shorter frame sequence configurations from the beginning of the season. With 3D-CNN, we were able to achieve 218.9 kg/ha mean absolute error (MAE) and 5.51% mean absolute percentage error (MAPE) performance with full length sequences. The best shorter length sequence performance with the same model was 292.8 kg/ha MAE and 7.17% MAPE with four weekly frames from the beginning of the season.

Reliability Analysis of Multi-Objective Spatio-Temporal Segmentation of Human Motion in Video Sequences

In view of the problem of uneven distribution of edge contour of multi-target human motion image in video sequence, which leads to the decline of target detection ability, an algorithm of multi-target spatial-temporal segmentation of human motion in video sequence based on edge contour feature detection and block fusion is proposed. Firstly, a multi-target spatial-temporal detection model of human motion in video sequence was constructed, extracting video image frame sequence, using discrete frame fusion method to segment and fuse moving target image, matching moving multi-target in video sequence, secondly segmenting motion features in moving target image, combining with SURF algorithm (speeded up robust features, accelerated robust features) to detect and extract human motion objects in video sequence. The experimental results show that the gray histogram of human motion multi-target space-time segmentation is close to the original image histogram, and the detection and recognition ability of human motion target is improved.

Exploring biological motion perception in two-stream convolutional neural networks

Visual recognition of biological motion recruits form and motion processes supported by both dorsal and ventral pathways. This neural architecture inspired the two-stream convolutional neural network (CNN) model, which includes a spatial CNN to process appearance information in a sequence of image frames, a temporal CNN to process optical flow information, and a fusion network to integrate the features extracted by the two CNNs and make final decisions about action recognition. In five simulations, we compared the CNN model's performance with classical findings in biological motion perception. The CNNs trained with raw RGB action videos showed weak performance in recognizing point-light actions. Additional transfer training with actions shown in other display formats (e.g., skeletal) was necessary for CNNs to recognize point-light actions. The CNN models exhibited largely viewpoint-dependent recognition of actions, with a limited ability to generalize to viewpoints close to the training views. The CNNs predicted the inversion effect in the presence of global body configuration, but failed to predict the inversion effect driven solely by local motion signals. The CNNs provided a qualitative account of some behavioral results observed in human biological motion perception for fine discrimination tasks with noisy inputs, such as point-light actions with disrupted local motion signals, and walking actions with temporally misaligned motion cues. However, these successes are limited by the CNNs’ lack of adaptive integration for form and motion processes, and failure to incorporate specialized mechanisms (e.g., a life detector) as well as top-down influences on biological motion perception.

Monitoring ambient vibration pollution based on visual information perception and neural network analysis

As a type of pollution, low-frequency vibrations in the environment have a negative effect on human life. To monitor pollution without extra complications, we propose a novel method based on visual information perception and neural network analysis. Image frame sequences can capture the state of objects, and changes in pixel values at salient vibration points may indirectly reflect ambient vibrations. In this paper, a test environment is established to simulate the influence of vibrations on daily life. A CCD camera is used to continuously sample objects via image frame sequences to monitor potential vibration pollution in the current environment through image processing and neural network analysis. During the image processing stage, a combination of image filtering and edge extraction can accurately locate the test objects’ contours. During the low-frequency vibration monitoring stage, the neural network analyzes the changes in pixel values at the contour points to monitor ambient vibration pollution. The network combines local and global features for vibration frequency classification and prediction. The proposed method's superiority is verified by the test results and a comparison of four different networks. The results demonstrate that this method accurately locates salient vibration points to monitor ambient vibration pollution.

Survey on Object Tracking using Deep Learning Paradigms

The field of object tracking extends across different domains. It is a major key player in the field of image processing and pattern recognition. Object tracking is the process of tracking an object over a continuous sequence of image frames to determine over time the relative movements or changes. With the massive advancements in the field of deep learning, the use of deep neural networks has risen due to their impressive accomplishments in object detection and tracking. In this Survey, the objective is to give a comprehensive overview of the recent attempts in the field of object tracking with a focus on the use of deep learning techniques and algorithms. The paper is divided into four sections; at first, we will give an overview of the recent work to highlight the techniques and methods which have been used in object tracking using deep learning. The second section focus is on the object tracking that uses convolutional networks techniques. The third section focuses on some of the recurrent neural networks to tack objects. The final section is concentrated on auto-encoders object tracking.

Optical Flow Methods for Lung Nodule Segmentation on LIDC-IDRI Images.

Lung nodule segmentation is an essential step in any CAD system for lung cancer detection and diagnosis. Traditional approaches for image segmentation are mainly morphology based or intensity based. Motion-based segmentation techniques tend to use the temporal information along with the morphology and intensity information to perform segmentation of regions of interest in videos. CT scans comprise of a sequence of dicom 2-D image slices similar to videos which also comprise of a sequence of image frames ordered on a timeline. In this work, Farneback, Horn-Schunck and Lucas-Kanade optical flow methods have been used for processing the dicom slices. The novelty of this work lies in the usage of optical flow methods, generally used in motion-based segmentation tasks, for the segmentation of nodules from CT images. Since thin-sliced CT scans are the imaging modality considered, they closely approximate the motion videos and are the primary motivation for using optical flow for lung nodule segmentation. This paper also provides a detailed comparative analysis and validates the effectiveness of using optical flow methods for segmentation. Finally, we propose methods to further improve the efficiency of segmentation using optical flow methods on CT scans.

Recent Developments in Dual-Laser Digital Holography for Plasma-Facing Surface Characterization

A digital holography device is currently undergoing development at the Oak Ridge National Laboratory for the purpose of measuring surface topography, with the goal of deployment as a real-time plasma-facing component diagnostic for the study of materials that could be utilized in a nuclear fusion device. The holography system utilizes one or two lasers depending on the scale of surface features under measurement. Measurements of surface roughness were performed in a single-laser mode and compared with the data from profilometry, with a linear correlation of increased holographic measurement fidelity as surfaces became smoother. Characterization of the dual-laser operating mode has been performed via surface measurement of stainless steel targets with “stair-step” features in various sizes. Results demonstrated that surface features with known sizes as small as 25.4 μm could be resolved. Measurements were within ~55 μm or less deviation from the actual sizes, and measurement accuracy was improved as feature size was increased, corresponding to the effect of noise becoming less pronounced. A target exposed to plasma generated by an electrothermal (ET) arc source was analyzed with flatfield correction and averaging of sequential image frames to demonstrate the improved measurement quality in preparation for future use of holography on ET arc-exposed targets.

Vision-based Tracking by a Quadrotor on ROS

Vision-based tracking of a moving target is achieved in three steps: image processing, coordinate recovery and trajectory following. A testing platform containing two spawned quadrotors as the target and the tracker is developed on ROS, from which sequences of image frames are generated by an on-board camera while the target moves along a given path on the ground. A simple image processing algorithm is developed that achieves visual tracking and is robust to image blur and other background noises. Coordinate recovery recovers the original target trajectory as the desired path of the tracker in the real-world. A target following control law moves the tracker along the desired trajectory point-by-point. An analysis of the robustness of the tracking approach with respect to image blur and measurement noise of the tracker position is carried out to assure the applicability of the approach to real-world scenarios.

Kernelized Fuzzy Modal Variation for Local Change Detection From Video Scenes

Background subtraction (BGS) is a popular scheme epitomized in the state-of-the-art literature on video processing. In this context, a novel online kernelized fuzzy modal variation based background subtraction scheme for detecting local changes from the sequences of image frames is proposed. In the proposed scheme, the time varying background at different instances of time are modeled using fuzzy set theory. The proposed background subtraction scheme, utilizes the fuzzy modal variation as the cost function for fitting the pixel values of the image frames. The use of kernel based modal variation helps in projecting the pixel values in a higher dimensional space, linearly separating them into object and background classes. The results of the proposed technique is verified on different challenging sequences including dynamic background, camera jitter, noise, blurred scene, etc. The proposed technique is successfully tested over several test sequences with two major databases (all sequences) and it provides better results compared to the twenty one existing state-of-the-art techniques.

A Deep Learning System for Recognizing Facial Expression in Real-Time

This article presents an image-based real-time facial expression recognition system that is able to recognize the facial expressions of several subjects on a webcam at the same time. Our proposed methodology combines a supervised transfer learning strategy and a joint supervision method with center loss, which is crucial for facial tasks. A newly proposed Convolutional Neural Network (CNN) model, MobileNet, which has both accuracy and speed, is deployed in both offline and in a real-time framework that enables fast and accurate real-time output. Evaluations towards two publicly available datasets, JAFFE and CK+, are carried out respectively. The JAFFE dataset reaches an accuracy of 95.24%, while an accuracy of 96.92% is achieved on the 6-class CK+ dataset, which contains only the last frames of image sequences. At last, the average run-time cost for the recognition of the real-time implementation is around 3.57ms/frame on a NVIDIA Quadro K4200 GPU.

Assessment of In Vivo Lumbar Inter-Vertebral Motion: Reliability of a Novel Dynamic Weight-Bearing Magnetic Resonance Imaging Technique Using a Side-Bending Task.

Study DesignBetween-session reliability of a magnetic resonance imaging (MRI) based experimental technique to quantify lumbar inter-vertebral motion in humans.PurposeWe have developed a novel, dynamic, MRI-based approach for quantifying in vivo lumbar inter-vertebral motion. In this study, we present the protocol’s reliability results to quantify inter-vertebral spine motion.Overview of LiteratureMorphometric studies on intervertebral displacements using static, supine MRI and quantification of dynamic spine motion using different X-ray based radiography techniques are commonly found in the literature. However, reliability testing of techniques assessing real-time lumbar intervertebral motion using weight-bearing MRI has rarely been reported.MethodsTen adults without a history of back pain performed a side-bending task on two separate occasions, inside an open-MRI, in a weight-bearing, upright position. The images were acquired during the task using a dynamic magnetic resonance (MR) sequence. The MRI imaging space was externally calibrated before the study to recreate the imaging volume for subsequent use in an animation software. The dynamic MR images were processed to create side-bending movement animations in the virtual environment. Participant-specific three-dimensional models were manually superimposed over vertebral image silhouettes in a sequence of image frames, representing the motion trials. Inter-vertebral axes and translation and rotational displacements of vertebrae were quantified using the animation software.ResultsQuantification of inter-vertebral rotations and translations shows high reliability. Between-session reliability results yielded high values for the intra-class correlation coefficient (0.86–0.93), coefficient of variation (13.3%–16.04%), and Pearson’s correlation coefficients (0.89–0.98).ConclusionsThis technique may be developed further to improve its speed and accuracy for diagnostic applications, to study in vivo spine stability, and to assess outcomes of surgical and non-surgical interventions applied to manage pathological spine motion.

Resting nailfold capillary blood flow in primary open-angle glaucoma

Background/AimsAn altered haemodynamic profile for various ocular posterior segment capillary beds has been documented in primary open-angle glaucoma (POAG). POAG may also involve abnormal non-ocular blood flow, and the nailfold...

Robust Optical Flow Estimation Using Tchebichef Moment Invariant Feature

In this paper, orthogonal moment invariant-based features are used to compute 2D optical flow from sequence of images. The gray level of pixel is described in terms of its local neighborhood using Tchebichef moment invariants rather than its individual intensity value. The description is then normalized in order to make it insensitive to intensity fluctuations due to noise or other perturbations such as varying illumination conditions. The principle of conservation of moment invariance is used to derive overdetermined system of 2D motion constraint equations in local neighborhood of each pixel. The velocity field is then estimated using the least square method. Experimental results are performed on sequential video and thermal image frames under varying environmental conditions. The run time, robustness against noisy and varying illumination conditions and rotation invariance of proposed method are compared with already existing moment and non-moment-based techniques.

A Method for Localizing the Eye Pupil for Point-of-Gaze Estimation

The estimation of the point of gaze in a scene presented on a digital screen has many applications, such as fatigue detection and attention tracking. Some popular applications of eye tracking through gaze estimation are depicted in Fig. 1. When estimating the point of gaze, indentifying the visual focus of a person within a scene is required. This is known as the eye fix or point of fixation. Finding the point of gaze involves tracking different features of human eyes. Various methods are available for eye tracking, some of which use special contact lenses, whereas others focus on electrical potential measurements. Optical tracking is a nonintrusive technique that uses a sequence of image frames of eyes that have been recorded using video-capturing devices. This technique is popularly known as video oculography.

Infrared moving point target detection based on an anisotropic spatial-temporal fourth-order diffusion filter

Infrared moving point target detection is a challenging task in infrared tracking systems. In this paper, an anisotropic spatial-temporal fourth-order diffusion filter (ASTFDF) is proposed for background prediction from infrared frame images. Using it, we can detect infrared point targets by subtracting the predicted background from the original image. Experiments in detecting targets from several different infrared image frame sequences indicated that the proposed ASTFDF can predict the background effectively and that the target can be detected precisely while achieving a higher background suppression factor (BSF) and signal-to-clutter ratio gain (SCRG) compared with state-of-the-art methods.

Development and research of the algorithm search of the singular points position in the aircraft vision systems

The paper deals with the feature detection problem in aircraft machine vision systems. We developed a fast detection algorithm. We determined feature locations in an image sequence depicting fast-moving objects as registered by aircraft machine vision systems. We analysed existing feature detection algorithms. The input image frame sequence undergoes a geometric transformation, that is, rotation. We simulated how our algorithm processed a sequence of frames depicting the target. We compared efficiency and speed of feature detection in the well-known SIFT algorithm with that of our algorithm. We demonstrate that the algorithm developed ensures faster feature detection that is more resistant to geometric transformations of the target image as compared to the SIFT algorithm.

Experimental Investigation of Particle Agglomeration Effects on Slurry Settling in Viscous Fluid

This paper presents fundamental analysis and micromechanical understanding of dense slurry behavior during settling in narrow smooth and rough slots. Particularly, this study seeks to contribute toward better understanding of dynamics of particle–particle and particle–wall interactions in viscous fluids using simple experiments. The findings of this study are applicable in a wide variety of problems, for example sediment transport, flow and transport of slurry in pipes, and industrial applications. However, the results interpretation focuses on better understanding of proppant flow and transport in narrow fractures. A sequence of experiments image frames captured by video camera is analyzed with particle image velocimetry (GeoPIV). The measurements include vertical velocities and displacement vectors of singular and agglomerated particles and larger area of formed slurry. Results present novel insights into the formation and effects of agglomerates on general slurry settling, and are supplemented with a comparison with previously published theoretical and empirical relationships. This work also emphasizes a role of particle–particle interactions in promoting agglomeration in viscous fluid. Particularly, a thin layer of viscous fluid between approaching particles dissipates particle kinetic energy due to lubrication effect. Lubrication effect is more pronounced when particles are constrained between two narrow walls and interact frequently with each other. Fluid tends to flow around agglomerated particles, and agglomerates remain stable for prolonged time periods gravitationally moving downward. The relative amount and size of agglomerated affects general settling of the slurry. It was found that fluid viscosity due to lubrication effect promotes agglomeration, and therefore, the overall slurry settling relatively increases at higher fluid viscosities. The results of the presented work have impact on various industrial and engineering processes, such as proppant flow and transport in hydraulic fractures, sand production in oil reservoirs, piping failure of dams and scour of foundation bridges.

Optical-flow analysis toolbox for characterization of spatiotemporal dynamics in mesoscale optical imaging of brain activity

Wide-field optical imaging techniques constitute powerful tools to investigate mesoscale neuronal activity. The sampled data constitutes a sequence of image frames in which one can investigate the flow of brain activity starting and terminating at source and sink locations respectively. Approaches to the analyses of information flow include qualitative assessment to identify sources and sinks of activity as well as their trajectories, and quantitative measurements based on computing the temporal variation of the intensity of pixels. Furthermore, in a few studies estimates of wave motion have been reported using optical-flow techniques from computer vision. However, a comprehensive toolbox for the quantitative analyses of mesoscale brain activity data is still lacking. We present a graphical-user-interface toolbox based in Matlab® for investigating the spatiotemporal dynamics of mesoscale brain activity using optical-flow analyses. The toolbox includes the implementation of three optical-flow methods namely Horn-Schunck, Combined Local-Global, and Temporospatial algorithms for estimating velocity vector fields of flow of mesoscale brain activity. From the velocity vector fields we determined the locations of sources and sinks as well as the trajectories and temporal velocities of flow of activity. Using simulated data as well as experimentally derived sensory-evoked voltage and calcium imaging data from mice, we compared the efficacy of the three optical-flow methods for determining spatiotemporal dynamics. Our results indicate that the combined local-global method we employed, yields the best results for estimating wave motion. The automated approach permits rapid and effective quantification of mesoscale brain dynamics and may facilitate the study of brain function in response to new experiences or pathology.

Analysis of Cardiovascular Tissue Components for the Diagnosis of Coronary Vulnerable Plaque from Intravascular Ultrasound Images.

The purpose of this study was to characterize cardiovascular tissue components and analyze the different tissue properties for predicting coronary vulnerable plaque from intravascular ultrasound (IVUS) images. For this purpose, sequential IVUS image frames were obtained from human coronary arteries using 20 MHz catheters. The plaque regions between the intima and media-adventitial borders were manually segmented in all IVUS images. Tissue components of the plaque regions were classified into having fibrous tissue (FT), fibrofatty tissue (FFT), necrotic core (NC), or dense calcium (DC). The media area and lumen diameter were also estimated simultaneously. In addition, the external elastic membrane (EEM) was computed to predict the vulnerable plaque after the tissue characterization. The reliability of manual segmentation was validated in terms of inter- and intraobserver agreements. The quantitative results found that the FT and the media as well as the NC would be good indicators for predicting vulnerable plaques in IVUS images. In addition, the lumen was not suitable for early diagnosis of vulnerable plaque because of the low significance compared to the other vessel parameters. To predict vulnerable plaque rupture, future study should have additional experiments using various tissue components, such as the EEM, FT, NC, and media.