Successive Frames Research Articles

Landmark tracking in liver US images using cascade convolutional neural networks with long short-term memory

Accurate tracking of anatomic landmarks is critical for motion management in liver radiation therapy. Ultrasound (US) is a safe, low-cost technology that is broadly available and offer real-time imaging capability. This study proposed a deep learning-based tracking method for the US image-guided radiation therapy. The proposed cascade deep learning model is composed of an attention network, a mask region-based convolutional neural network (mask R-CNN), and a long short-term memory (LSTM) network. The attention network learns a mapping from an US image to a suspected area of landmark motion in order to reduce the search region. The mask R-CNN then produces multiple region-of-interest proposals in the reduced region and identifies the proposed landmark via three network heads: bounding box regression, proposal classification, and landmark segmentation. The LSTM network models the temporal relationship among the successive image frames for bounding box regression and proposal classification. To consolidate the final proposal, a selection method is designed according to the similarities between sequential frames. The proposed method was tested on the liver US tracking datasets used in the medical image computing and computer assisted interventions 2015 challenges, where the landmarks were annotated by three experienced observers to obtain their mean positions. Five-fold cross validation on the 24 given US sequences with ground truths shows that the mean tracking error for all landmarks is 0.65 ± 0.56 mm, and the errors of all landmarks are within 2 mm. We further tested the proposed model on 69 landmarks from the testing dataset that have the similar image pattern with the training pattern, resulting in a mean tracking error of 0.94 ± 0.83 mm. The proposed deep-learning model was implemented on a graphics processing unit (GPU), tracking 47–81 frames s−1. Our experimental results have demonstrated the feasibility and accuracy of our proposed method in tracking liver anatomic landmarks using US images, providing a potential solution for real-time liver tracking for active motion management during radiation therapy.

A Python-OpenCV based software for processing single-bacterium tracking microscopy videos.

Real-time single-bacterium tracking (SBT) is a powerful microscopy technique that allows for direct observation of the motility of individual bacteria. This technique requires microscopy video processing methods to localize individual bacteria from microscopy videos, followed by obtaining bacterial trajectories by linking the localizations in successive video frames. These analysis steps are usually labor-intensive. In addition, many existing analysis softwares are developed using commercial programming languages, which makes it difficult to be further developed and used for larger contexts of usage. In this regard, a Python-OpenCV based software is presented which is capable of obtaining bacterial trajectories from SBT microscopy videos. The developed software does not require additional commercial software licenses. Also, the software can perform video processing and bacterial trajectory extraction steps in an automated way. The developed open-source software could thus be listed as one of the useful tools for related quantitative investigations. This work is supported by the National Natural Science Foundation of China (NSFC) under grant number 12104405.

Adaptive chaotic sampling particle filter to handle occlusion and fast motion in visual object tracking

We present a new particle filter method for visual object tracking to effectively handle occlusion and fast motion. The proposed approach uses a chaotic local search to model irregular motion and, compared to ordinary particle filter approaches, requires a lower number of particles. Furthermore, a new chaotic sampling procedure is used to force particles to specific areas with high values of the likelihood function with maximum diversity and a histogram of dynamical information is introduced to represent the motion over successive frames based on state space reconstruction. Finally, a new criterion is proposed to distinguish occlusion and out-of-view for appearance updating. We present numerical experiments demonstrating that the developed framework outperforms other state-of-the-art approaches dealing with irregular motions and uncertainties. According to the results on BOBOT, OTB100, OTB2013, and VOT2018, compared with traditional approaches, including methods based on deep and reinforcement learning, correlation filters, and Siamese neural networks, the proposed strategy leads to much closer convergence to the true target state, increasing the tracking accuracy. Finally, we prove analytically the convergence of the proposed method.

CNN-Based Temporal Video Segmentation Using a Nonlinear Hyperbolic PDE-Based Multi-Scale Analysis

An automatic temporal video segmentation framework is introduced in this article. The proposed cut detection technique performs a high-level feature extraction on the video frames, by applying a multi-scale image analysis approach combining nonlinear partial differential equations (PDE) to convolutional neural networks (CNN). A nonlinear second-order hyperbolic PDE model is proposed and its well-posedness is then investigated rigorously here. Its weak and unique solution is determined numerically applying a finite difference method-based numerical approximation algorithm that quickly converges to it. A scale-space representation is then created using that iterative discretization scheme. A CNN-based feature extraction is performed at each scale and the feature vectors obtained at multiple scales are concatenated into a final frame descriptor. The feature vector distance values between any two successive frames are then determined and the video transitions are identified next, by applying an automatic clustering scheme on these values. The proposed PDE model, its mathematical investigation and discretization, and the multi-scale analysis based on it represent the major contributions of this work. Some temporal segmentation experiments and method comparisons that illustrate the effectiveness of the proposed framework are finally described in this research paper.

A Markovian Error Model for False Negatives in DNN-based Perception-Driven Control Systems

This paper presents an improved Markovian error model for Deep Neural Network (DNN) based perception in autonomous vehicles and other perception-driven control systems. Many modern autonomous systems rely on DNN-driven perception-based control/planning methodologies such as autonomous navigation, where the perception errors significantly affect the control/planning performance and the systems’ safety. The traditional independent, identically-distributed (IID) perception error model is inadequate for perception-based control/planning applications because image sequences supplied to a DNN-based perception module are not independent in the real world. Based on this observation, we develop a novel Markov model to describe the error behavior of a DNN perception model—an error in one frame is likely to signal errors in successive frames, effectively reducing sample rate for the control command. We evaluate the effect of Markovian perception errors on a drone-control simulator and show that the Markovian error model provides a better estimate of control performance than does a traditional independent, identically-distributed (IID) model.

An Efficient Low Complexity Region-of-Interest Detection for Video Coding in Wireless Visual Surveillance

Following its use in several applications, including video coding in wireless surveillance, moving object detection (MOD) has become a popular video analysis topic. Despite the considerable progress in the accuracy of MOD for video coding, its implementation in constrained sensors is a real challenge owing to their high complexity and energy consumption. Therefore, there is a great need to address the trade-off between the accuracy and the energy efficiency of MOD approaches for video coding in constrained systems. In this work, an energy-efficient region-of-interest (ROI) detection algorithm as a pre-encoder for wireless visual surveillance (WVS) is proposed. The algorithm ensures a trade-off between detection accuracy and computational complexity. To this end, we propose constructing an activity map by measuring each block activity between successive frames. The map scores are processed using a combination of a fast Gaussian smoother and a rank-order filter to improve accuracy. Only the blocks in motion are coded and counted for transmission. The accuracy of our approach has been evaluated on a large dataset using key performance metrics. It has been found that our algorithm outperforms other state-of-the-art techniques in terms of true positive rate (TPR), with 80.84% on sensitivity metric, while exhibiting a well-balanced accuracy for all categories. A careful examination of the computational complexity confirms the low overhead. The energy and bitrate savings could achieve nearly 90% and 98%, respectively.

Deep learning-based multiple moving vehicle detection and tracking using a nonlinear fourth-order reaction-diffusion based multi-scale video object analysis

<p style='text-indent:20px;'>A novel automatic vehicle detection and tracking framework is proposed in this research work. First, the moving vehicles are detected in the frames of the video sequence by combining some deep learning and Gaussian Mixture Model-based object detection techniques. Then, the correspondence between the video objects detected in successive frames is determined using a multi-scale analysis of those objects. A scale-space representation is created by applying the numerical approximation algorithm that solves a nonlinear fourth-order reaction-diffusion based model whose mathematical validity is rigorously investigated here. A color image feature extraction is then performed at each scale using SURF and HOG-based features and the feature vectors determined at multiple scales are next concatenated into a final descriptor. A novel instance matching-based vehicle tracking technique using the distances between these feature vectors is then proposed. The results of the performed detection and tracking simulations are finally discussed.</p>

Enhanced Spatio-Temporal Interaction Learning for Video Deraining: Faster and Better.

Video deraining is an important task in computer vision as the unwanted rain hampers the visibility of videos and deteriorates the robustness of most outdoor vision systems. Despite the significant success which has been achieved for video deraining recently, two major challenges remain: 1) how to exploit the vast information among successive frames to extract powerful spatio-temporal features across both the spatial and temporal domains, and 2) how to restore high-quality derained videos with a high-speed approach. In this paper, we present a new end-to-end video deraining framework, dubbed Enhanced Spatio-Temporal Interaction Network (ESTINet), which considerably boosts current state-of-the-art video deraining quality and speed. The ESTINet takes the advantage of deep residual networks and convolutional long short-term memory, which can capture the spatial features and temporal correlations among successive frames at the cost of very little computational resource. Extensive experiments on three public datasets show that the proposed ESTINet can achieve faster speed than the competitors, while maintaining superior performance over the state-of-the-art methods. https://github.com/HDCVLab/Enhanced-Spatio-Temporal-Interaction-Learning-for-Video-Deraining.

A Deep Learning Method for Motion Artifact Correction in Intravascular Photoacoustic Image Sequence.

In vivo application of intravascular photoacoustic (IVPA) imaging for coronary arteries is hampered by motion artifacts associated with the cardiac cycle. Gating is a common strategy to mitigate motion artifacts. However, a large amount of diagnostically valuable information might be lost due to one frame per cycle. In this work, we present a deep learning-based method for directly correcting motion artifacts in non-gated IVPA pullback sequences. The raw signal frames are classified into dynamic and static frames by clustering. Then, a neural network named Motion Artifact Correction (MAC)-Net is designed to correct motion in dynamic frames. Given the lack of the ground truth information on the underlying dynamics of coronary arteries, we trained and tested the network using a computer-generated dataset. Based on the results, it has been observed that the trained network can directly correct motion in successive frames while preserving the original structures without discarding any frames. The improvement in the visual effect of the longitudinal view has been demonstrated based on quantitative evaluation of the inter-frame dissimilarity. The comparison results validated the motion-suppression ability of our method comparable to gating and image registration-based non-learning methods, while maintaining the integrity of the pullbacks without image preprocessing. Experimental results from in vivo intravascular ultrasound and optical coherence tomography pullbacks validated the feasibility of our method in the in vivo intracoronary imaging scenario.

Optical Measurement of Roller Slip, Tilt, and Skew in a Spherical Roller Bearing

AbstractIn this study, a novel test rig was designed and developed to investigate roller slip, tilt, and skew in a spherical roller bearing (SRB). The test rig utilized a double-row 22313 SRB and was designed to allow for direct visual access to each row. A high-speed camera was used to capture the motion and angular position of the various rollers as they traversed the bearing. Successive frames captured from the videos were analyzed to determine roller slip and the SRB load zone. Roller tilt and skew angles were also measured by inserting a nearly weightless pin into the center of a roller. In a similar manner, high-speed videography was used to assess the tilt and skew of the roller for a complete revolution of the roller around the inner race of the SRB. The dynamic behavior of the rollers was then corroborated with a previously developed SRB dynamic bearing model (DBM). The experimental and analytical results indicate that the roller tilt angle increases with axial load, remains constant with speed, and decreases with increasing radial load when the roller is located in the load zone. Further, roller skew in the load zone increases with axial load and shaft speed; however, it decreases with the radial load. The results indicate that when the radial-to-axial load ratio is greater than 4, roller tilt and skew are minimized. Due to roller intermittent slip and roller-cage pocket collision in the unload zone, tilt and skew become chaotic. The magnitude of the tilt and skew in the unload zone is directly related to the roller-race and roller-cage pocket clearances, respectively.

CRVC-GAN: combining cross-scale fusion and recursion for video colorization adversarial generative networks

We proposed an innovative end-to-end generative adversarial network structure (CRVC-GAN) for video colorization, which is dedicated to solving the temporal consistency and coloring quality problems in the video coloring domain. For the temporal consistency problem, we utilize the network structure of recurrent neural network (RNN) because the model shape of RNN does not change with the input length and the calculation takes into account the history information to ensure the degree of correlation between successive frames. This better ensures the continuity between frames, so the consecutive frames get good temporal consistency in coloring. For the coloring quality problem, we utilize the UNet++ coding method to decode the video frames, integrating different levels of features to improve accuracy, and then enhance the object boundary information in the video frames, thus improving the coloring quality of grayscale video frames. In addition, we innovatively utilize the style transformation loss function in terms of images for model training to ensure temporal consistency and coloring quality by exploiting the slight difference in style variation from frame to frame. The experimental results show that, compared with the existing methods and through its own designed balance index, the colorization and temporary consistency balance index, CRVC-GAN obtained certain advantages on the DAVIS and Videvo public datasets with the results of 0.56176 and 0.74286, respectively, and achieved a better balance between time consistency and color quality.

Improved Segmentation of Echocardiography With Orientation-Congruency of Optical Flow and Motion-Enhanced Segmentation.

Quantification of left ventricular (LV) ejection fraction (EF) from echocardiography depends upon the identification of endocardium boundaries as well as the calculation of end-diastolic (ED) and end-systolic (ES) LV volumes. It's critical to segment the LV cavity for precise calculation of EF from echocardiography. Most of the existing echocardiography segmentation approaches either only segment ES and ED frames without leveraging the motion information, or the motion information is only utilized as an auxiliary task. To address the above drawbacks, in this work, we propose a novel echocardiography segmentation method which can effectively utilize the underlying motion information by accurately predicting optical flow (OF) fields. First, we devised a feature extractor shared by the segmentation and the optical flow sub-tasks for efficient information exchange. Then, we proposed a new orientation congruency constraint for the OF estimation sub-task by promoting the congruency of optical flow orientation between successive frames. Finally, we design a motion-enhanced segmentation module for the final segmentation. Experimental results show that the proposed method achieved state-of-the-art performance for EF estimation, with a Pearson correlation coefficient of 0.893 and a Mean Absolute Error of 5.20% when validated with echo sequences of 450 patients.

Video Polyp Segmentation: A Deep Learning Perspective

We present the first comprehensive video polyp segmentation (VPS) study in the deep learning era. Over the years, developments in VPS are not moving forward with ease due to the lack of a large-scale dataset with fine-grained segmentation annotations. To address this issue, we first introduce a high-quality frame-by-frame annotated VPS dataset, named SUN-SEG, which contains 158 690 colonoscopy video frames from the well-known SUN-database. We provide additional annotation covering diverse types, i.e., attribute, object mask, boundary, scribble, and polygon. Second, we design a simple but efficient baseline, named PNS+, which consists of a global encoder, a local encoder, and normalized self-attention (NS) blocks. The global and local encoders receive an anchor frame and multiple successive frames to extract long-term and short-term spatial-temporal representations, which are then progressively refined by two NS blocks. Extensive experiments show that PNS+ achieves the best performance and real-time inference speed (170 fps), making it a promising solution for the VPS task. Third, we extensively evaluate 13 representative polyp/object segmentation models on our SUN-SEG dataset and provide attribute-based comparisons. Finally, we discuss several open issues and suggest possible research directions for the VPS community. Our project and dataset are publicly available at https://github.com/GewelsJI/VPS.

Similarity-Aware CNN for Efficient Video Recognition at the Edge

Convolutional neural networks (CNNs) often extract similar features from successive video frames due to having identical appearances. In contrast, conventional CNNs for video recognition process individual frames with a fixed computational effort. Each video frame is independently processed, resulting in numerous redundant computations and an inefficient use of limited energy resources, particularly for edge computing applications. To alleviate the high energy requirements associated with video frame processing, this article presented similarity-aware CNNs that recognize similar feature pixels across frames and avoid computations on them. First, with a loss of less than 1% in recognition accuracy, a proposed similarity-aware quantization technique increases the average number of unchanged feature pixels across frame pairs by up to 85%. Then, a proposed similarity-aware dataflow improves energy consumption by minimizing redundant computations and memory accesses across frame pairs. According to simulation experiments, the proposed dataflow decreases the energy consumed by video frame processing by up to 30%.

A multimodal deep neural network for prediction of the driver’s focus of attention based on anthropomorphic attention mechanism and prior knowledge

The prediction of the driver’s focus of attention (DFoA) is becoming essential research for the driver distraction detection and intelligent vehicle. Therefore, this work makes an attempt to predict DFoA. However, traffic driving environment is a complex and dynamic changing scene. The existing methods lack full utilization of driving scene information and ignore the importance of different objects or regions of the driving scene. To alleviate this, we propose a multimodal deep neural network based on anthropomorphic attention mechanism and prior knowledge (MDNN-AAM-PK). Specifically, a more comprehensive information of driving scene (RGB images, semantic images, optical flow images and depth images of successive frames) is as the input of MDNN-AAM-PK. An anthropomorphic attention mechanism is developed to calculate the importance of each pixel in the driving scene. A graph attention network is adopted to learn semantic context features. The convolutional long short-term memory network (ConvLSTM) is used to achieve the transition of fused features in successive frames. Furthermore, a training method based on prior knowledge is designed to improve the efficiency of training and the performance of DFoA prediction. These experiments, including experimental comparison with the state-of-the-art methods, the ablation study of the proposed method, the evaluation on different datasets and the visual assessment experiment in vehicle simulation platform, show that the proposed method can accurately predict DFoA and is better than the state-of-the-art methods.

Online Multiplayer Tracking by Extracting Temporal Contexts with Transformer

Sports competition is one of the most popular programs for many audiences. Tracking the players in sports game videos from broadcasts is a nontrivial challenge for computer vision researchers. In sports videos, the direction of an athlete’s movement changes quickly and unpredictably. Mutual occlusion between athletes is also more frequent in team competitions. However, the rich temporal contexts among the adjacent frames have been excluded from consideration. To address this dilemma, we propose an online transformer-based learnable framework in an end-to-end fashion. We use a transformer architecture to extract the temporal contexts between the successive frames and add them to the network training, which is robust to occlusion and complex direction changes in multiplayer tracking. We demonstrate the effectiveness of our method on three sports video datasets by comparing them with recently advanced multiplayer trackers.

Quantifying the Structure and Chemistry of Semiconductor Devices with the Latest Advancements in Scanning Transmission Electron Microscopy

Electron microscopy has been revolutionized by the advent of the aberration corrector and advances in detector both for imaging and spectroscopy. Aberration correction dramatically improved spatial resolution into the sub-ångstrom regime, while new detectors have unlocked information about materials that was previously only imaginable.While these recent advances have proven essential to the atomic scale characterization of materials, many measurements have remained largely semi-quantitative. In particular, accuracy and precision for scanning transmission electron microscopy (STEM) was significantly hampered by the presence of sample drift and scan distortion. Until recently, this limitation has obscured the capabilities to characterize minute changes to the atomic structure that can ultimately define material properties.In this talk, we will introduce revolving scanning transmission electron microscopy (RevSTEM) for the characterization of semiconductors. The method uses a series of fast-acquisition STEM images, but with the scan coordinates rotated between successive frames. This scan rotation introduces a concomitant change in image distortion that is then used to analyze the sample drift rate and direction. Multiple case studies will be presented to demonstrate the power of this technique to characterize electronic materials. For example, we will show how picometer level precision and accuracy enabled the capability to directly quantify ferroelectric HfO2 thin films. We will also show how RevSTEM images can be used to accurately determine crystallographic parameters in real-space, and to determine the structural origins of polarization.When combined with modern spectroscopy tools, the technique can directly inform growth mechanisms in AlN based materials for fabrication of quantum well structures. Specifically, We will show the interface quality between various AlGaN/AlN quantum wells using a combination of high-angle annular dark-field (HAADF) imaging and energy dispersive X-ray spectroscopy (EDS). We will show that we can directly measure the lattice parameters for each layer and how they vary across each AlN/AlGaN interface. Using EDS, we correlate the lattice parameters and lattice parameter values to the chemical composition in each layer. We show that throughout multiple quantum well structures, there is an asymmetrical interface with a diffusion of Ga on one side of the AlGaN layer. Further, we will discuss fluctuation in the composition present throughout each layer and the role on the local strain distribution.

Driver Distraction Detection Based on the True Driver’s Focus of Attention

Effective driver distraction detection (DDD) can significantly improve driving safety. Inspired by the definition of driver distraction, this work aims to detect driver distraction based on the true driver’s focus of attention (TDFoA). This is accomplished based on two stages, the prediction of TDFoA and DDD based on the driver’s focus of attention (DFoA) and the TDFoA. However, this process is challenging due to the complex and dynamic traffic environment, momentary glimpses not related to the TDFoA in normal driving, the interference of noise (information unrelated to the TDFoA), the ignoring of multi-scale information from driving scenarios, and the variability of thresholds used to distinguish between driver distraction and non-distraction. To solve these problems, we design a deep 3D residual network with attention mechanism and encoder-decoder (D3DRN-AMED) based on successive frames with convolution LSTM for the prediction of TDFoA, where successive frames are used to eliminate the impact of momentary glimpses. The convolution LSTM achieves the transition of features in successive frames, considering the historical variation of driving scenarios. Attention mechanism based on soft thresholding is inserted into the D3DRN-AMED as nonlinear transformation layers to eliminate the noise-related features, and the encoder-decoder module is introduced to extract multi-scale features. A method based on a neural network is then proposed to detect driver distraction according to the difference or similarity between DFoA and TDFoA, which can accurately detect driver distraction without determining the threshold. Experimental results show that the proposed method can effectively detect driver distraction.

Deep convolutional network based real time fatigue detection and drowsiness alertness system

<span>Fatigue and drowsiness detection techniques based on the external features are under progress, and the methods of facial feature extraction require further development. This paper discusses the innovative processes, efficient methods, and recent advancements in the field of drowsiness and fatigue detection. In this proposed model, a wide application is planned in the field of artificial intelligence by defining the fundamentals of human-computer interaction, facial expression recognition and driver fatigue-sleepiness determination. This research outlines an efficient and effective three-phase strategy for detecting drowsiness. Viola Jones is used to detect facial traits in these three phases. Detection of yawning and tracking once the face has been identified, the segmenting the skin, the system becomes lighting invariant portion by itself, focusing on the chromatic components based on skin, and to reject most of non-face image backdrops. The color eye tracking and yawning detection are carried out by template matching with the correlation coefficient. The vectors of features based on each of the above phases is concatenated, and a binary result is obtained. The analysis of sound and successive frames into fatigue and non-fatigue states has been classified. If the time in fatigue state exceeds the threshold, the system will sound an alarm. </span>

End-to-End Plankton Database Collection System

Monitoring living plankton status has greatly developed owing to machine vision technology. One of these developments is the automatic video acquisition system, which can capture high-resolution information and record considerable details of a scene. However, existing optical sensors cannot usually obtain a whole image, in which all details of plankton are fully clear, due to the influence of many factors, such as depth-of-field limitation of lenses, movement of plankton, and large difference in plankton scale. The captured video also needs to be filtered to eliminate the redundancy before using for the sparsity of the sample. This procedure is time-consuming and costly. Therefore, in this article, we develop an end-to-end plankton database collection system that can directly generate complete, clear plankton images from video. First, the regions of interest of the plankton are extracted. Then, the same plankton which appears in successive frames is identified to reconstruct the clearest morphological and structure features through their detailed information. Experimental results indicate that the proposed system can effectively compress the original data. The proposed fusion method also outperforms the state-of-the-art methods, especially for images with anisotropic blur. Furthermore, this system can monitor the abundance and distribution of marine plankton with the help of an embedded computing platform.