Consecutive Image Frames Research Articles

Extended kernel correlation filter for abrupt motion tracking

The Kernelized Correlation Filters (KCF) tracker has caused the extensive concern in recent years because of the high efficiency. Numerous improvements have been made successively. However, due to the abrupt motion between the consecutive image frames, these methods cannot track object well. To cope with the problem, we propose an extended KCF tracker based on swarm intelligence method. Unlike existing KCF-based trackers, we firstly introduce a swarm-based sampling method to KCF tracker and design a unified framework to track smooth or abrupt motion simultaneously. Secondly, we propose a global motion estimation method, where the exploration factor is constructed to search the whole state space so as to adapt abrupt motion. Finally, we give an adaptive threshold in light of confidence map, which ensures the accuracy of the motion estimation strategy. Extensive experimental results in both quantitative and qualitative measures demonstrate the effectiveness of our proposed method in tracking abrupt motion.

Data rate enhancement of optical camera communications by compensating inter-frame gaps

Optical camera communications (OCC) is a convenient way of transmitting data between LED lamps and image sensors that are included in most smart devices. Although many schemes have been suggested to increase the data rate of the OCC system, it is still much lower than that of the photodiode-based LiFi system. One major reason of this low data rate is attributed to the inter-frame gap (IFG) of image sensor system, that is, the time gap between consecutive image frames. In this paper, we propose a way to compensate for this IFG efficiently by an interleaved Hamming coding scheme. The proposed scheme is implemented and the performance is measured.

Abnormality tracking during video capsule endoscopy using an affine triangular constraint based on surrounding features

The precise tracking of an abnormality in the gastrointestinal tract is useful for medical training purposes. However, the gastrointestinal wall deforms continuously in an unpredictable manner, while abnormalities lack distinctive features, making them difficult to track over continuous frames. To address this problem, we propose a tracking method for capsule endoscopy using the surrounding features of abnormalities. By applying triangular constraints using an affine transformation, we are able to track abnormalities that do not have distinctive features over consecutive image frames. We demonstrate the efficacy of our approach using eight common types of gastrointestinal abnormalities.

Diffeomorphic Multi-Frame Non-Rigid Registration of Cell Nuclei in 2D and 3D Live Cell Images.

To gain a better understanding of cellular and molecular processes, it is important to quantitatively analyze the motion of subcellular particles in live cell microscopy image sequences. Since, generally, the subcellular particles move and cell nuclei move as well as deform, it is important to decouple the movement of particles from that of the cell nuclei using non-rigid registration methods. We have developed a diffeomorphic multi-frame approach for non-rigid registration of cell nuclei in 2D and 3D live cell fluorescence microscopy images. Our non-rigid registration approach is based on local optic flow estimation, exploits information from multiple consecutive image frames, and determines diffeomorphic transformations in the log-domain, which allows efficient computation of the inverse transformations. To register single images of an image sequence to a reference image, we use a temporally weighted mean image, which is constructed based on inverse transformations and multiple consecutive frames. Using multiple consecutive frames improves the registration accuracy compared to pairwise registration, and using a temporally weighted mean image significantly reduces the computation time compared with previous work. In addition, we use a flow boundary preserving method for regularization of computed deformation vector fields, which prevents from over-smoothing compared to standard Gaussian filtering. Our approach has been successfully applied to 2D and 3D synthetic as well as real live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise, multi-frame, and temporal groupwise registration has been carried out.

Online dynamic cardiac imaging based on the elastic-net model

Purpose: The purpose of this work was to develop an online dynamic cardiac MRI model to reconstruct image frames from partial acquisition of the Cartesian k-space data, which utilizes structural knowledge of consecutive image frames. Materials and methods: Using an elastic-net model, the proposed algorithm reconstructs dynamic images using both L1 and L2 norm operations. The L1 norm enforces the sparsity of the frame difference, while the L2 norm with motion-adaptive weights catches the internal structure of frame differences. Unlike other online methods such as the Kalman filter (KF) technique, the new model requires no assumption of Gaussian noise, and can faithfully reconstruct the dynamic images within a compressive sensing framework. Results: The proposed method was evaluated using simulated dynamic phantoms with 40 frames of images (128 × 128) and a cardiac MRI cine of 25 frames (256 × 256). Both results showed that the new model offered a better performance than the online KF method in depicting simulated phantom and cardiac dynamics. Conclusion: It is concluded that the proposed imaging model can be used to capture a large variety of objects in motion from highly under-sampled k-space data, and being particularly useful for improving temporal resolution of cardiac MRI.

Moving Object Classification Using a Combination of Static Appearance Features and Spatial and Temporal Entropy Values of Optical Flows

This paper proposes a new approach for classifying four types of moving objects in an intelligent transportation system. Pedestrians, cars, motorcycles, and bicycles are classified based on their side views from a fixed camera. A moving object is segmented and tracked using background subtraction, silhouette projection, an area ratio, a Kalman filter, and appearance correlation operations. For the classification of a segmented object, a combination of static and spatiotemporal features based on the cooccurrence of its appearance and the movements of its local parts is proposed. To extract the static appearance features, adaptive block-based gradient intensities and histograms of oriented gradients are proposed. For the spatiotemporal features, the optical-flow-based entropy values of instantaneous and short-term movements are proposed. The former finds the spatial entropy values of the orientations and the amplitudes of optical flows in a block to extract the local movement information from two consecutive image frames. The latter finds the temporal entropy values of the tracked optical flows in different orientation bins to extract the short-term movement information from several consecutive frames. Linear support vector machines with batch incremental learning are proposed to classify the four classes of objects. Experimental results from 12 test video sequences and comparisons with several feature descriptors show the effect of the proposed classification system and the advantage of the proposed features in classification.

특징 점 기반의 ICP 알고리즘을 이용한 2차원 격자지도 보정

This paper suggests a feature point-based Iterative Closest Point (ICP) algorithm to compensate for the disparity error in building a two-dimensional map. The ICP algorithm is a typical algorithm for matching a common object in two different images. In the process of building a two-dimensional map using the laser scanner data, warping and distortions exist in the map because of the disparity between the two sensor values. The ICP algorithm has been utilized to reduce the disparity error in matching the scanned line data. For this matching process in the conventional ICP algorithm, pre-known reference data are required. Since the proposed algorithm extracts characteristic points from laser-scanned data, reference data are not required for the matching. The laser scanner starts from the right side of the mobile robot and ends at the left side, which causes disparity in the scanned line data. By finding the matching points between two consecutive frame images, the motion vector of the mobile robot can be obtained. Therefore, the disparity error can be minimized by compensating for the motion vector caused by the mobile robot motion. The validity of the proposed algorithm has been verified by comparing the proposed algorithm in terms of map-building accuracy to conventional ICP algorithm real experiments.

Thermal image based fault diagnosis for rotating machinery

Infrared imaging is crucial for condition monitoring as the thermographic patterns will differ depending on the fault or machine condition. Currently, a limited number of machine faults have been studied using thermal imaging. Therefore, this paper proposes a novel automatic fault detection system using infrared imaging, focussing on bearings of rotating machinery. The set of bearing faults monitored contain faults for which state-of-the-art techniques have no general solutions such as bearing-lubricant starvation. For each fault, several recordings are made using different bearings to ensure generalization of the fault-detection system. The system contains two image-processing pipelines, each with their own respective purposes. The first pipeline focusses on detecting rotor imbalance, regardless of the bearing faults. The second pipeline focusses on the bearing faults, regardless of whether the machine is balanced or not. Within the first pipeline, imbalance is detected by differencing the consecutive image frames which are subsequently summarized by their distribution along the image axes. For the second pipeline, three features are introduced which are the standard deviation of the temperature, the Gini coefficient, and the Moment of Light. The final system is able to distinguish between all eight different conditions with an accuracy of 88.25%.

A novel obstacle detection method based on deformable grid for the visually impaired

To assist visually impaired people, a variety of obstacle detection (OD) methods with the monocular vision have been developed. Most conventional OD methods detect the obstacle by using the variation of the motion vector or the object size between two consecutive image frames. However, the OD performance using the short-term variation is significantly affected by the tracking error, which leads to inaccurate OD results. To cope with this problem, this paper presents a novel OD method based on a new structure, called Deformable Grid (DG). The DG is initially a regular gridshaped, but it can be deformed gradually depending on the motion of the object in the scene. The proposed method detects the object at risk of collision based on the degree of the deformation of the DG. Experimental results demonstrate that the proposed OD method outperforms the conventional ones in terms of processing time as well as accuracy.

Extracting Graphics Information for Better Video Compression

Cloud gaming services are heavily dependent on the efficiency of real-time video streaming technology owing to the limited bandwidths of wire or wireless networks through which consecutive frame images are delivered to gamers. Video compression algorithms typically take advantage of similarities among video frame images or in a single video frame image. This paper presents a method for computing and extracting both graphics information and an object's boundary from consecutive frame images of a game application. The method will allow video compression algorithms to determine the positions and sizes of similar image blocks, which in turn, will help achieve better video compression ratios. The proposed method can be easily implemented using function call interception, a programmable graphics pipeline, and off-screen rendering. It is implemented using the most widely used Direct3D API and applied to a well-known sample application to verify its feasibility and analyze its performance. The proposed method computes various kinds of graphics information with minimal overhead.

Local Binary Pattern Based Fast Digital Image Stabilization

In this letter, a fast digital image stabilization method based on local binary pattern (LBP) for real time applications is presented. The LBP approach utilized in this work enables efficient representation of the image frames in 1-bit depth resolution. A simple Boolean exclusive-OR (XOR) based matching criterion is employed to decide global motion vector between consecutive image frames. The constant velocity model based Kalman filtering is executed on the global motion vectors to obtain smoothed frame positions. Experiments show that the proposed approach provides comparable or better performance against the methods at the same category by requiring lower computational complexity.

Simulating Organogenesis

Recent advances in imaging technology now provide us with 3D images of developing organs. These can be used to extract 3D geometries for simulations of organ development. To solve models on growing domains, the displacement fields between consecutive image frames need to be determined. Here we develop and evaluate different landmark-free algorithms for the determination of such displacement fields from image data. In particular, we examine minimal distance, normal distance, diffusion-based, and uniform mapping algorithms and test these algorithms with both synthetic and real data in 2D and 3D. We conclude that in most cases, the normal distance algorithm is the method of choice and wherever it fails, diffusion-based mapping provides a good alternative.

A Biologically Inspired Vision-Based Approach for Detecting Multiple Moving Objects in Complex Outdoor Scenes

In the human brain, independent components of optical flows from the medial superior temporal area are speculated for motion cognition. Inspired by this hypothesis, a novel approach combining independent component analysis (ICA) with principal component analysis (PCA) is proposed in this paper for multiple moving objects detection in complex scenes—a major real-time challenge as bad weather or dynamic background can seriously influence the results of motion detection. In the proposed approach, by taking advantage of ICA’s capability of separating the statistically independent features from signals, the ICA algorithm is initially employed to analyze the optical flows of consecutive visual image frames. As a result, the optical flows of background and foreground can be approximately separated. Since there are still many disturbances in the foreground optical flows in the complex scene, PCA is then applied to the optical flows of foreground components so that major optical flows corresponding to multiple moving objects can be enhanced effectively and the motions resulted from the changing background and small disturbances are relatively suppressed at the same time. Comparative experimental results with existing popular motion detection methods for challenging imaging sequences demonstrate that our proposed biologically inspired vision-based approach can extract multiple moving objects effectively in a complex scene.

Real-time velocity measurement to linear motion of a rigid object with monocular image sequence analyses

This paper presents a methodology and all procedures used to validate it, which were executed in a physics laboratory under controlled and known conditions. The validation was based on the analyses of registered data in an image sequence and the measurements acquired by high precision sensors. This methodology intended to measure the velocity of a rigid object in linear motion with the use of an image sequence acquired by commercial digital video camera. The proposed methodology does not need a stereo pair of images to calculate the object position in the 3D space: it needs only images sequence acquired for one, only one, angle view (monocular vision). To do so, these objects need to be detected while in movement, which is conducted by the application of a segmentation technique based on the temporal average values of each pixel registered in N consecutive image frames. After detecting and framing these objects, specific points belonging to the object (pixels), on the plane image (2D coordinates or space image), are automatically chosen, which are then transformed into corresponding points in the space object (3D coordinates) by the application of collinearity equations or rational functions (proposed in this work). After obtaining the coordinates of these points in the space object that are registered in the sequence of images, the distance, in meters, covered by the object in a particular time interval may be measured and, consequently, its velocity can be calculated. The system is low cost, use only a computer (architecture Intel I3), and a webcam used to acquire the images (640 × 480, 30 fps). The complexity of the algorithm is linear, fact that allows the system to operate in real time. The results of the analyses are discussed and the advantages and disadvantages of the method are presented.

Development of a crack growth predictor for geomaterials using detrended fluctuation analysis and optical flow method

Several conventional methods like seismic, micro-seismic, acoustic, electromagnetic etc. have been proposed to investigate the crack initiation and growth in rocks during static and dynamic loading conditions. These conventional sensors are prone to acquire noise and moreover these methods do not provide comprehensive knowledge about failure process since data are collected at a few points attached to a rock sample. This paper presents the application of detrended fluctuation analysis method and optical flow method of image analysis for analyzing failure mechanism of rock specimen under uniaxial loading condition. Two predictors, normalized cumulative fluctuation coefficient (NCFC) and normalized cumulative standard deviation of strain (NCSS) have been developed for forecasting the same. Both the predictors are developed by analyzing consecutive image frames under incremental loading conditions and are found to be powerful markers for categorizing the crack initiation period, stable crack growth period and collapse period in the rock failure process. It is also obtained that NCFC and NCSS produce almost similar results even if they are derived using different image analysis concepts. Numerous laboratory experiments have been conducted to check the applicability of the both the predictors. Based on the experimental data, it is envisaged that both the predictors can be used as precursors for monitoring rock failure mechanism.

SIFT flow for large-displacement object tracking.

Traditional tracking methods place an emphasis on how to cope with the variations in target appearance effectively. However, when the motion displacement of the target between image frames becomes larger, these methods may be unstable. This paper presents a novel (to our knowledge) visual object tracking method. In this method, we first introduce scale-invariant feature transform (SIFT) flow into the tracking problem and develop a real-time motion prediction method to capture large displacement between consecutive image frames. Then we use belief propagation (BP) to convert the problem of finding maximum a posteriori probability (MAP) to globally minimizing an energy function to get the best matching pairs of points for producing good candidate regions of the target. And last, the refined point trajectories are obtained according to the bidirectional flow field consistency estimation and covariance region descriptor matching, which can update model states efficiently so as to achieve enhanced robustness for visual tracking. Compared with the state-of-art tracking methods, the experimental results demonstrate that the proposed algorithm shows favorable performance when the object undergoes large motion displacement between image frames.

Streamline-based method for intra-day solar forecasting through remote sensing

This work presents an enhanced deterministic solar irradiance forecasting approach that relies on satellite images and ground measurements as inputs. The proposed method is based on a ground-truth improved satellite-to-irradiance model for the prediction of global horizontal irradiance (GHI). This approach relies on cloud tracking and advection with an optical flow algorithm. The application of the optical flow algorithm between two consecutive satellite image frames allows for the calculation of a vector field covering each pixel in the satellite image. This cloud motion vector (CMV) field determines the streamline passing through the location of interest. The estimated cloud advection along the quasi-steady streamline to the location of interest is than computed, and this information is translated into an irradiance forecast for the location of interest. In order to reduce the error associated with a linear satellite-to-irradiance model, a novel approach employing ground measurements is proposed. Additionally, decision heuristics are identified and implemented to issue a forecast based on CMV or persistence, depending on the current sky conditions. The overall method is tested for over 110days of operational 1-, 2- and 3-h ahead GHI forecasts, implemented and evaluated for San Diego, California. The continual forecasting skill of this method for the 110days ranges between 8% and 19% over persistence, depending on the forecast horizon. While previously proposed methods achieve similar skills, the completely deterministic approach combined with comparably low computation and implementation costs makes the proposed method suitable for applications with limited availability of data.

Video Segmentation Algorithm Based on Improved Kirsch Edge Operator and Three-Frame Difference

For the most commonly used method of are susceptible to noise, it is difficult to extract a complete moving target . Aiming at this problem, we propose a video segmentation algorithm through the combination of an improved Kirsch edge operator and the three-frame difference. Firstly, we got a moving region through a differential by the three consecutive image frames, and then got the edge on the current frame using an improved Kirsch edge detection operator, Synthesis of the two test results, we got a more accurate moving object edges. Then using the edge connector algorithm to complete the connection of the fracture edge, and finally, got moving target mask image through regional filling, thereby dividing the complete moving target, thus dividing the complete moving target. Experimental results show that the algorithm has a certain robustness, and can accurately detect moving targets.

Autonomous Star Camera Calibration and Spacecraft Attitude Determination

This paper presents two methods of star camera calibration to determine camera calibrating parameters (like principal point, focal length etc) along with lens distortions (radial and decentering). First method works autonomously utilizing star coordinates in three consecutive image frames thus independent of star identification or biased attitude information. The parameters obtained in autonomous self-calibration technique helps to identify the imaged stars with the cataloged stars. Least Square based second method utilizes inertial star coordinates to determine satellite attitude and star camera parameters with lens radial distortion, both independent of each other. Camera parameters determined by the second method are more accurate than the first method of camera self calibration. Moreover, unlike most of the attitude determination algorithms where attitude of the satellite depend on the camera calibrating parameters, the second method has the advantage of computing spacecraft attitude independent of camera calibrating parameters except lens distortions (radial). Finally Kalman filter based sequential estimation scheme is employed to filter out the noise of the LS based estimation.

Content-based video copy detection based on motion vectors estimated using a lower frame rate

We propose a motion vector-based video content-based copy detection method. One of the signatures of a given video is motion vectors extracted from image sequences. However, when consecutive image frames are used, the resulting motion vectors are not descriptive enough because most vectors are either too small or they appear to scatter in all directions. We calculate motion vectors in a lower frame rate than the actual frame rate of the video to overcome this problem. As a result, we obtain large vectors and they represent a given video in a robust manner. We carry out experiments for various parameters and present the results.