Previous Image Frames Research Articles

Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns

Dynamic patterns are characterized by complex spatial and motion patterns. Understanding dynamic patterns requires a disentangled representational model that separates the factorial components. A commonly used model for dynamic patterns is the state space model, where the state evolves over time according to a transition model and the state generates the observed image frames according to an emission model. To model the motions explicitly, it is natural for the model to be based on the motions or the displacement fields of the pixels. Thus in the emission model, we let the hidden state generate the displacement field, which warps the trackable component in the previous image frame to generate the next frame while adding a simultaneously emitted residual image to account for the change that cannot be explained by the deformation. The warping of the previous image is about the trackable part of the change of image frame, while the residual image is about the intrackable part of the image. We use a maximum likelihood algorithm to learn the model parameters that iterates between inferring latent noise vectors that drive the transition model and updating the parameters given the inferred latent vectors. Meanwhile we adopt a regularization term to penalize the norms of the residual images to encourage the model to explain the change of image frames by trackable motion. Unlike existing methods on dynamic patterns, we learn our model in unsupervised setting without ground truth displacement fields or optical flows. In addition, our model defines a notion of intrackability by the separation of warped component and residual component in each image frame. We show that our method can synthesize realistic dynamic pattern, and disentangling appearance, trackable and intrackable motions. The learned models can be useful for motion transfer, and it is natural to adopt it to define and measure intrackability of a dynamic pattern.

IRON-MAN: An Approach to Perform Temporal Motionless Analysis of Video Using CNN in MPSoC

This paper proposes a novel human-inspired methodology called IRON-MAN ( I ntegrated R ati ON al prediction and M otionless AN alysis ) for mobile multi-processor systems-on-chips (MPSoCs). The methodology integrates analysis of the previous image frames of the video to represent the analysis of the current frame in order to perform Temporal Motionless Analysis of the Video ( TMAV ). This is the first work on TMAV using Convolutional Neural Network (CNN) for scene prediction in MPSoCs. Experimental results show that our methodology outperforms state-of-the-art. We also introduce a metric named, Energy Consumption per Training Image ( ECTI ) to assess the suitability of using a CNN model in mobile MPSoCs with a focus on energy consumption and lifespan reliability of the device.

Shifting Matrix를 이용한 DCT 기반 부화소 단위 움직임 예측 알고리즘

The present invention relates to a method to estimate subpixel unit motion based on discrete cosine transform (DCT) by using a shifting matrix, to use a shifting matrix in a frequency domain, so as to estimate motion of subpixel accuracy based on DCT. According to the present invention, a method to estimate motion of a 1/4 subpixel unit for moving image encoding comprises the following steps: performing DCT about an inputted image frame; and estimating motion in a frequency domain, in order to acquire an estimation block having a minimum matching error with respect to a current block through block matching between a DCT-performed current image frame and a previous image frame. In the step of estimating the motion in the frequency domain, the estimation block is acquired through a horizontal/vertical shifting matrix applying image interpolation of a 1/4 subpixel unit using a three dimensional interpolation function, and matrix operation using a DCT coefficient matrix of four blocks around the estimation block in a search range of the previous image frame.

Human Tracking Using Convolutional Neural Networks

In this paper, we treat tracking as a learning problem of estimating the location and the scale of an object given its previous location, scale, as well as current and previous image frames. Given a set of examples, we train convolutional neural networks (CNNs) to perform the above estimation task. Different from other learning methods, the CNNs learn both spatial and temporal features jointly from image pairs of two adjacent frames. We introduce multiple path ways in CNN to better fuse local and global information. A creative shift-variant CNN architecture is designed so as to alleviate the drift problem when the distracting objects are similar to the target in cluttered environment. Furthermore, we employ CNNs to estimate the scale through the accurate localization of some key points. These techniques are object-independent so that the proposed method can be applied to track other types of object. The capability of the tracker of handling complex situations is demonstrated in many testing sequences.

Real-time camera tracking for marker-less and unprepared augmented reality environments

For three-dimensional video-based augmented reality applications, accurate measurements of the 6DOF camera pose relative to the real world are required for proper registration of the virtual objects. This paper presents an accurate and robust system for real-time 6DOF camera pose tracking based on natural features in an arbitrary scene. Crucially, the calculation is based on pre-captured reference images. This prevents a gradual increase in the camera position error. Point features in the current image frame are first matched to two spatially separated reference images. This wide baseline correspondence problem is overcome by constructing (1) a global homography between current and previous image frame and (2) local affine transforms derived from known matches between previous frame and reference images. Chaining these two mappings constrains the search for potential matches in the reference images and allows the warping of corner intensity neighborhoods so that a viewpoint invariant similarity measure for assessing potential point matches can be defined. We then minimize deviations from the two-view and three-view constraints between the reference images and current frame as a function of the camera motion parameters to obtain an estimate of the current camera pose relative to the reference images. This calculation is stabilized using a recursive form of temporal regularization similar in spirit to the Kalman filter. We can track camera pose reliably over hundreds of image frames and realistically integrate three-dimensional virtual objects with only slight jitter. This paper also tries to simplify the above described algorithm and present a real-time, robust tracking system based on computing homographies. Homography can exactly describe the image motion between two frames when the camera motion is pure rotation, or it is viewing a planar scene. For outdoor registration applications, the system is robust under small translations as long as the majority of the scene contents are distant.

A Pipelined Temporal Difference Imager

A 189/spl times/182 active pixel sensor (APS) for temporal difference computation is presented. The temporal difference imager (TDI), fabricated in 0.5-/spl mu/m CMOS process, contains in-pixel storage elements for a previous image frame. Difference double-sampling circuits are used to suppress the fixed pattern noise in both images and to compute the difference between the corrected images. The pixel area occupies 25 /spl mu/m by 25 /spl mu/m (using 0.7-/spl mu/m scalable rules), with fill factor of 30%. A novel pipelined readout technique is described, which is used to improve the accuracy of the temporal difference computation. With this pipelined readout architecture, >8-bit precision for the difference image and low spatial droop across the difference image is achieved. The chip consumes 30 mW at 50 fps from a 5-V power supply.

A neural fuzzy system for image motion estimation

Many methods for computing optical flow (image motion vector) have been proposed while others continue to appear. Block-matching methods are widely used because of their simplicity and easy implementation. The motion vector is uniquely defined, in block-matching methods, by the best fit of a small reference subblock from a previous image frame in a larger, search region from the present image frame. Hence, this method is very sensitive to the real environments (involving occlusion, specularity, shadowing, transparency, etc.). In this paper, a neural fuzzy system with robust characteristics and learning ability is incorporated with the block-matching method to make a system adaptive for different circumstances. In the neural fuzzy motion estimation system, each subblock in the search region is assigned a similarity membership contributing different degrees to the motion vector. This system is more reliable, robust, and accurate in motion estimation than many other methods including Horn and Schunck's optical flow, fuzzy logic motion estimator (FME), best block matching, NR, and fast block matching. Since fast block-matching algorithms can be used to reduce search time, a three-step fast search method is employed to find the motion vector in our system. However, the candidate motion vector is often trapped by the local minimum, which makes the motion vector undesirable. An improved three-step fast search method is tested to reduce the effect from local minimum and some comparisons about fast search algorithms are made. In addition, a Quarter Compensation Algorithm for compensating the interframe image to tackle the problem that the motion vector is not an integer but rather a floating point is proposed. Since our system can give the accurate motion vector, we may use the motion information in many different applications such as motion compensation, CCD camera auto-focusing or zooming, moving object extraction, etc. Two application examples will be illustrated in this paper.

Recursive Estimation of Illuminant Motion from Flow Field and Simultaneous Recovery of Shape

We present a scheme based on the total least squares estimator to track the motion of a point source, by observing a sequence of images of an object illuminated by it. The initial shape of the object is estimated from any one of the existing shape and source from shading schemes from a single image frame. The total least squares formulation is used that considers this estimate of shape as being a noisy measurement of the actual shape. The method tracks the illuminant motion between successive frames from the flow field. Using the data only from two successive frames, an approach is developed to estimate the underlying illuminant motion between those two frames. This approach is then extended and a recursive method which uses the data from all previous image frames is developed to track the illuminant motion. An incremental shape recovery scheme using the proposed illuminant motion estimator is also presented. The method is then extended to deal with the presence of ambient light and the motion of multiple illuminants. We present typical simulation results to validate the proposed illuminant motion and shape update schemes. No feature point correspondence is required between the successive frames. Also the method is applicable to any general reflectance map.

Object tracking through adaptive correlation

Current Air Force interests include a desire to track an object based on its shape once it has been designated as a target. The use of a correlation-based system to track an object through a series of images based on templates derived from previous image frames is discussed. The ability to track is extended to sequences that include multiple objects of interest within the field of view. This is accomplished by comparing the height and shape of the template autocorrelation to the peaks in the correlation of the template with the next scene. The result is to identify the region in the next scene that best matches the designated target. In addition to correlation plane postprocessing, an adaptive window is used to determine the template size to reduce the effects of correlator walkoff. The image sequences used were taken from a forward-looking infrared sensor mounted on board a DC-3 aircraft. The images contain a T-55 tank and both an M-113 and a TAB-71 armored personnel carrier moving in a columnized formation along a dirt road. This research presents techniques to (1) track targets in the presence of other, and sometimes brighter, targets of similar shape, (2) to maintain small tracking errors, and (3) to reduce the effects of correlator walk-off.