Inter-frame Motion Research Articles

Optimizing the frame duration for data-driven rigid motion estimation in brain PET imaging.

Data-driven rigid motion estimation for PET brain imaging is usually performed using data frames sampled at low temporal resolution to reduce the overall computation time and to provide adequate signal-to-noise ratio in the frames. In recent work it has been demonstrated that list-mode reconstructions of ultrashort frames are sufficient for motion estimation and can be performed very quickly. In this work we take the approach of using image-based registration of reconstructions of very short frames for data-driven motion estimation, and optimize a number of reconstruction and registration parameters (frame duration, MLEM iterations, image pixel size, post-smoothing filter, reference image creation, and registration metric) to ensure accurate registrations while maximizing temporal resolution and minimizing total computation time. Data from 18 F-fluorodeoxyglucose (FDG) and 18 F-florbetaben (FBB) tracer studies with varying count rates are analyzed, for PET/MR and PET/CT scanners. For framed reconstructions using various parameter combinations interframe motion is simulated and image-based registrations are performed to estimate that motion. For FDG and FBB tracers using 4×105 true and scattered coincidence events per frame ensures that 95% of the registrations will be accurate to within 1mm of the ground truth. This corresponds to a frame duration of 0.5-1sec for typical clinical PET activity levels. Using four MLEM iterations with no subsets, a transaxial pixel size of 4mm, a post-smoothing filter with 4-6mm full width at half maximum, and averaging two or more frames to create the reference image provides an optimal set of parameters to produce accurate registrations while keeping the reconstruction and processing time low. It is shown that very short frames (≤1sec) can be used to provide accurate and quick data-driven rigid motion estimates for use in an event-by-event motion corrected reconstruction.

Group Abnormal Behaviour Detection Algorithm Based on Global Optical Flow

Abnormal behaviour detection algorithm needs to conduct behaviour analysis on the basis of continuous video inclination tracking, and the robustness of the algorithm is reduced for the occlusion of moving targets, the occlusion of the environment, and the movement of targets with the same colour. For this reason, the optical flow information between RGB (red, green, and blue) images and video frames is used as the input of the network in view of group behaviour. Then, the direction, velocity, acceleration, and energy of the crowd were weighted and fused into a global optical flow descriptor. At the same time, the crowd trajectory map is extracted from the original image of a single frame. Following, in order to realize the detection of large displacement moving target and solve the problem that the traditional optical flow algorithm is only suitable for the detection of displacement moving target, a video abnormal behaviour detection algorithm based on the double-flow convolutional neural network is proposed. The network uses two network branches to learn spatial dimension information and temporal dimension information, respectively, and uses short- and long-time neural network to model the dependency relationship between long-time video frames, so as to obtain the final behaviour classification results. Simulation test results show that the proposed method can achieve good recognition effect on multiple datasets, and the performance of abnormal behaviour detection can be significantly improved by using interframe motion information.

Continuous Scale-Space Direct Image Alignment for Visual Odometry From RGB-D Images

In this letter, we propose a novel dense 3D image alignment algorithm that estimates the Euclidean transformation between pairs of camera poses from pixel intensities. The novelty consists in the automatic scale adaptation within each level of a multi-resolution image pyramid, using the scale-space representation of images. This is done through the continuous optimization of a scale parameter along with camera pose parameters in the same optimization framework. The proposed approach permits to significantly improve the robustness of the direct image alignment to large inter-frame motion. Various experiments on the TUM RGB-D dataset show that the proposed algorithm outperforms a fixed scale pyramid-based state-of-the-art alignment method.

Two-stream deep spatial-temporal auto-encoder for surveillance video abnormal event detection

With the improvement of public security awareness, video anomaly detection has become an indispensable demand in surveillance videos. To improve the accuracy of video anomaly detection, this paper proposes a novel two-stream spatial-temporal architecture called Two-Stream Deep Spatial-Temporal Auto-Encoder (Two-Stream DSTAE), which is composed of a spatial stream DSTAE and a temporal stream DSTAE. Firstly, the spatial stream extracts appearance characteristics whereas the temporal stream extracts the motion patterns, respectively. Then, based on the novel policy joint reconstruction error, this model fuses the spatial stream and the temporal stream to extract spatial-temporal characteristics to detect anomalies. Furthermore, since the optical flow is invariant to appearances such as color or light, we introduce optical flow to enhance the capability of extracting continuity between adjacent frames and inter-frame motion information. We demonstrate the accuracy of the proposed method on the publicly available standard datasets: UCSD, Avenue and UMN datasets. Our experiments demonstrate high accuracy, which is superior to the state-of-the-art methods.

Simultaneous Video Stabilization and Rolling Shutter Removal.

Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

HRSiam: High-Resolution Siamese Network, Towards Space-Borne Satellite Video Tracking.

Tracking moving objects from space-borne satellite videos is a new and challenging task. The main difficulty stems from the extremely small size of the target of interest. First, because the target usually occupies only a few pixels, it is hard to obtain discriminative appearance features. Second, the small object can easily suffer from occlusion and illumination variation, making the features of objects less distinguishable from features in surrounding regions. Current state-of-the-art tracking approaches mainly consider high-level deep features of a single frame with low spatial resolution, and hardly benefit from inter-frame motion information inherent in videos. Thus, they fail to accurately locate such small objects and handle challenging scenarios in satellite videos. In this article, we successfully design a lightweight parallel network with a high spatial resolution to locate the small objects in satellite videos. This architecture guarantees real-time and precise localization when applied to the Siamese Trackers. Moreover, a pixel-level refining model based on online moving object detection and adaptive fusion is proposed to enhance the tracking robustness in satellite videos. It models the video sequence in time to detect the moving targets in pixels and has ability to take full advantage of tracking and detecting. We conduct quantitative experiments on real satellite video datasets, and the results show the proposed HIGH-RESOLUTION SIAMESE NETWORK (HRSiam) achieves state-of-the-art tracking performance while running at over 30 FPS.

Neural Video Coding Using Multiscale Motion Compensation and Spatiotemporal Context Model

Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals, respectively. Novel features of NVC include: 1) To estimate and compensate motion over a large range of magnitudes, we propose an unsupervised multiscale motion compensation network (MS-MCN) together with a pyramid decoder in the VAE for coding motion features that generates multiscale flow fields, 2) we design a novel adaptive spatiotemporal context model for efficient entropy coding for motion information, 3) we adopt nonlocal attention modules (NLAM) at the bottlenecks of the VAEs for implicit adaptive feature extraction and activation, leveraging its high transformation capacity and unequal weighting with joint global and local information, and 4) we introduce multi-module optimization and a multi-frame training strategy to minimize the temporal error propagation among P-frames. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods following the common test conditions, demonstrating consistent gains across all popular test sequences for both PSNR and MS-SSIM distortion metrics.

FIFNET: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames

We present a novel motion-based multiframe image super-resolution (SR) algorithm using a convolutional neural network (CNN) that fuses multiple interpolated input frames to produce an SR output. We refer to the proposed CNN and associated preprocessing as the Fusion of Interpolated Frames Network (FIFNET). We believe this is the first such CNN approach in the literature to perform motion-based multiframe SR by fusing multiple input frames in a single network. We study the FIFNET using translational interframe motion with both fixed and random frame shifts. The input to the network is a sequence of interpolated and aligned frames. One key innovation is that we compute subpixel interframe registration information for each interpolated pixel and feed this into the network as additional input channels. We demonstrate that this subpixel registration information is critical to network performance. We also employ a realistic camera-specific optical transfer function model that accounts for diffraction and detector integration when generating training data. We present a number of experimental results to demonstrate the efficacy of the proposed FIFNET using both simulated and real camera data. The real data come directly from a camera and are not artificially downsampled or degraded. In the quantitative results with simulated data, we show that the FIFNET performs favorably in comparison to the benchmark methods tested.

A Novel 2-D Speckle Tracking Method for High-Frame-Rate Echocardiography

Speckle tracking echocardiography (STE) is a clinical tool to noninvasively assess regional myocardial function through the quantification of regional motion and deformation. Even if the time resolution of STE can be improved by high-frame-rate (HFR) imaging, dedicated HFR STE algorithms have to be developed to detect very small interframe motions. Therefore, in this article, we propose a novel 2-D STE method, purposely developed for HFR echocardiography. The 2-D motion estimator consists of a two-step algorithm based on the 1-D cross correlations to separately estimate the axial and lateral displacements. The method was first optimized and validated on simulated data giving an accuracy of ~3.3% and ~10.5% for the axial and lateral estimates, respectively. Then, it was preliminarily tested in vivo on ten healthy volunteers showing its clinical applicability and feasibility. Moreover, the extracted clinical markers were in the same range as those reported in the literature. Also, the estimated peak global longitudinal strain was compared with that measured with a clinical scanner showing good correlation and negligible differences (-20.94% versus -20.31%, p -value = 0.44). In conclusion, a novel algorithm for STE was developed: the radio frequency (RF) signals were preferred for the axial motion estimation, while envelope data were preferred for the lateral motion. Furthermore, using 2-D kernels, even for 1-D cross correlation, makes the method less sensitive to noise.

Video Face Super-Resolution with Motion-Adaptive Feedback Cell

Video super-resolution (VSR) methods have recently achieved a remarkable success due to the development of deep convolutional neural networks (CNN). Current state-of-the-art CNN methods usually treat the VSR problem as a large number of separate multi-frame super-resolution tasks, at which a batch of low resolution (LR) frames is utilized to generate a single high resolution (HR) frame, and running a slide window to select LR frames over the entire video would obtain a series of HR frames. However, duo to the complex temporal dependency between frames, with the number of LR input frames increase, the performance of the reconstructed HR frames become worse. The reason is in that these methods lack the ability to model complex temporal dependencies and hard to give an accurate motion estimation and compensation for VSR process. Which makes the performance degrade drastically when the motion in frames is complex. In this paper, we propose a Motion-Adaptive Feedback Cell (MAFC), a simple but effective block, which can efficiently capture the motion compensation and feed it back to the network in an adaptive way. Our approach efficiently utilizes the information of the inter-frame motion, the dependence of the network on motion estimation and compensation method can be avoid. In addition, benefiting from the excellent nature of MAFC, the network can achieve better performance in the case of extremely complex motion scenarios. Extensive evaluations and comparisons validate the strengths of our approach, and the experimental results demonstrated that the proposed framework is outperform the state-of-the-art methods.

A moving vehicles extraction method in Satellite Videos

With the continuous innovation of optical remote sensing technology and the increasing demand for spatial information, satellite videos, which can provide higher spatial and temporal resolution, have been paid a lot of attention. And moving vehicles extraction in satellite videos is one of the most important tasks. By analyzing the shortcomings of current satellite video moving vehicles extraction algorithms, and combining with the characteristics of satellite videos and moving vehicles, this paper proposes an algorithm to extract moving vehicles in satellite videos, that some vehicles are firstly separated from the background by using image extreme points and mean differences, and then the moving vehicles are extracted by joint detection of inter-frame vehicles motion. At the same time, based on the extracted moving vehicles, we also propose a method that can extract road masks by using only three frames. Finally, we use Jilin-1 satellite video data to prove the proposed methods in the experiment. And also this paper has compared the propose methods with another two algorithms, where the results show that the proposed methods greatly improve the accuracy and quality of moving vehicles detection in satellite videos.

Locally Low-Rank Regularized Video Stabilization With Motion Diversity Constraints

This paper presents a novel motion aware regularization model with diversity constraints for motion smoothing in video stabilization. Differing from the global path optimization methods, the proposed model puts an emphasis on the relations of inter-frame motions and incorporates sliding windowed low-rank and smoothness constraints. The rationale behind it is cinematography rules which assume that camera motions can be divided into diverse patterns: zero velocity, constant velocity, and acceleration motion. Firstly, a locally motion aware fidelity term is adopted in light of the local motion stationarity. Secondly, to improve the robustness of the model for different motion patterns, a locally low-rank constrained regularization term is further introduced by considering the motion correlation in a local temporal window. Moreover, to cope with the over-smoothing problem in rapid motion situations with extreme acceleration, a motion steering kernel and varying window length are employed to enhance the flexibility of the proposed model. The experimental results demonstrate the superiority of the proposed optimization model and the efficiency to suppress over-smoothing when rapid motions occur. Meanwhile, we compare the results with some state-of-the-art methods quantitatively and qualitatively, and our method can achieve a comparable or even better stabilization effect.

Accurate Tracking of Manoeuvring Target using Scale Estimation and Detection

Camera zoom operation and fast approaching/receding target causes scaling of acquired target in video frames. Fast moving target manifests in large inter-frame motion. In general, non-uniform background degrades performance of tracking algorithms. Fast Fourier transform (FFT)-based Correlation algorithms improve tracking in this scenario, but their applications is limited to small inter-frame motion. Increasing search region has implication on execution speed of the algorithms. Rapid target scaling, non-uniform background and large inter-frame motion of target hinder accurate and long term visual tracking. These challenges have been addressed for extended target tracking by augmenting fast discriminative scale space tracking (fDSST) algorithm with probable target location prediction and target detection. Localisation of fast motion has been achieved by applying fused outputs of Kalman filter and quadratic regression based prediction before applying fDSST. It has helped in accurate localisation of fast motion without increasing search region. In each frame, target location and size have been estimated using fDSST and further refined by target detection near this location. Smoothing and limiting of trajectory and size of detected target has enhanced tracking performance. Experimental results show considerable improvement of precision, success rate and centre location error tracking performance against state-of-the-art trackers in stringent conditions.

Motion-Compensated Spatio-Temporal Filtering for Multi-Image and Multimodal Super-Resolution

The classical multi-image super-resolution model assumes that the super-resolved image is related to the low-resolution frames by warping, convolution and downsampling. State-of-the-art algorithms either use explicit registration to fuse the information for each pixel in its trajectory or exploit spatial and temporal similarities. We propose to combine both ideas, making use of inter-frame motion and exploiting spatio-temporal redundancy with patch-based techniques. We introduce a non-linear filtering approach that combines patches from several frames not necessarily belonging to the same pixel trajectory. The selection of candidate patches depends on a motion-compensated 3D distance, which is robust to noise and aliasing. The selected 3D volumes are then sliced per frame, providing a collection of 2D patches which are finally averaged depending on their similarity to the reference one. This makes the upsampling strategy robust to flow inaccuracies and occlusions. Total variation and nonlocal regularization are used in the deconvolution stage. The experimental results demonstrate the state-of-the-art performance of the proposed method for the super-resolution of videos and light-field images. We also adapt our approach to multimodal sequences when some additional data at the desired resolution is available.

Vehicle tracking based on shape information and inter-frame motion vector

Vehicle tracking is one of the most fundamental aspects in traffic surveillance systems. However, there is always a trade–off between tracking accuracy and speed. Tracking algorithms with excellent tracking accuracy tend to have low processing speeds, whereas fast tracking algorithms usually have low tracking accuracy. This paper proposes a vehicle tracking algorithm based on the simple motion of vehicles in traffic surveillance videos. In the proposed algorithm, an inter–frame motion vector is defined using vehicle speed and acceleration in the current frame to forecast the vehicle speed in subsequent frames. Further, speed and shape error functions are defined to match the target. The results of experiments conducted comparing the proposed method with kernelized correlation filter, multiple instance learning, and Kalman filter indicate that only the proposed algorithm achieves a balance in terms of both speed and accuracy, resulting in satisfactory tracking accuracy and meeting real–time requirements.

Dynamic cardiac MRI reconstruction using motion aligned locally low rank tensor (MALLRT)

Various sparse transform models have been explored for compressed sensing-based dynamic cardiac MRI reconstruction from vastly under-sampled k-space data. Recently emerged low rank tensor model using Tucker decomposition could be viewed as a special form of sparse model, where the core tensor, which is obtained using high-order singular value decomposition, is sparse in the sense that only a few elements have dominantly large magnitude. However, local details tend to be over-smoothed when the entire image is conventionally modeled as a global tensor. Moreover, low rankness is sensitive to motion as spatiotemporal correlation is corrupted by spatial misalignment between temporal frames. To overcome these limitations, this paper presents a novel motion aligned locally low rank tensor (MALLRT) model for dynamic MRI reconstruction. In MALLRT, low rank constraint is enforced on image patch-based local tensors, which correspond to overlapping blocks extracted from the reconstructed high-dimensional image after group-wise inter-frame motion registration. For solving the proposed model, this paper presents an efficient optimization algorithm by using variable splitting and alternating direction method of multipliers (ADMM). MALLRT demonstrated promising performance as validated on one cardiac perfusion MRI dataset and two cardiac cine MRI datasets using retrospective under-sampling with various acceleration factors, as well as one prospectively under-sampled cardiac perfusion MRI dataset. Compared to four state-of-the-art methods, MALLRT achieved substantially better image reconstruction quality in terms of both signal to error ratio (SER) and structural similarity index (SSIM) metrics, and visual perception in preserving spatial details and capturing temporal variations.

Attitude-Trajectory Estimation for Forward-Looking Multibeam Sonar Based on Acoustic Image Registration

This work considers the processing of acoustic data from a multibeam forward-looking sonar (FLS) on a moving underwater platform to estimate the platform's attitude and trajectory. We propose an algorithm to produce an estimate of the attitude trajectory for an FLS based on the optical flow between consecutive sonar frames. The attitude trajectory can be used to locate an underwater platform, such as an autonomous underwater vehicle, to a degree of accuracy suitable for navigation. It can also be used to build a mosaic of the underwater scene. The estimation is performed in three steps. First, a selection of techniques based on the optical flow model is used to estimate a pixel displacement map (DM) between consecutive sonar frames represented in the native polar (range/bearing) format. The second step finds the best match between the estimated DM and DMs for a set of modeled sonar sensor motions. To reduce complexity, it is proposed to describe the DM with a small parameter vector derived from the displacement distribution. Thus, an estimate of the incremental sensor motion between frames is made. Finally, using a weighted regularized spline technique, the incremental interframe motions are integrated into an attitude trajectory for the sonar sensor. To assess the accuracy of the attitude-trajectory estimate, it is used to register FLS frames from a field experiment data set and build a high-quality mosaic of the underwater scene.

Mixed reality and remote sensing application of unmanned aerial vehicle in fire and smoke detection

Abstract This paper proposes the development of a system incorporating inertial measurement unit (IMU), a consumer-grade digital camera and a fire detection algorithm simultaneously with a nano Unmanned Aerial Vehicle (UAV) for inspection purposes. The video streams are collected through the monocular camera and navigation relied on the state-of-the-art indoor/outdoor Simultaneous Localisation and Mapping (SLAM) system. It implements the robotic operating system (ROS) and computer vision algorithm to provide a robust, accurate and unique inter-frame motion estimation. The collected onboard data are communicated to the ground station and used the SLAM system to generate a map of the environment. A robust and efficient re-localization was performed to recover from tracking failure, motion blur, and frame lost in the data received. The fire detection algorithm was deployed based on the color, movement attributes, temporal variation of fire intensity and its accumulation around a point. The cumulative time derivative matrix was utilized to analyze the frame-by-frame changes and to detect areas with high-frequency luminance flicker (random characteristic). Color, surface coarseness, boundary roughness, and skewness features were perceived as the quadrotor flew autonomously within the clutter and congested area. Mixed Reality system was adopted to visualize and test the proposed system in a physical environment, and the virtual simulation was conducted through the Unity game engine. The results showed that the UAV could successfully detect fire and flame, autonomously fly towards and hover around it, communicate with the ground station and simultaneously generate a map of the environment. There was a slight error between the real and virtual UAV calibration due to the ground truth data and the correlation complexity of tracking real and virtual camera coordinate frames.

Motion Compensated Dynamic MRI Reconstruction With Local Affine Optical Flow Estimation.

This paper proposes a novel framework to reconstruct dynamic magnetic resonance imaging (DMRI) with motion compensation (MC). Specifically, by combining the intensity-based optical flow constraint with the traditional compressed sensing scheme, we are able to jointly reconstruct the DMRI sequences and estimate the interframe motion vectors. Then, the DMRI reconstruction can be refined through MC with the estimated motion field. By employing the coarse-to-fine multi-scale resolution strategy, we are able to update the motion field in different spatial scales. The estimated motion vectors need to be interpolated to the finest resolution scale to compensate the DMRI reconstruction. Moreover, the proposed framework is capable of handling a wide class of prior information (regularizations) for DMRI reconstruction, such as sparsity, low rank, and total variation. The formulated optimization problem is solved by a primal-dual algorithm with linesearch due to its efficiency when dealing with non-differentiable problems. Experiments on various DMRI datasets validate the reconstruction quality improvement using the proposed scheme in comparison to several state-of-the-art algorithms.

The need 4 speed in real-time dense visual tracking

The advent of consumer depth cameras has incited the development of a new cohort of algorithms tackling challenging computer vision problems. The primary reason is that depth provides direct geometric information that is largely invariant to texture and illumination. As such, substantial progress has been made in human and object pose estimation, 3D reconstruction and simultaneous localization and mapping. Most of these algorithms naturally benefit from the ability to accurately track the pose of an object or scene of interest from one frame to the next. However, commercially available depth sensors (typically running at 30fps) can allow for large inter-frame motions to occur that make such tracking problematic. A high frame rate depth camera would thus greatly ameliorate these issues, and further increase the tractability of these computer vision problems. Nonetheless, the depth accuracy of recent systems for high-speed depth estimation [Fanello et al. 2017b] can degrade at high frame rates. This is because the active illumination employed produces a low SNR and thus a high exposure time is required to obtain a dense accurate depth image. Furthermore in the presence of rapid motion, longer exposure times produce artifacts due to motion blur, and necessitates a lower frame rate that introduces large inter-frame motion that often yield tracking failures. In contrast, this paper proposes a novel combination of hardware and software components that avoids the need to compromise between a dense accurate depth map and a high frame rate. We document the creation of a full 3D capture system for high speed and quality depth estimation, and demonstrate its advantages in a variety of tracking and reconstruction tasks. We extend the state of the art active stereo algorithm presented in Fanello et al. [2017b] by adding a space-time feature in the matching phase. We also propose a machine learning based depth refinement step that is an order of magnitude faster than traditional postprocessing methods. We quantitatively and qualitatively demonstrate the benefits of the proposed algorithms in the acquisition of geometry in motion. Our pipeline executes in 1.1ms leveraging modern GPUs and off-the-shelf cameras and illumination components. We show how the sensor can be employed in many different applications, from [non-]rigid reconstructions to hand/face tracking. Further, we show many advantages over existing state of the art depth camera technologies beyond framerate, including latency, motion artifacts, multi-path errors, and multi-sensor interference.