Sub-pixel Motion Estimation Research Articles

Target-specified reference-based deep learning network for joint image deblurring and resolution enhancement in surgical zoom lens camera calibration

Background and objectiveFor the augmented reality of surgical navigation, which overlays a 3D model of the surgical target on an image, accurate camera calibration is imperative. However, when the checkerboard images for calibration are captured using a surgical microscope having high magnification, blur owing to the narrow depth of focus and blocking artifacts caused by limited resolution around the fine edges occur. These artifacts strongly affect the localization of corner points of the checkerboard in these images, resulting in inaccurate calibration, which leads to a large displacement in augmented reality. To solve this problem, in this study, we proposed a novel target-specific deep learning network that simultaneously enhances both the blur and spatial resolution of an image for surgical zoom lens camera calibration. MethodsAs a scheme of an end-to-end convolutional deep neural network, the proposed network is specifically intended for the checkerboard image enhancement used in camera calibration. Through the symmetric architecture of the network, which consists of encoding and decoding layers, the distinctive spatial features of the encoding layers are transferred and merged with the output of the decoding layers. Additionally, by integrating a multi-frame framework including subpixel motion estimation and ideal reference image with the symmetric architecture, joint image deblurring and enhanced resolution were efficiently achieved. ResultsFrom experimental comparisons, we verified the capability of the proposed method to improve the subjective and objective performances of surgical microscope calibration. Furthermore, we confirmed that the augmented reality overlap ratio, which quantitatively indicates augmented reality accuracy, from calibration with the enhanced image of the proposed method is higher than that of the previous methods. ConclusionsThese findings suggest that the proposed network provides sharp high-resolution images from blurry low-resolution inputs. Furthermore, we demonstrate superior performance in camera calibration by using surgical microscopic images, thus showing its potential applications in the field of practical surgical navigation.

Subpixel motion artifacts correction and motion estimation for 3D-OCT.

A number of hardware-based and software-based strategies have been suggested to eliminate motion artifacts for improvement of 3D-optical coherence tomography (OCT) image quality. However, the hardware-based strategies have to employ additional hardware to record motion compensation information. Many software-based strategies have to need additional scanning for motion correction at the expense of longer acquisition time. To address this issue, we propose a motion artifacts correction and motion estimation method for OCT volumetric imaging of anterior segment, without requirements of additional hardware and redundant scanning. The motion correction effect with subpixel accuracy for in vivo 3D-OCT has been demonstrated in experiments. Moreover, the physiological information of imaging object, including respiratory curve and respiratory rate, has been experimentally extracted using the proposed method. The proposed method offers a powerful tool for scientific research and clinical diagnosis in ophthalmology and may be further extended for other biomedical volumetric imaging applications.

Fast Subpixel Motion Estimation Based on Human Visual System

More than 80% of video coding times are consumed by motion estimation calculations, which are the most complex aspect of the process. This method eliminates temporal redundancies in a video sequence to achieve maximum compression. Numerous efforts have been made to bring calculations closer to real time, yielding fruitful results. This study proposes a fast subpixel motion estimation algorithm for video encoding with fewer search points. This method employs the capabilities of human visual systems (HVSs), physical motion characteristics of real‐world objects, and special image information from successive frames. The number of search points (NSP) using the statistical data of the movement of the blocks in the frames of video sequences is reduced to apply fewer calculations to the system while maintaining the quality of images. Therefore, it is possible to approach fast and real‐time calculations instead of time‐consuming algorithms by accurately modeling this algorithm.

Fraction Execution Resolver Using a Hybrid Multi-CPU/GPU Encoding Scheme

Modern video coding standards make use of sub-pixel motion estimation to improve the video quality and reduce the bitrate. It is known that the fraction motion estimation (FME) part follows the integer motion estimation (IME) and adds an extra computational overhead due to the interpolation and the additional motion searches. In this paper, we propose a fraction execution resolver (FER) algorithm that lets the encoder skip the fraction part when specific criteria are met by introducing a preliminary fast test decision point (pFTDP) function for the IME part. If the pFTDP returns zero motion vectors (MVs) and the displacement search area center is also zero, then the fraction part is skipped. The pFTDP decision maker is executed only once, when a 2N × 2N block is first met, while all subsequent blocks follow this initial decision either by receiving the necessary MVs and RD from the pFTDP function or by using the precalculated IME values from the GPU kernel. For our experiments, we use a multithreaded CPU environment that also makes use of GPUs only for the integer part. Our evaluations provide a greater than 1600% encoding time saving at its peak in comparison with the default HEVC sequential mode and ideally a saving of greater than 2286% for still video frame sequences. The total average speedup for both Class A and Class B video sequences is ×13.45. The gain of the FER itself is more than ×3.9 compared with the same multithreaded setup environment. The PSNR and bitrate overhead observed are proportional to the tiling scheme used and are more related to the way CABAC works internally. The FER’s negative effects on coding efficiency are proven to be negligible. A balance between speed and quality achieved by using a lower tiling pattern is shown to minimize the negative effects of the encoding scheme pattern. The experimental results confirm the validity of our motivation, namely, that we can benefit from a software fraction execution resolver without any extra hardware costs. The gain is further increased when video sequences have more static blocks than others.

Utilizing Spatio Temporal Gait Pattern and Quadratic SVM for Gait Recognition

This study aimed to develop a vision-based gait recognition system for person identification. Gait is the soft biometric trait recognizable from low-resolution surveillance videos, where the face and other hard biometrics are not even extractable. The gait is a cycle pattern of human body locomotion that consists of two sequential phases: swing and stance. The gait features of the complete gait cycle, referred to as gait signature, can be used for person identification. The proposed work utilizes gait dynamics for gait feature extraction. For this purpose, the spatio temporal power spectral gait features are utilized for gait dynamics captured through sub-pixel motion estimation, and they are less affected by the subject’s appearance. The spatio temporal power spectral gait features are utilized for a quadratic support vector machine classifier for gait recognition aiming for person identification. Spatio temporal power spectral preserves the spatiotemporal gait features and is adaptable for a quadratic support vector machine classifier-based gait recognition across different views and appearances. We have evaluated the gait features and support vector machine classifier-based gait recognition on a locally collected gait dataset that captures the effect of view variance in high scene depth videos. The proposed gait recognition technique achieves significant accuracy across all appearances and views.

Extracting Small Flying Airplane With Spatially Accurate and Temporally Consistent Foreground Modeling

Extracting the flying airplane from the satellite video has received a great deal of attention in remote sensing applications. Nevertheless, the challenge arises in two aspects: the limited spatial resolution of space-borne imaging devices, and the small size of the airplane with low contrast to the background. This paper presents a novel spatially accurate and temporally consistent model for the foreground airplane extraction. Specifically, each frame is decomposed as a linear combination of the airplane and background with a kind of mixture ratios. This enables more accurate spatial representation of the small airplane of low resolution. Then, the airplanes from different frames obey the common mixture ratios based on the sub-pixel motion estimation. This enables the temporally consistent extraction through frames. The effectiveness of our method is demonstrated by the qualitative and quantitative evaluation on both synthetic and real data.

Blind MV-based video steganalysis based on joint inter-frame and intra-frame statistics

Despite all its irrefutable benefits, the development of steganography methods has sparked ever-increasing concerns over steganography abuse in recent decades. To prevent the inimical usage of steganography, steganalysis approaches have been introduced. Since motion vector manipulation leads to random and indirect changes in the statistics of videos, MV-based video steganography has been the center of attention in recent years. In this paper, we propose a 54-dimentional feature set exploiting spatio-temporal features of motion vectors to blindly detect MV-based stego videos. The idea behind the proposed features originates from two facts. First, there are strong dependencies among neighboring MVs due to utilizing rate-distortion optimization techniques and belonging to the same rigid object or static background. Accordingly, MV manipulation can leave important clues on the differences between each MV and the MVs belonging to the neighboring blocks. Second, a majority of MVs in original videos are locally optimal after decoding concerning the Lagrangian multiplier, notwithstanding the information loss during compression. Motion vector alteration during information embedding can affect these statistics that can be utilized for steganalysis. Experimental results have shown that our features’ performance far exceeds that of state-of-the-art steganalysis methods. This outstanding performance lies in the utilization of complementary spatio-temporal statistics affected by MV manipulation as well as feature dimensionality reduction applied to prevent overfitting. Moreover, unlike other existing MV-based steganalysis methods, our proposed features can be adjusted to various settings of the state-of-the-art video codec standards such as sub-pixel motion estimation and variable-block-size motion estimation.

Subpixel and On-Line Motion Correction for Photoacoustic Dermoscopy

Photoacoustic dermoscopy (PAD) has been proven to visualize the microvascular network within the dermis noninvasively, and has great application potential and advantages. In practice, some subjects are children or unwell and may have difficulty in controlling their breathing or trembling. The artifacts caused by the uncontrollable trembling will degrade the image quality, resulting in deviation of the pathophysiological features. This work introduces a subpixel and on-line motion correction method for clinical application of PAD. The high-accuracy motion correction was realized by the subpixel motion estimation, which was achieved by five-fold upsampling the cross-correlation matrix between A-lines or B-scan images. Since the motion correction algorithm only takes the acquired data as a priori, it can be processed immediately when a B-scan is input, so it can be embedded in the acquisition program, leading to the motion correction during the data acquisition. Moreover, the algorithm was verified by the experimental data of human skin. The results demonstrated that the images after sub-pixel motion correction are obviously improved visually, and the structural similarity index measurement (SSIM) and peak signal-to-noise ratio (PSNR) between adjacent B-scan images after correction are increased by 31.6% and 47.7% compared to the uncorrected images. To conclude, the proposed algorithm can make it robust in terms of accuracy and reliability and will help PAD fulfill its clinical promise.

Motion estimation using maximum sub-image and sub-pixel phase correlation on a DSP platform

Motion estimation is a key step for many video process systems, but it usually suffers from low precision in complex imaging scenarios, as well as the problem of real-time processing. This paper proposes an accurate and fast motion estimation algorithm using maximum sub-image and sub-pixel phase correlation implemented on a DSP platform. Firstly, a maximum sub-image is extracted to contain the image content as much as possible, which both satisfies the requirement of Fourier precondition and the wide coverage of complete background. And then, the extracted sub-image is down-sampled with median filter to increase the signal-noise-ratio and decrease the computation load. Secondly, a sub-pixel motion estimation is used to compensate the losing precision due to down-sampling, and keep the range of motion estimation. Finally, the proposed motion estimation algorithm is implemented on a single core TMS320C6678 DSP platform, and it is accelerated by applying multistage data cache and advanced data access. Experiments demonstrate the accuracy of the proposed motion estimation algorithm in complex scenarios. Meanwhile, it can achieve 7.4 ms/frame for sub-image with 512 × 512 pixels size and 29.6 ms/frame for sub-image with 1024 × 1024 pixels size, respectively.

Phase Amplified Correlation for Improved Sub-pixel Motion Estimation.

Phase correlation (PC) is widely employed by several sub-pixel motion estimation techniques in an attempt to accurately and robustly detect the displacement between two images. To achieve sub-pixel accuracy, these techniques employ interpolation methods and function-fitting approaches on the cross-correlation function derived from the PC core. However, such motion estimation techniques still present a lower bound of accuracy that cannot be overcome. To allow room for further improvements, we propose in this paper the enhancement of the sub-pixel accuracy of motion estimation techniques by employing a completely different approach: the concept of motion magnification. To this end, we propose the novel phase amplified correlation (PAC) that integrates motion magnification between two compared images inside the phase correlation part of frequencybased motion estimation algorithms and thus directly substitutes the PC core. The experimentation on magnetic resonance (MR) images and real video sequences demonstrates the ability of the proposed PAC core to make subtle motions highly distinguishable and improve the sub-pixel accuracy of frequency-based motion estimation techniques.

A sub-pixel motion estimation skipping method for fast HEVC encoding

Abstract While motion estimation (ME) efficiently compresses video data by reducing temporal redundancy between frames, it does so at the cost of computational complexity. In particular, the complexity of sub-pixel ME (SME) greatly limits fast video encoding because the SME requires a complexity-intensive interpolation process to generate imaginary pixels. Thus, this paper proposes a method that conditionally skips SME, including its interpolation process, by utilizing previously encoded motion information. Results show that the proposed method efficiently reduces both the total encoding time and memory access compared to the previous high efficiency video coding (HEVC) reference model.

Directional Transforms for Video Coding Based on Lifting on Graphs

In this paper, we describe and optimize a general scheme based on lifting transforms on graphs for video coding. A graph is constructed to represent the video signal. Each pixel becomes a node in the graph and links between nodes represent similarity between them. Therefore, spatial neighbors and temporal motion-related pixels can be linked, while nonsimilar pixels (e.g., pixels across an edge) may not be. Then, a lifting-based transform, in which filtering operations are performed using linked nodes, is applied to this graph, leading to a 3D (spatio-temporal) directional transform, which can be viewed as an extension of wavelet transforms for video. The design of the proposed scheme requires four main steps: 1) graph construction; 2) graph splitting; 3) filter design; and 4) extension of the transform to different levels of decomposition. We focus on the optimization of these steps in order to obtain an effective transform for video coding. Furthermore, based on this scheme, we propose a coefficient reordering method and an entropy coder leading to a complete video encoder that achieves better coding performance than a motion-compensated temporal filtering wavelet-based encoder, and a simple encoder derived from H.264/AVC that makes use of similar tools as our proposed encoder (reference software JM15.1 configured to use one reference frame, no subpixel motion estimation, and $16 \times 16$ inter and $4 \times 4$ intra modes).

Block unshifting high-accuracy motion estimation: A new method adapted to super-resolution enhancement

Sub-pixel motion estimation plays a vital role in a multitude of video applications, including encoding, audiovisual archiving/heritage and super-resolution enhancement. Most existing block-based methods rely on the implicit assumption that blocks can be accurately predicted through appropriate shifts. In particular, shifted blocks in the target frame are estimated from the associated anchor frame blocks. The present paper introduces a different strategy, which discards this assumption and treats anchor and target frame blocks equally, as sub-pixel shifted versions of an unavailable implied block. The new method attempts to construct this implied block and, by calculating the “imaginary” motion vectors that relate it to the two existing blocks, it estimates the wanted motion vectors more accurately. This approach aims at extracting motion vectors that more accurately represent the actual movements of objects, minimizing the interpolation error that is associated with sub-pixel shifting, which manifests as blurring and a lowering of contrast. The new method focuses on accurate motion estimation, paying less attention to the associated computational load. Hence, the approach is both inspired from, and proposed for, super-resolution enhancement scenarios, where higher definition motion image sequences are estimated from their available lower definition counterparts. In order to implement the new strategy, an algorithm for reversing the bilinear sub-pixel shift of a block (unshifting) is implemented and validated. Comparisons between original blocks of images and blocks that have been shifted and unshifted back to their original coordinates showcase the accuracy of the unshifting process. The proposed motion estimation method is evaluated through a number of different experimental assessment procedures and metrics, comparing it to existing high-accuracy state-of-the-art motion estimation methods.

Implementation and optimization of sub-pixel motion estimation on BWDSP platform

Sub-pixel Motion estimation algorithm is a key technology in video coding inter-frame prediction algorithm, which has important influence on video coding performance. In the latest video coding standard H.265/HEVC, interpolation filters based on DCT are used to Sub-pixel motion estimation, but it has very high computation complexity. In order to ensure the real-time performance of hardware coding, we combine the characteristics of BWDSP architecture, using code level optimization techniques to realize the sub-pixel motion estimation algorithm. Experimental results demonstrate that In the BWDSP simulation environment, the proposed method significantly decreases the running clock cycle and thus improves the performance of the encoder.

High accuracy block-matching sub-pixel motion estimation through detection of error surface minima

The present paper focuses on high-accuracy block-based sub-pixel motion estimation utilizing a straightforward error minimization approach. In particular, the mathematics of bilinear interpolation are utilized for the selection of the candidate motion vectors that minimize the error criterion, by estimating local minima in the error surface with arbitrary accuracy. The implemented approach favors optimum accuracy over computational load demands, making it ideal as a benchmark for faster methods to compare against; however, it is not best suited to real-time critical applications (i.e. video compression). Other video processing needs relying on motion vectors and requiring high-resolution/accuracy can also take advantage of the proposed solution (and its simplified nature in terms of underlying theoretical complexity), such as motion-compensation filtering for super resolution image enhancement, motion analysis in sensitive areas (e.g. high-speed video monitoring, medical imaging, motion analysis in sport science, big-data visual surveillance, etc.). The proposed method is thoroughly evaluated using both real video and synthetic motion sequences from still images, adopting well-tested block-based motion estimation evaluation procedures. Assessment includes comparisons to a number of existing block-based methods with respect to PSNR and SSIM metrics over ground-truth samples. The conducted evaluation takes into consideration both the original (arbitrary-accuracy) and the truncated motion vectors (after rounding them to the nearest half, quarter, or eighth of a pixel), where superior performance with more accurate motion vector estimation is revealed. In this context, the degree to which sub-pixel motion estimation methods actually produce sub-pixel motion vectors is investigated, and the implications thereof are discussed.

Multidirectional parabolic prediction-based interpolation-free sub-pixel motion estimation

Motion estimation is one of the most effective key components in video codecs to eliminate temporal redundancies between successive frames. Sub-pixel motion estimation, including sub-pixel interpolation and sub-pixel motion vector prediction, has been employed in many recent international video coding standards to further improve the rate-distortion performance. Although sub-pixel motion estimation plays an important role in improving coding efficiency, it simultaneously brings significant computational complexity, especially in sub-pixel interpolation and motion vector prediction. In this paper, we propose a multidirectional parabolic prediction-based interpolation-free sub-pixel motion estimation scheme. First, four different directional parabolic prediction patterns are developed to estimate the candidate optimal sub-pixel positions, making full use of the available rate-distortion costs of neighboring positions in the integer-pixel motion estimation. Second, an improvement for each directional parabolic prediction pattern is realized by projection operations to generate a more precise candidate optimal sub-pixel position. Third, an optimal sub-pixel position decision method is presented to select the most likely predictive position from the four candidate sub-pixel positions derived in the previous steps. Finally, by fully reusing the information of the powerful advanced motion vector prediction (AMVP) in HEVC, a prediction enhancement mechanism is presented in which the final optimal sub-pixel motion vector (MV) is derived between the MV corresponding to the most likely predictive position and the predictive motion vector (PMV), without requiring extra interpolation operations. Experimental results show that the proposed algorithm significantly reduces the computational complexity of sub-pixel motion estimation and outperforms state-of-the-art interpolation-free sub-pixel motion estimation algorithms with much improved encoding performance and comparable computational complexity reduction.

Sub-pixel motion estimation using phase correlation: comparisons and evaluations

Purpose Effective and robust motion estimation with sub-pixel accuracy is essential in many image processing and computer vision applications. Due to its computational efficiency and robustness in the presence of intensity changes as well as geometric distortions, phase correlation in the Fourier domain provides an attractive solution for global motion estimation and image registration. The paper aims to discuss these issues. Design/methodology/approach In this paper, relevant sub-pixel strategies are categorized into three classes, namely, single-side peak interpolation, dual-side peak interpolation and curve fitting. The well-known images “Barbara” and “Pentagon” were used to evaluate the performance of eight typical methods, in which Gaussian noise was attached in the synthetic data. Findings For eight such typical methods, the tests using synthetic data have suggested that considering dual-side peaks in interpolation or fitting helps to produce better results. In addition, dual-side interpolation outperforms curve fitting methods in dealing with noisy samples. Overall, Gaussian-based dual-side interpolation seems the best in the experiments. Originality/value Based on the comparisons of eight typical methods, the authors can have a better understanding of the phase correlation for motion estimation. The evaluation can provide useful guidance in this context.

Low-Complexity Sub-Pixel Motion Estimation Utilizing Shifting Matrix in Transform Domain

Motion estimation (ME) algorithms supporting quarter-pixel accuracy have been recently introduced to retain detailed motion information for high quality of video in the state-of-the-art video compression standard of H.264/AVC. Conventional sub-pixel ME algorithms in the spatial domain are faced with a common problem of computational complexity because of embedded interpolation schemes. This paper proposes a low-complexity sub-pixel motion estimation algorithm in the transform domain utilizing shifting matrix. Simulations are performed to compare the performances of spatial-domain ME algorithms and transform-domain ME algorithms in terms of peak signal-to-noise ratio (PSNR) and the number of bits per frame. Simulation results confirm that the transform-domain approach not only improves the video quality and the compression efficiency, but also remarkably alleviates the computational complexity, compared to the spatial-domain approach.

Super‐resolution reconstruction based on multisource bidirectional similarity and non‐local similarity matching

A novel super-resolution (SR) reconstruction algorithm based on multisource bidirectional similarity and non-local similarity matching for multi-exposure dynamic image sequences is proposed in this study. First, luminance compensation for multi-exposure dynamic images is implemented based on multisource bidirectional similarity. Then, combining a self-adaptive regional correlation evaluation strategy and a weighting strategy based on pseudo-Zernike moment feature similarity and structural similarity yields a novel and robust non-local similarity matching scheme (PZ-NSM) to learn non-local similarity priors between low- and high-resolution dynamic image patches at different spatio-temporal scales without additional training. Finally, SR reconstruction is implemented based on PZ-NSM and spatio-temporal trans-scale fusion of non-local similarities between dynamic images. The proposed algorithm does not rely on accurate estimation of subpixel motion and can therefore be adapted to more complex motion patterns. It has high rotation invariance effectiveness and is robust to noise and illumination. Experimental results demonstrate that the proposed algorithm outperforms existing algorithms in terms of both subjective and objective evaluations.

Temporally coherent superresolution of textured video via dynamic texture synthesis.

This paper addresses the problem of hallucinating the missing high-resolution (HR) details of a low-resolution (LR) video while maintaining the temporal coherence of the reconstructed HR details using dynamic texture synthesis (DTS). Most existing multiframe-based video superresolution (SR) methods suffer from the problem of limited reconstructed visual quality due to inaccurate subpixel motion estimation between frames in an LR video. To achieve high-quality reconstruction of HR details for an LR video, we propose a texture-synthesis (TS)-based video SR method, in which a novel DTS scheme is proposed to render the reconstructed HR details in a temporally coherent way, which effectively addresses the temporal incoherence problem caused by traditional TS-based image SR methods. To further reduce the complexity of the proposed method, our method only performs the TS-based SR on a set of key frames, while the HR details of the remaining nonkey frames are simply predicted using the bidirectional overlapped block motion compensation. After all frames are upscaled, the proposed DTS-SR is applied to maintain the temporal coherence in the HR video. Experimental results demonstrate that the proposed method achieves significant subjective and objective visual quality improvement over state-of-the-art video SR methods.