Motion Estimation Algorithm Research Articles

Video Superresolution via Motion Compensation and Deep Residual Learning

Video superresolution (SR) techniques are of essential usages for high-resolution display devices due to the current lack of high-resolution videos. Although many algorithms have been proposed, video SR still remains a very challenging inverse problem under different conditions. In this paper, we propose a new method for video SR named motion compensation and residual net (MCResNet). We use optical flow algorithm for motion estimation and motion compensation as a preprocessing step. Then, we employ a novel deep residual convolutional neural network (CNN) to predict a high-resolution image using multiple motion compensated observations. The new residual CNN model preserves the low-frequency contents and facilitates the restoration of high-frequency details. Our method is able to handle large and complex motions adaptively. Extensive experimental results validate that our proposed method outperforms state-of-the-art single-image-based and multi-frame-based algorithms for video SR quantitatively and qualitatively.

A hardware-oriented concurrent TZ search algorithm for High-Efficiency Video Coding

High-Efficiency Video Coding (HEVC) is the latest video coding standard, in which the compression performance is double that of its predecessor, the H.264/AVC standard, while the video quality remains unchanged. In HEVC, the test zone (TZ) search algorithm is widely used for integer motion estimation because it effectively searches the good-quality motion vector with a relatively small amount of computation. However, the complex computation structure of the TZ search algorithm makes it difficult to implement it in the hardware. This paper proposes a new integer motion estimation algorithm which is designed for hardware execution by modifying the conventional TZ search to allow parallel motion estimations of all prediction unit (PU) partitions. The algorithm consists of the three phases of zonal, raster, and refinement searches. At the beginning of each phase, the algorithm obtains the search points required by the original TZ search for all PU partitions in a coding unit (CU). Then, all redundant search points are removed prior to the estimation of the motion costs, and the best search points are then selected for all PUs. Compared to the conventional TZ search algorithm, experimental results show that the proposed algorithm significantly decreases the Bjøntegaard Delta bitrate (BD-BR) by 0.84%, and it also reduces the computational complexity by 54.54%.

A perceptually temporal adaptive quantization algorithm for HEVC

Adaptive quantization (AQ) proves to be an effective coding tool to improve the performance of video coding. This paper presents a perceptually temporal AQ method to improve the subjective coding performance for High Efficiency Video Coding (HEVC). We first put forward a perceptual quality oriented motion estimation algorithm, which is conducted with a spatial-temporal just noticeable distortion (JND) model. Then one perceptual feature in temporal domain is proposed to develop our AQ method, which can generate different quantization parameter (QP) offsets for each coding unit (CU). The proposed method fully utilizes the temporal and perceptual characteristics of each CU, which can produce more visual-friendly QP offsets distribution. Experiments are conducted on HM16.0 (HEVC reference software), and with SSIM (Structure Similarity Index Metric) as the distortion metric, more than 8.08% and 7.95% rate savings can be obtained for Low-Delay-P (LDP) and Low-Delay-B (LDB) configurations on average, respectively. The subjective quality evaluation demonstrates that the proposed AQ method can achieve comparable visual quality as the HM16.0 while the proposed method can yield remarkable bitrate reductions.

Real-time motion estimation diamond search algorithm for the new high efficiency video coding on FPGA

High efficiency video coding (HEVC) is the latest video coding standard aimed to replace the H.264/AVC standard according to its high coding performance, which allows it to be mostly suitable for application in high definition videos. However, this performance is accompanied by a high computational complexity due principally to the motion estimation (ME) algorithm. As in H.264/AVC, the ME in HEVC is a highly computational demanding part that takes the largest part of the whole encoding time. Hence, many fast algorithms have been proposed in order to reduce computation, but, the majority, do not study how they can be effectively implemented by hardware. In this paper, two hardware architectures of the diamond pattern search algorithm for HEVC video coding with sequential and parallel techniques, are proposed. These architectures are based on parallel processing techniques. The sequential and parallel VHDL codes have been verified and can achieve at a high frequency on a Virtex-7 field-programmable gate-array design (FPGA) circuit. Compared to other designs, our parallel design provides better efficient implementation of available resources on FPGA. Our architecture can meet the real-time processing of the FHD @ 30 frames per second.

Motion Detection in High Resolution Enhancement

Shifted Superposition (SSPOS) is a resolution enhancement methodwhere apparent high-resolution content is displayed using a lowresolutionprojection system with an opto-mechanical shifter. WhileSSPOS-enhanced projectors have been showing promising resultsin still images, they still suffer from motion artifacts in video contents.Motivated by this, we present a novel approach to apparentprojector resolution enhancement for videos via motion-basedblurring module. We propose the use of a motion detection moduleand a blurring module to compensate for both SSPOS-resulted andnatural motion artifacts in the video content. To accomplish this,we combine both local and global motion estimation algorithms togenerate accurate dense flow fields. The detected motion regionsare enhanced using directional Gaussian filters. Preliminary resultsshow that the proposed method can produce accurate densemotion vectors and significantly reduce the artifacts in videos.

Syris: a flexible and efficient framework for X-ray imaging experiments simulation

An open-source framework for conducting a broad range of virtual X-ray imaging experiments, syris, is presented. The simulated wavefield created by a source propagates through an arbitrary number of objects until it reaches a detector. The objects in the light path and the source are time-dependent, which enables simulations of dynamic experiments, e.g. four-dimensional time-resolved tomography and laminography. The high-level interface of syris is written in Python and its modularity makes the framework very flexible. The computationally demanding parts behind this interface are implemented in OpenCL, which enables fast calculations on modern graphics processing units. The combination of flexibility and speed opens new possibilities for studying novel imaging methods and systematic search of optimal combinations of measurement conditions and data processing parameters. This can help to increase the success rates and efficiency of valuable synchrotron beam time. To demonstrate the capabilities of the framework, various experiments have been simulated and compared with real data. To show the use case of measurement and data processing parameter optimization based on simulation, a virtual counterpart of a high-speed radiography experiment was created and the simulated data were used to select a suitable motion estimation algorithm; one of its parameters was optimized in order to achieve the best motion estimation accuracy when applied on the real data. syris was also used to simulate tomographic data sets under various imaging conditions which impact the tomographic reconstruction accuracy, and it is shown how the accuracy may guide the selection of imaging conditions for particular use cases.

Adaptive Search Range for HEVC Motion Estimation Based on Depth Information

High Efficiency Video Coding achieves twofold coding efficiency improvement compared with its predecessor H.264/MPEG-4 Advanced Video Coding. However, it suffers from high computational complexity due to its quad-tree structure in motion estimation (ME). This paper exposes the use of depth maps in the multiview video plus depth format for relieving the computational burden. The depth map provides an intimation of the objects' distance from the projected screen in a 3D scene, which is explored in adaptive search range determination in this paper. The proposed algorithm exploits the high temporal correlation between the depth map and the motion in texture. By utilizing this correlation, a depth/motion relationship map is built for a mapping process. For each block, this forms a tailor-made search range with a motion-aware asymmetric shape to skip unnecessary search points in ME. The obtained search range can be further adjusted by taking the influence of 3D-to-2D projection into consideration. Simulation results reveal that, compared to the full search approach, the proposed algorithm can reduce the complexity by 93% on average, whereas the coding efficiency can be maintained. Besides, the proposed search range determination can work well with other fast search ME algorithms in the literature.

An error-based micro-sensor capture system for real-time motion estimation**Research supported by the National Natural Science Foundation of China (Nos. 61431017, 81272166).

A wearable micro-sensor motion capture system with 16 IMUs and an error-compensatory complementary filter algorithm for real-time motion estimation has been developed to acquire accurate 3D orientation and displacement in real life activities. In the proposed filter algorithm, the gyroscope bias error, orientation error and magnetic disturbance error are estimated and compensated, significantly reducing the orientation estimation error due to sensor noise and drift. Displacement estimation, especially for activities such as jumping, has been the challenge in micro-sensor motion capture. An adaptive gait phase detection algorithm has been developed to accommodate accurate displacement estimation in different types of activities. The performance of this system is benchmarked with respect to the results of VICON optical capture system. The experimental results have demonstrated effectiveness of the system in daily activities tracking, with estimation error 0.16 ± 0.06 m for normal walking and 0.13 ± 0.11 m for jumping motions.

Enhanced zonal search algorithm for motion estimation in depth-map coding

This paper presents a novel motion estimation scheme for depth-map coding. Depth map has a couple of different characteristics compared with the common color scene, and therefore, careful investigation of the conventional fast motion search schemes is required. In this paper, I first provide the necessity of the depth-oriented motion search scheme based on experiments and then analyze the problems of the conventional methods. On the basis of the analysis, I propose to use the initial position refinement step during a motion search. In detail, the modification of the one-at-a-time search scheme is proposed for improvement of the prediction accuracy, and an initial direction selection and a 1-bit transform scheme follow for complexity reduction. The experimental results show that the proposed scheme can achieve coding gains of more than 3% on average over the state-of-the-art zonal search methods while keeping the overall complexity increase low.

Vectors through a cross-sectional image (VCI): A visualization method for four-dimensional motion analysis for cardiac computed tomography

BackgroundCardiac computed tomography (CT) has the potential for fully four-dimensional (4D for 3D plus time) motion analysis of the heart. We aimed at developing a method for assessment and presentation of the 4D motion for multi-phase, contrast-enhanced cardiac CT data sets and demonstrating its clinical applicability. MethodsFour patients with normal cardiac function, old myocardial infarction (OMI), takotsubo cardiomyopathy, and hypertrophic cardiomyopathy (HCM) underwent contrast-enhanced cardiac CT for one heartbeat using a 320-row CT scanner with no tube current modulation. CT images for 10 cardiac phases (with a 10%-increment of the R-R interval) were reconstructed with the isotropic effective resolution of (0.5 mm)3 An image-based motion-estimation (iME) algorithm, developed previously, has been used to estimate a time series of 3D cardiac motion, from the end-systole to the other nine phases. The iME uses down-sampled images with a resolution of (1.0 mm)3 deforms the end-systole images non-rigidly to match images at other phases. Once the agreement is maximized, iME outputs a 3D motion vector defined for each voxel for each phase, that smoothly changes over voxels and phases. The proposed visualization method, which is called “vectors through a cross-sectional image (VCI),” presents 3D vectors from the end-diastole to the end-systole as arrows with an end-diastole CT slice. We performed visual assessment of the VCI with calculated the mean vector lengths to evaluate regional left ventricular (LV) contraction. ResultsThe VCI images showed the magnitude and direction of systolic 3D vectors, including the through-plane motion, and successfully visualized the relations of LV wall segments and abnormal regional wall motion. Decreased regional motion and asymmetric motion due to hypokinetic infarct segment, takotsubo cardiomyopathy, and hyper trophic cardiomyopathy was clearly observed. It was easy to appreciate the relation of the abnormal regional wall motion to the affected LV wall segments. The mean vector lengths of the affected segments with pathologies were clearly smaller than the other unaffected segments (1.2–1.7 mm versus 2.5–4.7 mm). ConclusionsVCI images could capture the magnitude and direction of through-plane motion and show the relations of LV wall segments and abnormal wall motion.

A novel search method based on artificial bee colony algorithm for block motion estimation

The large amount of bandwidth that is required for the transmission or storage of digital videos is the main incentive for researchers to develop algorithms that aim at compressing video data while keeping their quality as high as possible. Block matching has been extensively utilized in compression algorithms for motion estimation as they reduce the memory requirements of any video file. A novel method by using modified artificial bee colony algorithm (MABCA) is proposed, which is the combination of basic ABCA and several techniques—initialization based on temporal-spatial correlation, duplicate searching avoidance technique, adaptive search criteria for iteration, and early termination of block matching. Experimental results show that the proposed method has achieved significant improvement over the existing popular block-matching methods in terms of estimation accuracy and computational complexity, especially when dealing with sequences with violent motion. This method can also adaptively adjust the average search point number according to the motion intensity of the different sequences in order to get the best search result at the lowest possible cost.

Fast Motion Estimation Algorithm using Hybrid Search Patterns for Video Streaming Application

The objective of the paper is to develop new block matching Motion Estimation (ME) algorithm using hybrid search patterns along the direction of best match. The search efficiency for sequences with fast motions and high resolutions is improved by proposing New Cross Diagonal-Hexagon Search (NCDHS) algorithm which involves a novel multi half-hexagon grid global search pattern and a cross diagonal-hexagon local search pattern. The new search pattern enables the proposed algorithm to perform better search using 9.068 search points on an average, to obtain optimal motion vector with slight improvement in quality. This inturn reduces ME Time upto 50.11%, 47.12%, 32.99% and 43.28% on average when compared to the existing Diamond Search (DS), Hexagon Search (HS), New Cross Hexagon Search (NHEXS) and Enhanced Diamond Search (EDS) algorithms respectively. The novelty of the algorithm is further achieved by applying the algorithm proposed for live streaming application. The NCDHS algorithm is run on two MATLAB sessions on the same computer by establishing the connection using Transmission Control Protocol (TCP) /Internet Protocol (IP) network. The ME Time obtained is 14.5986 seconds for a block size 16x16, is less when compared to existing algorithms and that makes the NCDHS algorithm suitable for real time streaming application.

Dynamic initial search pattern defined on Cartesian product of neighboring motion vectors for fast block-based motion estimation

Block-based motion estimation is widely used in video compression for reducing the temporal data redundancy. However, it is still a main problem to effectively reduce the computational complexity of motion estimation. The median predictor is usually used for initial search center prediction, however it is not always accurate enough, especially for fast motion sequences. In this paper, a novel dynamic initial search pattern algorithm for fast block-based motion estimation is proposed. Based on the observation that the components of the current motion vector are very similar to the corresponding components of its neighboring motion vectors, Cartesian product of neighboring motion vectors is introduced to generate the proposed dynamic initial search pattern (DISP). And then the cross search pattern is employed to search for the best matching block. The number of search points of the proposed DISP is adaptive to the neighboring correlation of the current block. In fact, the proposed DISP can be considered as a generalization of median prediction scheme and it performs better in capturing the best matching block than median prediction. Experiment results show that the proposed DISP method with small cross search pattern can save about 1.71 search points on average compared with adaptive rood pattern search (ARPS) algorithm and can achieve the similar PSNR to full search (FS) algorithm by combining large cross search pattern.

In brief

PAGE 1192 Researchers in Korea have developed a bi-directional deformable block-based motion-estimation algorithm for accurate motion prediction in 360-degree videos. A significant improvement in visual quality is demonstrated with this approach in comparison to conventional motion estimation techniques. The proposed algorithm can be used for virtual reality applications, motion estimation and frame-rate up conversion. PAGE 1180 Researchers in Australia have fabricated a non-invasive optical sensor for measuring knee movement in 3D space. The triangular sensor works by measuring the angle-of-arrival of optical signals and is integrated with a 2D image processing procedure. The reported sensor is shown to measure knee movement with a high precision that is within the clinically acceptable limit. Improved visual quality in 360-degree videos using a new motion-estimation algorithm. PAGE 1200 The development of sensors for rapid detection of hydrogen gas is currently a hot topic in renewable fuel research. Work from Korea proposes a Pt-functionalised hydrogen sensor developed on an AlGaN/GaN-on-Si heterostructure platform. The sensor was demonstrated to have high sensitivity and a rapid detection response time of 3 seconds in 4% hydrogen concentration. A novel non-invasive sensor design for measuring knee movement. PAGE 1203 A Decoupled Empty-Substrate Integrated Waveguide (DESIW) has been made by researchers in Spain. The DESIW design is a continuous conductive metal structure that enables AC/DC decoupling with integrated devices, reducing unwanted power losses. Measured scattering and transmission parameters show that the design is highly competitive with alternative technologies. The authors present a new hydrogen sensor with rapid detection response time. PAGE 1226 Lens-based millimeter-wave massive MIMO is a key technique for improving data transfer rates and efficiencies in 5G wireless communications. Work from China introduces an adaptive neighbourhood search algorithm for beam selection, which reduces the energy consumption and hardware costs in mm wave MIMO based systems. The proposed beam selection approach achieves near optimal performance with low complexity. A novel DESIW has been designed and manufactured to improve AC/DC decoupling efficiencies. PAGE 1178 The ability to control a wheelchair's movement through the act of thinking has been achieved by researchers in China. The design is controlled by a wireless dry EEG headset acquisition system and has five steering behaviours controlled by the brain-computer interface. A new adaptive neighbourhood search algorithm for efficient beam selection in 5G wireless technology is proposed. PAGE 1169 Work from China proposes a new approach to providing an antenna for slim electronic devices. Using the existing infrastructure of a laptop computer, the design integrates ground-planes and metal hinges to achieve high radiation efficiency without the physical presence of an antenna structure. The proposed design reduces the physical space taken up by an antenna with reduced manufacturing costs. Five steering behaviours are reported in a newly designed brain-controlled wheelchair. PAGE 1196 The reconstruction of images using compressive sensing can produce restoration precision errors resulting in blurring and low resolution. Researchers in China have developed a compressive sensing algorithm that uses a vector quantisation error to suppress the described precision errors. Experimental results demonstrate the algorithm outperforms previously reported compressive sensing methods in aspects of perception and peak signal-to-noise ratios. Designing an antenna within the hinges of slim electronic devices. PAGE 1209 The miniaturisation of a wideband bandpass filter is achieved using stepped impedance by researchers in India. The device also incorporates dual-mode resonators and three types of inter digital capacitors to achieve frequency selectivity. The authors demonstrate the device to display wide stopband properties suitable for terrestrial trunked radio applications. Vector quantisation error parameters are used to remove precision errors in compressive sensing methods. PAGE 1174 Work from Turkey presents a novel design of C-band Fabry-Perot resonator antenna with improved impedance and bandwidth gaining radiation characteristics. The active element of the antenna is a square patch with four crossing points. The antenna is reported to be suitable for a broad range of applications including satellite communication. A miniaturised wideband bandpass filter has been designed with functional frequency selectivity. A new Fabry-Perot resonator antenna design is proposed for satellite communication applications.

Implementation and optimization of sub-pixel motion estimation on BWDSP platform

Sub-pixel Motion estimation algorithm is a key technology in video coding inter-frame prediction algorithm, which has important influence on video coding performance. In the latest video coding standard H.265/HEVC, interpolation filters based on DCT are used to Sub-pixel motion estimation, but it has very high computation complexity. In order to ensure the real-time performance of hardware coding, we combine the characteristics of BWDSP architecture, using code level optimization techniques to realize the sub-pixel motion estimation algorithm. Experimental results demonstrate that In the BWDSP simulation environment, the proposed method significantly decreases the running clock cycle and thus improves the performance of the encoder.

A Hybrid System for Measuring Solid and Liquid Contaminants in Steam Generators

This paper presents a hybrid device for real-time measurement and imaging of solid and liquid contaminants that may occur in steam generators. The device uses a dedicated near infrared device to determine the type of contaminants (i.e., water droplets and iron oxide particles) and a THz imaging system, which measures the amount of contaminants as well as its flow rate. The near infrared response (NIR) device can also determine the concentration of contaminants at sub-mg accuracy when its value is relatively low using spectrometry technique combined with principal component analysis (PCA). Three principal components (PC1, PC2, and PC3) were enough for this purpose. The PCA classification was performed using the least square support vector machine method. In case of relatively high concentration, the THz imaging system which uses block-based motion estimation algorithm can determine the velocity of individual contaminant particles to compute the global motion vector, the intensity and direction of which represents the overall flow rate and flow regime of the contaminants. The usage of image processing techniques together with an NIR spectrometry constitutes a new promising step in flow metering. This is demonstrated by the extensive experiments which have been conducted for different scenario where the NIR subsystem could determine the concentration of water droplets and solid contaminants with a maximum uncertainty of +/– 1.45% and +/– 1.16%, respectively. With the NIR subsystem, pixel-level accuracy of motion vector was achieved, whereas the concentration of solid contaminants showed consisted proportionality with the average pixel intensity.

Bi‐directional deformable block‐based motion estimation for frame rate‐up conversion of 360‐degree videos

A bi-directional deformable block-based motion-estimation algorithm to improve the accuracy of a motion prediction in a 360-degree video, used for motion compensated frame-rate up conversion (MC-FRUC) is proposed. The block shapes are adaptively deformed to predict a motion based on the coordinate of the current block, considering the geometric property of the equi-rectangular projection of 360-degree videos. Experimental results demonstrate that the proposed technique can significantly improve the visual quality of the videos as compared with the conventional MC-FRUC.

Context-aware block-based motion estimation algorithm for multimedia internet of things (IoT) platform

Shaping video data into fast-responding transmission and high resolution output video using cost-effective video processing is desirable in many applications including Internet of Things (IoT) applications. In association with rapid development of IoT smart sensor applications, real-time processing of huge-amount of data for a video signal has become necessary leading to video compression technology. Motion estimation (ME) is necessary for improving the quality, but it has high computational complexity in video compression system. The present article, therefore, proposes a context-aware adaptive pattern-based ME algorithm for multimedia IoT platform to improve video compression. In the proposed algorithm, the motions are classified into large or small based on distortion value. Accordingly, the search pattern is chosen either small diamond search pattern (SDSP) or large diamond search pattern (LDSP) in each and every step of ME; allowing adaptive processing of large and small abstract information. Compared to conventional fast algorithms, the experimental results demonstrate up to 40 and 36% reduction in encoding time for low-delay main (LB-main) and random access main (RA-main) profiles respectively in HEVC test model 16.10 encoder with bit-rate loss of 0.071 and 0.246% for both the profiles, ensuring quality video and searching precision.

Pattern Based Motion Estimation using Zero Motion Pre-judgement and Quantum behaved Particle Swarm Optimization

Motion estimation is a fundamental and resource hungry operation in most of the video coding applications. The most popular method used in any video coding application is block matching motion estimation (BMME). This conventional fast motion estimation algorithm adopts a monotonic error surface for faster computation. However, these search techniques may trap at local minima resulting in erroneous motion estimation. To overcome this issue, various evolutionary swarm intelligence based algorithms were proposed. In this paper, a pattern based motion estimation using zero motion prejudgment and Quantum behaved Particle Swarm Optimization (QPSO) algorithms is proposed, referred to as the Pattern Based Motion Estimation (PBME) algorithm. The notion of QPSO improves the diversity in the search space, which enhances the search efficiency and helps in reduction of the computational burden. At the same time, QPSO needs fewer parameters to control. Therefore, the proposed algorithm enhances the estimation accuracy. An initial search pattern (Hexagonal Based Search) was used which speeds the convergence rate of the algorithm. From the simulation results, it was found that the proposed method outperformed the existing fast block matching (BMA) algorithms of the search point reduction by 40–75%

Video-based arm motion estimation and interaction with fuzzy predictive control

Multi-projector display systems are gaining popularity for use in immersive virtual reality applications and scientific visualization. While recent work has addressed the issues of human interfaces to hide the distributed nature of those systems, there has been relatively little work on natural interactive modalities. In this paper, based on the discrete characteristics of nodes distribution and the spatio-temporal coherence of the user’s movement, we propose a non-contact interaction solution for multi-projector display system. Utilizing a virtual three-dimensional interactive rectangular parallelepiped, we establish correspondence between the virtual scene and the user’s arm position information. For robustly tracking the user’s arm position, an arm motion estimation method is designed based on the fuzzy predictive control theory. To verify the efficiency and accuracy of the proposed method, various motion estimation algorithms were tested with and without the fuzzy predictive control theory to stabilize the output.