Block-based Motion Estimation Research Articles

Block based motion estimation model using CNN with representative point matching algorithm for object tracking in videos

Motion estimation is considered significant for tracking the movement of an object in video sequences, and it is widely used in various video processing applications. Traditionally, many researchers focus on pixel-based motion estimation for object tracking, but it experienced increased computation time and cost. To reduce computation time, the utilization of a block-based motion estimation approach for object tracking is a recent trend. The existing block-based approach faces difficulty in finding representative points within the intensity domain. Therefore, this current research merged the deep learning approach with a block-matching algorithm for achieving efficient object tracking. In this proposed work, initially, video sequences are collected from a benchmark video dataset. Then, the acquired video sequences are segmented into frames. From the segmented frames, current and previous frames are considered for motion estimation. Frames are sent for the data augmentation process in which the process of flipping, cropping, and rotation is carried out. Then, the augmented frames are sent into Convolutional Neural Network (CNN) for feature extraction. Representative Point Matching (RPM) is used to estimate the motion vector based on the extracted features. After estimating the motion vector, the similarity between two consecutive frames is found using Structural Similarity Index (SSIM) technique. Finally, based on the similarity score, the movement of an object in the video is tracked effectively. Simulation analysis of the proposed block-based motion estimation model is done by evaluating some performance metrics. RMSE, PSNR, Execution Time, SSIM, and accuracy obtained for the proposed model are 27.5, 26.5 db, 31 sec, 0.91, and 94 %. This analysis suggested that the proposed CNN-RPM motion estimation model performs better in tracking the movement of the object.

Advanced Patch-Based Affine Motion Estimation for Dynamic Point Cloud Geometry Compression.

The substantial data volume within dynamic point clouds representing three-dimensional moving entities necessitates advancements in compression techniques. Motion estimation (ME) is crucial for reducing point cloud temporal redundancy. Standard block-based ME schemes, which typically utilize the previously decoded point clouds as inter-reference frames, often yield inaccurate and translation-only estimates for dynamic point clouds. To overcome this limitation, we propose an advanced patch-based affine ME scheme for dynamic point cloud geometry compression. Our approach employs a forward-backward jointing ME strategy, generating affine motion-compensated frames for improved inter-geometry references. Before the forward ME process, point cloud motion analysis is conducted on previous frames to perceive motion characteristics. Then, a point cloud is segmented into deformable patches based on geometry correlation and motion coherence. During the forward ME process, affine motion models are introduced to depict the deformable patch motions from the reference to the current frame. Later, affine motion-compensated frames are exploited in the backward ME process to obtain refined motions for better coding performance. Experimental results demonstrate the superiority of our proposed scheme, achieving an average 6.28% geometry bitrate gain over the inter codec anchor. Additional results also validate the effectiveness of key modules within the proposed ME scheme.

Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression.

Block based motion estimation is integral to inter prediction processes performed in hybrid video codecs. Prevalent block matching based methods that are used to compute block motion vectors (MVs) rely on computationally intensive search procedures. They also suffer from the aperture problem, which tends to worsen as the block size is reduced. Moreover, the block matching criteria used in typical codecs do not account for the resulting levels of perceptual quality of the motion compensated pictures that are created upon decoding. Towards achieving the elusive goal of perceptually optimized motion estimation, we propose a search-free block motion estimation framework using a multi-stage convolutional neural network, which is able to conduct motion estimation on multiple block sizes simultaneously, using a triplet of frames as input. This composite block translation network (CBT-Net) is trained in a self-supervised manner on a large database that we created from publicly available uncompressed video content. We deploy the multi-scale structural similarity (MS-SSIM) loss function to optimize the perceptual quality of the motion compensated predicted frames. Our experimental results highlight the computational efficiency of our proposed model relative to conventional block matching based motion estimation algorithms, for comparable prediction errors. Further, when used to perform inter prediction in AV1, the MV predictions of the perceptually optimized model result in average Bjontegaard-delta rate (BD-rate) improvements of -1.73% and -1.31% with respect to the MS-SSIM and Video Multi-Method Assessment Fusion (VMAF) quality metrics, respectively, as compared to the block matching based motion estimation system employed in the SVT-AV1 encoder.

Special issue on International Conference on Computing and Communication Networks

Special issue on International Conference on Computing and Communication Networks

Improved Hybrid Block-Based Motion Estimation for Inter-frame Coding

Digital video technology has been increasingly needed in various fields, such as telecommunications, entertainment, medicine. Therefore, video compression is required. Motion estimation methods help in improving video compression efficiency by effectively removing the temporal redundancy between successive frames. Several block-based motion estimation (BME) algorithms are being suggested to reduce the coding process’s computational complexity. This paper proposes a new rapid hybrid (BME) algorithm established on the primary search point prediction and advance ending search point strategies. It combines rough adaptive search and effective local search. The coarse search introduces a new motion vector (MV) prediction technique that utilizes the macro-blocks (MBs) Spatio-temporal correlations to optimize the traditional adaptive-rood-pattern search algorithm (ARPS) and speeding up the whole process without affecting the accuracy. In the accurate local search, the cross-formed search pattern using a one-step search (OSS) block matching algorithm is employed, to estimate the actual (MV) with less computation time and further speed up the search efficiency. Exhaustive experiments are performed to demonstrate the present algorithm’s performance over the benchmark schemes concerning specific assessment criteria for results, including the peak signal-to-noise ratio (PSNR), computational complexity and computational gain. The results show that the proposed algorithm is efficient and reliable; it can always give better performance over diamond search (DS) and (ARPS). The conducted test shows an increased performance of search speed while preserving the visual quality of the motion-compensated images, and it achieves 59.76–88.03 speed improvement over (DS) and 20.98–72.06 over (ARPS) for different video sequences. Besides, the suggested method (ARP-OSS) provides the best result compared to (DS) and (ARPS) in terms of time complexity for analyzing all the video samples.

RETRACTED: Optimization of Motion Estimation Algorithm Based on FPGA Hardware System and Video Tracking

Motion Estimation (ME) is a computationally expensive step in video tracking. A thorough search technique produces the best execution time and the best accuracy. Computational burden, many fast search algorithms limit and report the number of places to search. Motion estimation, analysis of image sequences, computer vision, tracking a target, and is a critical process in a wide range of fields and applications such as video coding. Precision motion estimation lowers significantly bit rate, but can require a complexity of higher computational. This is different reference frames, sub-pixel estimation, and video compression standards, including techniques such as variable block size, MPEG, especially true for a new generation. Only the motion estimation in H.261 is limited to an integer pixel accuracy, rather than the pixel grid, not in the object is often located, move to move between pixels. The system proposed in the motion vector optimization motion estimation (OME) pixel accuracy. By estimating the amount of displacement at a finer resolution, improve prediction, it can expect better performance than motion estimation using an integer pixel precision. Optimization motion estimation (OME) and the compensation method is very similar to that employed in other standards. The main is that block-based motion estimation and compensation is suitable for optimization structures in MPEG-4. Motion estimation (ME) is one of the most computationally-intensive companies in the video compression technology. Video compression algorithm, MPEG1, MPEG4, such as H.261 and H.264, a lot of the standard used. Improved compression performance. Energy effective motion estimation technique is reduced to the rest of the frame involved in motion compensation.

WITHDRAWN: Developing new search pattern in fast motion estimation algorithm used for compress a video

Due to rapid growing of digital video, it was very complicated task for storing, processing and transferring a sequence of video. Motion Estimation (ME) is a primary tool for video compression. This paper comprises new pattern i.e. rectangular search pattern used in place of diamond search pattern in block-based motion estimation algorithm. The performance can be measured in terms of number of points searching in searching area per macro-block and quality of video by using PSNR value.

Dynamic Scene Change Detection in Video Coding

Video compression has become a source of different research studies. It is necessary in order to address channel bandwidth limitations and growing video demand, including digital libraries and streaming media delivery via the Internet. A video is a number of frames captured by a camera while a scene is a series of consecutive frames captured from a specific narrative viewpoint. To compress a video, firstly, the intra frames are separated from inter frames using scene change detection methods. Then, block-based motion estimation algorithms are used to eliminate the temporal redundancy between successive frames. This paper describes some scene change detection methods for use on the uncompressed video to detect scene types such as cut, dissolve, wipe, etc. Absolute Frame Difference (AFD), Mean Absolute Frame Differences (MAFD), Mean Histogram Absolute Frame Difference (MHAFD), and Maximum Gradient Value (MGV) techniques are adaptively tested on different video types to identify accurate scene change in both low and high object motion scenes. Test results show that the proposed approach (MHAFD) obtains a better accuracy, F1-scor measure of 100%, especially for cuts and gradual transitions (wipe) video types. Dissolve scene change is detected with a high precision of 100% (i.e., no false detection) with the (MAFD) detector. Besides, in terms of time complexity for analyzing all the video samples, the proposed method (MHAFD ) provides the best result compared to the selected detectors.

Deep Network-Based Frame Extrapolation With Reference Frame Alignment

Frame extrapolation is to predict future frames from the past (reference) frames, which has been studied intensively in the computer vision research and has great potential in video coding. Recently, a number of studies have been devoted to the use of deep networks for frame extrapolation, which achieves certain success. However, due to the complex and diverse motion patterns in natural video, it is still difficult to extrapolate frames with high fidelity directly from reference frames. To address this problem, we introduce reference frame alignment as a key technique for deep network-based frame extrapolation. We propose to align the reference frames, e.g. using block-based motion estimation and motion compensation, and then to extrapolate from the aligned frames by a trained deep network. Since the alignment, a preprocessing step, effectively reduces the diversity of network input, we observe that the network is easier to train and the extrapolated frames are of higher quality. We verify the proposed technique in video coding, using the extrapolated frame for inter prediction in High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). We investigate different schemes, including whether to align between the target frame and the reference frames, and whether to perform motion estimation on the extrapolated frame. We conduct a comprehensive set of experiments to study the efficiency of the proposed method and to compare different schemes. Experimental results show that our proposal achieves on average 5.3% and 2.8% BD-rate reduction in Y component compared to HEVC, under low-delay P and low-delay B configurations, respectively. Our proposal performs much better than the frame extrapolation without reference frame alignment.

BlockNet: A Deep Neural Network for Block-Based Motion Estimation Using Representative Matching

Owing to the limitations of practical realizations, block-based motion is widely used as an alternative for pixel-based motion in video applications such as global motion estimation and frame rate up-conversion. We hereby present BlockNet, a compact but effective deep neural architecture for block-based motion estimation. First, BlockNet extracts rich features for a pair of input images. Then, it estimates coarse-to-fine block motion using a pyramidal structure. In each level, block-based motion is estimated using the proposed representative matching with a simple average operator. The experimental results show that BlockNet achieved a similar average end-point error with and without representative matching, whereas the proposed matching incurred 18% lower computational cost than full matching.

Effective Video Saliency Mapping for Object Detection using Deep Learning Neural Networks

In this paper, we propose a new novel DNN-based video saliency prediction method.The main aim of this project is to separating foreground and background images and also finding the motion of the objects.The methodogy used in this project is to seperating the foreground images using background modelling techniques(Pixel searching algorithm).The another technique is Homography based search algorithm for block based motion estimation is used to finding the movement of the objects.Saliency object detection is also used to highlighting the particular region in the images.In our base paper[1],context subtraction algorithm is introduced.But in this paper,Convolutional neural networks is used to classifying the foreground objects from background images.We propose an deep convolutional neural network (CNN) Classifier is used to predict intra-frame salience by exploring object and object movement details .The CNN classifier is also used to separating the new objects from background. We also find from our database that a temporary correlation of human attention exists, with a smooth transition of saliency across video frames.Therefore, in our DNN-based approach we create a two-layer convolutionary long-term memory (2C-LSTM) network, using the extracted CNN features as the data.The fuzzy clustering is used to store the information as the processed images in this project.This project can be implemented in real time applications such as defence,forest and miltrary area.Saliency Mapping is also used to reduce the Zipper effect,it mainly occurs in edges.It gives High resolution too. Finally, the proposed system is developed using matlab simulation.The PSNR value is attained above 95%.

VLSI Architecture of Block Matching Algorithms for Motion Estimation in High Efficiency Video Coding

High-efficiency video coding (HEVC) is a latest video coding standard and the motion estimation unit is the most important block. The work presents the different types of Matching Criteria for Block-Based Motion Estimation technique in HEVC standard. HEVC requires fast motion estimation algorithms to have better real time performance. The hardware implementation of motion estimation helps to achieve high speed though parallel processing. An improved block matching technique is designed with reduced blocks for HEVC. The proposed method has less execution time where only blocks having motion are compared for prediction computation. The searching time complexity is dependent on the number of blocks that are having motion. The searching time of frame having small motion can be reduced to 80–85% as compared to the traditional full search algorithm. In the paper, sum of absolute difference, mean square error and mean absolute difference are computed to find the best matching algorithm for HEVC. However, SAD has less computational complexity with compare to other matching criteria. The results suggest that proposed motion estimation algorithm has better performance with compare to similar previous work.

Low Power Oriented Full Search Block Based Motion Estimation (LP-FSBME) Architecture Using Power Efficient Adder Compressor For H.265 Coding Techniques.

H.265 coding is known as HIGH efficiency video coding (HEVC). This is most successful video compression standard and extended from H.264/MPEG-4 advanced video coding (AVC) for same level of video quality. However, H.265 improved better video quality for same bit rate. In video coding, motion estimation (ME) is determined the motion vector from adjacent frames. Various algorithms have been introduced by many researchers to accomplish low power oriented ME. However, low power oriented full search block based motion estimation (LP-FSBME) algorithm gives accurate results. Architecture of sum of absolute difference (SAD) is used an adder tree to accumulate the processing elements. Power efficient 16:2 adder compressors in SAD architecture reduce the power dissipation rather than convention adders in SAD architecture. The hardware implementation of proposed method is done in Xilinx Virtex 7 FPGA XC7VX1140T device with speed grade 1 in Xilinx software version 14.5 tool, developed in Verilog Hardware Description Language (Verilog-HDL), and simulated in ISE simulator for tennis, BQ terrace and Kimono videos with the resolution of 1080x720 pixels with 30fps.

Motion segmentation-based surveillance video compression using adaptive particle swarm optimization

Video surveillance is one of the widely used and most active research applications of computer vision. Although lots of works have been done in the area of smart surveillance, but still there is a need of effective compression technique for compact archival and efficient transmission of vast amount of surveillance video data. In this work, we propose a hybrid video compression approach with the help of foreground motion compensation for the above application. This method works effectively by including the advantages of both block-based and object-based coding techniques as well as reducing the drawbacks of both. The proposed method first segments the foreground moving objects from the background with the help of adaptive thresholding-based optical flow techniques. Next, it determines the contour of the segmented foreground regions with the help of Freeman chain code. Subsequently, block-based motion estimation and compensation using variants of particle swarm optimization are computed. After that, motion failure areas are detected using change detection method, and finally, DCT and Huffman coding-based entropy encoding are done to compactly represent the data. Experimental results and analyses on different surveillance video sequences using Wilcoxon’s rank-sum test, PSNR and SSID show that our method outperforms other recent and relevant existing techniques.

A New Pentagon Search Algorithm for Fast Block-Matching Motion Estimation

The impact of search pattern is a crucial part in the block-based motion estimation for finding the motion vector. An issue of distortion performance and search speed heavily depends upon the size and shape of search strategy applied. Performing the deep analysis for motion vector distribution on standard test videos, it is desirable to have such type of algorithm that meets the requirement of searching motion vector in less time. Hence, a new kind of Pentagon algorithm is proposed in this paper for fast block-matching motion estimation (BMME). It is an easy and efficient technique for finding motion vector. Experimental results expose the proposed Pentagon algorithm sparsely surpass the noted Diamond search (DS) algorithm. The new Pentagon search algorithm is examined with the previously proposed Diamond search algorithm in terms of performance measure; the proposed algorithm attains better performance with the less complexity. The experimental examination also depicts that the pentagon algorithm is better than the previously proposed Diamond search (DS) in terms of mean-square error performance and required the number of search points. The overall speed improvement rate (SIR) is about 31% with respect to the DS.

Motion Estimation for Fisheye Video With an Application to Temporal Resolution Enhancement

Surveying wide areas with only one camera is a typical scenario in surveillance and automotive applications. Ultra wide-angle fisheye cameras employed to that end produce video data with characteristics that differ significantly from conventional rectilinear imagery as obtained by perspective pinhole cameras. Those characteristics are not considered in typical image and video processing algorithms such as motion estimation, where translation is assumed to be the predominant kind of motion. This contribution introduces an adapted technique for use in block-based motion estimation that takes into the account the projection function of fisheye cameras and thus compensates for the non-perspective properties of fisheye videos. By including suitable projections, the translational motion model that would otherwise only hold for perspective material is exploited, leading to improved motion estimation results without altering the source material. In addition, we discuss extensions that allow for a better prediction of the peripheral image areas, where motion estimation falters due to spatial constraints, and further include calibration information to account for lens properties deviating from the theoretical function. Simulations and experiments are conducted on synthetic as well as real-world fisheye video sequences that are part of a data set created in the context of this paper. Average synthetic and real-world gains of 1.45 and 1.51 dB in luminance PSNR are achieved compared against conventional block matching. Furthermore, the proposed fisheye motion estimation method is successfully applied to motion compensated temporal resolution enhancement, where average gains amount to 0.79 and 0.76 dB.

A hybrid block-based motion estimation algorithm using JAYA for video coding techniques

In any video coding solution, motion estimation (ME) and motion compensation (MC) techniques are widely used and they play an inevitable role in reducing the temporal redundancies between successive frames. Block-based motion estimation (BME) is one of the widely utilized techniques in the recently developed video compression standards by JCT-VC including HEVC due to its efficacy and ease of implementation. In BME, the frames in a video sequence are partitioned into a number of non-overlapping blocks. Then, for each of the blocks, a best-matched block is obtained within a definite search region in the reference frame to minimize certain cost function or fitness value such as the sum of absolute difference (SAD), mean of absolute difference (MAD), or mean square error (MSE). However, the key challenge lies in the evaluation of these cost functions, since they are computationally expensive and involves the most time-taking operations in the BME process. Hence, BME-based approaches can be viewed as an optimization problem and meta-heuristic algorithms can be effectively exploited for the problem under consideration. In this paper, a hybrid BME technique using a recently developed optimization algorithm, namely, JAYA algorithm, is proposed. Besides this proposal, several other modules, namely, the fitness approximation technique, search history preservation, and early & adaptive termination criterion are utilized. The main objective behind the use of the aforementioned modules is to avoid the unnecessary evaluation of the fitness function. Exhaustive simulations are carried out to demonstrate the efficacy of the proposed method over that of the benchmark schemes with respect to different performance measures, namely, the peak signal-to-noise ratio (PSNR), PSNR degradation ratio (DPSNR), search efficiency, structural similarity index measure (SSIM), and computation time. Comparative analysis and quantitative evaluation clearly show that the present technique produces more reliable motion vectors with competitive time complexity as compared to that of the benchmark schemes.

Meteosat image sequence coding in the Radon field

The high demand for Meteosat image sequences for various fields and the high rate of presentation of methods dealing with these latter is facing with the low rate of presentation of remote sensing image sequence compression methods generally for different imagers. To tackle this issue, a novel Meteosat image sequence coding method in the Radon field is presented in this paper. The idea behind this paper is to show the contribution of the coding in Radon field to an efficient lossless compression. The Radon field, represented here by a new fast Mojette transform (FMT), gives a homogeneous field through a Mojette matrix engendered by the intra-projections and inter-projections redundancy. As a consequence, the use of an efficient block-based motion estimation (ME) algorithm such as adaptive rood pattern search (ARPS) becomes possible in a such environment, which can help to avoid many problems encountered when using the same techniques for remote sensing image as a block effect by way of example. In order to achieve a high lossless compression, the Sorted Run Length Coding (SRLC) is applied to encode besides the motion vectors (MVs), the residual error represented by separate long sequences of zeros in addition to other repeated symbols. The test phase showed that our proposed scheme exceeds the performances of the JPEG-LS standard. The proposed scheme can also be used as pre-technique for methods using Meteosat images in general and their sequences in particular after the decompression process such as meteorological phenomena tracking, segmentation, and so on.

Medical Image(s) Watermarking and its Optimization using Genetic Programming

In this paper, an medical image watermarking technique has been proposed, where intelligence has been incorporated into the encoding and decoding structure. The motion vectors of the medical image sequence are used for embedding the watermark. Instead of a manual selection of the candidate motion vectors, a generalized approach is used to select the most suitable motion vectors for embedding the watermark. Genetic programming (GP) module has been employed to develop a function in accordance with imperceptibility and watermarking capacity. Employment of intelligence in the system improves its imperceptibility, capacity, and resistance toward different attacks that can occur during communication and storing. The motion vectors are generated by applying a block-based motion estimation algorithm. In this work, Full-Search method has been used for its better performance as compared to the other methods. Experimental results show marked improvement in capacity and visual similarity as compared to the conventional approaches.

Protection of Ultrasound Image Sequence: Employing Motion Vector Reversible Watermarking

In healthcare information systems, medical data is very important for diagnosis. Most of the health institutions store their patients’ data on third-party servers. Therefore, its security is very important, since the advent of advanced multimedia and communication technology, whereby digital contents are manipulated, copied, and duplicated without leaving any trace. In this paper, a reversible watermarking technique is applied to the patients’ data (ultrasound image sequence). Since the traditional watermarking schemes can experience some permanent distortions that are not acceptable in the medical application. Thus, a reversible watermarking technique has been used, which can not only secure the ultrasound image sequence but also restore the original sequence back. For watermark embedding, the magnitude and phase angles of motion vectors of the image sequence are used that are obtained by using Full-Search block-based motion estimation algorithm. Before applying the motion estimation algorithm and watermark embedding, the histogram pre-processing is performed to avoid underflow/overflow. Unlike other state-of-the-art watermarking schemes that are reported in the last decades, the experimental results show that the proposed algorithm is simple, provides a much larger embedding capacity and better quality of the watermarked image sequence.