Local Motion Compensation Research Articles

Flexible and accurate total variation and cascaded denoisers-based image reconstruction algorithm for hyperspectrally compressed ultrafast photography

Hyperspectrally compressed ultrafast photography (HCUP) based on compressed sensing and time- and spectrum-to-space mappings can simultaneously realize the temporal and spectral imaging of non-repeatable or difficult-to-repeat transient events with a passive manner in single exposure. HCUP possesses an incredibly high frame rate of tens of trillions of frames per second and a sequence depth of several hundred, and therefore plays a revolutionary role in single-shot ultrafast optical imaging. However, due to ultra-high data compression ratios induced by the extremely large sequence depth, as well as limited fidelities of traditional algorithms over the image reconstruction process, HCUP suffers from a poor image reconstruction quality and fails to capture fine structures in complex transient scenes. To overcome these restrictions, we report a flexible image reconstruction algorithm based on a total variation (TV) and cascaded denoisers (CD) for HCUP, named the TV-CD algorithm. The TV-CD algorithm applies the TV denoising model cascaded with several advanced deep learning-based denoising models in the iterative plug-and-play alternating direction method of multipliers framework, which not only preserves the image smoothness with TV, but also obtains more priori with CD. Therefore, it solves the common sparsity representation problem in local similarity and motion compensation. Both the simulation and experimental results show that the proposed TV-CD algorithm can effectively improve the image reconstruction accuracy and quality of HCUP, and may further promote the practical applications of HCUP in capturing high-dimensional complex physical, chemical and biological ultrafast dynamic scenes.

Fusion of Global and Local Motion Estimation Using Foreground Objects for Distributed Video Coding

The side information (SI) in Distributed Video Coding (DVC) is estimated using the available decoded frames and exploited for the decoding and reconstruction of other frames. The quality of the SI has a strong impact on the performance of DVC. Here, we propose a new approach that combines both global and local SI to improve coding performance. Since the background pixels in a frame are assigned to global estimation and the foreground objects to local estimation, one needs to estimate foreground objects in the SI using the backward and forward foreground objects, the background pixels are directly taken from the global SI. Specifically, elastic curves and local motion compensation are used to generate the foreground objects masks in the SI. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.39 dB for a group of picture (GOP) size of 2, and up to 4.73 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

Fusion of Global and Local Motion Estimation for Distributed Video Coding

The quality of side information plays a key role in distributed video coding. In this paper, we propose a new approach that consists of combining global and local motion compensation at the decoder side. The parameters of the global motion are estimated at the encoder using scale invariant feature transform features. Those estimated parameters are sent to the decoder in order to generate a globally motion compensated side information. Conversely, a locally motion compensated side information is generated at the decoder based on motion-compensated temporal interpolation of neighboring reference frames. Moreover, an improved fusion of global and local side information during the decoding process is achieved using the partially decoded Wyner-Ziv frame and decoded reference frames. The proposed technique improves significantly the quality of the side information, especially for sequences containing high global motion. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.9 dB for a Group of Pictures (GOP) size of 2, and up to 4.65 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

Turbulence compensation enhances long-range imagery

In military scenarios it is of crucial importance to observe over long distances, or in other words, to have a large recognition range. Visible light and IR cameras are those mainly used for this task. However, the recognition range—especially for ‘landto-land’ observation (i.e., along horizontal paths at ground/sea level)—is hampered by the optical effects of atmospheric turbulence, which are caused by local variations in the refractive index of air. These in turn are caused by temperature and humidity variations along the observation path. In images, atmospheric turbulence is observed as locally varying blur and beam wander. If we could reduce these effects the recognition range would be extended significantly. Many hardware and software solutions have been developed in the past. Some hardware solutions use adaptive optics1 and are used for astronomical applications. However, these are expensive and have only one mode, that of performing the same processing on the whole image. Software solutions are much cheaper and can be used in combination with existing cameras. However, a lot of these software methods2, 3 also use a global approach (i.e., applying a uniform process across the whole image), whereas for land-to-land observation the effects of atmospheric turbulence are local. We propose a software solution that performs local processing to reduce these effects. Figure 1 shows a flowchart of our method, which can be divided into three parts: first, frame selection to select the sharpest frames; second, local motion compensation to stabilize the observed scene; and third, multi-frame deblurring to increase the resolution and remove the blur in each frame. This processing results in a more stable image, which is more comfortable and less tiresome for the operator to watch. In addition, more details will be visible, because the resolution is enhanced and the image is deblurred. Our solution is flexible and can be implemented in real time. Figure 1. Flowchart of our turbulence compensation method.

Edge-Based Blotch Identification Algorithm In Archive Film Restoration

Abstract Previous blotch identification algorithms perform identification process for each pixel in current frame, which are time-consuming. To deal with this issue, a blotch identification algorithm based on edge detection is proposed in this work. Above all, edges of current frame are extracted by edge detectors, blotch identification area is limited to edge pixels and pixels nearby. To further reduce time complexity, anon-blotch edge removal approach is proposed to remove partial non-blotch edge points.T hen, a scanning method combined with local motion compensation (MC)-based SROD blotch identification algorithm is used to identify all blotches from remaining edge points. Abundant experiments have been done to verify our method and compared our method to other two typical previous algorithms. Experimental results show that the proposed approach gives successful identification performances, provides the computational simplicity and outperforms classical algorithms.

Missing-data recovery from dirt sparkles on degraded color films

A novel spatiotemporal method is proposed for detection of and recovery from dirt sparkles on degraded color films. Firstly, a confidence measurement of dirt is extracted by comparing pixel values per color component after global motion compensation. Then, candidate dirt is detected by filtering and thresholding this confidence measurement. For each candidate region of dirt, bidirectional local motion compensation is employed, and motion-compensated pixels are selected according to their confidence values, using an improved ML3Dex filter to preserve details and avoid oversmoothing of images. Experiments on real data demonstrate that our method outperforms several well-established algorithms in accuracy, efficiency, and robustness.

Temporal shape error concealment by global motion compensation with local refinement

This paper presents an original temporal shape error concealment technique based on a combination of global and local motion compensation. For this technique, which is especially useful for object-based video applications in error-prone environments (e.g., mobile networks), it is assumed that the shape of the corrupted object at hand is in the form of a binary alpha plane and some of the shape data is missing due to channel errors. To conceal the corrupted shape, the decoder first assumes that a global motion model can describe the shape changes in consecutive time instants. This way, based on locally estimated global motion parameters, the decoder attempts to conceal the corrupted alpha plane by global motion compensating the shape data from the previous time instant. Afterwards, since a global motion model cannot perfectly describe all alpha plane changes, a local motion refinement is applied to improve the concealment in areas of the object with significant local motion.

Video de-interlacing by adaptive 4-field global/local motion compensated approach

A de-interlacing algorithm using adaptive 4-field global/local motion compensated approach is presented. It consists of block-based directional edge interpolation, same-parity 4-field motion detection, global/local motion estimation and compensation. The edges are sharper when the directional edge interpolation is adopted. The same parity 4-field motion detection and the 4-field local motion estimation detect the static areas and fast motion by four reference fields, and the global motion estimation detects the camera panning and zooming motions. The global and local motion compensation recover the interlaced videos to the progressive ones. Experimental results show that the peak signal-to-noise ratio of our proposed algorithm is 2/spl sim/3 dB higher than that of previous studies and attain the best quality of subjective view.

Fast local motion-compensation algorithm for video sequences with brightness variations

Proposes a fast local motion-compensation algorithm that improves motion-compensation performance for video sequences with brightness variations. The brightness variation parameters, a multiplier and an offset field for image intensity, are robustly estimated and local motions are compensated. We also propose the frame classification method based on the cross entropy between histograms of two successive frames, to detect the frame with brightness variations. Simulation results show that the proposed method yields a higher peak signal-to-noise ratio than the conventional methods, with a low computational load, when the video scene contains large brightness changes.

Hybrid global–local motion compensated frame interpolation for low bit rate video coding

A hybrid global–local motion compensated frame interpolation (HMCI) scheme for low bit rate video coding is proposed in this paper. Different from most other global–local motion compensation algorithms, which were performed at the encoder end, the proposed scheme works solely on the decoded video stream. Thus, it can be applied to any MCP (motion compensated prediction) based standard coders without bit stream syntax modification. Along with the proposed HMCI scheme, a motion-based foreground–background segmentation method is presented. Traditional intensity-based segmentation methods can only segment moving objects from still background. In contrast, the proposed approach can segment foreground objects from both moving and still background by exploiting information carried by block-based motion vectors. Both the six-parameter affine or the eight-parameter perspective models can be applied to the foreground–background segmentation and the moving background reconstruction; while the six-parameter affine model with triangular patch warping is adopted to reconstruct the foreground blocks due to its low complexity. Experiments show that the proposed HMCI scheme achieves higher subjective (visual) quality compared to the conventional block-based frame interpolation without global motion compensation, where artifacts in moving backgrounds are significantly reduced.

Video coding using global motion and brightness‐variation compensation with complexity reduction

AbstractGlobal motion compensation and global brightness‐variation compensation are proposed in order to deal with motion or brightness variation over the whole image. It is expected that the coding efficiency will be greatly improved by these techniques. The purpose of this investigation is to reduce the tremendous amount of computation which has especially hindered global brightness‐variation compensation. A simple method is introduced for blockwise motion estimation processing and brightness‐variation parameter estimation processing. A coding scheme is proposed, with the motion estimation procedure shared between global motion compensation and local motion compensation, so that the two functions are realized simultaneously with smaller complexity. In the case of the block size and the range of motion search in the experiment, the number of multiplications is reduced to about 1/400 as many as when the conventional compensation procedures are simply combined. As regards the coding efficiency, we demonstrate improvement integrating the effects obtained when global motion compensation and brightness‐variation compensation are separately applied. © 2002 Wiley Periodicals, Inc. Electron Comm Jpn Pt 1, 86(2): 80–93, 2003; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ecja.1156

Embedded video coding using invertible motion compensated 3-D subband/wavelet filter bank

Abstract In this paper, we present a new embedded video coding system with local motion compensation. An invertible motion compensated 3-D subband filter bank is utilized for video analysis/synthesis. The efficient embedded image coding scheme EZBC is extended to 3-D coding of video subbands. The coded bitstream is rate scalable and fully embedded thus meeting the increasing need for fine granular scalability in video streaming applications. Our experimental results show the new algorithm outperforms nonscalable standard MPEG-2 by 0.8– 3.3 dB (luminance component) over a broad range of bitrates, while providing an embedded video bitstream. Unlike conventional SNR hybrid coders, this multirate property is achieved without a loss in compression efficiency. Comparisons are also made to two other competing embedded video coders, LZC and 3D-SPIHT, both based on 3-D subband coding as well. The new coder demonstrates a significant advantage for encoding video with high local motion content, thanks to the efficient inclusion of local motion information in the temporal filter bank of our proposed system.

Two-stage motion compensation using adaptive global MC and local affine MC

This paper describes a high-efficiency video coding method based on ITU-T H.263. To improve the coding efficiency of H.263, a two-stage motion compensation (MC) method is proposed, consisting of global MC (GMC) for predicting camera motion and local MC (LMC) for macroblock prediction. First, global motion such as panning, tilting, and zooming is estimated, and the global-motion-compensated image is produced for use as a reference in LMC. Next, LMC is performed both for the global-motion-compensated reference image and for the image without GMC. LMC employs an affine motion model in the context of H.263's overlapped block motion compensation. Using the overlapped block affine MC, rotation and scaling of small objects can be predicted, in addition to translational motion. In the proposed method, GMC is adaptively turned on/off for each macroblock since GMC cannot be used for prediction in all regions in a frame. In addition, either an affine or a translational motion model is adaptively selected in LMC for each macroblock. Simulation results show that the proposed video coding technique using the two-stage MC significantly outperforms H.263 under identical conditions, especially for sequences with fast camera motion. The performance improvements in peak-to-peak SNR (PSNR) are about 3 dB over the original H.263, which does not use the two-stage MC.

1.15 Mbit/s coding of video signals including global motion compensation

A coder for television video signals working at a bit-rate of 1.15 Mbit/s is presented. It uses a special coding structure facilitating trick modes and a separated global and local motion compensation. Global motion is described by a central zoom and pan model. For the estimation of global motion a frame matching algorithm is explained. The advantages of using global motion parameters are given by the reduction of the amount of coded motion information as well as the reduction of regions to be replenished in the predicted pictures. Experimental results show that for the given bit-rate of 1.15 Mbit/s the quantization step size for prediction error coding can be reduced by the factor three in case of existing global motion. Also at the decoder side global motion parameters are used for a synthesis of omitted frames resulting in a higher resolution in time for global motions.