dB Higher PSNR Research Articles

NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution

In recent years, image and video delivery systems have begun integrating deep learning super-resolution (SR) approaches, leveraging their unprecedented visual enhancement capabilities while reducing reliance on networking conditions. Nevertheless, deploying these solutions on mobile devices still remains an active challenge as SR models are excessively demanding with respect to workload and memory footprint. Despite recent progress on on-device SR frameworks, existing systems either penalize visual quality, lead to excessive energy consumption or make inefficient use of the available resources. This work presents NAWQ-SR, a novel framework for the efficient on-device execution of SR models. Through a novel hybrid-precision quantization technique and a runtime neural image codec, NAWQ-SR exploits the multi-precision capabilities of modern mobile NPUs in order to minimize latency, while meeting user-specified quality constraints. Moreover, NAWQ-SR selectively adapts the arithmetic precision at run time to equip the SR DNN's layers with wider representational power, improving visual quality beyond what was previously possible on NPUs. Altogether, NAWQ-SR achieves an average speedup of 7.9×, 3× and 1.91× over the state-of-the-art on-device SR systems that use heterogeneous processors (MobiSR), CPU (SplitSR) and NPU (XLSR), respectively. Furthermore, NAWQ-SR delivers an average of 3.2× speedup and 0.39 dB higher PSNR over status-quo INT8 NPU designs, but most importantly mitigates the negative effects of quantization on visual quality, setting a new state-of-the-art in the attainable quality of NPU-based SR.

Parallel-stream fusion of scan-specific and scan-general priors for learning deep MRI reconstruction in low-data regimes

Magnetic resonance imaging (MRI) is an essential diagnostic tool that suffers from prolonged scan times. Reconstruction methods can alleviate this limitation by recovering clinically usable images from accelerated acquisitions. In particular, learning-based methods promise performance leaps by employing deep neural networks as data-driven priors. A powerful approach uses scan-specific (SS) priors that leverage information regarding the underlying physical signal model for reconstruction. SS priors are learned on each individual test scan without the need for a training dataset, albeit they suffer from computationally burdening inference with nonlinear networks. An alternative approach uses scan-general (SG) priors that instead leverage information regarding the latent features of MRI images for reconstruction. SG priors are frozen at test time for efficiency, albeit they require learning from a large training dataset. Here, we introduce a novel parallel-stream fusion model (PSFNet) that synergistically fuses SS and SG priors for performant MRI reconstruction in low-data regimes, while maintaining competitive inference times to SG methods. PSFNet implements its SG prior based on a nonlinear network, yet it forms its SS prior based on a linear network to maintain efficiency. A pervasive framework for combining multiple priors in MRI reconstruction is algorithmic unrolling that uses serially alternated projections, causing error propagation under low-data regimes. To alleviate error propagation, PSFNet combines its SS and SG priors via a novel parallel-stream architecture with learnable fusion parameters. Demonstrations are performed on multi-coil brain MRI for varying amounts of training data. PSFNet outperforms SG methods in low-data regimes, and surpasses SS methods with few tens of training samples. On average across tasks, PSFNet achieves 3.1 dB higher PSNR, 2.8% higher SSIM, and 0.3 × lower RMSE than baselines. Furthermore, in both supervised and unsupervised setups, PSFNet requires an order of magnitude lower samples compared to SG methods, and enables an order of magnitude faster inference compared to SS methods. Thus, the proposed model improves deep MRI reconstruction with elevated learning and computational efficiency.

EFID: Edge-Focused Image Denoising Using a Convolutional Neural Network

In this paper, we propose an edge-focused image denoising convolutional neural network for the restoration of noisy images corrupted with additive white Gaussian noise (AWGN). First, the edge information for an input image is obtained for each RGB channel using the simple Laplacian edge operator on a smoothened image using the smoothen mask. Second, residual convolutional blocks, where each block comprises two parallel depth layers (one for an image channel and the other for respective edge channels), perform image and edge processing. Finally, the processed image and edge features are mixed and mapped using a single convolutional layer into the restored image. Our proposed parallel image and edge processing blocks can recover edges and fine structures while smoothing out the noise. In addition, our proposed network is efficiently designed such that the total number of weight parameters can be considerably reduced compared with conventional methods. Experimental results show that the proposed model is more effective in preserving textures and edges while removing noise and achieves up to 0.8 dB higher PSNR and 0.05 higher SSIM on a real image dataset than conventional methods.

Compression loss-based spatial-temporal attention module for compressed video quality enhancement

Recently, deep learning technology has achieved remarkable progress in compressed video quality enhancement. However, the existing methods fail to consider the fact that the regions with different compression losses contain varied effective information. To address this issue, this paper proposes a Compression Loss-based Spatial-Temporal Attention (CLSTA) module that predicts the compression loss-based attention of each pixel when restoring target frames. Based on this, the pixels with lower compression losses contribute more to restoring target frames. Meanwhile, the compression-domain information in the compressed video stream is exploited to overcome the difficulty in inferring compression loss-based attention directly from the pixel-domain. With a slight modification, the CLSTA module can be easily integrated into the existing methods for end-to-end training. A series of experiments have been conducted in this study to validate the effectiveness and generality of the CLSTA module for compressed video quality enhancement. By adding the CLSTA module to the state-of-the-art method STDF, the proposed method achieves an average PSNR improvement of 0.96 dB and a BD-rate reduction of 26.85% under the Low-Delay P configuration, which is respectively 0.06 dB and 1.83% better than the baseline STDF. In the best case, the proposed method obtains 0.19 dB higher PSNR than STDF for a single frame.

A novel hyperspectral compressive sensing framework of plant leaves based on multiple arbitrary-shape regions of interest.

Massive plant hyperspectral images (HSIs) result in huge storage space and put a heavy burden for the traditional data acquisition and compression technology. For plant leaf HSIs, useful plant information is located in multiple arbitrary-shape regions of interest (MAROIs), while the background usually does not contain useful information, which wastes a lot of storage resources. In this paper, a novel hyperspectral compressive sensing framework for plant leaves with MAROIs (HCSMAROI) is proposed to alleviate these problems. HCSMAROI only compresses and reconstructs MAROIs by discarding the background to achieve good reconstructed performance. But for different plant leaf HSIs, HCSMAROI has the potential to be applied in other HSIs. Firstly, spatial spectral decorrelation criterion (SSDC) is used to obtain the optimal band of plant leaf HSIs; Secondly, different leaf regions and background are distinguished by the mask image of the optimal band; Finally, in order to improve the compression efficiency, after discarding the background region the compressed sensing technology based on blocking and expansion is used to compress and reconstruct the MAROIs of plant leaves one by one. Experimental results of soybean leaves and tea leaves show that HCSMAROI can achieve 3.08 and 5.05 dB higher PSNR than those of blocking compressive sensing (BCS) at the sampling rate of 5%, respectively. The reconstructed spectra of HCSMAROI are especially closer to the original ones than that of BCS. Therefore, HCSMAROI can achieve significantly higher reconstructed performance than that of BCS. Moreover, HCSMAROI can provide a flexible way to compress and reconstruct different MAROIs with different sampling rates, while achieving good reconstruction performance in the spatial and spectral domains.

Supremo: Cloud-Assisted Low-Latency Super-Resolution in Mobile Devices

We present Supremo, a cloud-assisted system for low-latency image Super-Resolution (SR) in mobile devices. As SR is extremely compute-intensive, we first further optimize state-of-the-art DNN to reduce the inference latency. Furthermore, we design a mobile-cloud cooperative execution pipeline composed of specialized data compression algorithms to minimize end-to-end latency with minimal image quality degradation. Finally, we extend Supremo to video applications by formulating a dynamic optimal control algorithm to design Supremo-Opt, which aims to maximize the impact of SR while satisfying latency and resource constraints under practical network conditions. Supremo upscales 360p image to 1080p in 122 ms, which is 43.68 x faster than on-device GPU execution. Compared to cloud offloading-based solutions, Supremo reduces wireless network bandwidth consumption and end-to-end latency by 15.23 x and 4.85 x compared to baseline approach of sending and receiving whole images, and achieves 2.39 dB higher PSNR compared to using conventional JPEG to achieve similar data size compression. Furthermore, Supremo-Opt guarantees robust performance in practical scenarios.

An Efficient Accuracy Reconfigurable CLA Adder Designs Using Complementary Logic

High computational complexity of multimedia applications on the portable devices demands a high speed and energy efficient processing cores. The performance of arithmetic unit within these cores significantly affects the overall performance of the devices. Therefore, this paper presents carry look ahead (CLA) adder based two efficient accuracy reconfigurable (AR) adder designs namely AR-CLA-I and AR-CLA-II. The AR-CLA-I design is developed using 4-bit CLA segments with novel approximate sum generation approach. Whereas, the AR-CLA-II design is developed using a novel complementary logic based CLA segment which utilizes proposed new logic formulation. The proposed AR-CLA designs can be reconfigured to achieve high energy-efficiency at cost of acceptable loss in quality. The synthesis results on TSMC 65nm CMOS technology library show that the proposed AR-CLA-I (AR-CLA-II) provides on average 76.52% (80.62%) and 71.65% (74.16%) less area and energy respectively over the best available CLA based approximate adder. The quality metrics of the proposed AR-CLA adder designs as standalone unit demonstrates significant improvement over the existing approximate adder designs. Finally, Sobel edge detectors (SED) embedded with proposed AR-CLA adders provide 71.89 dB higher PSNR over SED embedded with best known approximate adder.

A process-tolerant low-power adder architecture for image processing applications

The aggressive CMOS technology scaling in the sub-100-nm regime leads to highly challenging VLSI design due to the presence of unreliable components. The delay failures in arithmetic units are increasing rapidly due to the increased effect of process variation (PV) in scaled technology. This paper introduces a novel process-tolerant low-power adder (Prot-LA) architecture for error-tolerant applications. The proposed Prot-LA architecture segments the operands into two parts and computes addition of the upper parts in carry-propagate, whereas it computes the lower parts in a carry-free manner. In the Prot-LA, the number of bits in carry-propagate and carry-free additions can be reconfigured based on the amount of PV. An on-chip PV detector is embedded to determine the PV severity. Because of this reconfigurability, the proposed adder completes the carry propagation with minimum error even under severe process variation. The simulation results show that proposed Prot-LA provides 19.9 % reduced power consumption over the state-of-the-art approximate adder. The efficacy of the proposed adder is demonstrated in the real application by designing an image scaling processor (ISP). The simulation results show that the Prot-LA embedded ISP consumes 7.75 % reduced energy with 2.43 dB higher PSNR over the existing approximate adder embedded ISP.

Adaptive pattern selection strategy for diamond search algorithm in fast motion estimation

In this paper, an adaptive pattern selection strategy for diamond search (DS) algorithm is proposed. DS is one of state-of-the-art motion estimation algorithms, while the fixed search strategy and the singular termination strategy lead to lots of redundancy of search points. The proposed search strategy is based on the observation that more than 75% motion vector differences are around the initial predicted search centre in the range of 1. In our search strategy, small diamond search pattern and large diamond search pattern are adaptively used according to the distribution of motion vector differences and the matching error information of initial search centre. Our search strategy focuses on how to use small diamond search pattern and large diamond search pattern more efficiently than diamond search algorithm without introducing additional search patterns. Experimental results show that the proposed algorithm can save about 10.81 search points and achieve 0.12 dB higher PSNR on average compared to DS.

Optimized Predictive Zonal Search (OPZS) for block-based motion estimation

Predictive Motion Estimation (ME) techniques are fast ME algorithms that have been widely used in video CODECs due to their performance efficiency and low computational complexity. In this paper, a new approach named Optimized Predictive Zonal Search (OPZS) is proposed, which improves the state-of-the-art Enhanced Predictive Zonal Search (EPZS) algorithm. The new initial predicted motion vectors employ fewer, but more optimized predictors, as well as improve the termination procedure. The proposed OPZS not only achieves better output quality and significantly higher compression ratio, but also reduces the number of search points and ME processing time even further. Simulation results after encoding 6 different video test sequences with OPZS along with EPZS and three other algorithms implemented under the same conditions has shown up to 33% compression ratio enhancement. This is achieved by searching up to 14% less points, which means less computational complexity and up to 0.4dB higher PSNR values. The simplicity and high performance of OPZS make it more practical for implementation on both hardware and software platforms. Since OPZS provides a combination of low computational complexity and high quality encoded images, it is energy efficient and suitable for portable video processing in computation- or power-constrained applications. Furthermore, because of the higher compression ratio this newly proposed algorithm is more compatible with limited capacity storage media and transmission networks with limited band-width.

Spatiotemporal deinterlacing using a maximuma posterioriestimator based on multiple-field registration

This paper proposes an accurate deinterlacing algorithm using a maximum a posteriori (MAP) estimator. First, we produce accurate motion vector fields between the current field and adjacent fields by employing an advanced motion compensation scheme that is suitable for an interlaced format. Next, the progressive frame corresponding to the current field is found via the MAP estimator based on the derived motion vector fields. Here, in order to obtain a stable solution, well-known bilateral total variation–based regularization is applied. Then, at a specific mode decision step, it is decided whether the result from the aforementioned temporal deinterlacing is acceptable or not. Finally, if the temporal deinterlacing is determined to be inappropriate by the mode decision, a typical spatial deinterlacing is applied instead of the MAP estimator-based temporal deinterlacing. Experimental results show that the proposed algorithm provides at maximum 2 dB higher PSNR than a cutting-edge deinterlacing algorithm, while providing better visual quality than the latter.

基于弱选择正则化正交匹配追踪的图像重构算法

Regularized Orthogonal Match Pursuit(ROMP) is widely applied as a signal reconstruction algorithm.Despite its high efficiency,ROMP requires the prior knowledge of signal sparsity,and would be unstable if the sparsity level is improperly estimated.To overcome this drawback,a weak selection strategy was introduced to adaptively determine the number of atoms and the candidate atoms by estimating the relevance between iterative residue and measurement matrix of the original ROMP algorithm.Thus,an optimal atom set for the signal reconstruction procedure could be selected from the candidate atoms according to the regularization principle.Numerical results demonstrate that the proposed method outperforms other greedy algorithms with 0.5~1.5 dB higher PSNR and much lower MSE.

A High-Capacity Lossless Data Hiding Scheme for Binary Images

Many previous binary image lossless data hiding schemes often cause serious perceptible distortion and have poor visual quality of stego-images. To obtain better invisibility and higher hiding capacity, a novel reversible data hiding scheme for binary images is proposed by combining pattern substitution and human visual perception. Firstly, an improved binary image boundary detection method is presented by avoiding successive boundary pixel flipping. Secondly, secret messages are embedded into binary image boundaries by employing interlaced block partition. Finally, to achieve lossless recovery of original binary image, the side-information is lossless compressed by using the Golomb Code and the occurrence frequency of each pattern. Extensive experiments show that the proposed scheme is effective, and obtain at least 3.6 dB higher PSNR values of stego-image and 92.5% higher embedding capacity compared with Ho et al’s method.

Efficient Motion Field Interpolation Method for Wyner-Ziv Video Coding

Wyner-Ziv video coding has the capability to reduce video encoding complexity by shifting motion estimation procedure from encoder to decoder. Amongst many motion estimation methods, e xpectation m aximization algorithm is the most effective one. Unfortunately, the implementation of block-based motion estimation in this algorithm causes motion field profile bounded by granularity of block size. Nearest-neighbor and bilinear interpolation methods have already applied in multiview image coding to handle similar problem. This paper aims to evaluate performance of both interpolation methods in transform-domain Wyner-Ziv video codec. Results showed that bilinear interpolation effective only for high motion video sequences. In this scenario, it has bitrate saving up to 3.29 %, 0.2 dB higher PSNR, and 12.30 % higher decoding complexity compared to nearest-neighbor. In low motion video content, bitrate saving only gained up to 0.82%, with almost the same PSNR, while decoding complexity increase up to 10.32%.

Iteration-free fractal image coding based on efficient domain pool design

The domain pool design is one of the dominant issues which affect the coding performance of fractal image compression. In this paper, we employ the LBG algorithm and propose a block averaging method to design the efficient domain pools based on a proposed iteration-free fractal image codec. The redundancies between the generated domain blocks are reduced by the proposed methods. Therefore, we can obtain the domain pools that are more efficient than those in the conventional fractal coding schemes and thus the coding performance is improved. On the other hand, the iteration process in the conventional fractal coding scheme not only requires a large size of memory and a high computation complexity but also prolongs the decoding process. The proposed iteration-free fractal codec can overcome the problems above. In computer simulation, both the LBG-based and block-averaging methods for the domain pool design in the proposed iteration free scheme achieve excellent performances. For example, based on the proposed block-averaging method, the decoded Lena image has at least a 0.5 dB higher PSNR (under the same bit rate) and an eight-time faster decoding speed than the conventional fractal coding schemes that require iterations.

L ∞-Constrained near-lossless image compression using weighted finite automata encoding

In this paper we study high-fidelity image compression with a given tight bound on the maximum error magnitude. We propose an L ∞-constrained weighted finite automata recursive encoding scheme on the adaptive context modelling based quantizing prediction residue images. By incorporating the proposed L ∞-constrained WFA encoding techniques into the context modelling based nearly-lossless CALIC, we were able to increase its PSNR by 2 dB or more and obtain bit rates 29% lower than the original CALIC. When applying the wavelet transform for the residue images and then applying the WFA encoding on the wavelet coefficients, we were able to obtain competitive PSNR results against the best wavelet coders in L ∞ metrics, while obtaining much smaller maximum error magnitude than the latter. In particular, when the error bound is <3, the L ∞-constrained wavelet WFA coder were able to obtain 1.34–16.75 dB higher PSNR on the standard ISO test benchmarks than the SPIHT, one of the best wavelet coder.