Blocking Artifacts Research Articles

Pyramid Attention Network for Image Restoration

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks

EARN: toward efficient and robust JPEG compression artifact reduction

EARN: toward efficient and robust JPEG compression artifact reduction

JPEG Decoding With Nonlinear Inverse Transform Network and Progressive Recurrent Residual Network

JPEG is the most widely used image compression format, especially with the popularity of mobile or portable devices. However, the quality of a decoded JPEG image is usually degraded by compression artifacts such as blocking effect and ringing effect, especially at low bit rates. Recently, some convolutional neural network (CNN) based methods have been designed to solve the above problem. These methods take the problem as post-processing and only add a CNN-based post-processing network after a JPEG decoder to improve the image quality. In this paper, the JPEG decoding with nonlinear inverse transform network and progressive recurrent residual network (dubbed as JDNet) is proposed. JDNet can reconstruct JPEG images of different quality factors (QF) with only one model. In JDNet, first, the CNN-based inverse transform network (iTNet) is proposed to learn the nonlinear mapping from DCT coefficients to their corresponding original pixels, which is against the linear mapping in previous DCT networks. iTNet can reduce error propagation during inverse DCT and obtain more accurate reconstruction. Furthermore, iTNet can be combined with any JPEG post-processing method to improve its performance. Second, the progressive recurrent residual network (PRRN) is proposed for local feature extraction in the designed post-processing network which utilizes local and nonlocal similarities in multi-scale space (LNLMS) to further enhance the decoded image quality. The experimental results show that compared with JPEG, ARCNN, DnCNN and STRRN, the average gains of JDNet are 1.88dB, 0.63dB, 0.35dB, 0.24dB on Live1 dataset, 2.61dB, 1.11dB, 0.69dB and 0.29dB on Urban100 dataset, and 1.91dB, 0.69dB, 0.37dB and 0.13dB on BSD500 dataset, respectively.

Deep steerable pyramid wavelet network for unified JPEG compression artifact reduction

Although numerous methods have been proposed to remove blocking artifacts in JPEG-compressed images, one important issue not well addressed so far is the construction of a unified model that requires no prior knowledge of the JPEG encoding parameters to operate effectively on different compression-level images (grayscale/color) while occupying relatively small storage space to save and run. To address this issue, in this paper, we present a unified JPEG compression artifact reduction model called DSPW-Net, which employs (1) the deep steerable pyramid wavelet transform network for Y-channel restoration, and (2) the classic U-Net architecture for CbCr-channel restoration. To enable our model to work effectively on images with a wide range of compression levels, the quality factor (QF) related features extracted by the convolutional layers in the QF-estimation network are incorporated in the two restoration branches. Meanwhile, recursive blocks with shared parameters are utilized to drastically reduce model parameters and shared-source residual learning is employed to avoid the gradient vanishing/explosion problem in training. Extensive quantitative and qualitative results tested on various benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art deblocking methods.

MoReLab: A Software for User-Assisted 3D Reconstruction.

We present MoReLab, a tool for user-assisted 3D reconstruction. This reconstruction requires an understanding of the shapes of the desired objects. Our experiments demonstrate that existing Structure from Motion (SfM) software packages fail to estimate accurate 3D models in low-quality videos due to several issues such as low resolution, featureless surfaces, low lighting, etc. In such scenarios, which are common for industrial utility companies, user assistance becomes necessary to create reliable 3D models. In our system, the user first needs to add features and correspondences manually on multiple video frames. Then, classic camera calibration and bundle adjustment are applied. At this point, MoReLab provides several primitive shape tools such as rectangles, cylinders, curved cylinders, etc., to model different parts of the scene and export 3D meshes. These shapes are essential for modeling industrial equipment whose videos are typically captured by utility companies with old video cameras (low resolution, compression artifacts, etc.) and in disadvantageous lighting conditions (low lighting, torchlight attached to the video camera, etc.). We evaluate our tool on real industrial case scenarios and compare it against existing approaches. Visual comparisons and quantitative results show that MoReLab achieves superior results with regard to other user-interactive 3D modeling tools.

A truncated generalized Huber prior for image smoothing

Image smoothing is a fundamental task in computer vision and graphics. This paper presents a new image smoothing method based on a truncated generalized Huber prior. The proposed model is neither convex nor concave and is hard to optimize. We first transform the prior into a concave one, then utilize the technique of half-quadratic minimization to get an equivalent convex surrogate function. Thus the numerical algorithm is obtained by solving a weighted least square problem and iteratively updating the weights. The convergence of the algorithm is theoretically guaranteed. The proposed method is flexible and powerful in preserving edge/structure and eliminating undesired information. The effectiveness of the proposed method is demonstrated by several applications, including scale space filtering, texture removal, and clip-art JPEG artifacts removal.

Video Compression Artifact Reduction by Fusing Motion Compensation and Global Context in a Swin-CNN Based Parallel Architecture

Video Compression Artifact Reduction aims to reduce the artifacts caused by video compression algorithms and improve the quality of compressed video frames. The critical challenge in this task is to make use of the redundant high-quality information in compressed frames for compensation as much as possible. Two important possible compensations: Motion compensation and global context, are not comprehensively considered in previous works, leading to inferior results. The key idea of this paper is to fuse the motion compensation and global context together to gain more compensation information to improve the quality of compressed videos. Here, we propose a novel Spatio-Temporal Compensation Fusion (STCF) framework with the Parallel Swin-CNN Fusion (PSCF) block, which can simultaneously learn and merge the motion compensation and global context to reduce the video compression artifacts. Specifically, a temporal self-attention strategy based on shifted windows is developed to capture the global context in an efficient way, for which we use the Swin transformer layer in the PSCF block. Moreover, an additional Ada-CNN layer is applied in the PSCF block to extract the motion compensation. Experimental results demonstrate that our proposed STCF framework outperforms the state-of-the-art methods up to 0.23dB (27% improvement) on the MFQEv2 dataset.

CAIR: Combining integrated attention with iterative optimization learning for sparse-view CT reconstruction

Sparse-view CT is an efficient way for low dose scanning but degrades image quality. Inspired by the successful use of non-local attention in natural image denoising and compression artifact removal, we proposed a network combining integrated attention and iterative optimization learning for sparse-view CT reconstruction (CAIR). Specifically, we first unrolled the proximal gradient descent into a deep network and added an enhanced initializer between the gradient term and the approximation term. It can enhance the information flow between different layers, fully preserve the image details, and improve the network convergence speed. Secondly, the integrated attention module was introduced into the reconstruction process as a regularization term. It adaptively fuses the local and non-local features of the image which are used to reconstruct the complex texture and repetitive details of the image, respectively. Note that we innovatively designed a one-shot iteration strategy to simplify the network structure and reduce the reconstruction time while maintaining image quality. Experiments showed that the proposed method is very robust and outperforms state-of-the-art methods in terms of both quantitative and qualitative, greatly improving the preservation of structures and the removal of artifacts.

Crafting the MPEG metrics for objective and perceptual quality assessment of volumetric videos.

Efficient objective and perceptual metrics are valuable tools to evaluate the visual impact of compression artifacts on the visual quality of volumetric videos (VVs). In this paper, we present some of the MPEG group efforts to create, benchmark and calibrate objective quality assessment metrics for volumetric videos represented as textured meshes. We created a challenging dataset of 176 volumetric videos impaired with various distortions and conducted a subjective experiment to gather human opinions (more than 5896 subjective scores were collected). We adapted two state-of-the-art model-based metrics for point cloud evaluation to our context of textured mesh evaluation by selecting efficient sampling methods. We also present a new image-based metric for the evaluation of such VVs whose purpose is to reduce the cumbersome computation times inherent to the point-based metrics due to their use of multiple kd-tree searches. Each metric presented above is calibrated (i.e., selection of best values for parameters such as the number of views or grid sampling density) and evaluated on our new ground-truth subjective dataset. For each metric, the optimal selection and combination of features is determined by logistic regression through cross-validation. This performance analysis, combined with MPEG experts' requirements, lead to the validation of two selected metrics and recommendations on the features of most importance through learned feature weights.

Impedance-Based Ventilation Detection and Signal Quality Control During Out-of-Hospital Cardiopulmonary Resuscitation.

Feedback on ventilation could help improve cardiopulmonary resuscitation quality and survival from out-of-hospital cardiac arrest (OHCA). However, current technology that monitors ventilation during OHCA is very limited. Thoracic impedance (TI) is sensitive to air volume changes in the lungs, allowing ventilations to be identified, but is affected by artifacts due to chest compressions and electrode motion. This study introduces a novel algorithm to identify ventilations in TI during continuous chest compressions in OHCA. Data from 367 OHCA patients were included, and 2551 one-minute TI segments were extracted. Concurrent capnography data were used to annotate 20724 ground truth ventilations for training and evaluation. A three-step procedure was applied to each TI segment: First, bidirectional static and adaptive filters were applied to remove compression artifacts. Then, fluctuations potentially due to ventilations were located and characterized. Finally, a recurrent neural network was used to discriminate ventilations from other spurious fluctuations. A quality control stage was also developed to anticipate segments where ventilation detection could be compromised. The algorithm was trained and tested using 5-fold cross-validation, and outperformed previous solutions in the literature on the study dataset. The median (interquartile range, IQR) per-segment and per-patient F 1-scores were 89.1 (70.8-99.6) and 84.1 (69.0-93.9), respectively. The quality control stage identified most low performance segments. For the 50% of segments with highest quality scores, the median per-segment and per-patient F 1-scores were 100.0 (90.9-100.0) and 94.3 (86.5-97.8). The proposed algorithm could allow reliable, quality-conditioned feedback on ventilation in the challenging scenario of continuous manual CPR in OHCA.

A joint sparse representation and low rank prior regularization model for image deblocking

When the compression quality factor (QF) is low, JPEG images are usually accompanied by very severe block artifacts. The conventional image deblocking models based on low-rank regularization focus on eliminating block artifacts, but lack the capacity of recovering texture details on image edges. In this paper, an image deblocking model which combines sparse representation and low-rank regularization based on fractional norm is proposed. To avoid the abrupt increase in computational complexity, multiple orthogonal dictionary learning is proposed to enhance the ability to sparsify diverse image structures and eliminate correlations among the atoms in the overcomplete dictionary. To further improve the image deblocking effect, a quantization constraint and fractional norm are used to more effectively enforce the low-rank property. Finally, an alternating minimization strategy is applied to solve the derived optimization problem. The model proposed in this paper preserves more texture details while effectively removing image block artifacts compared with the mainstream image deblocking model utilizing low-rank regularization alone. Experimental results show that the proposed method outperforms the current state-of-the-art image deblocking models in terms of both subjective metrics and objective perception.

Adaptive rate image compressive sensing based on the hybrid sparsity estimation model

In some actual image compressive sensing (CS) systems, the sparsity of the original image signals is unknown on the sampling side. Resources on the sampling side are also limited. Therefore, it is a challenging task to implement adaptive rate allocation on the sampling side of a CS system. In this paper, a novel adaptive rate allocation block CS scheme for images based on a hybrid sparsity estimation model is proposed. First, a cosine similarity sparsity estimation model is constructed by directly modeling the measurement results, and on the reconstruction side, a thresholding cosine transform sparsity estimation model is constructed according to the initial reconstructed image. Then, the sparsity estimates generated by the above two models are fused according to their respective characteristics using a Bayesian probability fusion algorithm. The advantage of the proposed scheme is that it can significantly improve the accuracy of the final sparsity estimates while limiting the computational complexity on the sampling side, which makes it exceptionally suitable for wireless sensor network applications. On the reconstruction side, we propose an energy-based adaptive global reconstruction model, that can further reduce block artifacts caused by block-based CS. The experiment results show that, compared with previous state-of-the-art schemes, the proposed scheme can achieve a significant improvement in the peak signal-to-noise ratio at the same sampling rate.

Improving Compressed Video Using Single Lightweight Model with Temporal Fusion Module.

Video compression algorithms are commonly used to reduce the number of bits required to represent a video with a high compression ratio. However, this can result in the loss of content details and visual artifacts that affect the overall quality of the video. We propose a learning-based restoration method to address this issue, which can handle varying degrees of compression artifacts with a single model by predicting the difference between the original and compressed video frames to restore video quality. To achieve this, we adopted a recursive neural network model with dilated convolution, which increases the receptive field of the model while keeping the number of parameters low, making it suitable for deployment on a variety of hardware devices. We also designed a temporal fusion module and integrated the color channels into the objective function. This enables the model to analyze temporal correlation and repair chromaticity artifacts. Despite handling color channels, and unlike other methods that have to train a different model for each quantization parameter (QP), the number of parameters in our lightweight model is kept to only about 269 k, requiring only about one-twelfth of the parameters used by other methods. Our model applied to the HEVC test model (HM) improves the compressed video quality by an average of 0.18 dB of BD-PSNR and -5.06% of BD-BR.

A method supporting fault-tolerant optical text recognition from video sequences recorded with handheld cameras

In the paper a method supporting the optical character recognition from video sequences recorded with cameras without good stabilization is proposed. Due to the presence of various distortions, such as motion blur, shadows, lossy compression artifacts, auto-focusing errors, etc., the quality of individual video frames, e.g., recorded by a smartphone camera, differs noticeably, influencing the results of text recognition, causing the presence of additional errors that may be unacceptable even in fault-tolerant applications. Although the quality of individual video frames may be assessed using state-of-the-art no-reference image quality metrics, the obtained results, particularly using the relatively fast metrics, are poorly correlated with recognition results. Therefore, a novel hybrid approach to image quality assessment supporting fault-tolerant optical character recognition is proposed, making it possible to select automatically the video frames leading to the smallest number of errors. Since the data extraction is rarely perfect, the proposed method may be particularly useful in systems where users should identify the faults defined as a too high number of text recognition errors. The proposed method makes it possible to eliminate low quality fragments of video sequences from further analysis, reducing the overall computational burden. It may also be useful in industrial OCR applications based on offline video recording with memory constraints where the use of online OCR services is not possible.

Advanced Image Compression Techniques for Medical Applications: Survey

The field of artificial intelligence has grown significantly in the past decade, with deep learning being a particularly promising technique due to its ability to learn complex feature representations from data. One area where deep learning has shown promise is in image compression, which is important for applications such as medical imaging, remote sensing, and video streaming. Traditional compression methods such as JPEG, JPEG200 and PNG have been used for decades, but recent advances in deep learning have led to innovative techniques that involve training deep neural networks to learn a compressed representation of the image data. This survey paper focuses on research-based applications of deep learning in image compression and reviews recent works on using deep learning techniques to compress and accelerate deep neural networks, including pruning, quantization, and low-rank factorization methods. In addition, we discuss the use of deep learning techniques to minimize compression defects, such as block artifacts and ringing, which can degrade image quality. The paper provides an overview of popular methods and recent works in the field, highlighting their characteristics, advantages, and shortcomings. We also discuss challenges and open research questions, such as the trade-off between compression efficiency and reconstruction quality, and the need for standardized evaluation metrics for comparing different compression methods. Overall, this survey aims to provide a comprehensive understanding of current trends in image compression using deep learning techniques and their potential to revolutionize the field. By exploring the advantages and limitations of these methods, we hope to facilitate further research and development in this exciting area.

Shearlet-based intensity classification loop filter for video coding

The new Versatile Video Coding standard (VVC) exhibits a significantly higher coding efficiency in comparison to its predecessor High Efficiency Video Coding (HEVC). Since the finalization of VVC in July 2020 many new in-loop compression tools were suggested, especially with data-driven approaches. The question arises if non-data-driven in-loop filtering approaches can further increase coding efficiency beyond the capability of VVC. In this paper we approach the task of filtering images corrupted by quantization noise from the perspective of applied harmonic analysis, an area of applied mathematics focusing on the efficient representation, analysis and encoding of data. The shearlet-based intensity classification loop filter (SCLF) is a novel non-data-driven approach using an overcomplete and sparsifying transform, the shearlet transform. The basic idea is to achieve a signal-noise separation by applying a shearlet transform. Shearlets can identify important structures of natural images such as edges in the sparse transform domain. Each shearlet transform coefficient is classified into different classes. All coefficients with similar intensities are grouped into the same class, which eventually gives a partition of the set of all shearlet coefficients. This separates important information from noise and ensures an accurate reconstruction of original information by performing an inverse shearlet transform combined with Wiener filtering separately for each class. SCLF effectively removes compression artifacts and therefore restores subjective visual quality. Simulation results show that by adding SCLF to the in-loop stage yet further average bit rate reductions of up to 1.5% are achieved for JVET common test sequences.

Adaptive-weighted high order TV algorithm for sparse-view CT reconstruction.

With the development of low-dose computed tomography (CT), incomplete data reconstruction has been widely concerned. The total variation (TV) minimization algorithm can accurately reconstruct images from sparse or noisy data. However, the traditional TV algorithm ignores the direction of structures in images, leading to the loss of edge information and block artifacts when the object is not piecewise constant. Since the anisotropic information can facilitate preserving the edge and detail information in images, we aim to improve the TV algorithm in terms of reconstruction accuracy via this approach. In this paper, we propose an adaptive-weighted high order total variation (awHOTV) algorithm. We construct the second order TV-norm using the second order gradient, adapt the anisotropic edge property between neighboring image pixels, adjust the local image-intensity gradient to keep edge information, and design the corresponding Chambolle-Pock (CP) solving algorithm. Implementing the proposed algorithm, comprehensive studies are conducted in the ideal projection data experiment where the Structural similarity (SSIM), Root Mean Square Error (RMSE), Contrast to noise ratio (CNR), and modulation transform function (MTF) curves are utilized to evaluate the quality of reconstructed images in statism, structure, spatial resolution, and contrast, respectively. In the noisy data experiment, we further use the noise power spectrum (NPS) curve to evaluate the reconstructed images and compare it with other three algorithms. We use the 2D slice in the XCAT phantom, 2D slice in TCIA Challenge data and FORBILD phantom as simulation phantoms and use real bird data for real verification. The results show that, compared with the traditional TV and FBP algorithms, the awHOTV has better performance in terms of RMSE, SSIM, and Pearson correlation coefficient (PCC) under the projected data with different sparsity. In addition, the awHOTV algorithm is robust against the noise of different intensities. The proposed awHOTV method can reconstruct the images with high accuracy under sparse or noisy data. The awHOTV solves the strip artifacts caused by sparse data in the FBP method. Compared with the TV method, the awHOTV can effectively suppress block artifacts and has good detail protectionability.

AutoBCS: Block-Based Image Compressive Sensing With Data-Driven Acquisition and Noniterative Reconstruction.

Block compressive sensing (CS) is a well-known signal acquisition and reconstruction paradigm with widespread application prospects in science, engineering, and cybernetic systems. However, state-of-the-art block-based image CS (BCS) methods generally suffer from two issues. The sparsifying domain and the sensing matrices widely used for image acquisition are not data driven and, thus, both the features of the image and the relationships among subblock images are ignored. Moreover, it requires to address a high-dimensional optimization problem with extensive computational complexity for image reconstruction. In this article, we provide a deep learning (DL) strategy for BCS, called AutoBCS, which automatically takes the prior knowledge of images into account in the acquisition step and establishes a reconstruction model for performing fast image reconstruction. More precisely, we present a learning-based sensing matrix to accomplish image acquisition, thereby capturing and preserving more image characteristics than those captured by the existing methods. In addition, we build a noniterative reconstruction network, which provides an end-to-end BCS reconstruction framework to maximize image reconstruction efficiency. Furthermore, we investigate comprehensive comparison studies with both traditional BCS approaches and newly developed DL methods. Compared with these approaches, our proposed AutoBCS can not only provide superior performance in terms of image quality metrics (SSIM and PSNR) and visual perception but also automatically benefit reconstruction speed.

Deep In-Loop Filtering via Multi-Domain Correlation Learning and Partition Constraint for Multiview Video Coding

The deep learning-based in-loop filtering methods have greatly improved the coding efficiency for High Efficiency Video Coding (HEVC). However, directly applying these HEVC-orientated in-loop filtering methods to multiview video coding may not obtain satisfactory performance due to the characteristics of multiview video. In this paper, a deep in-loop filtering method based on multi-domain correlation learning and partition constraint network (MDP-Net) is proposed to boost the multiview video coding performance. To the best of our knowledge, this work is the first attempt at deep in-loop filtering for multiview video coding. Specifically, a multi-domain correlation learning module is presented to restore the high-frequency details of the distorted frame by exploring the multi-domain correlations. Besides, based on the block partition information generated in video coding, a partition-constrained reconstruction module is proposed to better attenuate the compression artifacts by designing a partition loss. Finally, the proposed MDP-Net is integrated into 3D-HEVC reference software, and the experimental results demonstrate that the proposed method achieves considerable performance improvement compared with 3D-HEVC.

99m-Tc TRODAT Single-Photon Emission Computerized Tomography Scan Image Compression using Singular Value Decomposition.

The objective of the study was to compress 99m-Tc TRODAT single-photon emission computerized tomography (SPECT) scan image using Singular Value Decomposition (SVD) into an acceptable compressed image and then calculate the compression factor. The SVD of every image from the image dataset of 2256 images (of forty-eight 99m-Tc TRODAT SPECT studies [48 studies X 47 trans-axial images = 2256 trans-axial images]) was computed and after truncating singular values smaller than a threshold, the compressed image was reconstructed. The SVD computation time and percentage compression achieved were calculated for each image. Two nuclear medicine physicians visually compared compressed image with its original image, and labeled it as either acceptable or unacceptable. Compressed image having loss of clinical details or presence of compression artifact was labeled unacceptable. The quality of compressed image was also assessed objectively using the following image quality metrics: Error, structural similarity (SSIM), brightness, global contrast factor (GCF), contrast per pixel (CPP), and blur. We also compared the TRODAT uptake in basal ganglia estimated from the compressed image and original image. Nuclear Medicine Physician labeled each image acceptable, as they found compressed image identical to its original image. The values of brightness, GCF, CPP, and blur metrics show that compressed images are less noisy, brighter, and sharper than its original image. The median values of error (0.0006) and SSIM (0.93) indicate that the compressed images were approximately identical to its original image. In 39 out of 48 studies, the percentage difference in TRODAT uptake (in basal ganglia from compressed and original image) was negligible (approximately equal to zero). In remaining 9 studies, the maximum percentage difference was 13%. The SVD computation time and percentage compression achieved for a TRODAT study were 0.17398 s and up to 54.61%, respectively. The compression factor up to 54.61% was achieved during 99m-Tc TRODAT SPECT scan image compression using SVD, for an acceptable compressed image.