Blocking Artifacts Research Articles

Perceptual quality assessment for fine-grained compressed images

Recent years have witnessed the rapid development of image storage and transmission systems, in which image compression plays an important role. Generally speaking, image compression algorithms are developed to ensure good visual quality at limited bit rates. However, due to the different compression optimization methods, the compressed images may have different levels of quality, which needs to be evaluated quantificationally. Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of compressed images are obvious), however, they may perform poorly for fine-grained compressed images whose bit rates differences are quite subtle. Therefore, to better improve the Quality of Experience (QoE) and provide useful guidance for compression algorithms, we propose a full-reference image quality assessment (FR-IQA) method for compressed images of fine-grained levels. Specifically, the reference images and compressed images are first converted to YCbCr color space. The gradient features are extracted from regions that are sensitive to compression artifacts. Then we employ the Log-Gabor transformation to further analyze the texture difference. Finally, the obtained features are fused into a quality score. The proposed method is validated on the fine-grained compression image quality assessment (FGIQA) database, which is especially constructed for assessing the quality of compressed images with close bit rates. The experimental results show that our metric outperforms mainstream FR-IQA metrics on the FGIQA database. We also test our method on other commonly used compression IQA databases and the results show that our method obtains competitive performance on the coarse-grained compression IQA databases as well.

Super-resolving compressed images via parallel and series integration of artefacts removal and resolution enhancement

In real-world applications, such as sharing photos on social media platforms, images are always not only sub-sampled but also heavily compressed thus often containing various artefacts. Simple methods for enhancing the resolution of such images will exacerbate the artefacts, rendering them visually objectionable. In spite of its high practical values, super-resolving compressed images is not well studied in the literature. In this paper, we propose a novel compressed image super resolution (CISR) framework based on parallel and series integration of artefacts removal and resolution enhancement. Based on a mathematical inference model for estimating a clean low-resolution (LR) image and a clean high-resolution (HR) image from a down-sampled and compressed observation, we have designed a CISR architecture consisting of two deep neural network modules: the artefacts removal module (ARM) and the resolution enhancement module (REM). The ARM and the REM work in parallel with both taking the compressed LR image as their inputs, at the same time they also work in series with the REM taking the output of the ARM as one of its inputs and the ARM taking the output of the REM as its other input. A technique called unfolding is introduced to recursively suppress the compression artefacts and restore the image resolution. A unique feature of our CISR system is that it exploits the parallel and series connections between the ARM and the REM, and recursive optimization to reduce the model’s dependency on specific types of degradation thus making it possible to train a single model to super-resolve images compressed by different methods to different qualities. Experiments are conducted on a mixture of JPEG and WebP compressed images without assuming apriori compression type and compression quality factor. To demonstrate our technique’s real-world application value, we have also applied the trained models directly to restore social media images which have undergone scaling and compression by unknown algorithms. Visual and quantitative comparisons demonstrate the superiority of our method over state-of-the-art super resolution methods, especially for heavily compressed images.

A Model-Driven Deep Unfolding Method for JPEG Artifacts Removal.

Deep learning-based methods have achieved notable progress in removing blocking artifacts caused by lossy JPEG compression on images. However, most deep learning-based methods handle this task by designing black-box network architectures to directly learn the relationships between the compressed images and their clean versions. These network architectures are always lack of sufficient interpretability, which limits their further improvements in deblocking performance. To address this issue, in this article, we propose a model-driven deep unfolding method for JPEG artifacts removal, with interpretable network structures. First, we build a maximum posterior (MAP) model for deblocking using convolutional dictionary learning and design an iterative optimization algorithm using proximal operators. Second, we unfold this iterative algorithm into a learnable deep network structure, where each module corresponds to a specific operation of the iterative algorithm. In this way, our network inherits the benefits of both the powerful model ability of data-driven deep learning method and the interpretability of traditional model-driven method. By training the proposed network in an end-to-end manner, all learnable modules can be automatically explored to well characterize the representations of both JPEG artifacts and image content. Experiments on synthetic and real-world datasets show that our method is able to generate competitive or even better deblocking results, compared with state-of-the-art methods both quantitatively and qualitatively.

Effective Tensor Completion via Element-Wise Weighted Low-Rank Tensor Train With Overlapping Ket Augmentation

Tensor completion methods based on the tensor train (TT) have the issues of inaccurate weight assignment and ineffective tensor augmentation pre-processing. In this work, we propose a novel tensor completion approach via the element-wise weighted technique. Accordingly, a novel formulation for tensor completion and an effective optimization algorithm, called tensor completion by parallel weighted matrix factorization via tensor train (TWMac-TT), is proposed. In addition, we specifically consider the recovery quality of edge elements from adjacent blocks. Different from traditional reshaping and ket augmentation, we utilize a new tensor augmentation technique called overlapping ket augmentation, which can further avoid blocking artifacts. We then conduct extensive performance evaluations on synthetic data and several real image data sets. Our experimental results demonstrate that the proposed algorithm TWMac-TT outperforms several other competing tensor completion methods. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yzcv/</uri> TWMac-TT-OKA

Video restoration based on deep learning: a comprehensive survey

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

A quality enhancement network with coding priors for constant bit rate video coding

Video coding can effectively compress data, while the introduction of compression artifacts degrades the visual quality and the performance of artificial intelligence video applications. Video quality enhancement (VQE) methods can improve compressed video quality without modifying the coding standard’s main modules. The existing VQE works mainly aim to improve video quality in the constant quantization parameter (CQP) coding mode. However, constant bit rate (CBR) coding mode is widely used in some streaming playback video applications, and VQE at CBR is more challenging than that at CQP. This article presents a novel Constant bit rate Video quality enhancement Network combined with Coding priors (CVCN). The proposed CVCN can be inserted into High Efficiency Video Coding (HEVC) codec as a CNN-based in-loop filter (LF) module or a post-processing module out of the codec. Moreover, we design a two-pass training strategy for the LF module to overcome multiple filtering. To adapt the QP diversity of CBR videos, we utilize the coding unit (CU)-wise QP prior by constructing a CU-wise QP adaptive module (CQAM) and a QP adaptive multi-scale residual block (QAMSRB) based on CQAM. We construct the CU-partition prior (CPP) by exploiting the relationship between the compression noise and CU partition information. A novel CPP spatial-attention block is proposed to combine the CPP with the self-attention module. Furthermore, the proposed CVCN can effectively enhance CBR videos at different bits per pixel (BPP) via a single model. Extensive experimental results show the superiority of the CVCN over state-of-the-art VQE approaches for CBR videos.

Atmospheric light veil correction dehazing algorithm based on regional detection

To address the problems that the dehazing algorithm is prone to distortion, block artifacts, and edge pixel loss from the minimum filter in the highlighted area of an image, a dehazing algorithm with an atmospheric veil correction model under regional detection is proposed. First, the initial atmospheric light veil is obtained by using the improved dark channel, and the atmospheric light veil correction model is established for it. Then, the color distance and color distance histogram fitting curve of the haze image are used to accurately detect the region where the initial atmospheric light veil is corrected, and the correction term is established. Finally, the atmospheric light is estimated by combining the image information entropy with quad-tree subdivision algorithm. It is proved that the proposed method effectively solves the distorted problem in the highlighted region and the block artifacts produced by minimum filtering and can recover image with rich detailed information.

Compression artifact reduction of low bit‐rate videos via deep neural networks using self‐similarity prior

This paper presents a hybrid scheme that integratedly uses self-similarity prior and deep convolutional neural network (CNN) fusion for compression artifact reduction in low bit-rate video applications. Based on the temporal correlation hypothesis, the self-similarity prior is extended to the temporal domain by using as references not only the current decoded frame but also its neighbouring frames. Furthermore, being cognizant of that the bicubic downsampling process can typically improve the perceptual quality of a video coded at low bit-rate, for each small patch in the current frame, we search for similar patches in down-scaled versions of these references, and then form several self-similarity prior based predictions by tiling these similar patches at corresponding positions. To further exploit information flow across scales, a deep CNN model is constructed that contains two sub-networks to estimate the final output. One sub-network takes the self-similarity prior based predictions along with the decoded frame itself; and the other takes the down-scaled versions of these frames as network input. Experimental results demonstrate that the proposed method can remarkably improve, both subjectively and objectively, quality of compressed video sequences of low bit-rates.

Robust Estimation of Upscaling Factor on Double JPEG Compressed Images.

As one of the most important topics in image forensics, resampling detection has developed rapidly in recent years. However, the robustness to JPEG compression is still challenging for most classical spectrum-based methods, since JPEG compression severely degrades the image contents and introduces block artifacts in the boundary of the compression grid. In this article, we propose a method to estimate the upscaling factors on double JPEG compressed images in the presence of image upscaling between the two compressions. We first analyze the spectrum of scaled images and give an overall formulation of how the scaling factors along with the parameters of JPEG compression and image contents influence the appearance of tampering artifacts. The expected positions of five kinds of characteristic peaks are analytically derived. Then, we analyze the features of double JPEG compressed images in the block discrete cosine transform (BDCT) domain and present an inverse scaling strategy for the upscaling factor estimation with a detailed proof. Finally, a fusion method is proposed that through frequency-domain analysis, a candidate set of upscaling factors is given, and through analysis in the BDCT domain, the optimal estimation from all candidates is determined. The experimental results demonstrate that the proposed method outperforms other state-of-the-art methods.

A Generalized Framework for Edge-preserving and Structure-preserving Image Smoothing.

Image smoothing is a fundamental procedure in applications of both computer vision and graphics. The required smoothing properties can be different or even contradictive among different tasks. Nevertheless, the inherent smoothing nature of one smoothing operator is usually fixed and thus cannot meet the various requirements of different applications. In this paper, we first introduce the truncated Huber penalty function which shows strong flexibility under different parameter settings. A generalized framework is then proposed with the introduced truncated Huber penalty function. When combined with its strong flexibility, our framework is able to achieve diverse smoothing natures where contradictive smoothing behaviors can even be achieved. It can also yield the smoothing behavior that can seldom be achieved by previous methods, and superior performance is thus achieved in challenging cases. These together enable our framework capable of a range of applications and able to outperform the state-of-the-art approaches in several tasks. In addition, an efficient numerical solution is provided and its convergence is theoretically guaranteed even the optimization framework is non-convex and non-smooth. A simple yet effective approach is further proposed to reduce the computational cost of our method while maintaining its performance. The effectiveness and superior performance of our approach are validated through comprehensive experiments in a range of applications.

L0 image smoothing via iterating truncated L1 gradient regularization

Edge-preserving image smoothing is a fundamental tool in computational photography and graphics. It aims to suppress insignificant details while maintaining salient structures. The classical L0 filter provides an elegant framework for a variety of applications. However, it inclines to sharpen the salient edges, thus suffering from gradient reversals and color deviations. We propose a solution toward the objective of optimizing summed squared error regularized by the L0-norm of the gradients. The proposed solution explores an iterative strategy, where each iteration is an optimization problem based on the truncated L1 regularization, which can be solved efficiently with the combination of alternating direction method of multipliers and Fourier domain optimization. Leveraging L1-norm limits the aggressive modifications of gradients during the iterative process, which alleviates various artifacts in classical L0 filter. Experiment results indicate that the proposed method achieves superior performance in various applications, including texture removal, detail enhancement, HDR tone mapping, and compression artifact removal. The proposed filter can be conveniently implemented on GPU. It takes our filter 0.51 s to process a 720P color image on a NVIDIA GTX 1080 GPU.

A no-reference perceptual image quality assessment database for learned image codecs

The drastic growth of research in image compression, especially deep learning-based image compression techniques, poses new challenges to objective image quality assessment (IQA). Typical artifacts encountered in the emerging image codecs are significantly different from that produced by traditional block-based codecs, leading to inapplicability of the existing objective IQA algorithms. Towards advancing the development of objective IQA algorithms for recent compression artifacts, we built a learning-based compressed image quality assessment (LCIQA) database involving traditional block-based image codecs, hybrid neural network based image codecs, convolutional neural network based and generative adversarial network (GAN) based end-to-end optimized image coding approaches. Our study confirms the statistical difference and human perception difference between reconstructions of learned compression and traditional block-based compression. We propose a two-step deep learning model for learning-based compressed image quality assessment. Extensive experiments on LCIQA database demonstrate that our proposed model performs better than other counterparts on learning-based compressed images, especially on GAN compressed images, and achieves competitive performance to the state-of-the-art IQA metrics on traditional compressed images.

Gated fusion network for SAO filter and inter frame prediction in Versatile Video Coding

In order to achieve higher coding efficiency, the Versatile Video Coding (VVC) standard includes several new components at the expense of an increase in decoder computational complexity. These technologies often create ringing and contouring effects on the reconstructed frames at a low bit rate and introduce blurring and distortion. To smooth those visual artifacts, the H.266/VVC framework supports four post-processing filter operations. The state-of-the-art CNN-based in-loop filters prefer to deploy multiple networks for various quantization parameters and frame resolutions, which increases training resources and subsequently becomes overhead at decoder frame reconstruction. This paper presents a single deep-learning-based model for sample adaptive off-set (SAO) non-linear filtering operation on the decoder side, uses feature correlation among adjacent frames, and substantiates the merits of intra–inter frame quality enhancement. We introduced a variable filter size dual multi-scale convolutional neural network (D-MSCNN) to attenuate the compression artifact and incorporated strided deconvolution to restore the high-frequency details on the distorted frame. Our model follows sequential training across all QP values and updates the model weights. Using data augmentation, weight fusion, and residual learning, we demonstrated that our model could be trained effectively by transferring the convolution prior feature indices to the decoder to produce a dense output map. The Objective measurements demonstrate that the proposed method outperforms the baseline VVC method in PSNR, MS-SSIM, and VMAF metrics and achieves an average of 5.16% bit rate saving on different test sequence categories.

A 1Mb Mixed-Precision Quantized Encoder for Image Classification and Patch-Based Compression

Even if Application-Specific Integrated Circuits (ASIC) have proven to be a relevant choice for integrating inference at the edge, they are often limited in terms of applicability. In this paper, we demonstrate that an ASIC neural network accelerator dedicated to image processing can be applied to multiple tasks of different levels: image classification and compression, while requiring a very limited hardware. The key component is a reconfigurable, mixed-precision (3b/2b/1b) encoder that takes advantage of proper weight and activation quantizations combined with convolutional layer structural pruning to lower hardware-related constraints (memory and computing). We introduce an automatic adaptation of linear symmetric quantizer scaling factors to perform quantized levels equalization, aiming at stabilizing quinary and ternary weights training. In addition, a proposed layer-shared Bit-Shift Normalization significantly simplifies the implementation of the hardware-expensive Batch Normalization. For a specific configuration in which the encoder design only requires 1Mb, the classification accuracy reaches 87.5% on CIFAR-10. Besides, we also show that this quantized encoder can be used to compress image patch-by-patch while the reconstruction can performed remotely, by a dedicated full-frame decoder. This solution typically enables an end-to-end compression almost without any block artifacts, outperforming patch-based state-of-the-art techniques employing a patch-constant bitrate.

Research on Optimization Scheme for Blocking Artifacts after Patch-Based Medical Image Reconstruction.

Due to limitations of computer resources, when utilizing a neural network to process an image with a high resolution, the typical processing approach is to slice the original image. However, because of the influence of zero-padding in the edge component during the convolution process, the central part of the patch often has more accurate feature information than the edge part, resulting in image blocking artifacts after patch stitching. We studied this problem in this paper and proposed a fusion method that assigns a weight to each pixel in a patch using a truncated Gaussian function as the weighting function. In this method, we used the weighting function to transform the Euclidean-distance between a point in the overlapping part and the central point of the patch where the point was located into a weight coefficient. With increasing distance, the value of the weight coefficient decreased. Finally, the reconstructed image was obtained by weighting. We employed the bias correction model to evaluate our method on the simulated database BrainWeb and the real dataset HCP (Human Connectome Project). The results show that the proposed method is capable of effectively removing blocking artifacts and obtaining a smoother bias field. To verify the effectiveness of our algorithm, we employed a denoising model to test it on the IXI-Guys human dataset. Qualitative and quantitative evaluations of both models show that the fusion method proposed in this paper can effectively remove blocking artifacts and demonstrates superior performance compared to five commonly available and state-of-the-art fusion methods.

Data-Driven Supervised Compression Artifacts Detection on Continuous Glucose Sensors.

Continuous Glucose Monitoring (CGM) sensors micro-invasively provide frequent glucose readings, improving the management of Type 1 diabetic patients' life and making available reach data-sets for retrospective analysis. Unlikely, CGM sensors are subject to failures, such as compression artifacts, that might impact on both real-time and respective CGM use. In this work is focused on retrospective detection of compression artifacts. An in-silico dataset is generated using the T1D UVa/Padova simulator and compression artifacts are subsequently added in known position, thus creating a dataset with perfectly accurate faulty/not-faulty labels. The problem of compression artifact detection is then faced with supervised data-driven techniques, in particular using Random Forest algorithm. The detection performance guaranteed by the method on in-silico data is satisfactory, opening the way for further analysis on real-data.

Rethinking Deinterlacing for Early Interlaced Videos

In recent years, high-definition restoration of early videos have received much attention. Real-world interlaced videos usually contain various degradations mixed with interlacing artifacts, such as noises and compression artifacts. Unfortunately, traditional deinterlacing methods only focus on the inverse process of interlacing scanning, and cannot remove these complex and complicated artifacts. Hence, this paper proposes an image deinterlacing network (DIN), which is specifically designed for joint removal of interlacing mixed with other artifacts. The DIN is composed of two stages, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , a cooperative vertical interpolation stage for splitting and fully using the information of adjacent fields, and a field-merging stage to perceive movements and suppress ghost artifacts. Experimental results demonstrate the effectiveness of the proposed DIN on both synthetic and real-world test sets.

Transcoded Video Restoration by Temporal Spatial Auxiliary Network

In most video platforms, such as Youtube, Kwai, and TikTok, the played videos usually have undergone multiple video encodings such as hardware encoding by recording devices, software encoding by video editing apps, and single/multiple video transcoding by video application servers. Previous works in compressed video restoration typically assume the compression artifacts are caused by one-time encoding. Thus, the derived solution usually does not work very well in practice. In this paper, we propose a new method, temporal spatial auxiliary network (TSAN), for transcoded video restoration. Our method considers the unique traits between video encoding and transcoding, and we consider the initial shallow encoded videos as the intermediate labels to assist the network to conduct self-supervised attention training. In addition, we employ adjacent multi-frame information and propose the temporal deformable alignment and pyramidal spatial fusion for transcoded video restoration. The experimental results demonstrate that the performance of the proposed method is superior to that of the previous techniques. The code is available at https://github.com/icecherylXuli/TSAN.

A robust facemask forgery detection system in video

An in-depth fake video uses an Artificial Intelligent (AI), AI programming, and a Personal computer (PC) mix to create a deep fake video of the action. Deep-faking can also be used to represent images and sounds. We provide insights into our reviews in this document. We're showing our dataset to start. At this point, we present the subtleties and reproductively of exploratory settings to evaluate the discovered effects finally. It is no surprise to find deep fake videos, which only monitor a tiny section of the video (e.g., the target face appears quickly on the video; hence the time is limited). We remove our system's fixed duration's persistent effects as each video contributes to the preparation, approval, and testing sections to reflect this. The edge groups are isolated from each video successively (without outline skips). The entire pipeline is ready to be finished when the approval stage is ten years old. Convolutional Neural Network (CNN) was the best and most reliable of the classification systems. Fake videos typically use low-quality pictures to mask faults or insist that the general public regard camera defects as unexplainable phenomena. 'This is a common trope with Unidentified Flying Object (UFO) videos: ghostly orbs are lenses; snakes are compression artifacts on one's face. In this study, we have implemented a sophisticated, knowledgeable method to recognize false images. Our test results using various monitored videos have shown that we can reliably predict whether videos are monitored through with simple co-evolutionary Long Short-Term Memory (LSTM) structure.

Abstract 5056: Automated Gleason grading of digitized frozen section prostate tissue slide images

Abstract Frozen sections are used in intraoperative consultations which need rapid microscopic analysis and in certain pathological procedures that require fresh tissue. Due to the various artifacts present on frozen section tissue slides, it is difficult to automatically analyze them. We tried automatic analysis of frozen sections. Two types of segmentation deep learning models detecting Gleason patterns were used. Both models were trained from hematoxylin and eosin (H&amp;E) stained formalin-fixed, paraffin-embedded prostate tissue slide images. Each slide image was reviewed by an experienced pathologist, and all prostate cancer lesions were annotated with corresponding Gleason grades. One model was trained with H&amp;E stained images (RS model), and the other model was trained with Hematoxylin-only (H-only) stained images (H-only model). A color deconvolution method was used to generate H-only stained images from original H&amp;E stained images. In evaluation, frozen section prostate cancer slide images from The Cancer Genome Atlas (TCGA) were used. To diagnose the slide, we splitted whole slide images (WSI) into patches and analyze those patches to segment the Gleason pattern area. For the WSI result, we reconstructed the result as a whole size heatmap, then we counted the Gleason pattern to diagnose the slide as the prostate grade group. In detecting malignant cases, the RS model achieved a sensitivity of 98% and the H only model of 97%, respectively. In detecting clinically significant risk cases (grade group 2 or over), sensitivities of 99% and 96% were achieved, respectively. Most of the errors were all false-positive cases. In particular, the RS model mainly had false positives because of the ice crystal artifact, and the H-only model had many false positives for the folded part and the part suspected of being a prostatic intraepithelial neoplasia (PIN). False negatives were common in the compression artifacts for both models. We confirmed that both models showed high performance in cancer detection. In addition, they showed high accuracy in the classification of high-risk and low-risk groups of cancer. Further performance improvement can be expected with additional training of data related to artifacts or PIN. Citation Format: Joonyoung Cho, Tae-Yeong Kwak, Sun Woo Kim, Hyeyoon Chang. Automated Gleason grading of digitized frozen section prostate tissue slide images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 5056.