Blocking Artifacts Research Articles

Super-Resolution of Compressed Images Using Residual Information Distillation Network

Super-Resolution (SR) is a fundamental computer vision task, which reconstructs high-resolution images from low-resolution ones. Existing SR methods mainly recover images from clear low-resolution images, leading to unsatisfactory results when processing compressed low-resolution images. In the paper, we propose a two-stage SR method for compressed images, which consists of the Compression Artifact Removal Module (CARM) and Super-Resolution Module (SRM). The compressed low-resolution image is used to reconstruct the clear low-resolution image by CARM, and the high-resolution image is obtained by SRM. In addition, we propose a residual information distillation block to learn the texture details which are lost during the compression process. The proposed method has been validated and compared with the state of the art, and experimental results show that the proposed method outperforms other super-resolution methods in terms of visual effects and objective evaluation metrics.

Multi-Scale Inter-Communication Spatio-Temporal Network for Video Compression Artifacts Reduction

Video compression artifacts are widespread in online videos, which greatly affects the quality of the videos. In recent years, the deep learning-based methods for video compression artifacts reduction have made impressive achievements. However, most of existing methods ignore the rich multi-scale information contained in video frames. In this brief, we propose a new multi-scale inter-communication spatio-temporal network (MSICSTN) for video compression artifacts reduction by fully exploring contextual multi-scale information. MSICSTN is constructed on the basis of the multi-scale feature inter-communication module (MSFICM) and the source feature selection enhancement module (SFSEM). Specifically, the MSFICM adaptively generates respective attention weights using channel statistical information after the inter-communication of different scale features, and recalibrates the input multi-scale features to enhance the multi-scale features. The SFSEM utilizes the statistical information of the shallow features of the target frame to fuse the shallow features with the deep features for better dense pixel predictions. The experimental results on test videos demonstrate that our method outperforms the existing methods.

Rectified Wasserstein Generative Adversarial Networks for Perceptual Image Restoration.

Wasserstein generative adversarial network (WGAN) has attracted great attention due to its solid mathematical background, i.e., to minimize the Wasserstein distance between the generated distribution and the distribution of interest. In WGAN, the Wasserstein distance is quantitatively evaluated by the discriminator, also known as the critic. The vanilla WGAN trained the critic with the simple Lipschitz condition, which was later shown less effective for modeling complex distributions, like the distribution of natural images. We try to improve the WGAN training by introducing pairwise constraint on the critic, oriented to image restoration tasks. In principle, pairwise constraint is to suggest the critic assign a higher rating to the original (real) image than to the restored (generated) image, as long as such a pair of images are available. We show that such pairwise constraint may be implemented by rectifying the gradients in WGAN training, which leads to the proposed rectified Wasserstein generative adversarial network (ReWaGAN). In addition, we build interesting connections between ReWaGAN and the perception-distortion tradeoff. We verify ReWaGAN on two representative image restoration tasks: single image super-resolution (4× and 8×) and compression artifact reduction, where our ReWaGAN not only beats the vanilla WGAN consistently, but also outperforms the state-of-the-art perceptual quality-oriented methods significantly. Our code and models are publicly available at https://github.com/mahaichuan/ReWaGAN.

CUR Transformer: A Convolutional Unbiased Regional Transformer for Image Denoising

Image denoising is a fundamental problem in computer vision and multimedia computation. Non-local filters are effective for image denoising. But existing deep learning methods that use non-local computation structures are mostly designed for high-level tasks, and global self-attention is usually adopted. For the task of image denoising, they have high computational complexity and have a lot of redundant computation of uncorrelated pixels. To solve this problem and combine the marvelous advantages of non-local filter and deep learning, we propose a Convolutional Unbiased Regional (CUR) transformer. Based on the prior that, for each pixel, its similar pixels are usually spatially close, our insights are that (1) we partition the image into non-overlapped windows and perform regional self-attention to reduce the search range of each pixel, and (2) we encourage pixels across different windows to communicate with each other. Based on our insights, the CUR transformer is cascaded by a series of convolutional regional self-attention (CRSA) blocks with U-style short connections. In each CRSA block, we use convolutional layers to extract the query, key, and value features, namely Q , K , and V , of the input feature. Then, we partition the Q , K , and V features into local non-overlapped windows and perform regional self-attention within each window to obtain the output feature of this CRSA block. Among different CRSA blocks, we perform the unbiased window partition by changing the partition positions of the windows. Experimental results show that the CUR transformer outperforms the state-of-the-art methods significantly on four low-level vision tasks, including real and synthetic image denoising, JPEG compression artifact reduction, and low-light image enhancement.

DRRU-Net: DCT-Coefficient-Learning RRU-Net for Detecting an Image-Splicing Forgery

In this paper, we propose a lightweight deep learning network (DRRU-Net) for image-splicing forgery detection. DRRU-Net is an architecture that combines RRU-Net for learning the visual content of images and image acquisition artifacts, and a JPEG artifact learning module for learning compression artifacts in the discrete cosine transform (DCT) domain. The backbone model of a network based on pre-training, such as CAT-Net, a representative network for image forgery detection, has a relatively large number of parameters, resulting in overfitting in a small dataset, which hinders generalization performance. Therefore, in this paper, the learning module is designed to learn the characteristics according to the DCT domain in real time without pre-training. In the experiments, the proposed network architecture and training method of DRRU-Net show that the network parameters are smaller than CAT-Net, the forgery detection performance is better than that of RRU-Net, and the generalization performance for various datasets can be improved.

Vision Transformers in Image Restoration: A Survey.

The Vision Transformer (ViT) architecture has been remarkably successful in image restoration. For a while, Convolutional Neural Networks (CNN) predominated in most computer vision tasks. Now, both CNN and ViT are efficient approaches that demonstrate powerful capabilities to restore a better version of an image given in a low-quality format. In this study, the efficiency of ViT in image restoration is studied extensively. The ViT architectures are classified for every task of image restoration. Seven image restoration tasks are considered: Image Super-Resolution, Image Denoising, General Image Enhancement, JPEG Compression Artifact Reduction, Image Deblurring, Removing Adverse Weather Conditions, and Image Dehazing. The outcomes, the advantages, the limitations, and the possible areas for future research are detailed. Overall, it is noted that incorporating ViT in the new architectures for image restoration is becoming a rule. This is due to some advantages compared to CNN, such as better efficiency, especially when more data are fed to the network, robustness in feature extraction, and a better feature learning approach that sees better the variances and characteristics of the input. Nevertheless, some drawbacks exist, such as the need for more data to show the benefits of ViT over CNN, the increased computational cost due to the complexity of the self-attention block, a more challenging training process, and the lack of interpretability. These drawbacks represent the future research direction that should be targeted to increase the efficiency of ViT in the image restoration domain.

Multi-Modality Deep Restoration of Extremely Compressed Face Videos.

Arguably the most common and salient object in daily video communications is the talking head, as encountered in social media, virtual classrooms, teleconferences, news broadcasting, talk shows, etc. When communication bandwidth is limited by network congestions or cost effectiveness, compression artifacts in talking head videos are inevitable. The resulting video quality degradation is highly visible and objectionable due to high acuity of human visual system to faces. To solve this problem, we develop a multi-modality deep convolutional neural network method for restoring face videos that are aggressively compressed. The main innovation is a new DCNN architecture that incorporates known priors of multiple modalities: the video-synchronized speech signal and semantic elements of the compression code stream, including motion vectors, code partition map and quantization parameters. These priors strongly correlate with the latent video and hence they are able to enhance the capability of deep learning to remove compression artifacts. Ample empirical evidences are presented to validate the superior performance of the proposed DCNN method on face videos over the existing state-of-the-art methods.

Image and video compression of fluid flow data

We study the compression of spatial and temporal features in fluid flow data using multimedia compression techniques. The efficacy of spatial compression techniques, including JPEG and JPEG2000 (JP2), and spatiotemporal video compression techniques, namely H.264, H.265, and AV1, in limiting the introduction of compression artifacts and preserving underlying flow physics are considered for laminar periodic wake around a cylinder, two-dimensional turbulence, and turbulent channel flow. These compression techniques significantly compress flow data while maintaining dominant flow features with negligible error. AV1 and H.265 compressions present the best performance across a variety of canonical flow regimes and outperform traditional techniques such as proper orthogonal decomposition in some cases. These image and video compression algorithms are flexible, scalable, and generalizable holding potential for a wide range of applications in fluid dynamics in the context of data storage and transfer.Graphic abstract

(Compress and Restore) N : A Robust Defense Against Adversarial Attacks on Image Classification

Modern image classification approaches often rely on deep neural networks, which have shown pronounced weakness to adversarial examples: images corrupted with specifically designed yet imperceptible noise that causes the network to misclassify. In this article, we propose a conceptually simple yet robust solution to tackle adversarial attacks on image classification. Our defense works by first applying a JPEG compression with a random quality factor; compression artifacts are subsequently removed by means of a generative model Artifact Restoration GAN. The process can be iterated ensuring the image is not degraded and hence the classification not compromised. We train different AR-GANs for different compression factors, so that we can change its parameters dynamically at each iteration depending on the current compression, making the gradient approximation difficult. We experiment with our defense against three white-box and two black-box attacks, with a particular focus on the state-of-the-art BPDA attack. Our method does not require any adversarial training, and is independent of both the classifier and the attack. Experiments demonstrate that dynamically changing the AR-GAN parameters is of fundamental importance to obtain significant robustness.

Image demosaicing: Subjective analysis and evaluation of image quality metrics

Most cameras use a single-sensor arrangement with Color Filter Array (CFA). Color interpolation techniques performed during image demosaicing are normally the reason behind visual artifacts generated in a captured image. While the severity of the artifacts depends on the demosaicing methods used, the artifacts themselves are mainly zipper artifacts (block artifacts across the edges) and false-color distortions. In this study and to evaluate the performance of demosaicing methods, a subjective pair-comparison method with 15 observers was performed on six different methods (namely Nearest Neighbours, Bilinear interpolation, Laplacian, Adaptive Laplacian, Smooth hue transition, and Gradient-Based image interpolation) and nine different scenes. The subjective scores and scene images are then collected as a dataset and used to evaluate a set of no-reference image quality metrics. Assessment of the performance of these image quality metrics in terms of correlation with the subjective scores show that many of the evaluated no-reference metrics cannot predict perceived image quality.

Image Compressed Sensing Using Non-Local Neural Network

Deep network-based image Compressed Sensing (CS) has attracted much attention in recent years. However, the existing deep network-based CS schemes either reconstruct the target image in a block-by-block manner that leads to serious block artifacts or train the deep network as a black box that brings about limited insights of image prior knowledge. In this paper, a novel image CS framework using non-local neural network (NL-CSNet) is proposed, which utilizes the non-local self-similarity priors with deep network to improve the reconstruction quality. In the proposed NL-CSNet, two non-local subnetworks are constructed for utilizing the non-local self-similarity priors in the measurement domain and the multi-scale feature domain respectively. Specifically, in the subnetwork of measurement domain, the long-distance dependencies between the measurements of different image blocks are established for better initial reconstruction. Analogically, in the subnetwork of multi-scale feature domain, the affinities between the dense feature representations are explored in the multi-scale space for deep reconstruction. Furthermore, a novel loss function is developed to enhance the coupling between the non-local representations, which also enables an end-to-end training of NL-CSNet. Extensive experiments manifest that NL-CSNet outperforms existing state-of-the-art CS methods, while maintaining fast computational speed.

JPEG Artifact Reduction Based on Deformable Offset Gating Network Controlled by a Variational Autoencoder

For the reduction of JPEG compression artifacts, there have been many methods using deep neural networks. Most of them use the JPEG compression quality factor (QF) as prior knowledge in designing and training the networks. However, since the images we get from the Internet are often recompressed, the given QF is not so informative or misleading. Also, early works validated their methods on low QFs less than 50, while recent smartphones use high QFs larger than or equal to 90. In this paper, we propose a new JPEG artifacts reduction network considering the above-stated problems. Specifically, to extract quality information from the input image itself instead of the QF provided in the header of the JPEG file, we use a variational autoencoder (VAE) and regard its latent vector as quality information. In designing the artifact reduction network, we let the network change flexibly according to the input image quality by employing a deformable offset gating (DOG) network. The gating network and VAE are merged as our overall network, dubbed DOG-VAE, where the information from the VAE is used to adjust the DOG network according to the input quality. The DOG-VAE is trained end-to-end with the QFs in the range of [10,90]. Extensive experiments validate that our method achieves comparable results to the state-of-the-art method for monochrome images and better results for color images. Our codes are available at https://github.com/yunjh410/DOGNet.

Toward Fast and Accurate Violence Detection for Automated Video Surveillance Applications

Surveillance cameras are increasingly being used worldwide due to the proliferation of digital video capturing, storage, and processing technologies. However, the large volume of video data generated makes it difficult for humans to perform real-time analysis, and even manual approaches can result in delayed detection of events. Automatic violence detection in surveillance footage has therefore gained significant attention in the scientific community as a way to address this challenge. With the advancement of machine learning algorithms, automatic video recognition tasks such as violence detection have become increasingly feasible. In this study, we investigate the use of smart networks that model the dynamic relationships between actors and/or objects using 3D convolutions to capture both the spatial and temporal structure of the data. We also leverage the knowledge learned by a pre-trained action recognition model for efficient and accurate violence detection in surveillance footage. We extend and evaluate several public datasets featuring diverse and challenging video content to assess the effectiveness of our proposed methods. Our results show that our approach outperforms state-of-the-art methods, achieving approximately a 2% improvement in accuracy with fewer model parameters. Additionally, our experiments demonstrate the robustness of our approach under common compression artifacts encountered in remote server processing applications.

Blind JPEG Compression Artifacts Removal by Integrating Channel Regulation With Exit Strategy

Compression artifacts removal methods based on convolutional neural networks have attracted great attention. However, most existing methods require a specific trained model for a specific compression quality factor (QF), which inevitably leads to resource-consuming. Unfortunately, the QF is unknown in most practical applications, so it is intractable to choose a suitable model. In this work, we experimentally analyze the relationship between compression index estimation and compression artifacts removal. Based on the connection between them, we couple compression index estimation with compression artifacts removal into a unified network. A network named CRESNet is proposed, working for a wide range of QFs by integrating channel regulation with an exit strategy. Specifically, CRESNet adopts a multi-stage progressive structure with an exit strategy embedded to automatically select the optimal exit stage according to the estimated compression index reflecting the difficulty of the input sample. Benefiting from the exit strategy, CRESNet removes artifacts from slightly compressed images through a simple process while doing an elaborate process for severely compressed images. Furthermore, a compression-information-guided channel regulation (CICR) mechanism is developed to adaptively regulate feature maps based on the estimated compression index. CRESNet achieves a more elegant trade-off between artifacts removal and detail preservation in a resource-efficient manner. Experiments demonstrate that CRESNet achieves state-of-the-art performance.

CLASSIFICATION REVIEW OF RAW SUBJECTIVE SCORES TOWARDS STATISTICAL ANALYSIS

We proposed a no reference model for quality estimation based on compression artifacts which are principal factor for determining the QoE compared to varying depth range within compression. Moreover, in our past research work, contents with strong compression have been rejected by the Subjects who graded quality of distorted videos and judged unacceptable. The results of our proposed approach conrm that our subjective scores of 120 videos are related to control of visible spatial artifacts of reconstructed videos not distorted and with consideration of User Experience (UX), It was traced out by rening raw subjective scores, it happened due to coding of spatial and temporal perceptual information according to H.264 standards mentioned by ITU only for video quality assessment, i.e., instead of coding spatial of chrominance and temporal information of luminance plane, selected test sequences were coded within luminance plane for both spatial and temporal information

DWDN: Deep Wiener Deconvolution Network for Non-Blind Image Deblurring.

We present a simple and effective approach for non-blind image deblurring, combining classical techniques and deep learning. In contrast to existing methods that deblur the image directly in the standard image space, we propose to perform an explicit deconvolution process in a feature space by integrating a classical Wiener deconvolution framework with learned deep features. A multi-scale cascaded feature refinement module then predicts the deblurred image from the deconvolved deep features, progressively recovering detail and small-scale structures. The proposed model is trained in an end-to-end manner and evaluated on scenarios with simulated Gaussian noise, saturated pixels, or JPEG compression artifacts as well as real-world images. Moreover, we present detailed analyses of the benefit of the feature-based Wiener deconvolution and of the multi-scale cascaded feature refinement as well as the robustness of the proposed approach. Our extensive experimental results show that the proposed deep Wiener deconvolution network facilitates deblurred results with visibly fewer artifacts and quantitatively outperforms state-of-the-art non-blind image deblurring methods by a wide margin.

Nonlinear kernel based feature maps for blur‐sensitive unsharp masking of JPEG images

In this paper, a method for estimating the blur regions of an image is first proposed, resorting to a mixture of linear and nonlinear convolutional kernels. The blur map obtained is then utilized to enhance images such that the enhancement strength is an inverse function of the amount of measured blur. The blur map can also be used for tasks such as attention-based object classification, low light image enhancement, and more. A CNN architecture is trained with nonlinear upsampling layers using a standard blur detection benchmark dataset, with the help of blur target maps. Further, it is proposed to use the same architecture to build maps of areas affected by the typical JPEG artifacts, ringing and blockiness. The blur map and the artifact map pair permit to build an activation map for the enhancement of a (possibly JPEG compressed) image. Extensive experiments on standard test images verify the quality of the maps obtained using the algorithm and their effectiveness in locally controlling the enhancement, for superior perceptual quality. Last but not least, the computation time for generating these maps is much lower than the one of other comparable algorithms.

Digital Image Anti-Forensic Model Using Exponential Chaotic Biogeography-Based Optimization Algorithm

Abstract The innovation in visual imagery has led to massive growth in technologies, wherein digital cameras are obtainable at affordable prices. Thus, the digital images are easily captured and processed due to the internet connectivity. On the other hand, the development of strong image editing software facilitated the forgers to manipulate the accessible images with different tampering operations. Several techniques are devised for detecting the forgeries. Accordingly, this paper devises an anti-forensic model, namely Exponentially Weighted Moving Average-Chaotic Biography Based Optimization (E-CBBO) for joint photographic experts group (JPEG) compression to mitigate the forgeries occurred on the internet while transmitting data. The proposed E-CBBO is designed by integrating the properties of the exponentially weighted moving average (EWMA) with the chaotic biography-based optimization (CBBO). The suggested JPEG anti-forensic model is used to eliminate JPEG compression artifacts through the use of unique deblocking, smoothing with dither and decalibration operations. In addition, the goal is to balance visual quality and forensic undetectability when compressing the JPEG image. The fitness function is developed using the structural similarity index (SSIM), universal image quality index (UIQI) and histogram deviation parameters. With a maximum accuracy of 93.2%, a minimal MSE of 0.110, a maximum SSIM of 0.932 and a maximum UIQI of 0.890, the suggested E-CBBO beat existing approaches.

Perceptual quality assessment for fine-grained compressed images

Recent years have witnessed the rapid development of image storage and transmission systems, in which image compression plays an important role. Generally speaking, image compression algorithms are developed to ensure good visual quality at limited bit rates. However, due to the different compression optimization methods, the compressed images may have different levels of quality, which needs to be evaluated quantificationally. Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of compressed images are obvious), however, they may perform poorly for fine-grained compressed images whose bit rates differences are quite subtle. Therefore, to better improve the Quality of Experience (QoE) and provide useful guidance for compression algorithms, we propose a full-reference image quality assessment (FR-IQA) method for compressed images of fine-grained levels. Specifically, the reference images and compressed images are first converted to YCbCr color space. The gradient features are extracted from regions that are sensitive to compression artifacts. Then we employ the Log-Gabor transformation to further analyze the texture difference. Finally, the obtained features are fused into a quality score. The proposed method is validated on the fine-grained compression image quality assessment (FGIQA) database, which is especially constructed for assessing the quality of compressed images with close bit rates. The experimental results show that our metric outperforms mainstream FR-IQA metrics on the FGIQA database. We also test our method on other commonly used compression IQA databases and the results show that our method obtains competitive performance on the coarse-grained compression IQA databases as well.

Super-resolving compressed images via parallel and series integration of artefacts removal and resolution enhancement

In real-world applications, such as sharing photos on social media platforms, images are always not only sub-sampled but also heavily compressed thus often containing various artefacts. Simple methods for enhancing the resolution of such images will exacerbate the artefacts, rendering them visually objectionable. In spite of its high practical values, super-resolving compressed images is not well studied in the literature. In this paper, we propose a novel compressed image super resolution (CISR) framework based on parallel and series integration of artefacts removal and resolution enhancement. Based on a mathematical inference model for estimating a clean low-resolution (LR) image and a clean high-resolution (HR) image from a down-sampled and compressed observation, we have designed a CISR architecture consisting of two deep neural network modules: the artefacts removal module (ARM) and the resolution enhancement module (REM). The ARM and the REM work in parallel with both taking the compressed LR image as their inputs, at the same time they also work in series with the REM taking the output of the ARM as one of its inputs and the ARM taking the output of the REM as its other input. A technique called unfolding is introduced to recursively suppress the compression artefacts and restore the image resolution. A unique feature of our CISR system is that it exploits the parallel and series connections between the ARM and the REM, and recursive optimization to reduce the model’s dependency on specific types of degradation thus making it possible to train a single model to super-resolve images compressed by different methods to different qualities. Experiments are conducted on a mixture of JPEG and WebP compressed images without assuming apriori compression type and compression quality factor. To demonstrate our technique’s real-world application value, we have also applied the trained models directly to restore social media images which have undergone scaling and compression by unknown algorithms. Visual and quantitative comparisons demonstrate the superiority of our method over state-of-the-art super resolution methods, especially for heavily compressed images.