Low-level Image Processing Tasks Research Articles

CDLNet: Noise-Adaptive Convolutional Dictionary Learning Network for Blind Denoising and Demosaicing

Deep learning based methods hold state-of-the-art results in low-level image processing tasks, but remain difficult to interpret due to their black-box construction. Unrolled optimization networks present an interpretable alternative to constructing deep neural networks by deriving their architecture from classical iterative optimization methods without use of tricks from the standard deep learning tool-box. So far, such methods have demonstrated performance close to that of state-of-the-art models while using their interpretable construction to achieve a comparably low learned parameter count. In this work, we propose an unrolled convolutional dictionary learning network (CDLNet) and demonstrate its competitive denoising and joint denoising and demosaicing (JDD) performance both in low and high parameter count regimes. Specifically, we show that the proposed model outperforms state-of-the-art fully convolutional denoising and JDD models when scaled to a similar parameter count. In addition, we leverage the model's interpretable construction to propose a noise-adaptive parameterization of thresholds in the network that enables state-of-the-art blind denoising performance, and near-perfect generalization on noise-levels unseen during training. Furthermore, we show that such performance extends to the JDD task and unsupervised learning.

Watermarking Deep Neural Networks in Image Processing.

Publishing/sharing pretrained deep neural network (DNN) models is a common practice in the community of computer vision. The increasing popularity of pretrained models has made it a serious concern: how to protect the intellectual properties of model owners and avert illegal usages by malicious attackers. This article aims at developing a framework for watermarking DNNs, with a particular focus on low-level image processing tasks that map images to images. Using image denoising and superresolution as case studies, we develop a black-box watermarking method for pretrained models, which exploits the overparameterization of the DNNs in image processing. In addition, an auxiliary module for visualizing the watermark information is proposed for further verification. Extensive experiments show that the proposed watermarking framework has no noticeable impact on model performance and enjoys the robustness against the often-seen attacks.

IEEE Access Special Section Editorial: Deep Learning Technologies for Internet of Video Things

Recently, data-driven algorithms such as deep neural networks have attracted a lot of attention and have become a popular area of research and development. This interest is driven by several factors, such as recent advances in processing power (cheap and powerful hardware), the availability of large datasets (big data), and several small but important algorithmic advances (e.g., convolutional layers). Nowadays, deep neural networks are the state-of-the-art for several computer vision tasks, such as the ones that require high-level understanding of image semantics, e.g., image classification, object segmentation, saliency detection, and also in low level image processing tasks, such as image denoising, inpainting, and super-resolution, among others. Deep learning can handle large volumes of video data by making use of its powerful non-linear mapping and extracting high-level features with very deep networks. Incorporating deep learning into the IoVT can provide radical innovations in video sensing, coding, enhancing, understanding, and evaluation areas making it easier to handle the enormous growth in video data compared to traditional methods.

Dual-Stream Multi-Path Recursive Residual Network for JPEG Image Compression Artifacts Reduction

JPEG is the most widely used lossy image compression standard. When using JPEG with high compression ratios, visual artifacts cannot be avoided. These artifacts not only degrade the user experience but also negatively affect many low-level image processing tasks. Recently, convolutional neural network (CNN)-based compression artifact removal approaches have achieved significant success, however, at the cost of high computational complexity due to an enormous number of parameters. To address this issue, we propose a dual-stream recursive residual network (STRRN) which consists of structure and texture streams for separately reducing the specific artifacts related to high-frequency or low-frequency image components. The outputs of these streams are combined and fed into an aggregation network to further enhance the restored images. By using parameter sharing, the proposed network reduces the total number of training parameters significantly. Moreover, experiments conducted on five commonly used datasets confirm that the proposed STRRN can efficiently reduce the compression artifacts, while using up to 4.6 times less training parameters and 5 times less running time compared to the state-of-the-art approaches.

Recognition-Driven Compressed Image Generation Using Semantic-Prior Information

Image compression, as one of the fundamental low-level image processing tasks, is very essential for computer vision. Current image compression methods can maintain considerable visual quality even at relatively lower bit-rate, but pay little attention to their performance in high-level vision tasks, e.g, image recognition and semantic segmentation. In this letter, we aim to generate compressed images with similar visual quality as before, but with much higher recognition accuracy. To this end, we explore the significance of semantic-prior information in image compression and design a semantic-prior attention module to adaptively enhance the semantic-ware features. Moreover, a semantic perceptual loss combining semantic-prior map is employed to concentrate on the machine perceptual quality of semantic-ware content. Experiments on benchmarks demonstrate that the proposed algorithm has the effectiveness to improve recognition accuracy and maintain visual quality. In addition, performance improvement on different recognition networks and tasks shows the generality of our algorithm.

PNEN: Pyramid Non-Local Enhanced Networks.

Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We propose a novel non-local module, Pyramid Non-local Block, to build up connection between every pixel and all remain pixels. The proposed module is capable of efficiently exploiting pairwise dependencies between different scales of low-level structures. The target is fulfilled through first learning a query feature map with full resolution and a pyramid of reference feature maps with downscaled resolutions. Then correlations with multi-scale reference features are exploited for enhancing pixel-level feature representation. The calculation procedure is economical considering memory consumption and computational cost. Based on the proposed module, we devise a Pyramid Non-local Enhanced Networks for edge-preserving image smoothing which achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into convolution neural networks for other image restoration tasks. We integrate it into two existing methods for image denoising and single image super-resolution, achieving consistently improved performance.

FoGDbED: Fractional-order Gaussian derivatives-based edge-relevant structure detection using Caputo-Fabrizio definition

Edge-relevant structure features (ERSFs), e.g., object edges, boundaries and contours, junctions, etc. play an important role in low and middle level image processing tasks, such as image segmentation, as well as in higher-level computer vision tasks, such as scene analysis and content understanding. Commonly-used ERSF detection methods employ the integer-order differential-based methods, which are noise-sensitive and have less selectivity of ERSFs. Hence, they are difficult to effectively extract object edges or boundaries, especially in natural images with rich fractal-like structures. This paper presents a highly selective and noise-robust ERSF detection approach based on the fractional-order Gaussian derivatives (FoGDs) by using the definition of Caputo-Fabrizio derivative, termed FoGDbED. FoGDbED is constructed based on the concept of robust edge feature selection and inflexion point localization, whose detection mask can be designed with a close-form expression of FoGDs. Theoretical analysis and experimental results show that the proposed FoGDbED operator is capable of extracting complex ERSFs in natural images especially detecting object edges and junctions in natural images with serious noises to achieve a better visual effect.

A generalized nonlocal mean framework with object-level cues for saliency detection

Nonlocal mean (NM) is an efficient method for many low-level image processing tasks. However, it is challenging to directly utilize NM for saliency detection. This is because that conventional NM method can only extract the structure of the image itself and is based on regular pixel-level graph. However, saliency detection usually requires human perceptions and more complex connectivity of image elements. In this paper, we propose a novel generalized nonlocal mean (GNM) framework with the object-level cue which fuses the low-level and high-level cues to generate saliency maps. For a given image, we first use uniqueness to describe the low-level cue. Second, we adopt the objectness algorithm to find potential object candidates, then we pool the object measures onto patches to generate two high-level cues. Finally, by fusing these three cues as an object-level cue for GNM, we obtain the saliency map of the image. Extensive experiments show that our GNM saliency detector produces more precise and reliable results compared to state-of-the-art algorithms.

Library for model-based design of image processing algorithms on FPGAs

This paper describes a library (XSGImgLib) that includes parameterizable blocks to implement low-level image processing tasks on FPGAs. A modelbased design technique provided by Xilinx System Generator (XSG) has been used to design the blocks, which implement point operation (binarization) and neighborhood operations (linear and non-linear filtering) in grayscale images. The blocks are parameterizable for input/output data precision, window size, normalization strategy, and implementation options (area versus speed optimization). The paper includes the implementation results obtained after fixing these options and exemplifies the combination of several blocks of the library to build a complete design for image segmentation purposes.

The Weibull manifold in low-level image processing: An application to automatic image focusing

In this paper, we introduce a novel framework for low-level image processing and analysis. First, we process images with very simple, difference-based filter functions. Second, we fit the 2-parameter Weibull distribution to the filtered output. This maps each image to the 2D Weibull manifold. Third, we exploit the information geometry of this manifold and solve low-level image processing tasks as minimisation problems on point sets. For a proof-of-concept example, we examine the image autofocusing task. We propose appropriate cost functions together with a simple implicitly-constrained manifold optimisation algorithm and show that our framework compares very favourably against common autofocus methods from literature. In particular, our approach exhibits the best overall performance in terms of combined speed and accuracy.

Extraction of Objects from Terrestrial Laser Scans by Integrating Geometry Image and Intensity Data with Demonstration on Trees

Terrestrial laser scanning is becoming a standard for 3D modeling of complex scenes. Results of the scan contain detailed geometric information about the scene; however, the lack of semantic details still constitutes a gap in ensuring this data is usable for mapping. This paper proposes a framework for recognition of objects in laser scans; aiming to utilize all the available information, range, intensity and color information integrated into the extraction framework. Instead of using the 3D point cloud, which is complex to process since it lacks an inherent neighborhood structure, we propose a polar representation which facilitates low-level image processing tasks, e.g., segmentation and texture modeling. Using attributes of each segment, a feature space analysis is used to classify segments into objects. This process is followed by a fine-tuning stage based on graph-cut algorithm, which considers the 3D nature of the data. The proposed algorithm is demonstrated on tree extraction and tested on scans containing complex objects in addition to trees. Results show a very high detection level and thereby the feasibility of the proposed framework.

An FPGA-based processing pipeline for high-definition stereo video

This paper presents a real-time processing platform for high-definition stereo video. The system is capable to process stereo video streams at resolutions up to 1, 920 × 1, 080 at 30 frames per second (1080p30). In the hybrid FPGA-GPU-CPU system, a high-density FPGA is used not only to perform the low-level image processing tasks such as color interpolation and cross-image color correction, but also to carry out radial undistortion, image rectification, and disparity estimation. We show how the corresponding algorithms can be implemented very efficiently in programmable hardware, relieving the GPU from the burden of these tasks. Our FPGA implementation results are compared with corresponding GPU implementations and with other implementations reported in the literature.

Equivariant Holomorphic Filters for Contour Denoising and Rapid Object Detection

It is well known that linear filters are not powerful enough for many low-level image processing tasks. However, it is also very difficult to design robust nonlinear filters that respond exclusively to features of interest and that are, at the same time, equivariant with respect to translation and rotation. This paper proposes a new class of rotation-equivariant nonlinear filters that is based on the principle of group integration. These filters become efficiently computable by an iterative scheme based on repeated differentiation of products and summations of intermediate results. The relations of the proposed approach to Volterra filters and steerable filters are shown. In the context of detection problems, the filter may be interpreted as some kind of generalized Hough transform. The experiments show that the new filter can be used for enhancing noisy contours and rapid object detection in microscopical images. In the detection context, our experiments show that the proposed filter is definitely superior to alternative approaches, when high localization accuracy is required.

Reconfigurable Hybrid CMOS/Nanodevice Circuits for Image Processing

We have analyzed two options of using hybrid CMOS/nanodevice circuits with area-distributed (CMOL) interface for the low-level image processing tasks, on the simplest example of 2-D image convolution with a sizable filter window. The first option is to use digital, DSP-like circuits based on a reconfigurable CMOL fabric, while the second one is based on mixed-signal CMOL circuits with the analog presentation of input and output data and the binary presentation of the filter function. Estimates of the circuit performance have been carried out for the 45-nm CMOS technology and the 4.5-nm nanowire half-pitch, and the power consumption fixed at a manageable, ITRS-specified level. In the digital case, the circuit area per pixel is about 25times25 , and the time necessary for convolving a 1024times1024-pixel, 12-bit-accurate image with a 3232-pixel window function of similar accuracy is close to 25 , much shorter than that estimated for purely CMOS circuits with the same minimum feature size on 45 nm. For a mixed-signal CMOL circuit, the corresponding numbers are much better ( ~1 mum2 and 1mus, respectively), but this option requires a very high (~1%) reproducibility of on currents of the necessary crosspoint devices (programmable diodes), which has not yet been reached experimentally.

How close are we to solving the problem of automated visual surveillance?

The problem of automated visual surveillance has spawned a lively research area, with 2005 seeing three conferences or workshops and special issues of two major journals devoted to the topic. These alone are responsible for somewhere in the region of 240 papers and posters on automated visual surveillance before we begin to count those presented in more general fora. Many of these systems and algorithms perform one small sub-part of the surveillance task, such as motion detection. But even with low level image processing tasks it is often difficult to compare systems on the basis of published results alone. This review paper aims to answer the difficult question “How close are we to developing surveillance related systems which are really useful?” The first section of this paper considers the question of surveillance in the real world: installations, systems and practises. The main body of the paper then considers existing computer vision techniques with an emphasis on higher level processes such as behaviour modelling and event detection. We conclude with a review of the evaluative mechanisms that have grown from within the computer vision community in an attempt to provide some form of robust evaluation and cross-system comparability.

Performance Evaluation and Limitations of a Vision System on a Reconfigurable/Programmable Chip

This paper presents a survey of the characteristics of a vision system implemented in a reconfigurable/programmable chip (FPGA). System limitations and performance have been evaluated in order to derive specifications and constraints for further vision system synthesis. The system hereby reported has a conventional architecture. It consists in a central microprocessor (CPU) and the necessary peripheral elements for data acquisition, data storage and communications. It has been designed to stand alone, but a link to the programming and debugging tools running in a digital host (PC) is provided. In order to alleviate the computational load of the central microprocessor, we have designed a visual co-processor in charge of the low-level image processing tasks. It operates autonomously, commanded by the CPU, as another system peripheral. The complete system, without the sensor, has been implemented in a single reconfigurable chip as a SOPC. The incorporation of a dedicated visual co-processor, with specific circuitry for low-level image processing acceleration, enhances the system throughput outperforming conventional processing schemes. However, time- multiplexing of the dedicated hardware remains a limiting factor for the achievable peak computing power. We have quantified this effect and sketched possible solutions, like replication of the specific image processing hardware.

Image Analysis of Soil Micromorphology: Feature Extraction, Segmentation, and Quality Inference

We present an automated system that we have developed for estimation of the bioecological quality of soils using various image analysis methodologies. Its goal is to analyze soilsection images, extract features related to their micromorphology, and relate the visual features to various degrees of soil fertility inferred from biochemical characteristics of the soil. The image methodologies used range from low-level image processing tasks, such as nonlinear enhancement, multiscale analysis, geometric feature detection, and size distributions, to object-oriented analysis, such as segmentation, region texture, and shape analysis.

A pixel-parallel image processor using logic pitch-matched to dynamic memory

A pixel-parallel image processor provides the capability for desktop systems to perform low-level image processing tasks in real time. Compact logic units are pitch-matched to DRAM columns to form dense blocks of processing elements. The processing elements are interconnected to form a 64/spl times/64 array, with each processing element assigned to a single pixel. Operating with a 60-ns clock cycle in a complete system, fully functional devices dissipate 300 mW. Using the devices, low-level image processing tasks have been performed in real time with input images provided at rates exceeding 30 frames/s.

KNOWLEDGE-BASED SHAPE-FROM-SHADING

In this paper, we study the problem of recovering approximate shape from the shading of a three-dimensional object in a single image when knowledge about the object is available. The application of knowledge-based methods to low-level image processing tasks will help overcome problems that arise from processing images using a pixel-based approach. Shape-from-shading has generally been approached by precognitive vision methods where a standard operator is applied to the image based on assumptions about the imaging process and generic properties of what appears. This paper explores some advantages of applying knowledge and hypotheses about what appears in the image. The knowledge and hypotheses used here come from domain knowledge and edge-matching. Specifically, we are able to find solutions to some problems that cannot be solved by other methods and gain advantages in terms of computation speed over similar approaches. Further, we can fully automate the derivation of the approximate shape of an object. This paper demonstrates the efficacy of using knowledge in the basic operation of an early vision operator, and so introduces a new paradigm for computer vision that may be applied to other early vision operators.

A linear array processor with dynamic frequency clocking for image processing applications

The need for high-performance image processing systems has led to the design and development of several application-specific parallel processing systems. An SIMD linear array processor with dynamic frequency clocking is proposed for real-time image processing applications. The architecture uses a novel concept called dynamic frequency clocking which allows the processor to vary the clock frequency dynamically based on the operation being performed. A VLSI chip based on the proposed architecture has been designed and verified using the Cadence design tools. The chip will operate at between 400 and 50 MHz based on the operation being performed. Several low-level image processing tasks have been mapped onto the architecture to evaluate the system performance and to demonstrate the effectiveness of the dynamic frequency clocking scheme.