Image Processing Tasks Research Articles

Disentangling Light Fields for Super-Resolution and Disparity Estimation.

Light field (LF) cameras record both intensity and directions of light rays, and encode 3D scenes into 4D LF images. Recently, many convolutional neural networks (CNNs) have been proposed for various LF image processing tasks. However, it is challenging for CNNs to effectively process LF images since the spatial and angular information are highly inter-twined with varying disparities. In this paper, we propose a generic mechanism to disentangle these coupled information for LF image processing. Specifically, we first design a class of domain-specific convolutions to disentangle LFs from different dimensions, and then leverage these disentangled features by designing task-specific modules. Our disentangling mechanism can well incorporate the LF structure prior and effectively handle 4D LF data. Based on the proposed mechanism, we develop three networks (i.e., DistgSSR, DistgASR and DistgDisp) for spatial super-resolution, angular super-resolution and disparity estimation. Experimental results show that our networks achieve state-of-the-art performance on all these three tasks, which demonstrates the effectiveness, efficiency, and generality of our disentangling mechanism. Project page: https://yingqianwang.github.io/DistgLF/.

Learning to Bound: A Generative Cramér-Rao Bound

The Cramér-Rao bound (CRB), a well-known lower bound on the performance of any unbiased parameter estimator, has been used to study a wide variety of problems. However, to obtain the CRB, requires an analytical expression for the likelihood of the measurements given the parameters, or equivalently a precise and explicit statistical model for the data. In many applications, such a model is not available. Instead, this work introduces a novel approach to approximate the CRB using data-driven methods, which removes the requirement for an analytical statistical model. This approach is based on the recent success of deep generative models in modeling complex, high-dimensional distributions. Using a learned normalizing flow model, we model the distribution of the measurements and obtain an approximation of the CRB, which we call Generative Cramér-Rao Bound (GCRB). Numerical experiments on simple problems validate this approach, and experiments on two image processing tasks of image denoising and edge detection with a learned camera noise model demonstrate its power and benefits.

Deep Learning Framework of Convolutional Neural Network (CNN) and Attention CNN for Early Diagnosis of Alzheimer's Disease

One of the biggest killers in the industrialized world is Alzheimer's disease (AD). Although computer-aided techniques have shown promising outcomes in laboratory experiments, they have yet to be used in a clinical setting. Recently, deep neural networks have gained traction, particularly for image processing tasks. There has been a dramatic increase in the number of publications written on the topic of identifying AD using deep learning since 2017. It has been observed that deep networks are more efficient than standard machine learning methods for detecting AD. It remains difficult to identify AD because distinguishing between comparable brain signals during categorization needs an extremely discriminative depiction of features. This paper proposed a deep neural network method for prediction the AD. Low-level computer vision has been a hotspot for research into deep convolutional neural networks (CNNs). Studies often focus on enhancing performance through the use of very deep CNNs. Yet, as one goes deeper, the effect of the shallow layers on the deeper ones gradually diminishes. Prompted by reality. This paper compared with the CNN and attention CNN models. The proposed model applied in the AD dataset which contains 5121 images for the train set. The results showed the attention CNN model is better than the CNN model in accuracy, precision, recall, loss, and AUC.

Formation of a typical form of an object image in a series of digital frames

A computational method for the automated formation of a typical form of a digital image of the investigated objects on a series of digital frames has been developed. Due to the imperfection of the mounting of digital cameras, as well as their automated mounts, their immobility at shooting during exposure time can be disturbed, which leads to the formation of "blurred" images of objects of various forms. Due to such inaccuracies in the tracking of objects on digital frames, even in one series, the typical form of the image of objects can vary from frame to frame. This fact of the difference in the standard form significantly complicates the execution of various image processing tasks. In order to simplify the evaluation of the image parameters of objects in a series of digital frames, it has been proposed to use a typical image on a digital frame corresponding to the average image of objects as a model of object images. In this case, the appearance of the image of the object, its form, the distribution of brightness in the image will be determined only by the typical image. This paper proposes a computational method for the automated formation and evaluation of the typical form of the image of an object in a digital frame based on the initial data – the actual given digital frame. This computational method is based on the selection of single images of objects and the formation of their rectangular area. Next, the offset is evaluated, and the selected single images of objects are normalized to calculate the typical form of the object image. Using the method makes it possible to highlight objects against the background of noise and reduce the number of false detections. It is recommended to apply the method only in the case when the frames have defects and "blurs" during the shooting, otherwise there will be unreasonable additional computational costs. The developed computational method was successfully tested in practice within the framework of the CoLiTec project and implemented in the intraframe processing unit of the Lemur software.

Image Unprocessing: A Pipeline to Recover Raw Data from sRGB Images

Access to high quality datasets is an essential condition for data-driven methods as it is known that mismatches between the distributions of training and test data may cause learning-based methods to fail. This issue has led to one of the most active research subjects in learning-based image restoration. For instance neural networks trained on unrealistic synthetic data may not generalize to real data even if they perform well on those synthetic data. This is specially problematic for image and video processing tasks, such as denoising, which are performed on raw data, since acquiring real raw datasets is not straightforward and is even impossible in some cases (acquiring a video dataset of real noise with clean ground-truth, for instance). Consequently, CNNs are often trained on synthetic data. Synthesizing realistic raw data is a difficult task and requires to invert properly the image processing pipeline. This paper focuses on the backward pipeline proposed by Brooks et al. [Unprocessing images for learned raw denoising, CVPR 2019] which aims at producing raw data from sRGB images. **This is an MLBriefs article, the source code has not been reviewed!**<br> **The original source code is [[available here|https://github.com/timothybrooks/unprocessing]] (last checked 2022/12/30).**

Quaternion tensor singular value decomposition using a flexible transform-based approach

A flexible transform-based tensor product named ★QT-product for Lth-order (L≥3) quaternion tensors is proposed. Based on the ★QT-product, we define the corresponding singular value decomposition named TQt-SVD and the rank named TQt-rank of the Lth-order (L≥3) quaternion tensor. Furthermore, with orthogonal quaternion transformations, the TQt-SVD can provide the best TQt-rank-s approximation of any Lth-order (L≥3) quaternion tensor. On the other side, since a color pixel with RGB channels can be well encoded as a pure quaternion, the proposed TQt-SVD, therefore, becomes a new mathematical tool for many color image processing tasks. In the experiments, we have verified the effectiveness of the proposed TQt-SVD in the application of the best TQt-rank-s approximation for color videos represented by third-order quaternion tensors.

Rapid and robust endoscopic content area estimation: a lean GPU-based pipeline and curated benchmark dataset

ABSTRACT Endoscopic content area refers to the informative area enclosed by the dark, non-informative, border regions present in most endoscopic footage. The estimation of the content area is a common task in endoscopic image processing and computer vision pipelines. Despite the apparent simplicity of the problem, several factors make reliable real-time estimation surprisingly challenging. The lack of rigorous investigation into the topic combined with the lack of a common benchmark dataset for this task has been a long-lasting issue in the field. In this paper, we propose two variants of a lean GPU-based computational pipeline combining edge detection and circle fitting. The two variants differ by relying on handcrafted features, and learned features respectively to extract content area edge point candidates. We also present a first-of-its-kind dataset of manually annotated and pseudo-labelled content areas across a range of surgical indications. To encourage further developments, the curated dataset, and an implementation of both algorithms, has been made public (anonymised url https://doi.org/10.7303/syn32148000, https://github.com/charliebudd/torch-content-area). We compare our proposed algorithm with a state-of-the-art U-Net-based approach and demonstrate significant improvement in terms of both accuracy (Hausdorff distance: 6.3 px versus 118.1 px) and computational time (Average runtime per frame: 0.13 ms versus 11.2 ms).

Boundary-aware feature and prediction refinement for polyp segmentation

ABSTRACT Polyp segmentation from colonoscopy videos is an essential task in medical image processing for detecting early cancer. However, segmenting a precise boundary is still challenging, even with powerful deep neural networks. We consider the difficulty can be caused by: (1) the ambiguity boundary and (2) some complicated shape makes polyps hard to segment. To address these problems, we propose the Boundary-aware Feature and Prediction Refinement framework (BaFPR) for polyp segmentation. Specifically, we design a segmentation decoder for representation learning with boundary prior and propose a novel consistency loss to learn clues from the polar coordinate. The decoder mainly consists of a boundary prior module (BPM) and a bi-directional fusion module (BiFM). BPM is designed to learn the boundary prior, while BiFM learns to fuse representations of BPM and multi-scale representations from an encoder. To handle these complicated shapes of polyps, we maintain an extra segmentation network that learns with polar transformations of data to provide extra clues for the main segmentation network by our proposed consistency loss. We evaluated BaFPR with five challenging datasets for polyp segmentation and the results showed that our proposal consistently improves the segmentation performance of polyps. Code available at: https://github.com/MoriLabNU/BaFPR.

Aerial Image Denoising Using a Best-So-Far ABC-based Adaptive Filter Method

Nowadays, digital images play an increasingly important role in helping to explain phenomena and to attract people’s attention through various types of media rather than the use of text. However, the quality of digital images may be degraded due to noise that has occurred either during their recording or their transmission via a network. Therefore, removal of image noise, which is known as “image denoising”, is one of the primary required tasks in digital image processing. Various methods in earlier studies have been developed and proposed to remove the noise found in images. For example, the use of metric filters to eliminate noise has received much attention from researchers in recent literature. However, the convergence speed when searching for the optimal filter coefficient of these proposed algorithms is quite low. Previous research in the past few years has found that biologically inspired approaches are among the more promising metaheuristic methods used to find optimal solutions. In this work, an image denoising approach based on the best-so-far (BSF) ABC algorithm combined with an adaptive filter is proposed to enhance the performance of searching for the optimal filter coefficient in the denoising process. Experimental results indicate that the denoising of images employing the proposed BSF ABC technique yields good quality and the ability to remove noise while preventing the features of the image from being lost in the denoising process. The denoised image quality obtained by the proposed method achieves a 20% increase compared with other recently developed techniques in the field of biologically inspired approaches.

Mesh Total Generalized Variation for Denoising.

Recent studies have shown that the Total Generalized Variation (TGV) is highly effective in preserving sharp features as well as smooth transition variations for image processing tasks. However, currently there is no existing work that is suitable for applying TGV to 3D data, in particular, triangular meshes. In this article, we develop a novel framework for discretizing second-order TGV on triangular meshes. Further, we propose a TGV-based variational method for the denoising of face normal fields on triangular meshes. The TGV regularizer in our method is composed of a first-order term and a second-order term, which are automatically balanced. The first-order term allows our TGV regularizer to locate and preserve sharp features, while the second-order term allows our regularizer to recognize and recover smoothly curved regions. To solve the optimization problem, we introduce an efficient iterative algorithm based on variable-splitting and augmented Lagrangian method. Extensive results and comparisons on synthetic and real scanning data validate that the proposed method outperforms the state-of-the-art visually and numerically.

2D-FACE ALIGNMENT WITH CYCLEGAN FACE AGING IMAGE-TO-IMAGE TRANSLATION

Face alignment is one of the pre-processing processes where the face plays a crucial part in image tasks and computer vision. As part of the pre-processing step, it is the first step taken before implementing an image processing task. By aligning face, it is expected to improve the network model performance, because good input data is now represented in the network model. This research aims to see whether pre-processing the input data can improve the network model performance. A 2D-face alignment technique is used to align all the input images. All the input image that is already being aligned is used as the input image for the CycleGAN face aging image-to-image translation model. In this work, the CycleGAN network model is used to translate an image of a young face to their older version and vice versa. The result obtained shows that if the network model is presented with a properly aligned face, it can translate the image into a younger or older version better than when presented with a non-aligned face.

Predicting visual difference maps for computer‐generated images by integrating human visual system model and deep learning

The quality of images generated by computer graphics rendering algorithms is mainly affected by visible distortion at some pixel locations. Image quality assessment (IQA) metrics are commonly utilized to assess the quality of rendered images, but their results are a global difference value, which does not provide pixel-wise differences to optimize the renderings. In contrast, visibility difference models including visual perception models and deep learning models can calculate pixel-wise visibility difference between distorted images and reference images. However, they either are only applied to a single type of visible distortion or are seriously dependent on datasets. To this end, the authors propose a novel model, dubbed Human Visual Perception and Deep Learning Image Difference Metric (HPDL-IDM), which combines the Human Visual System (HVS) model and deep learning. HPDL-IDM primarily consists of two modules: (i) the visual perception feature calculation module, which calculates difference maps between various kinds of features extracted from the reference image and the distorted image according to the visual characteristics of human eyes and concatenates them, and (ii) the deep learning module, which utilizes a neural network of encoder–decoder structure to train on the LocvisVC and VisTexRes datasets whose input and output are these concatenated feature difference maps and the final image distortion visibility difference map respectively. Additionally, the authors pool the final difference map into a global difference value between 0 and 1 to apply their model to many image processing tasks related to Image quality metrics (IQMs). Experimental results show that HPDL-IDM's generalization capacity and accuracy are improved by a large margin compared to other models.

2D Image Object Detection Aided by Generative Adversarial Networks: A Literature Review

Object Detection (OD) is one of the most critical tasks in 2D image processing. The researchers proposed multiple math models and frameworks based on Deep Convolutional Networks, such as R-CNN, SSD, and YOLO are the most common. Generative Adversarial Nets (GAN) represent a prominent field of study in machine learning, and it has been applied to many tasks with exciting results. This work aims to assess the potential of GANs applied to OD tasks and the proposed frameworks as a field of study. The methodology used was a systemic review of 14 papers. The conclusion shows that although OD and GANs are popular themes, there are not many developments in the intersection of both subjects. Therefore, OD with GAN-applied tasks is an excellent field to explore in future works.

Hybrid non-convex regularizers model for removing multiplicative noise

Obtaining natural and realistic restorations from the noisy images contaminated by multiplicative noise is a challenging task in image processing. To get over this conundrum, by introducing the non-convex potential functions into the total variation and high-order total variation regularizers, we investigate a novel hybrid non-convex optimization model for image restoration. Numerically, to optimize the resulting high-order PDE system, a proximal linearized alternating minimization method, based on the classical iteratively reweighted ℓ 1 algorithm and variable splitting technique, is designed in detail. Meanwhile, the convergence of the constructed algorithm is also established on the basis of convex analysis. The provided numerical experiments point out that our new scheme shows superiorities in both visual effects and quantitative comparison, especially in terms of the staircase aspects suppression and edge details preservation, compared with some popular denoising methods.

Perceptual Hashing With Complementary Color Wavelet Transform and Compressed Sensing for Reduced-Reference Image Quality Assessment

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application.

Convergence of Neural Networks with a Class of Real Memristors with Rectifying Characteristics

The paper considers a neural network with a class of real extended memristors obtained via the parallel connection of an ideal memristor and a nonlinear resistor. The resistor has the same rectifying characteristic for the current as that used in relevant models in the literature to account for diode-like effects at the interface between the memristor metal and insulating material. The paper proves some fundamental results on the trajectory convergence of this class of real memristor neural networks under the assumption that the interconnection matrix satisfies some symmetry conditions. First of all, the paper shows that, while in the case of neural networks with ideal memristors, it is possible to explicitly find functions of the state variables that are invariants of motions, the same functions can be used as Lyapunov functions that decrease along the trajectories in the case of real memristors with rectifying characteristics. This fundamental property is then used to study convergence by means of a reduction-of-order technique in combination with a Lyapunov approach. The theoretical predictions are verified via numerical simulations, and the convergence results are illustrated via the applications of real memristor neural networks to the solution of some image processing tasks in real time.

MsRAN: a multi-scale residual attention network for multi-model image fusion.

Fusion is a critical step in image processing tasks. Recently, deep learning networks have been considerably applied in information fusion. But the significant limitation of existing image fusion methods is the inability to highlight typical regions of the source image and retain sufficient useful information. To address the problem, the paper proposes a multi-scale residual attention network (MsRAN) to fully exploit the image feature. Its generator network contains two information refinement networks and one information integration network. The information refinement network extracts feature at different scales using convolution kernels of different sizes. The information integration network, with a merging block and an attention block added, prevents the underutilization of information in the intermediate layers and forces the generator to focus on salient regions in multi-modal source images. Furthermore, in the phase of model training, we add an information loss function and adopt a dual adversarial structure, enabling the model to capture more details. Qualitative and quantitative experiments on publicly available datasets validate that the proposed method provides better visual results than other methods and retains more detail information.

Hyper-LGNet: Coupling Local and Global Features for Hyperspectral Image Classification

Hyperspectral sensors provide an opportunity to capture the intensity of high spatial/spectral information and enable applications for high-level earth observation missions, such as accurate land cover mapping and target/object detection. Currently, convolutional neural networks (CNNs) are good at coping with hyperspectral image processing tasks because of the strong spatial and spectral feature extraction ability brought by hierarchical structures, but the convolution operation in CNNs is limited to local feature extraction in both dimensions. In the meanwhile, the introduction of the Transformer structure has provided an opportunity to capture long-distance dependencies between tokens from a global perspective; however, Transformer-based methods have a restricted ability to extract local information because they have no inductive bias, as CNNs do. To make full use of these two methods’ advantages in hyperspectral image processing, a dual-flow architecture named Hyper-LGNet to couple local and global features is firstly proposed by integrating CNN and Transformer branches to deal with HSI spatial-spectral information. In particular, a spatial-spectral feature fusion module (SSFFM) is designed to maximally integrate spectral and spatial information. Three mainstream hyperspectral datasets (Indian Pines, Pavia University and Houston 2013) are utilized to evaluate the proposed method’s performance. Comparative results show that the proposed Hyper-LGNet achieves state-of-the-art performance in comparison with the other nine approaches concerning overall accuracy (OA), average accuracy (AA) and kappa index. Consequently, it is anticipated that, by coupling CNN and Transformer structures, this study can provide novel insights into hyperspectral image analysis.

A Multi-Purpose Shallow Convolutional Neural Network for Chart Images.

Charts are often used for the graphical representation of tabular data. Due to their vast expansion in various fields, it is necessary to develop computer algorithms that can easily retrieve and process information from chart images in a helpful way. Convolutional neural networks (CNNs) have succeeded in various image processing and classification tasks. Nevertheless, the success of training neural networks in terms of result accuracy and computational requirements requires careful construction of the network layers’ and networks’ parameters. We propose a novel Shallow Convolutional Neural Network (SCNN) architecture for chart-type classification and image generation. We validate the proposed novel network by using it in three different models. The first use case is a traditional SCNN classifier where the model achieves average classification accuracy of 97.14%. The second use case consists of two previously introduced SCNN-based models in parallel, with the same configuration, shared weights, and parameters mirrored and updated in both models. The model achieves average classification accuracy of 100%. The third proposed use case consists of two distinct models, a generator and a discriminator, which are both trained simultaneously using an adversarial process. The generated chart images are plausible to the originals. Extensive experimental analysis end evaluation is provided for the classification task of seven chart classes. The results show that the proposed SCNN is a powerful tool for chart image classification and generation, comparable with Deep Convolutional Neural Networks (DCNNs) but with higher efficiency, reduced computational time, and space complexity.

Convolutional Neural Network Based on Crossbar Arrays of (Co-Fe-B)x(LiNbO3)100-x Nanocomposite Memristors.

Convolutional neural networks (CNNs) have been widely used in image recognition and processing tasks. Memristor-based CNNs accumulate the advantages of emerging memristive devices, such as nanometer critical dimensions, low power consumption, and functional similarity to biological synapses. Most studies on memristor-based CNNs use either software models of memristors for simulation analysis or full hardware CNN realization. Here, we propose a hybrid CNN, consisting of a hardware fixed pre-trained and explainable feature extractor and a trainable software classifier. The hardware part was realized on passive crossbar arrays of memristors based on nanocomposite (Co-Fe-B)x(LiNbO3)100−x structures. The constructed 2-kernel CNN was able to classify the binarized Fashion-MNIST dataset with ~ 84% accuracy. The performance of the hybrid CNN is comparable to the other reported memristor-based systems, while the number of trainable parameters for the hybrid CNN is substantially lower. Moreover, the hybrid CNN is robust to the variations in the memristive characteristics: dispersion of 20% leads to only a 3% accuracy decrease. The obtained results pave the way for the efficient and reliable realization of neural networks based on partially unreliable analog elements.