Color Image Reconstruction Research Articles

Quaternion Nuclear Norm Minus Frobenius Norm Minimization for color image reconstruction

Color image restoration methods typically represent images as vectors in Euclidean space or combinations of three monochrome channels. However, they often overlook the correlation between these channels, leading to color distortion and artifacts in the reconstructed image. To address this, we present Quaternion Nuclear Norm Minus Frobenius Norm Minimization (QNMF), a novel approach for color image reconstruction. QNMF utilizes quaternion algebra to capture the relationships among RGB channels comprehensively. By employing a regularization technique that involves nuclear norm minus Frobenius norm, QNMF approximates the underlying low-rank structure of quaternion-encoded color images. Theoretical proofs are provided to ensure the method’s mathematical integrity. Demonstrating versatility and efficacy, the QNMF regularizer excels in various color low-level vision tasks, including denoising, deblurring, inpainting, and random impulse noise removal, achieving state-of-the-art results.

Optimizing Visual Sensor Node Image Compression for Resource-Constrained Environments

Abstract: An Approximate hybrid multiplier design is proposed to reduce the data-path width of lifting 2-D discrete wavelet transform (DWT) especially for wireless visual sensor node applications. The proposed multiplier design takes multiplier operand of size less by 3 bit and calculates full-width outputs with marginal loss of accuracy, which is significant. The proposed approximate 2-D DWT structure uses data-path DWT levels and produces reconstructed color images higher than the 70-dB peak-signal-to-noise ratio (PSNR). Compared with the existing fractional wavelet transform (FrWT)-based 2D DWT structures, the proposed structure has a smaller data-path width and free from overhead. Compared with the existing parallel structure

Joint Demosaicing and Denoising for Spike Camera

As a neuromorphic camera with high temporal resolution, spike camera can capture dynamic scenes with high-speed motion. Recently, spike camera with a color filter array (CFA) has been developed for color imaging. There are some methods for spike camera demosaicing to reconstruct color images from Bayer-pattern spike streams. However, the demosaicing results are bothered by severe noise in spike streams, to which previous works pay less attention. In this paper, we propose an iterative joint demosaicing and denoising network (SJDD-Net) for spike cameras based on the observation model. Firstly, we design a color spike representation (CSR) to learn latent representation from Bayer-pattern spike streams. In CSR, we propose an offset-sharing deformable convolution module to align temporal features of color channels. Then we develop a spike noise estimator (SNE) to obtain features of the noise distribution. Finally, a color correlation prior (CCP) module is proposed to utilize the color correlation for better details. For training and evaluation, we designed a spike camera simulator to generate Bayer-pattern spike streams with synthesized noise. Besides, we captured some Bayer-pattern spike streams, building the first real-world captured dataset to our knowledge. Experimental results show that our method can restore clean images from Bayer-pattern spike streams. The source codes and dataset are available at https://github.com/csycdong/SJDD-Net.

Learning a Deep Demosaicing Network for Spike Camera With Color Filter Array.

For capturing dynamic scenes with ultra-fast motion, neuromorphic cameras with extremely high temporal resolution have demonstrated their great capability and potential. Different from the event cameras that only record relative changes in light intensity, spike camera fires a stream of spikes according to a full-time accumulation of photons so that it can recover the texture details for both static areas and dynamic areas. Recently, color spike camera has been invented to record color information of dynamic scenes using a color filter array (CFA). However, demosaicing for color spike cameras is an open and challenging problem. In this paper, we develop a demosaicing network, called CSpkNet, to reconstruct dynamic color visual signals from the spike stream captured by the color spike camera. Firstly, we develop a light inference module to convert binary spike streams to intensity estimates. In particular, a feature-based channel attention module is proposed to reduce the noises caused by quantization errors. Secondly, considering both the Bayer configuration and object motion, we propose a motion-guided filtering module to estimate the missing pixels of each color channel, without undesired motion blur. Finally, we design a refinement module to improve the intensity and details, utilizing the color correlation. Experimental results demonstrate that CSpkNet can reconstruct color images from the Bayer-pattern spike stream with promising visual quality.

Color Sensing and Image Reconstruction Using Intelligent Machine Learning Algorithm with PINIP Radial Junction Imager

The development of a filterless imager has been eagerly awaited to overcome the diffraction limit when pixel sizes decrease to subwavelength scales. We propose an architecture for a filterless imager based on a symmetric inversely stacked radial junction (RJ) PINIP photodetector over silicon nanowires (SiNWs), whereby the diameter of which is less than 500 nm, which preliminarily displays the capability of bias-selected and tunable spectrum responses to the R, G, and B color bands. Assisted via suitably trained deep learning algorithms, the imager can provide more accurate color discrimination and imaging capabilities. Here, we used KNN (k-nearest neighbor) and convolution neural network (CNN) methods to retrieve the RGB ratios from the measured photocurrent value based on the pre-trained bias-tuned spectrum responses and reconstructed the images with high accuracy. Further, we demonstrated the capability of restoring sub-sampling pictures via CNN with a U-net architecture, and satisfactory reconstruction was obtained even with a sampling ratio as low as 20%. Our imaging scheme cannot only be used for high-resolution imaging but can also pave the way for application in single-pixel imaging and compressive sensing.

Joint optimization of splitter pattern and image reconstruction for metasurface-based color imaging systems

The metasurface-based splitters have demonstrated great potential in high-resolution, dark-scene, and ultra-fast imaging due to their excellent capabilities of preserving light intensity, attracting increasingly more research interests. This paper investigates the splitter pattern design, which determines the theoretical performance limit of metasurface-based color imaging systems, to guide their further development. We first establish a unified mathematical model for the splitter-based imaging process. This model reveals that the raw image detected by the splitters is degraded by noise, incomplete sampling, and blurring. Based on this, we develop a deep neural network capable of simultaneously tackling multiple image processing tasks to achieve color image reconstruction. We then conduct evaluation experiments for several existing patterns, through which we establish three design principles to guide the optimization of a splitter pattern. Furthermore, by jointly learning the parameters of our proposed imaging model and reconstruction network, we achieve an automatic optimization of a splitter pattern. The experiments demonstrate that the optimized pattern can achieve better quantitative and visual results than the original pattern, validating the effectiveness of this pattern optimization method.

Fast physic-informed mixer architecture for color Lensfree holographic reconstruction

Accurate color image reconstruction from multi-wavelength holograms is crucial in biomedical imaging applications. Current data-driven deep learning methods have made significant progress with promising reconstruction performance. Especially, untrained neural network methods have been developed to overcome data acquisition and the generalization issue. However, improving reconstruction quality requires more iterations, which can lead to longer convergence time. To address this challenge, we proposed an efficient complex-valued attention mixer (ECA-Mixer) architecture for fast and accurate physic-informed color holographic reconstruction. Our architecture consists of three core modules, an encoder, a nonlinear transformer and a decoder, each combining efficient attention mechanisms and Mixer layers for channel feature extraction and spatial information transformation. To preserve high-frequency information, we also introduced the use of 2D Haar wavelet and its inverse transform for encoding and decoding features. We then validate our method with extensive experiments on simulated and experimental samples and achieve state-of-art color reconstruction results in terms of computation time and image quality. Besides, we also demonstrate that our proposed lightweight network is capable of imaging large-size wide-field samples at a higher resolution in just a few minutes. The code is available and will be uploaded later at https://github.com/DeepPhysicVision/ECA-Mixer.git.

Compressive color ghost imaging based on pseudo-inverse matrix

In this paper, a compressive color ghost imaging method based on pseudo-inverse matrix is proposed, which improves the quality of imaging result via post-processing the measurement matrix. For a color ghost imaging utilizing compressive sensing algorithm, if we treat a color image as a form of grayscale image and perform pseudo-inverse operation on the measurement matrix, a new measurement model will be established and the preliminary ghost imaging reconstruction result can be obtained by the compressive sensing algorithm, then the preliminary result is converted into a superposition of three channels to form the final color image. The feasibility of this method is proved by numerical simulation and physical experiment, and comparations among our method and the latest typical improvement methods, i.e., the singular value decomposition compressive ghost imaging and the pseudo-inverse ghost imaging, are conducted. The results show that our method can achieve the better quality of reconstructed color image with a high structure similarity beyond 0.8.

Self-embedding reversible color-to-grayscale conversion with watermarking feature

This paper presents a self-embedding reversible color-to-grayscale conversion (RCGC) algorithm that makes good use of deep learning, vector quantization, and halftoning techniques to achieve its goals. By decoupling the luminance information of a pixel from its chrominance information, it explicitly controls the luminance error of both the conversion outputs and their corresponding reconstructed color images. It can also alleviate the burden of the deep learning network used to restore the embedded chrominance information during the reconstruction of the color image. Luminance-guided chrominance quantization and checkerboard-based halftoning are introduced in the paper to encode the chrominance information to be embedded while reference-guided inverse halftoning is proposed to restore the color image. Simulation results verify that its performance is remarkably superior to conventional state-of-art RCGC algorithms in various measures. In the aspect of authentication, embedding the watermark and chrominance information is realized with context-based pixel-wise encryption and a key-based watermark bit positioning mechanism, which makes us possible to locate tampered regions and prevent unauthorized use of the chrominance information.

High-quality and high-diversity conditionally generative ghost imaging based on denoising diffusion probabilistic model.

Deep-learning (DL) methods have gained significant attention in ghost imaging (GI) as promising approaches to attain high-quality reconstructions with limited sampling rates. However, existing DL-based GI methods primarily emphasize pixel-level loss and one-to-one mapping from bucket signals or low-quality GI images to high-quality images, tending to overlook the diversity in image reconstruction. Interpreting image reconstruction from the perspective of conditional probability, we propose the utilization of the denoising diffusion probabilistic model (DDPM) framework to address this challenge. Our designed method, known as DDPMGI, can not only achieve better quality but also generate reconstruction results with high diversity. At a sampling rate of 10%, our method achieves an average PSNR of 21.19 dB and an SSIM of 0.64, surpassing the performance of other comparison methods. The results of physical experiments further validate the effectiveness of our approach in real-world scenarios. Furthermore, we explore the potential application of our method in color GI reconstruction, where the average PSNR and SSIM reach 20.055 dB and 0.723, respectively. These results highlight the significant advancements and potential of our method in achieving high-quality image reconstructions in GI, including color image reconstruction.

Sequential color ghost imaging based on compressed sensing algorithm of post-processing measurement matrix

Compressed sensing algorithm can be used in color ghost imaging to restore the image of object at the less demand of measurements times. However, the quality of the reconstructed color image is usually not satisfactory. In this paper, we propose a sequential color ghost imaging method that can complete color ghost imaging in a simple architecture and improve the quality of color image, which is optimize compressed sensing that can get better result of compressed sensing algorithm by post-processing the measurement matrix and establishing a new compressed sensing process. Under the condition of the same measurement times and reconstruction algorithm, compared with the unpost-processing one, the quality and detail of reconstructed image by the post-processing measurement matrix is quite improved especially with the increase of measurement times. Discussions on factors affecting the quality of the new compressed sensing process, such as the number of measurement and the detection noise intensity, are also conducted. Numerical simulation and physical experiment verified our proposed method.

Deep learning method for pinhole array color image reconstruction.

The following paper proposes a combination of a supervised encoder-decoder neural network with coded apertures. Coded apertures provide improved sensitivity and signal-to-noise ratio (SNR) in planar images. The unique array design of this method overcomes the spatial frequency cutoff found in standard multi-pinhole arrays. In this design, the pinholes were positioned to minimize loss in spatial frequencies. The large number of pinholes results in significant overlapping on the detector. To overcome the overlapping issue, reconstruction of the object from the obtained image is done using inverse filtering methods. However, traces of duplications remain leading to a decline in SNR, contrast, and resolution. The proposed technique addresses the challenge of image distortion caused by the lack of accuracy in the inverse filter methods, by using a deep neural network. In this work, the coded aperture is combined with a deep convolutional neural network (CNN) to remove noise caused by pinhole imaging and inverse filter limitations. Compared to only using Wiener filtering, the proposed method delivers higher SNR, contrast, and resolution. The imaging system is presented in detail with experimental results that illustrate its efficiency.

Experimental and Simulated Study of the Relationship Between Color Camera Imaging and Color-Modeled Equivalence Ratio Measurement

A simulation methodology was proposed to reconstruct color images under different camera spectral sensitivity functions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSSFs</i> ) by integrating the hyperspectral images. Based on this proposed simulation method, the effects of different <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSSFs</i> , Bayer patterns, and demosaicing algorithms on color-modeled <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CH</i> * and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C₂</i> * measurement have been investigated. Firstly, initial full-color images were reconstructed under the CIE 1931 XYZ matching function ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CMF</i> ). Then, the Bayer pattern encoded images were obtained from the initial reconstructed images by removing the pixel values according to different Bayer patterns. Next, three different conventional demosaicing algorithms were employed to calculate the full interpolated color images, and last, the effects of demosaicing algorithms on color-modeled <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CH</i> * and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C₂</i> * measurement were analyzed. It was found that the effects of Bayer patterns and demosaicing algorithms on color-modeled global equivalence ratio (Φ) measurement were limited, whereas the most significant impact is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSSF</i> . Therefore, an improved color-modeled Φ measurement model was also proposed using a color correction algorithm under <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CMF</i> , which may provide a quick, efficient, and generalized method for color-modeled Φ measurement.

A Henon-like chaotic map and its application in image encryption combined with compressed sensing

A new method of color image encryption is proposed in this paper, with the combination of a two-dimensional (2D) Henon-like chaotic map and compressed sensing for better performance. A sinusoidal function is introduced into the 2D Henon map for increasing randomness. Image encryption algorithm includes three procedures: compression of a color image through the Bernoulli observation matrix, encryption of the compressed image by 2D-Henon map with sinusoidal factor, and reconstruction of color image based on compressed sensing algorithm of orthogonal matching pursuit (OMP). Numerical simulation shows that the proposed encryption algorithm has some advantages including being hardware-friendly, sufficient keyspace, and higher security.

Invertible Color-to-Grayscale Conversion Using Lossy Compression and High-Capacity Data Hiding

Invertible color-to-grayscale conversion is a research issue in grayscale image colorization, which is a complex problem resulting from information loss. This paper presents an innovative invertible color-to-grayscale conversion idea by independently compressing the chromaticity plane and hiding it into the corresponding luminance plane as a watermark. Since the chromaticity and luminance planes are orthogonal in the proposed method, they can be processed efficiently without mutual influence. By using an efficient lossy compression operation, we can save more chromatic information. By using two high-capacity data hiding techniques (reversible watermarking (RW) and least significant bit (LSB) substitution), we can embed the compressed chromatic information into the luminance plane perfectly. For the purpose of integrality authentication, the luminance plane is hashed as part of the embedded information before the embedding. Experimental results have shown that higher quality of reconstructed color images can be achieved by using RW, but the quality of the synthesized grayscale drops sharply. By using LSB substitution, we can obtain high-quality synthesized grayscale and reconstructed color images simultaneously. Furthermore, we have compared the proposed LSB-based scheme with several recently reported state-of-the-art methods to validate the superiority of the proposed approach.

Three-dimensional natural color imaging based on focus level correlation algorithm using structured illumination microscopy

Taking advantages of high-resolution, natural color restoration, and high imaging speed, optical sectioning structured illumination microscopy (OS-SIM) plays an important role in geology, biology, and material science. However, when encountering chromatic aberration or dealing with samples with semitransparent surface, the HSV (Hue, Saturation, and Value) decoding algorithm suffers intensity deviation and fading color. In this paper, we propose a focus level correlation algorithm for 3D color image reconstruction in OS-SIM. Simulations and experiments demonstrate that the algorithm can restore color of sample authentically, and improve the image processing speed by about 45%. This new algorithm successfully improves the results and the speed of optical sectioning reconstruction, expanding the application of OS-SIM.

Superpixel-guided locality quaternion representation for color face hallucination

Recently, the learning based methods have been well exploited to hallucinate grayscale face images. When facing color images, however, the previous approaches either suffering from the non-flexibility for arbitrary pattern shapes or ignoring the inherent color information. To address these concerns, in this paper we propose a new learning model named as Superpixel-guided Locality Quaternion Representation (SLQR) for color face hallucination. Rather than handling squared patches with fixed size, the proposed method handles superpixels with adaptive shapes segmented from face images according to semantic contents, which can well preserve the face spatial features. Moreover, the superpixels are mapped into quaternion space to exploit the inherent spectral information for color image reconstruction. In addition, considering that images are inevitably corrupted by noise in practice, we extend the SLQR to the robust version (W-SLQR) by introducing a proper reweighting strategy into the objective function to suppress noise. Compared to some state-of-the-art methods, various experiments have been conducted to verify the superiority of our proposed methods in hallucinating clean and noisy face images.

Fast bilateral complementary network for deep learning compressed sensing image reconstruction

Great progress has been made in deep learning image reconstruction, but there are still problems that need to be improved, such as the extraction of image texture details and the improvement of the overall contour quality of the image, and how to reduce the transmission cost is also the focus of research. This paper proposes a fast bilateral network suitable for both grayscale and color image reconstruction. In the compression network, bilinear interpolation, fully connected layer and convolutional neural network are selected for image compression. The reconstruction network is divided into two parts: the texture path and contour path, the former reconstructs the remaining texture details of the image, and the latter performs deep contour reconstruction. The bilateral complementary residual connection method transfers the texture information to the contour path and improves the contour quality, and the improvement of the contour quality can improve the learning of texture details. Through a large number of data tests, it shows that the network in this paper has achieved comparable or better results in terms of image reconstruction quality, time-consuming and robustness against noise. It solves the problem of memory consumption for storing a large number of images, and also provides convenience for shortening space and time during image transmission.

On the limits of Conditional Generative Adversarial Neural Networks to reconstruct the identification of inhabitants from IoT low-resolution thermal sensors

One of the main objectives of smart homes is to facilitate daily life by increasing user comfort, with the potential to play a key role in revolutionizing healthcare for the elderly, the disabled and people with functional limitations. To achieve this end, smart homes will have to be able to distinguish the identity of users, their location and the activities they are performing, while also being implemented in a non-invasive way that protects the privacy of these users. Computer vision is one of the main technologies included in smart homes. However, there are drawbacks to traditional cameras, given their dependence on light and privacy-related concerns. Thermal cameras provide a solution, as they operate regardless of light conditions (e.g. at night) while respecting users’ privacy. In this work, image reconstruction and identification of inhabitants from facial images collected by low-resolution thermal sensors has been carried out by using Conditional Generative Adversarial Neural Networks (CGANs). The system has been implemented through an IoT device with raspberry Pi and dual-vision thermal and visible-spectrum sensors installed in a real smart home to automatically collect paired visible-spectrum and thermal images. Thus, different configurations of CGANs have been implemented and analyzed to achieve the following outcomes: (1) inhabitant identification (normal and masked face) with enhanced user privacy and (2) transfer from thermal to color images in the visible spectrum. Results show that the proposed CGAN achieves a recognition rate of 95% and 94% for uncovered and masked faces. This enables user identification without registering accurate facial expressions in the color image reconstruction, protecting user privacy. Furthermore, the developed system outperformed similar approaches using low-resolution datasets and has demonstrated that the accuracy of image reconstruction depends on the resolution of the input visible-spectrum images. In addition, a contribution to high-performance computing has been made by designing a CGAN that runs efficiently on multiple GPUs, achieving increased performance and response speed of the network, as well as applicability to larger problems.

Joint diagonalization for a pair of Hermitian quaternion matrices and applications to color face recognition

A new joint diagonalization algorithm for a pair of Hermitian quaternion matrices is derived incorporating real structure-preserving strategy. The structure-preserving joint diagonalization algorithm leads to a novel two-dimensional quaternion linear discriminant analysis (2D-QLDA) method for color face recognition and image reconstruction. 2D-QLDA is mathematically characterized by Hermitian quaternion generalized eigenvalue problem. A weighted norm is obtained as a new measurement to determine the distances among Fisher feature matrices, which helps us avoid generating projected images explicitly. Numerical results based on the real face databases indicate that 2D-QLDA performs better than other 2D-LDA-like methods in color face recognition and is effective in image reconstruction.