Existing Super-resolution Methods Research Articles

Simultaneous single image super‐resolution and blind Gaussian denoising via slim ghost full‐frequency residual blocks

AbstractGiven that super‐resolution (SR) aims to recover lost information, and low‐resolution (LR) images in real‐world conditions might be corrupted with multiple degradations, considering basic bicubic down‐sampling as the sole degradation significantly limits the performance of most existing SR models. This paper presents a model for simultaneous super‐resolution and blind additive white Gaussian noise (AWGN) denoising with two components (netdeg and netSR) that is based on a generative adversarial network (GAN) to achieve detailed results. netdeg, featuring residual and innovative cost‐effective ghost residual blocks with a frequency separation module for obtaining long‐range information, blindly restores a clean version of the LR image. netSR leverages slim ghost full‐frequency residual blocks to process low‐frequency (LF) and high‐frequency (HF) information via static large convolutions and pixel‐wise highlighted input‐adaptive dynamic convolutions, respectively. To address the susceptibility of dynamic layers to noise and preserve feature diversity while reducing model’s costs, static and dynamic layer features are combined and highlighted. Diverse and non‐redundant features are then processed using ghost‐style blocks. The proposed model achieves comparable SR results in bicubic down‐sampling scenarios, outperform existing SR methods in the complex task of concurrent SR and AWGN denoising, and demonstrate robustness in handling images corrupted with varying levels of AWGN.

A novel GAN-based three-axis mutually supervised super-resolution reconstruction method for rectal cancer MR image

Background and objectiveThis study aims to enhance the resolution in the axial direction of rectal cancer magnetic resonance (MR) imaging scans to improve the accuracy of visual interpretation and quantitative analysis. MR imaging is a critical technique for the diagnosis and treatment planning of rectal cancer. However, obtaining high-resolution MR images is both time-consuming and costly. As a result, many hospitals store only a limited number of slices, often leading to low-resolution MR images, particularly in the axial plane. Given the importance of image resolution in accurate assessment, these low-resolution images frequently lack the necessary detail, posing substantial challenges for both human experts and computer-aided diagnostic systems. Image super-resolution (SR), a technique developed to enhance image resolution, was originally applied to natural images. Its success has since led to its application in various other tasks, especially in the reconstruction of low-resolution MR images. However, most existing SR methods fail to account for all anatomical planes during reconstruction, leading to unsatisfactory results when applied to rectal cancer MR images. MethodsIn this paper, we propose a GAN-based three-axis mutually supervised super-resolution reconstruction method tailored for low-resolution rectal cancer MR images. Our approach involves performing one-dimensional (1D) intra-slice SR reconstruction along the axial direction for both the sagittal and coronal planes, coupled with inter-slice SR reconstruction based on slice synthesis in the axial direction. To further enhance the accuracy of super-resolution reconstruction, we introduce a consistency supervision mechanism across the reconstruction results of different axes, promoting mutual learning between each axis. A key innovation of our method is the introduction of Depth-GAN for synthesize intermediate slices in the axial plane, incorporating depth information and leveraging Generative Adversarial Networks (GANs) for this purpose. Additionally, we enhance the accuracy of intermediate slice synthesis by employing a combination of supervised and unsupervised interactive learning techniques throughout the process. ResultsWe conducted extensive ablation studies and comparative analyses with existing methods to validate the effectiveness of our approach. On the test set from Shanxi Cancer Hospital, our method achieved a Peak Signal-to-Noise Ratio (PSNR) of 34.62 and a Structural Similarity Index (SSIM) of 96.34 %. These promising results demonstrate the superiority of our method.

MTLSC-Diff: Multitask learning with diffusion models for hyperspectral image super-resolution and classification

Hyperspectral image (HSIs) super-resolution (SR) can improve the spatial resolution of images for subsequent application tasks. In recent years, SR methods based on deep learning have gained widespread attention. However, most of the existing SR methods do not take into account the needs of specific application tasks when designing the network structure. These methods may not be able to efficiently generate high-quality images that satisfy the specific application tasks, leading to degradation of the performance of subsequent application tasks. To solve this problem, we propose a multi-task learning architecture based on the diffusion model, namely MTLSC-Diff. MTLSC-Diff combines the SR network and the classification network in a multi-task learning manner on the basis of the diffusion model. MTLSC-Diff achieves mutual guidance of the two tasks by iterating the image super-resolution and classification tasks, thus gradually reconstructing high-quality images and improving classification accuracy. The guided operations for each time step are performed by the specially designed Mutual-Guidance SR-Classification Synergy Module (M-GSCS). M-GSCS refines the multi-scale image obtained at the previous time step and uses the predicted high spatial resolution image for classification. Meanwhile, a class-guided SR dynamic refinement strategy (C-GSR) is proposed in M-GSCS, which uses multi-scale classification results to guide target scale images to learn new knowledge to further reconstruct high-quality images. Experimental results on relevant datasets show that our method significantly improves the super-resolution performance as well as the classification performance.

IRLF-SRNet: A super-resolution network based on local–global feature enhance-refine for camera-array based infrared light field images

Super resolution (SR) is essential for the camera-array based infrared light field (IRLF) images. However, the existing SR methods are mainly designed for visible LF images and are unsuitable for IRLF. This is because the sub-aperture images (SAIs) in IRLF hold blurred edges and pixel-level shifts. Artifacts and unbalanced SR performance may exist. In this paper, we propose a novel SR network, named IRLF-SRnet, to mitigate the above problems. It includes two designs: (1) local–global feature enhance-refine strategy, and (2) multi-branch cross-view correlation modeling block. The former uses global features from all SAIs to enhance the local feature from one SAI, which enhances the edges in each SAI implicitly. Then, it refines the global feature with the enhanced local feature, which further refines the edges. The latter first divides and sends the features of SAIs into multiple branches, and then models the cross-view correlation among features in each branch by bidirectional convolutional long-short term memory (BiConvLSTM), so that be aware of the pixel-level shifts. Qualitative and quantitative results show that our method outperforms state-of-the-art methods, with more balanced SR performance and fewer artifacts.

Heat Transfer-Inspired Network for Image Super-Resolution Reconstruction.

Image super-resolution (SR) is a critical image preprocessing task for many applications. How to recover features as accurately as possible is the focus of SR algorithms. Most existing SR methods tend to guide the image reconstruction process with gradient maps, frequency perception modules, etc. and improve the quality of recovered images from the perspective of enhancing edges, but rarely optimize the neural network structure from the system level. In this article, we conduct an in- depth exploration for the inner nature of the SR network structure. In light of the consistency between thermal particles in the thermal field and pixels in the image domain, we propose a novel heat-transfer-inspired network (HTI-Net) for image SR reconstruction based on the theoretical basis of heat transfer. With the finite difference theory, we use a second-order mixed-difference equation to redesign the residual network (ResNet), which can fully integrate multiple information to achieve better feature reuse. In addition, according to the thermal conduction differential equation (TCDE) in the thermal field, the pixel value flow equation (PVFE) in the image domain is derived to mine deep potential feature information. The experimental results on multiple standard databases demonstrate that the proposed HTI-Net has superior edge detail reconstruction effect and parameter performance compared with the existing SR methods. The experimental results on the microscope chip image (MCI) database consisting of realistic low-resolution (LR) and high-resolution (HR) images show that the proposed HTI-Net for image SR reconstruction can improve the effectiveness of the hardware Trojan detection system.

KernelFlexSR: a self-adaptive super-resolution algorithm with multi-path convolution and residual network for dynamic kernel enhancement

Machine learning-based image super-resolution (SR) has garnered increasing research interest in recent years. However, there are two issues that have not been adequately addressed. The first issue is that existing SR methods often overlook the importance of improving the quality of the training dataset, which is a crucial factor in determining SR performance, regardless of the training method employed. The second issue is that while some studies report high numerical metrics, the visual results remain unsatisfactory. To address the first problem, we propose a new image down-sampling method to obtain higher-quality training datasets. To tackle the second problem, we present a new image super-resolution model based on a large-size convolution kernel and a multi-path algorithm. Specifically, we use an adaptive large-size convolutional kernel to extract features from the image based on the size of the input image, and a residual network to generate a deeper model to retain more details of the original input image. Experimental results demonstrate that the proposed multilayer downsampling method (MDM) can significantly improve the visual quality compared to traditional downsampling methods. Moreover, our proposed method achieves the best peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values compared to several typical SR algorithms. Furthermore, subjective evaluation by human observers reveals that our method retains more details of the original image and produces smoother high-resolution images. Our proposed method effectively addresses the two aforementioned issues, which leads to improved SR performance in terms of both quantitative and qualitative measures.

Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective.

High perceptual quality and low distortion degree are two important goals in image restoration tasks such as super-resolution (SR). Most of the existing SR methods aim to achieve these goals by minimizing the corresponding yet conflicting losses, such as the l1 loss and the adversarial loss. Unfortunately, the commonly used gradient-based optimizers, such as Adam, are hard to balance these objectives due to the opposite gradient decent directions of the contradictory losses. In this paper, we formulate the perception-distortion trade-off in SR as a multi-objective optimization problem and develop a new optimizer by integrating the gradient-free evolutionary algorithm (EA) with gradient-based Adam, where EA and Adam focus on the divergence and convergence of the optimization directions respectively. As a result, a population of optimal models with different perception-distortion preferences is obtained. We then design a fusion network to merge these models into a single stronger one for an effective perception-distortion trade-off. Experiments demonstrate that with the same backbone network, the perception-distortion balanced SR model trained by our method can achieve better perceptual quality than its competitors while attaining better reconstruction fidelity. Codes and models can be found at https://github.com/csslc/EA-Adam.

Text-Enhanced Scene Image Super-Resolution via Stroke Mask and Orthogonal Attention

Low-resolution text images are very commonplace in real life and their information is hard to be extracted by using existing text recognition methods only. Although this problem can be solved by introducing super-resolution (SR) techniques, most existing SR methods fail to process stroke regions and background regions of input text images distinctively. In this paper, we propose a text-specific super-resolution network named Text Enhanced Attention Network (TEAN) to solve this problem. First of all, we compensate for disadvantages of traditional thresholding mask operation proposed in Text Super-Resolution Network (TSRN) by utilizing deep-learning based semantic segmentation method to get correct masks as prior semantic information and propose a Text-Segmented-Contextual-Attention (TSCA) branch on the basis of them. Besides, we design an Orthogonal Contextual Attention Module (OCAM) working with TSCA to implicitly enhance stroke regions of LR images. Secondly, to effectively fuse shallow features and deep features of SR model, we propose a convolutional structure named Weight Balanced Fusion Module (WBFM) to improve traditional feature fusion methods of SR network. Finally, extensive experiments on TextZoom dataset demonstrate that the proposed network can improve the recognition accuracy of text images on existing text recognition models. Using TEAN to process low-resolution text images improves the recognition accuracy by 25.4% on CRNN, by 17.4% on ASTER, by 17.3% on MORAN, by 20.7% on NRTR, by 17.3% on SAR and by 15.9% on MASTER compared with directly recognizing them, which attains competitive performances against state-of-the-art methods. Furthermore, cross-dataset experiments on IC15_2077 demonstrate that TEAN is helpful for scene text recognition task, especially for low-resolution images even with the cross-domain issue.

Arbitrary-scale Super-resolution via Deep Learning: A Comprehensive Survey

Super-resolution (SR) is an essential class of low-level vision tasks, which aims to improve the resolution of images or videos in computer vision. In recent years, significant progress has been made in image and video super-resolution techniques based on deep learning. Nevertheless, most of the methods only consider SR with a few integer scale factors, which limits the application of the SR techniques to real-world problems. Recently, the methods to achieve arbitrary-scale super-resolution via a single model have attracted much attention. However, there is no work to thoroughly analyze the arbitrary-scale methods based on deep learning. In this work, we present a comprehensive and systematic review of 45 existing deep learning-based methods for arbitrary-scale image and video SR. We first classify the existing SR methods according to the resolved scale factors. Furthermore, we propose an in-depth taxonomy for state-of-the-art methods based on the core problem of how to achieve arbitrary-scale super-resolution, i.e., how to perform arbitrary-scale upsampling. Moreover, the performance of existing arbitrary-scale SR methods is compared, and their advantages and limitations are analyzed. We also provide some guidance for the selection of these methods in different real-world applications. Finally, we briefly discuss the future directions of arbitrary-scale super-resolution, which shows some inspirations for the progress of subsequent works on arbitrary-scale image and video super-resolution tasks. The repository of this work is available at https://github.com/Weepingchestnut/Arbitrary-Scale-SR.

A lightweight distillation CNN-transformer architecture for remote sensing image super-resolution

ABSTRACT Remote sensing images exhibit rich texture features and strong autocorrelation. Although the super-resolution (SR) method of remote sensing images based on convolutional neural networks (CNN) can capture rich local information, the limited perceptual field prevents it from establishing long-distance dependence on global information, leading to the low accuracy of remote sensing image reconstruction. Furthermore, it is difficult for existing SR methods to be deployed in mobile devices due to their large network parameters and high computational demand. In this study, we propose a lightweight distillation CNN-Transformer SR architecture, named DCTA, for remote sensing SR, addressing the aforementioned issues. Specifically, the proposed DCTA first extracts the coarse features through the coarse feature extraction layer and then learns the deep features of remote sensing at different scales by fusing the feature distillation extraction module of CNN and Transformer. In addition, we introduce the feature fusion module at the end of the feature distillation extraction module to control the information propagation, aiming to select the informative components for better feature fusion. The extracted low-resolution (LR) feature maps are reorganized through the up-sampling module to obtain high-resolution (HR) feature maps with high accuracy to generate high-quality HR remote sensing images. The experiments comparing different methods demonstrate that the proposed approach performs well on multiple datasets, including NWPU-RESISC45, Draper, and UC Merced. This is achieved by balancing reconstruction performance and network complexity, resulting in both competitive subjective and objective results.

A super-resolution network using channel attention retention for pathology images.

Image super-resolution (SR) significantly improves the quality of low-resolution images, and is widely used for image reconstruction in various fields. Although the existing SR methods have achieved distinguished results in objective metrics, most methods focus on real-world images and employ large and complex network structures, which are inefficient for medical diagnosis scenarios. To address the aforementioned issues, the distinction between pathology images and real-world images was investigated, and an SR Network with a wider and deeper attention module called Channel Attention Retention is proposed to obtain SR images with enhanced high-frequency features. This network captures contextual information within and across blocks via residual skips and balances the performance and efficiency by controlling the number of blocks. Meanwhile, a new linear loss was introduced to optimize the network. To evaluate the work and compare multiple SR works, a benchmark dataset bcSR was created, which forces a model training on wider and more critical regions. The results show that the proposed model outperforms state-of-the-art methods in both performance and efficiency, and the newly created dataset significantly improves the reconstruction quality of all compared models. Moreover, image classification experiments demonstrate that the suggested network improves the performance of downstream tasks in medical diagnosis scenarios. The proposed network and dataset provide effective priors for the SR task of pathology images, which significantly improves the diagnosis of relevant medical staff. The source code and the dataset are available on https://github.com/MoyangSensei/CARN-Pytorch.

GDSSR: Toward Real-World Ultra-High-Resolution Image Super-Resolution

Although single image super-resolution (SR) has achieved great success, super-resolving the real-world Ultra-High-Resolution (UHR) image remains a challenging issue. Confronted with UHR images, most existing SR methods resort to patch-splitting so that the interconnections among the cropped patches are not attracted reasonable attention during the training and inference procedure. Rather than considering global image degradation levels and types, previous methods only focus on local degradation and unavoidably lead to inter-patch inconsistency, like blocking artifacts in the UHR image. To address this issue, we propose a real-world super-resolution framework to integrate the restoration of different patches through a Global Degradation Supervision Super-Resolution (GDSSR) method. Specifically, a lightweight Global Degradation Extractor is used for extracting global degradation features, which can facilitate restoring better local patches independently and enforce inter-patch consistency. Additionally, a joint training method of local and global patches is proposed to exercise global supervision during the training process, which enhances the degradation estimation and restores more natural results. Experiments show that our GDSSR method achieves superior restoration performance on real-world and UHR image SR datasets.

Lightweight Stepless Super-Resolution of Remote Sensing Images via Saliency-Aware Dynamic Routing Strategy

Deep learning-based algorithms have greatly improved the performance of remote sensing image (RSI) super-resolution (SR). However, increasing network depth and parameters cause a huge burden of computing and storage. Directly reducing the depth or width of existing models results in a large performance drop. We observe that the SR difficulty of different regions in an RSI varies greatly, and existing methods use the same deep network to process all regions in an image, resulting in a waste of computing resources. In addition, existing SR methods generally predefine integer scale factors and cannot perform stepless SR, i.e., a single model can deal with any potential scale factor. Retraining the model on each scale factor wastes considerable computing resources and model storage space. To address the above problems, we propose a saliency-aware dynamic routing network (SalDRN) for lightweight and stepless SR of RSIs. First, we introduce visual saliency as an indicator of region-level SR difficulty and integrate a lightweight saliency detector into the SalDRN to capture pixel-level visual characteristics. Then, we devise a saliency-aware dynamic routing strategy that employs path selection switches to adaptively select feature extraction paths of appropriate depth according to the SR difficulty of subimage patches. Finally, we propose a novel lightweight stepless upsampling module whose core is an implicit feature function for realizing mapping from low-resolution feature space to high-resolution feature space. Comprehensive experiments verify that the SalDRN can achieve a good tradeoff between performance and complexity. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hanlinwu/SalDRN</uri> .

Toward Real-World Super-Resolution via Adaptive Downsampling Models.

Most image super-resolution (SR) methods are developed on synthetic low-resolution (LR) and high-resolution (HR) image pairs that are constructed by a predetermined operation, e.g., bicubic downsampling. As existing methods typically learn an inverse mapping of the specific function, they produce blurry results when applied to real-world images whose exact formulation is different and unknown. Therefore, several methods attempt to synthesize much more diverse LR samples or learn a realistic downsampling model. However, due to restrictive assumptions on the downsampling process, they are still biased and less generalizable. This study proposes a novel method to simulate an unknown downsampling process without imposing restrictive prior knowledge. We propose a generalizable low-frequency loss (LFL) in the adversarial training framework to imitate the distribution of target LR images without using any paired examples. Furthermore, we design an adaptive data loss (ADL) for the downsampler, which can be adaptively learned and updated from the data during the training loops. Extensive experiments validate that our downsampling model can facilitate existing SR methods to perform more accurate reconstructions on various synthetic and real-world examples than the conventional approaches.

Gated Multi-Attention Feedback Network for Medical Image Super-Resolution

Medical imaging technology plays a crucial role in the diagnosis and treatment of diseases. However, the captured medical images are often in a low resolution (LR) due to the limited imaging condition. Super-resolution (SR) technology is a feasible solution to enhance the resolution of a medical image without increasing the hardware cost. However, the existing SR methods often ignore high-frequency details, which results in blurred edges and an unsatisfying visual perception. In this paper, a gated multi-attention feedback network (GAMA) is proposed for medical image SR. Specifically, a gated multi-feedback network is employed as the backbone to extract hierarchical features. Meanwhile, a layer attention feature extraction (LAFE) module is introduced to refine the feature map. In addition, a channel-space attention reconstruction (CSAR) module is built to enhance the representational ability of the semantic feature map. Furthermore, a gradient variance loss is tailored as the regularization in guiding the model learning to regularize the model in generating a faithful high-resolution image with rich textures and sharp edges. The experiments verify the effectiveness of the proposed GAMA compared with the state-of-the-art approaches.

Terrain feature-aware deep learning network for digital elevation model superresolution

Neural networks (NNs) have demonstrated the potential to recover finer textural details from lower-resolution images by superresolution (SR). Given similar grid-based data structures, some researchers have transferred image SR methods to digital elevation models (DEMs). These efforts have yielded better results than traditional spatial interpolation methods. However, terrain data present inherently different characteristics and practical meanings compared with natural images. This makes it unsuitable for existing SR methods on perceptually visual features of images to be directly adopted for extracting terrain features. In this paper, we argue that the problem lies in the lack of explicit terrain feature modeling and thus propose a terrain feature-aware superresolution model (TfaSR) to guide DEM SR towards the extraction and optimization of terrain features. Specifically, a deep residual module and a deformable convolution module are integrated to extract deep and adaptive terrain features, respectively. In addition, explicit terrain feature-aware optimization is proposed to focus on local terrain feature refinement during training. Extensive experiments show that TfaSR achieves state-of-the-art performance in terrain feature preservation during DEM SR. Specifically, compared with the traditional bicubic interpolation method and existing neural network methods (SRGAN, SRResNet, and SRCNN), the RMSE of our results is improved by 1.1% to 23.8% when recovering the DEM from 120 m to 30 m, by 4.9% to 22.7% when recovering the DEM from 60 m to 30 m, and by 7.8% to 53.7% when recovering the DEM from 30 m to 10 m. The source code that has been developed is shared on Figshare (https://doi.org/10.6084/m9.figshare.19597201).

Super-Resolution Network for Remote Sensing Images via Preclassification and Deep–Shallow Features Fusion

A novel super-resolution (SR) method is proposed in this paper to reconstruct high-resolution (HR) remote sensing images. Different scenes of remote sensing images have great disparities in structural complexity. Nevertheless, most existing SR methods ignore these differences, which increases the difficulty to train an SR network. Therefore, we first propose a preclassification strategy and adopt different SR networks to process the remote sensing images with different structural complexity. Furthermore, the main edge of low-resolution images are extracted as the shallow features and fused with the deep features extracted by the network to solve the blurry edge problem in remote sensing images. Finally, an edge loss function and a cycle consistent loss function are added to guide the training process to keep the edge details and main structures in a reconstructed image. A large number of comparative experiments on two typical remote sensing images datasets (WHURS and AID) illustrate that our approach achieves better performance than state-of-the-art approaches in both quantitative indicators and visual qualities. The peak signal-to-noise ratio (PSNR) value and the structural similarity (SSIM) value using the proposed method are improved by 0.5353 dB and 0.0262, respectively, over the average values of five typical deep learning methods on the ×4 AID testing set. Our method obtains satisfactory reconstructed images for the subsequent applications based on HR remote sensing images.

Visible-Assisted Infrared Image Super-Resolution Based on Spatial Attention Residual Network

Infrared images have a wide range of applications in military and civilian fields, including night vision, surveillance, and robotics. However, the most commonly used infrared images are low-resolution (LR), which lack texture details, and existing infrared image super-resolution (SR) algorithms are limited by the lack of spatial information utilization. To solve the above problems, a spatial attention residual network (SAResNet) is proposed. Specifically, the network consists of spatial attention residual block (SARB) with several short skip connections (SSCs). The SARB contains 20 spatial attention blocks (SAB), which adaptively adjusts weights of different spatial regions by considering interdependence between spatial features. Meanwhile, the visible images are considered as complementary sources; thus, a visible-assisted training strategy is designed for the infrared SR process, promoting details preservation. Furthermore, the spatial attention (SA) mechanism is utilized, which focuses more on spatial characteristics of the image and refines the main objects and target boundaries. Experimentally, the proposed method, SAResNet, is compared with existing SR methods, and the effectiveness of the proposed method is demonstrated based on both quantity and quality analyses.

NonRegSRNet: A Nonrigid Registration Hyperspectral Super-Resolution Network

Due to the limitations of imaging systems, satellite hyperspectral imagery (HSI), which yields rich spectral information in many channels, often suffers from poor spatial resolution. HSI super-resolution (SR) refers to the fusion of high spatial resolution multispectral imagery (MSI) and low spatial resolution HSI to generate HSI that has both a high spatial and high spectral resolution. However, most existing SR methods assume that the two original images used are perfectly registered: in reality, nonrigid deformation areas can exist locally in the two images even if prior registration of the control points has been carried out. To address this problem, we propose a novel unsupervised spectral unmixing and image deformation correction network&#x2014;NonRegSRNet&#x2014;with multimodal and multitask learning that can be used for the joint registration of HSI and MSI and to produce SR imagery. More specifically, NonRegSRNet integrates the dense registration and SR tasks into a unified model that includes a triplet convolutional neural network. This allows these two tasks to complement each other so that better registration and SR results can be achieved. Furthermore, because the point spread function (PSF) and spectral response function (SRF) are often unavailable, two special convolutional layers are designed to adaptively learn the parameters of the PSF and SRF, which makes the proposed model more adaptable. Experimental results demonstrate that the proposed method has the ability to produce highly accurate and stable reconstructed images under complex nonrigid deformation conditions. (Code available at <uri>https://github.com/saber-zero/NonRegSRNet</uri>)

Toward Real-World Super-Resolution Technique for Fringe Projection Profilometry

Deep learning technique has exhibited promising performance in achieving high-resolution (HR) image from their low-resolution (LR) image in super-resolution (SR) field. However, most of the existing SR methods have two underlying problems. Firstly, degraded datasets (i.e., bicubic downsampling) are usually used to train and evaluate network model, which maybe lead to less effective in practical scenarios. Secondly, 2D-to-3D SR technique is lacking. In this paper, a real-world 2D-to-3D technique is developed to realize SR 3D shape from 2D fringe images in fringe projection profilometry (FPP). A FPP system consisting of one projector and a dual-camera is applied to obtain the real-world dataset where paired LR-HR images on same scene are captured. The 3D geometrical constraints solved from FPP system are employed to align the image pairs by pixel-to-pixel mapping so that more accurate dataset can be obtained. In addition, a flexible multiple-to-two network structure is introduced to achieve SR 3D point cloud from multiple phase-shifting patterns. Experiments demonstrate the comparing between traditional degraded training and our training.