Similar Patches Research Articles

Fast SPECT/CT planar bone imaging enabled by deep learning enhancement.

The application of deep learning methods in rapid bone scintigraphy is increasingly promising for minimizing the duration of SPECT examinations. Recent works showed several deep learning models based on simulated data for the synthesis of high-count bone scintigraphy images from low-count counterparts. Few studies have been conducted and validated on real clinical pairs due to the misalignment inherent in multiple scan procedures. To generate high quality whole-body bone images from 2× and 3× fast scans using deep learning based enhancement method. Seventy-six cases who underwent whole-body bone scans were enrolled in this prospective study. All patients went through a standard scan at a speed of 20cm/min, which followed by fast scans consisting of 2× and 3× accelerations at speeds of 40 and 60cm/min. A content-attention image restoration approach based on Residual-in-Residual Dense Block (RRDB)is introduced to effectively recover high-quality images from fast scans with fine-details and less noise. Our approach is robust with misalignment introduced from patient's metabolism, and shows valid count-level consistency. Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID) are employed in evaluating the similarity to the standard bone images. To further prove our method practical in clinical settings, image quality of the anonymous images was evaluated by two experienced nuclear physicians on a 5-point Likert scale (5= excellent). The proposed method reaches the state-of-the-art performance on FID and LPIPS with 0.583 and 0.176 for 2× fast scans and 0.583 and 0.185 for 3× fast scans. Clinic evaluation further demonstrated the restored images had a significant improvement compared to fast scan in image quality, technetium 99m-methyl diphosphonate (Tc-99m MDP) distribution, artifacts, and diagnostic confidence. Our method was validated for accelerating whole-body bone scans by introducing real clinical data. Confirmed by nuclear medicine physicians, the proposed method can effectively enhance image diagnostic value, demonstrating potential for efficient high-quality fast bone imaging in practical settings.

Geo-guided deep learning for spatial downscaling of solute transport in heterogeneous porous media

Resolving solute transport in heterogeneous porous media is a complex task, because of the sparse experimental data and the high computational cost of numerical simulations. This work proposes a unique two-stage deep learning architecture comprising a dual-branch autoencoder and a geo-guided super-resolution generative adversarial network (Gg-SRGAN) to address this dual challenge. The dual-branch autoencoder addresses the issue of sparsity by constructing a continuous, but coarse representation of concentration and pressure profiles from a sparse, discontinuous profile with up to 85% missing data points. The Gg-SRGAN is then employed to generate a finer representation of field variables from the outputs generated by the dual-branch autoencoder (i.e., downscaling). We train and test our framework using six solute transport cases with varying levels of heterogeneity and compare the results with standalone methods, namely the vanilla autoencoder and vanilla SRGAN, in addition to ground truth profiles generated by the finite element method (FEM). The comparisons are performed based on several statistical metrics, such as absolute point error (APE), mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). The first four cases are used for training, evaluation, and testing. The last two cases are utilized for blind testing to determine the generalizability of the framework. Our results show that the dual-branch autoencoder outperforms the vanilla autoencoder, and the Gg-SRGAN outperforms the SRGAN during both the training and evaluation phases. Moreover, the proposed framework can successfully construct the fine representation of concentration profiles, compared to FEM, using the coarse representation of the pressure, concentration, and domain permeability fields. When tested using the two blind test cases, the proposed dual-branch autoencoder and Gg-SRGAN exhibit superior performance compared to their counterparts in terms of all evaluation metrics.

Denoising Diffusion Probabilistic Model with Adversarial Learning for Remote Sensing Super-Resolution

Single Image Super-Resolution (SISR) for image enhancement enables the generation of high spatial resolution in Remote Sensing (RS) images without incurring additional costs. This approach offers a practical solution to obtain high-resolution RS images, addressing challenges posed by the expense of acquisition equipment and unpredictable weather conditions. To address the over-smoothing of the previous SISR models, the diffusion model has been incorporated into RS SISR to generate Super-Resolution (SR) images with enhanced textural details. In this paper, we propose a Diffusion model with Adversarial Learning Strategy (DiffALS) to refine the generative capability of the diffusion model. DiffALS integrates an additional Noise Discriminator (ND) into the training process, employing an adversarial learning strategy on the data distribution learning. This ND guides noise prediction by considering the general correspondence between the noisy image in each step, thereby enhancing the diversity of generated data and the detailed texture prediction of the diffusion model. Furthermore, considering that the diffusion model may exhibit suboptimal performance on traditional pixel-level metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), we showcase the effectiveness of DiffALS through downstream semantic segmentation applications. Extensive experiments demonstrate that the proposed model achieves remarkable accuracy and notable visual enhancements. Compared to other state-of-the-art methods, our model establishes an improvement of 189 for Fréchet Inception Distance (FID) and 0.002 for Learned Perceptual Image Patch Similarity (LPIPS) in a SR dataset, namely Alsat, and achieves improvements of 0.4%, 0.3%, and 0.2% for F1 score, MIoU, and Accuracy, respectively, in a segmentation dataset, namely Vaihingen.

Tensor robust PCA with nonconvex and nonlocal regularization

Tensor robust principal component analysis (TRPCA) is a classical way for low-rank tensor recovery, which minimizes the convex surrogate of tensor rank by shrinking each tensor singular value equally. However, for real-world visual data, large singular values represent more significant information than small singular values. In this paper, we propose a nonconvex TRPCA (N-TRPCA) model based on the tensor adjustable logarithmic norm. Unlike TRPCA, our N-TRPCA can adaptively shrink small singular values more and shrink large singular values less. In addition, TRPCA assumes that the whole data tensor is of low rank. This assumption is hardly satisfied in practice for natural visual data, restricting the capability of TRPCA to recover the edges and texture details from noisy images and videos. To this end, we integrate nonlocal self-similarity into N-TRPCA, and further develop a nonconvex and nonlocal TRPCA (NN-TRPCA) model. Specifically, similar nonlocal patches are grouped as a tensor and then each group tensor is recovered by our N-TRPCA. Since the patches in one group are highly correlated, all group tensors have strong low-rank property, leading to an improvement of recovery performance. Experimental results demonstrate that the proposed NN-TRPCA outperforms existing TRPCA methods in visual data recovery. The demo code is available at https://github.com/qguo2010/NN-TRPCA.

Gramformer: Learning Crowd Counting via Graph-Modulated Transformer

Transformer has been popular in recent crowd counting work since it breaks the limited receptive field of traditional CNNs. However, since crowd images always contain a large number of similar patches, the self-attention mechanism in Transformer tends to find a homogenized solution where the attention maps of almost all patches are identical. In this paper, we address this problem by proposing Gramformer: a graph-modulated transformer to enhance the network by adjusting the attention and input node features respectively on the basis of two different types of graphs. Firstly, an attention graph is proposed to diverse attention maps to attend to complementary information. The graph is building upon the dissimilarities between patches, modulating the attention in an anti-similarity fashion. Secondly, a feature-based centrality encoding is proposed to discover the centrality positions or importance of nodes. We encode them with a proposed centrality indices scheme to modulate the node features and similarity relationships. Extensive experiments on four challenging crowd counting datasets have validated the competitiveness of the proposed method. Code is available at https://github.com/LoraLinH/Gramformer.

Abstract 892: Pan-cancer tumor microenvironment profiling with multiplexed immunofluorescence microscopy and self-supervised learning

Abstract Multiplexed immunofluorescence (mIF) microscopy reveals the spatial architecture of cancer tissue and its microenvironment that is not being fully explored with existing analysis methods. Most analysis approaches for mIF microscopy data focus on single-cell or region classification and rely on supervised machine learning. In this work, we present a self-supervised spatial profiling method for mIF images that takes local and global associations into account, and apply this method to profile cancer-associated fibroblasts (CAFs) in a pan-cancer dataset. We studied a 2-stage self-supervised training scheme to learn the representations of mIF tissue microarray (TMA) images in local and global scales (cellular and long-range associations). During the 1st self-supervised training stage, a Vision Transformer learns the local-scale representations by using small patches from TMA images. Then, the local representations are used as input to the 2nd stage where a similar self-supervised learning strategy is used to learn global patterns in the TMA images. We applied the method to profile multiple cohorts from three different solid tumors: prostate, renal, and lung cancer. In total, these cohorts include more than 5,000 TMA cores from over 1,750 patients extracted from the tumor center, tumor edge, and adjacent benign areas. The samples were stained with a CAF panel including FAP, aSMA, PDGFRB, pSTAT3/PDGFRA, nuclear and epithelial markers, and imaged with cyclic mIF microscopy. Samples were studied at the patch-level and TMA core-level. Small patches enable further analysis of associations in the local environment, whereas the core-level enables associations with patient clinical information. Clustering of patch-level (1st stage) and core-level (2nd stage) representations showed independently that self-supervised learning is capable of learning the representations of the mIF images. We were able to identify regions and cases with high pTNM staging from the prostate cancer samples and similar histological subtypes from renal cancer samples. We further validated the clustering using k-NN classification that showed high classification accuracy in all cohorts. Moreover, we developed a stopping criteria for the final model selection that balances between the similarities of samples and patches inside of samples to prevent overfitting. Our study shows that self-supervised learning enables unbiased discoveries from large-scale mIF microscopy imaging datasets. The developed method uncovers associations between imaging data and clinical information and highlights directly the patterns that are most meaningful for these associations. Citation Format: Gantugs Atarsaikhan, Isabel Mogollon, Katja Välimäki, Tuomas Mirtti, Teijo Pellinen, Lassi Paavolainen. Pan-cancer tumor microenvironment profiling with multiplexed immunofluorescence microscopy and self-supervised learning [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 892.

ALNet: An adaptive channel attention network with local discrepancy perception for accurate indoor visual localization

Visual localization, a fundamental component of several computer vision tasks, has been predominantly realized by scene coordinate regression (SCoRe) techniques. These methods leverage neural networks for scene coordinates prediction, followed by a PnP algorithm to recover the 6-DOF camera pose. However, similar image patches are prevalent in indoor scenes, which results in the extraction of comparable features for the regression of different scene coordinates. As a result, the localization accuracy is severely declined. In this work, we develop ALNet, a novel SCoRe method that incorporates a local discrepancy perception module (LDPM) and an adaptive channel attention module (ACAM) to address this challenge. For LDPM, our key insight lies in that scene attributes around different similar image patches are inconsistent. Technically, for each image patch, LDPM identifies a certain number of the most dissimilar patches around it and computes difference vectors to enrich its own features, thereby enabling the differentiation of similar image patches. Considering geometric attributes are beneficial for distinguishing similar patches while semantic context is conducive to encoding regression issues, integrating multi-level features is an effective approach to elevate the localization accuracy. Therefore, ACAM concatenates multi-level features together and leverages both average pooling and max pooling to generate reliable channel-wise weighting coefficient, thereby modeling the correlation among channels to integrate multi-level features effectively. Comprehensive experiments are conducted on mainstream indoor localization benchmarks and an actual environment, showing that ALNet achieves impressive performance. Source code and the experimental results video are available at https://github.com/DAMMONGAO/alnet.

A global patch similarity-based graph for unsupervised SAR image change detection

ABSTRACT This letter presents a novel method for synthetic aperture radar (SAR) image change detection using the global patch similarity-based graph (GPSG). First, the SAR image is divided into a number of square patches, which are then vectorized and stacked to form a global patch matrix (GPM). The GPSG is constructed by connecting the global similar patches that are selected through traversing the GPM. With the support of GPSG, the second-order random walk matrix is designed to widely aggregate the attributes of patches within the two-hop global neighbourhood. The difference image with good separability can be generated by comparing bitemporal patches and their aggregated attributes. Finally, the Otsu thresholding is adopted to obtain the change map showing changed and unchanged portions. Experiments conducted on three real SAR datasets demonstrate the superiority of the proposed GPSG method in terms of both robustness and accuracy for change detection.

Dual contrastive learning for synthesizing unpaired fundus fluorescein angiography from retinal fundus images.

Fundus fluorescein angiography (FFA) is an imaging method used to assess retinal vascular structures by injecting exogenous dye. FFA images provide complementary information to that provided by the widely used color fundus (CF) images. However, the injected dye can cause some adverse side effects, and the method is not suitable for all patients. To meet the demand for high-quality FFA images in the diagnosis of retinopathy without side effects to patients, this study proposed an unsupervised image synthesis framework based on dual contrastive learning that can synthesize FFA images from unpaired CF images by inferring the effective mappings and avoid the shortcoming of generating blurred pathological features caused by cycle-consistency in conventional approaches. By adding class activation mapping (CAM) to the adaptive layer-instance normalization (AdaLIN) function, the generated images are made more realistic. Additionally, the use of CAM improves the discriminative ability of the model. Further, the Coordinate Attention Block was used for better feature extraction, and it was compared with other attention mechanisms to demonstrate its effectiveness. The synthesized images were quantified by the Fréchet inception distance (FID), kernel inception distance (KID), and learned perceptual image patch similarity (LPIPS). The extensive experimental results showed the proposed approach achieved the best results with the lowest overall average FID of 50.490, the lowest overall average KID of 0.01529, and the lowest overall average LPIPS of 0.245 among all the approaches. When compared with several popular image synthesis approaches, our approach not only produced higher-quality FFA images with clearer vascular structures and pathological features, but also achieved the best FID, KID, and LPIPS scores in the quantitative evaluation.

Fabric defect image generation method based on the dual-stage W-net generative adversarial network

Due to the intricate and diverse nature of textile defects, detecting them poses an exceptionally challenging task. In comparison with conventional defect detection methods, deep learning-based defect detection methods generally exhibit superior precision. However, utilizing deep learning for defect detection requires a substantial volume of training data, which can be particularly challenging to accumulate for textile flaws. To augment the fabric defect dataset and enhance fabric defect detection accuracy, we propose a fabric defect image generation method based on Pix2Pix generative adversarial network. This approach devises a novel dual-stage W-net generative adversarial network. By increasing the network depth, this model can effectively extract intricate textile image features, thereby enhancing its ability to expand information sharing capacity. The dual-stage W-net generative adversarial network allows generating desired defects on defect-free textile images. We conduct quality assessment of the generated fabric defect images resulting in peak signal-to-noise ratio and structural similarity values exceeding 30 and 0.930, respectively, and a learned perceptual image patch similarity value no greater than 0.085, demonstrating the effectiveness of fabric defect data augmentation. The effectiveness of dual-stage W-net generative adversarial network is established through multiple comparative experiments evaluating the generated images. By examining the detection performance before and after data augmentation, the results demonstrate that mean average precision improves by 6.13% and 14.57% on YOLO V5 and faster recurrent convolutional neural networks detection models, respectively.

Self-supervised few-shot learning for real-time traffic sign classification

Although supervised approaches for traffic sign classification have demonstrated excellent performance, they are limited to classifying several traffic signs defined in the training dataset. This prevents them from being applied to different domains, i.e., different countries. Herein, we propose a self-supervised approach for few-shot learning-based traffic sign classification. A center-awareness similarity network is designed for the traffic sign problem and trained using an optical flow dataset. Unlike existing supervised traffic sign classification methods, the proposed method does not depend on traffic sign categories defined by the training dataset. It applies to any traffic signs from different countries. We construct a Korean traffic sign classification (KTSC) dataset, including 6000 traffic sign samples and 59 categories. We evaluate the proposed method with baseline methods using the KTSC, German traffic sign, and Belgian traffic sign classification datasets. Experimental results show that the proposed method extends the ability of existing supervised methods and can classify any traffic sign, regardless of region/country dependence. Furthermore, the proposed approach significantly outperforms baseline methods for patch similarity. This approach provides a flexible and robust solution for classifying traffic signs, allowing for accurate categorization of every traffic sign, regardless of regional or national differences.

A novel arc‐shaped four‐port wideband (21.8–29.1 GHz) MIMO antenna with improved characteristics for 5G NR networks

SummaryA novel antenna system that utilizes multiple‐input and multiple‐output (MIMO) technology to suit the requirements of 5G applications is proposed in this article. The antenna has a wide operating bandwidth and high isolation in n257/n258/n261 5G bands. The suggested arc‐shaped MIMO antenna is composed of four similar patch units with modified ground planes arranged in a four‐port configuration and is made on a Rogers RT/duroid 5880 substrate, which measures 31.891 × 37.5885 × 0.508 mm3. The prototype of the suggested MIMO antenna is realized to confirm the obtained simulated results and validate the proposed design. The four‐port MIMO antenna features a broad frequency range of 7.3 GHz from 21.8 to 29.1 GHz, a peak gain of 5.76 dBi at 29 GHz, and a good isolation less than −18 dB. The proposed antenna's MIMO performance is evaluated by analyzing several diversity parameters. The findings of the evaluation demonstrate an envelope correlation coefficient (ECC) value of less than 0.001, a diversity gain (DG) value exceeding 9.995 dB, and an average mean effective gain (MEG) of −3 dB within the operational frequency band. The suggested MIMO antenna's compact size and impressive performance make it a suitable choice for intended n257/n258/n261 5G NR networks.

GSSTU: Generative Spatial Self-Attention Transformer Unit for Enhanced Video Prediction.

Future frame prediction is a challenging task in computer vision with practical applications in areas such as video generation, autonomous driving, and robotics. Traditional recurrent neural networks have limited effectiveness in capturing long-range dependencies between frames, and combining convolutional neural networks (CNNs) with recurrent networks has limitations in modeling complex dependencies. Generative adversarial networks have shown promising results, but they are computationally expensive and suffer from instability during training. In this article, we propose a novel approach for future frame prediction that combines the encoding capabilities of 3-D CNNs with the sequence modeling capabilities of Transformers. We also propose a spatial self-attention mechanism and a novel neighborhood pixel intensity loss to preserve structural information and local intensity, respectively. Our approach outperforms existing methods in terms of structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and learned perceptual image patch similarity (LPIPS) scores on five public datasets. More precisely, our model exhibited an average improvement of 4.64%, 18.5%, and 42% concerning SSIM, PSNR, and LPIPS for the second most proficient method correspondingly, across all datasets. The results demonstrate the effectiveness of our proposed method in generating high-quality predictions of future frames.

MP-NeRF: More refined deblurred neural radiance field for 3D reconstruction of blurred images

Neural Radiance Fields (NeRF) has gained prominence in the domain of 3D reconstruction. Despite its popularity, NeRF algorithm typically require clear, static images to function effectively, leading to reduced performance when dealing with real-world scenarios that present non-ideal conditions such as complex reflections, low dynamic range, dark scenes and blurriness resulting from camera motion or defocus photography. The resilience of NeRF to such blurred inputs has not been sufficiently examined, presenting a gap in current research, which also tends to neglect the role of 3D scene context in image deblurring. Addressing these challenges, we introduce Multi-branch Fusion Network and Prior-based Learnable Weights NeRF (MP-NeRF), a novel approach engineered for the precise reconstruction of scenes from blurred images. MP-NeRF innovatively applies distinct priors for various blur types, thus better interpreting the blur formation mechanism. It integrates a Multi-branch Fusion Network (MBFNet) with a Prior-based Learnable Weights (PLW), which collectively enhance the capture of complex scene details, including textural and pattern information. Our experimental findings validate that MP-NeRF considerably enhances the visual accuracy of reconstructions achieved through NeRF, yielding substantial improvements in Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) compared to state-of-the-art models. The source code for MP-NeRF is made available at the following URL: https://github.com/luckhui0505/MP-NeRF.

Hide and track: Towards blind video watermarking network in frequency domain

Blind video watermarking (BVW) is the process of embedding visually imperceptible messages into a cover video, which can be retrieved even in the presence of distortion without requiring any reference to the original video. Compared to image watermarking, BVW encounters two critical challenges: 1) ensuring the visually imperceptible of the embedded information across sequential frames; 2) enabling quick localization of the watermarked frames in the event of temporal and spatial attacks such as frame dropping and video compression. This paper presents a robust blind video watermarking network by exploring a block-based selection mechanism in the frequency domain. The network comprises an encoding part that hides the watermark in a suitable location within the video to ensure visual imperceptibility of the watermark, and a decoding part that tracks the watermark in the frequency domain to ensure its robustness even when the video is distorted. Moreover, a plug-and-play watermark detector is designed to discover watermarking locations. This detector is readily applicable and can be easily integrated into most frequency watermarking embedding methods. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods on imperceptibility and robustness significantly. Our framework achieves an average peak signal-to-noise ratio (PSNR) of 37.59 dB and a Learned Perceptual Image Patch Similarity (LPIPS) of 1.12×10−2. Furthermore, the extraction accuracy reaches 99.99% under a variety of temporal attacks which demonstrates the robustness of the proposed method.

Refining intra-patch connectivity measures in landscape fragmentation and connectivity indices

ContextMeasuring intra-patch connectivity, i.e. the connectivity within a habitat patch, is important to evaluate landscape fragmentation and connectivity. However, intra-patch connectivity is mainly measured with patch size, which can conceal diverse intra-patch connectivity patterns for similar patch size distributions.ObjectivesWe suggest a method to refine the intra-patch connectivity component of fragmentation and connectivity indices. This method allows for distinguishing different intra-patch connectivity patterns for similar patch size distributions.MethodsWe used normalized patch complexity indices to weight patch size in common fragmentation and connectivity indices. Patch complexity indices included two existing geometrical indices (SHAPE and FRAC), and a new index derived from spatial network analysis, the mean detour index (MDI). We analyzed the behaviours of adjusted fragmentation and connectivity indices theoretically and empirically on both artificial and real landscapes.ResultsWhile maintaining the mathematical properties of fragmentation and connectivity indices, our method could distinguish landscapes with identical patch size distributions but different spatial configurations. The mean detour index had a different response than geometrical indices. This result indicates that, at the patch level, topological complexity can exhibit different patterns from geometrical complexity.ConclusionsMeasuring intra-patch connectivity with patch size in fragmentation and connectivity indices cannot distinguish landscapes having similar patch sizes distribution but different spatial configurations. This paper introduces a method to distinguish such patterns relying on geometrical and topological indices and shows to which extent it can impact conservation planning.

SynFAGnet: A Fully Automated Generative Network for Realistic Fire Image Generation

This paper proposes a fully automated generative network (“SynFAGnet”) for automatically creating a realistic-looking synthetic fire image. SynFAGnet is used as a data augmentation technique to create diverse data for training models, thereby solving problems related to real data acquisition and data imbalances. SynFAGnet comprises two main parts: an object-scene placement net (OSPNet) and a local–global context-based generative adversarial network (LGC-GAN). The OSPNet identifies suitable positions and scales for fires corresponding to the background scene. The LGC-GAN enhances the realistic appearance of synthetic fire images created by a given fire object-background scene pair by assembling effects such as halos and reflections in the surrounding area in the background scene. A comparative analysis shows that SynFAGnet achieves better outcomes than previous studies for both the Fréchet inception distance and learned perceptual image patch similarity evaluation metrics (values of 17.232 and 0.077, respectively). In addition, SynFAGnet is verified as a practically applicable data augmentation technique for training datasets, as it improves the detection and instance segmentation performance.

Deep Learning-Based Technique for Remote Sensing Image Enhancement Using Multiscale Feature Fusion.

The present study proposes a novel deep-learning model for remote sensing image enhancement. It maintains image details while enhancing brightness in the feature extraction module. An improved hierarchical model named Global Spatial Attention Network (GSA-Net), based on U-Net for image enhancement, is proposed to improve the model's performance. To circumvent the issue of insufficient sample data, gamma correction is applied to create low-light images, which are then used as training examples. A loss function is constructed using the Structural Similarity (SSIM) and Peak Signal-to-Noise Ratio (PSNR) indices. The GSA-Net network and loss function are utilized to restore images obtained via low-light remote sensing. This proposed method was tested on the Northwestern Polytechnical University Very-High-Resolution 10 (NWPU VHR-10) dataset, and its overall superiority was demonstrated in comparison with other state-of-the-art algorithms using various objective assessment indicators, such as PSNR, SSIM, and Learned Perceptual Image Patch Similarity (LPIPS). Furthermore, in high-level visual tasks such as object detection, this novel method provides better remote sensing images with distinct details and higher contrast than the competing methods.

Image enhancement with intensity transformation on embedding space

AbstractIn recent times, an image enhancement approach, which learns the global transformation function using deep neural networks, has gained attention. However, many existing methods based on this approach have a limitation: their transformation functions are too simple to imitate complex colour transformations between low‐quality images and manually retouched high‐quality images. In order to address this limitation, a simple yet effective approach for image enhancement is proposed. The proposed algorithm based on the channel‐wise intensity transformation is designed. However, this transformation is applied to the learnt embedding space instead of specific colour spaces and then return enhanced features to colours. To this end, the authors define the continuous intensity transformation (CIT) to describe the mapping between input and output intensities on the embedding space. Then, the enhancement network is developed, which produces multi‐scale feature maps from input images, derives the set of transformation functions, and performs the CIT to obtain enhanced images. Extensive experiments on the MIT‐Adobe 5K dataset demonstrate that the authors’ approach improves the performance of conventional intensity transforms on colour space metrics. Specifically, the authors achieved a 3.8% improvement in peak signal‐to‐noise ratio, a 1.8% improvement in structual similarity index measure, and a 27.5% improvement in learned perceptual image patch similarity. Also, the authors’ algorithm outperforms state‐of‐the‐art alternatives on three image enhancement datasets: MIT‐Adobe 5K, Low‐Light, and Google HDR+.

Robust HDR reconstruction using 3D patch based on two-scale decomposition

In this paper, we propose an effective 3D patch-based match and fusion method by taking account of dynamic or multi-view scenes in a multi-exposure image sequence using two-scale decomposition. As opposed to most multi-exposure image fusion methods, the proposed method does not require a pre-alignment step to reduce ghosting artifacts. Considering that pixel values are affected by multi-view or multi-exposure scenes, we use a uniform matching approach to match and find similar patches in different exposure images and then fuse them at each scale. By searching all similar patches (instead of the optimal patch) in the searching window to form the 3D patch when facing limited image information, we can fully use the complementary information of the multi-exposure images to preserve information about moving objects and scene details. The experimental results show that the proposed method not only performs well on dynamic scenes but also consistently generates high-quality fused images in multi-view scenes.