High-quality Reference Image Research Articles

A Collaborative Self-supervised Domain Adaptation for Low-Quality Medical Image Enhancement.

Medical image analysis techniques have been employed in diagnosing and screening clinical diseases. However, both poor medical image quality and illumination style inconsistency increase uncertainty in clinical decision-making, potentially resulting in clinician misdiagnosis. The majority of current image enhancement methods primarily concentrate on enhancing medical image quality by leveraging high-quality reference images, which are challenging to collect in clinical applications. In this study, we address image quality enhancement within a fully self-supervised learning setting, wherein neither high-quality images nor paired images are required. To achieve this goal, we investigate the potential of self-supervised learning combined with domain adaptation to enhance the quality of medical images without the guidance of high-quality medical images. We design a Domain Adaptation Self-supervised Quality Enhancement framework, called DASQE. More specifically, we establish multiple domains at the patch level through a designed rule-based quality assessment scheme and style clustering. To achieve image quality enhancement and maintain style consistency, we formulate the image quality enhancement as a collaborative self-supervised domain adaptation task for disentangling the low-quality factors, medical image content, and illumination style characteristics by exploring intrinsic supervision in the low-quality medical images. Finally, we perform extensive experiments on six benchmark datasets of medical images, and the experimental results demonstrate that DASQE attains state-of-the-art performance. Furthermore, we explore the impact of the proposed method on various clinical tasks, such as retinal fundus vessel/lesion segmentation, nerve fiber segmentation, polyp segmentation, skin lesion segmentation, and disease classification. The results demonstrate that DASQE is advantageous for diverse downstream image analysis tasks.

WaterPairs: a paired dataset for underwater image enhancement and underwater object detection

Due to its importance in marine engineering and aquatic robotics, underwater image enhancement works as a preprocessing step to improve the performance of high-level vision tasks such as underwater object detection and recognition. Although several studies exhibit that underwater image enhancement algorithms can boost the detection accuracy of detectors, no work has focused on studying the relationship between these two tasks. This is mainly because current underwater datasets lack either bounding box annotations or high-quality reference images, based on which detection accuracy or image quality assessment metrics are calculated. To examine how underwater image enhancement methods affect underwater object detection tasks, we provide a large-scale underwater object detection dataset with both bounding box annotations and high-quality reference images, namely, the WaterPairs dataset. The WaterPairs dataset offers a platform for researchers to comprehensively study the influence of underwater image enhancement algorithms on underwater object detection tasks. We will release our dataset at https://github.com/IanDragon/WaterPairs once this paper is accepted.

No-reference quality assessment for low-light image enhancement: Subjective and objective methods

No-Reference (NR) quality assessment is a crucial approach for evaluating the quality of low-light enhanced images, as it is often difficult to acquire high-quality reference images in applications such as night-time automatic driving. However, current NR evaluation methods for low-light enhanced images often lack consideration of important characteristics such as color, structure, and naturalness. This paper proposes a novel NR quality assessment method for NR low-light enhanced images from both subjective and objective aspects. On the subjective side, we construct the Low-light Enhanced Images Subjective Dataset (LEISD) containing 2040 images with 255 different image contents. Each image was evaluated based on the Single Stimulus (SS) method by 20 subjects. On the objective side, we propose Multi-Features Reconciliation-based Quality Assessment (MFRQA) methods for low-light enhanced images by observing the low-light enhanced images. The MFRQA summarized four key feature perspectives: brightness, color, structure and naturalness, and employed the traditional machine learning model to reconcile the multi-features. Experimental results on the LEISD dataset demonstrate competitive performance and low complexity of our method compared to the representative quality metrics.

Deep Multi-Exposure Image Fusion for Dynamic Scenes.

Recently, learning-based multi-exposure fusion (MEF) methods have made significant improvements. However, these methods mainly focus on static scenes and are prone to generate ghosting artifacts when tackling a more common scenario, i.e., the input images include motion, due to the lack of a benchmark dataset and solution for dynamic scenes. In this paper, we fill this gap by creating an MEF dataset of dynamic scenes, which contains multi-exposure image sequences and their corresponding high-quality reference images. To construct such a dataset, we propose a 'static-for-dynamic' strategy to obtain multi-exposure sequences with motions and their corresponding reference images. To the best of our knowledge, this is the first MEF dataset of dynamic scenes. Correspondingly, we propose a deep dynamic MEF (DDMEF) framework to reconstruct a ghost-free high-quality image from only two differently exposed images of a dynamic scene. DDMEF is achieved through two steps: pre-enhancement-based alignment and privilege-information-guided fusion. The former pre-enhances the input images before alignment, which helps to address the misalignments caused by the significant exposure difference. The latter introduces a privilege distillation scheme with an information attention transfer loss, which effectively improves the deghosting ability of the fusion network. Extensive qualitative and quantitative experimental results show that the proposed method outperforms state-of-the-art dynamic MEF methods. The source code and dataset are released at https://github.com/Tx000/Deep_dynamicMEF.

Method for developing and using high quality reference images to evaluate tone mapping operators.

With the advancement of imaging technology, high dynamic range (HDR) images can now be captured and displayed to produce realistic effects. Tone mapping operators (TMOs) are used to map HDR radiance to the displayable range. A reliable TMO would play a significant role in the accurate reproduction of HDR scenes. The present study aimed to establish an image quality metric based on external references to evaluate various TMOs. Two psychophysical experiments were conducted to develop reference images and to investigate the performance of TMOs. In experiment 1, a set of high quality reference images was developed by rendering image features in terms of contrast, sharpness, and colorfulness, to achieve good rendering for each image. The images were used as reference in experiment 2 to evaluate the performance of 14 TMOs using a six-point categorical judgment method. The TMOs were evaluated using four scales, i.e., contrast, sharpness, colorfulness, and overall performance. The hierarchical relationship among TMOs was established. The results were further compared with previous studies, and high correlation was found between the current experiments and previous studies.

The effect of image descriptors on the performance of classifiers of footwear outsole image pairs

Shoe prints are commonly found at the scene of a crime and can sometimes help link a suspect to the scene. Because prints tend to be partially observed or smudgy, comparing crime scene prints with reference images from a putative shoe can be challenging. Footwear examiners rely on guidelines such as those published by SWGTREAD [1] to visually assess the similarity between two or more footwear impressions, one reason being that reliable, quantitative methods have yet to be validated for use in real cases. To help in the development of such methods, we created a study dataset of images of outsole impressions that shared class characteristics and degree of wear and that were subject to a specific type of degradation. We also propose a method to quantify the similarity between two outsole images that extends the capabilities of MC-COMP [2]. The proposed method is composed of three steps; (1) extracting image descriptors, (2) aligning images using the maximum clique, (3) calculating similarity values using two different classifiers; (a) degree of overlap between the two images, and (b) a score produced by a random forest. To explore the performance of the algorithm we propose, we compared degraded, crime scene-like images to high-quality reference images produced by the same or by different shoes. Even though comparisons involved matches or very close non-matches, and one of the images was blurry, the algorithm shows good source classification performance.

Towards lower-dose PET using physics-based uncertainty-aware multimodal learning with robustness to out-of-distribution data

Radiation exposure in positron emission tomography (PET) imaging limits its usage in the studies of radiation-sensitive populations, e.g., pregnant women, children, and adults that require longitudinal imaging. Reducing the PET radiotracer dose or acquisition time reduces photon counts, which can deteriorate image quality. Recent deep-neural-network (DNN) based methods for image-to-image translation enable the mapping of low-quality PET images (acquired using substantially reduced dose), coupled with the associated magnetic resonance imaging (MRI) images, to high-quality PET images. However, such DNN methods focus on applications involving test data that match the statistical characteristics of the training data very closely and give little attention to evaluating the performance of these DNNs on new out-of-distribution (OOD) acquisitions. We propose a novel DNN formulation that models the (i) underlying sinogram-based physics of the PET imaging system and (ii) the uncertainty in the DNN output through the per-voxel heteroscedasticity of the residuals between the predicted and the high-quality reference images. Our sinogram-based uncertainty-aware DNN framework, namely, suDNN, estimates a standard-dose PET image using multimodal input in the form of (i) a low-dose/low-count PET image and (ii) the corresponding multi-contrast MRI images, leading to improved robustness of suDNN to OOD acquisitions. Results on in vivo simultaneous PET-MRI, and various forms of OOD data in PET-MRI, show the benefits of suDNN over the current state of the art, quantitatively and qualitatively.

Photo quality classification using deep learning

The detection of poor quality images for reasons such as focus, lighting, compression, and encoding is of great importance in the field of computer vision. The ability to quickly and automatically classify an image as poor quality creates opportunities for a multitude of applications such as digital cameras, phones, self-driving cars, and web search technologies. In this paper an end-to-end approach using Convolutional Neural Networks (CNN) is presented to classify images into six categories of bad lighting, Gaussian blur, motion blur, JPEG 2000, white-noise, and high quality reference images. A new dataset of images was produced and used to train and validate the model. Finally, the application of the developed model was evaluated using images from the German Traffic Sign Recognition Benchmark. The results show that the trained CNN can detect and correctly classify images into the aforementioned categories with high accuracy and the model can be easily re-calibrated for other applications with only a small sample of training images.

Image fusion network for dual-modal restoration

<p style='text-indent:20px;'>In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.</p>

Video Frame Interpolation and Enhancement via Pyramid Recurrent Framework.

Video frame interpolation aims to improve users' watching experiences by generating high-frame-rate videos from low-frame-rate ones. Existing approaches typically focus on synthesizing intermediate frames using high-quality reference images. However, the captured reference frames may suffer from inevitable spatial degradations such as motion blur, sensor noise, etc. Few studies have approached the joint video enhancement problem, namely synthesizing high-frame-rate and high-quality results from low-frame-rate degraded inputs. In this paper, we propose a unified optimization framework for video frame interpolation with spatial degradations. Specifically, we develop a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames. The pyramid module features adjustable spatial receptive field and temporal scope, thus contributing to controllable computational complexity and restoration ability. Besides, we propose an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship. The pyramid module integrates the recurrent module, thus can iteratively synthesize temporally smooth results. And the pyramid modules share weights across iterations, thus it does not expand the model's parameter size. Our model can be generalized to several applications such as up-converting the frame rate of videos with motion blur, reducing compression artifacts, and jointly super-resolving low-resolution videos. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods on various video frame interpolation and enhancement tasks.

A Tone Mapping Model Based on Receptive Field for HDR Images

High dynamic range (HDR) imaging has greater contrast reproduction capability than standard imaging techniques. It can achieve natural and pleasing appearance in terms of image quality. A tone mapping model (TMOz) is developed based on the center-surround properties of the mammalian ganglion cells of the human visual system for feature enhancement. The contrast of the HDR image is mapped adaptively to an SDR display range using a global method followed by contrast enhancement in local regions. A psychophysical experiment was conducted to refine the model for adaptivity of the contrast mapping function. Finally, the performance of the TMOz was evaluated using CIELAB (2:1) formula together with high quality reference images. The results showed that TMOz outperformed the other tone mapping operators (TMOs).

Combined application of extended depth of field imaging, image stitching and polarized microscopy techniques in identification of Spatholobus suberectus

ObjectiveAs traditional techniques for microscopic identification of Chinese medicines currently lack objective and high-quality reference images, here we developed a systemic procedure to be used in microscopic identification of Chinese medicines, which would lead to more objective, effective and accurate identification process. MethodsSpatholobi Caulis (Jixueteng in Chinese) was used as the specimen in the development of such procedure. Jixueteng samples were microscopically examined in bright- and dark-field microscopy. Microscopic images were obtained by regular, EDF, and image stitching techniques. ResultsThe microscopic images of the characteristics in pulverized Jixueteng were captured, thanks to EDF imaging and image stitching techniques which allowed the detailed and full sighting of each characteristic to be obtained simultaneously. Different layers in anatomical transverse section, including cork, phelloderm, cortex, phloem, cambium, xylem and pith, were distinctively observed. Moreover, by comparing images of bright- and dark-field microscopy, birefringent and non- birefringent components could readily be distinguished. ConclusionWith application of the developed procedure, high-definition, panoramic and microscopic images were acquired, which could be used as the reference images for microscopic identification of Chinese medicines.

Thick cloud removal in Landsat images based on autoregression of Landsat time-series data

Thick-cloud contamination causes serious missing data in Landsat images, which substantially limits applications of these images. To remove thick clouds from Landsat data, the most popular methods employ auxiliary data such as a cloud-free image of the same area acquired on another date (referred to as the “reference image”). However, the performance of most previous methods strongly depends on the usefulness of the specific reference image, but in some cases high-quality cloud-free reference images are rarely available. In addition, some of these methods ignore the use of partially cloud-contaminated reference images, but clear pixels in these images can be very useful. To address these issues, a new cloud-removal method (AutoRegression to Remove Clouds (ARRC)) has been developed in this study. The most important improvement of ARRC is that it considers autocorrelation of Landsat time-series data and employs multi-year Landsat images including partially cloud-contaminated images in the cloud-removing process. ARRC also addresses the cases in which autocorrelation of Landsat time series might be adversely affected by abrupt land cover changes over multiple years. We compared ARRC with the widely-used MNSPI (modified neighborhood similar pixel interpolator) method at four challenging sites, including an urban area in Beijing and three croplands, in the North China Plain, northeastern Vietnam, and Iowa, USA. Results from the cloud-simulated images showed that ARRC performed better than MNSPI and achieved lower RMSE values (e.g., 0.02129 vs. 0.03005, 0.03293 vs. 0.04725, 0.02740 vs. 0.03556, and 0.03303 vs. 0.03973 in the near-infrared band at the four testing sites, respectively). Moreover, the experiments demonstrated improved performance when clear pixels in partially cloud-contaminated images were used by ARRC. Furthermore, cloud-free images reconstructed by ARRC were visually better than those reconstructed by MNSPI when both approaches were applied to actual cloud-contaminated Landsat images.

Tone Mapping Operators Evaluation Based on High Quality Reference Images

High Dynamic Range (HDR) imaging applications have been commonly placed recently. Several tone mapping operators (TMOs) have been developed which project the HDR radiance range to that of a display. Currently, there is no agreement on a technique for evaluation of tone mapping operators. The goal of this study is to establish a method based on reference images to evaluate the TMOs. Two psychophysical experiments were carried out for the evaluation of tone mapping operators. In the first experiment, a set of high quality images were generated to possess right extents of image features including contrast, colourfulness and sharpness. These images were further used in the second experiment as reference images to evaluate different TMOs. It was found Reinhard's photographic reproduction based on local TMO gave an overall better performance. CIELAB(2:1) and S- CIELAB metrics were also used to judge colour image quality of the same TMOs. It was found that both metrics agreed well with the visual results.

Predicting the Quality of Images Compressed after Distortion in Two Steps.

In a typical communication pipeline, images undergo a series of processing steps that can cause visual distortions before being viewed. Given a high quality reference image, a reference (R) image quality assessment (IQA) algorithm can be applied after compression or transmission. However, the assumption of a high quality reference image is often not fulfilled in practice, thus contributing to less accurate quality predictions when using stand-alone R IQA models. This is particularly common on social media, where hundreds of billions of usergenerated photos and videos containing diverse, mixed distortions are uploaded, compressed, and shared annually on sites like Facebook, YouTube, and Snapchat. The qualities of the pictures that are uploaded to these sites vary over a very wide range. While this is an extremely common situation, the problem of assessing the qualities of compressed images against their precompressed, but often severely distorted (reference) pictures has been little studied. Towards ameliorating this problem, we propose a novel two-step image quality prediction concept that combines NR with R quality measurements. Applying a first stage of NR IQA to determine the possibly degraded quality of the source image yields information that can be used to quality-modulate the R prediction to improve its accuracy. We devise a simple and efficient weighted product model of R and NR stages, which combines a pre-compression NR measurement with a post-compression R measurement. This first-of-a-kind two-step approach produces more reliable objective prediction scores. We also constructed a new, first-of-a-kind dedicated database specialized for the design and testing of two-step IQA models. Using this new resource, we show that twostep approaches yield outstanding performance when applied to compressed images whose original, pre-compression quality covers a wide range of realistic distortion types and severities. The two-step concept is versatile as it can use any desired R and NR components. We are making the source code of a particularly efficient model that we call 2stepQA publicly available at https://github.com/xiangxuyu/2stepQA. We are also providing the dedicated new two-step database free of charge at http://live.ece.utexas.edu/research/twostep/index.html.

Reconstruction of a high-quality volumetric image and a respiratory motion model from patient CBCT projections.

To develop and evaluate a method of reconstructing a patient- and treatment day- specific volumetric image and motion model from free-breathing cone-beam projections and respiratory surrogate measurements. This Motion-Compensated Simultaneous Algebraic Reconstruction Technique (MC-SART) generates and uses a motion model derived directly from the cone-beam projections, without requiring prior motion measurements from 4DCT, and can compensate for both inter- and intrabin deformations. The motion model can be used to generate images at arbitrary breathing points, which can be used for estimating volumetric images during treatment delivery. The MC-SART was formulated using simultaneous image reconstruction and motion model estimation. For image reconstruction, projections were first binned according to external surrogate measurements. Projections in each bin were used to reconstruct a set of volumetric images using MC-SART. The motion model was estimated based on deformable image registration between the reconstructed bins, and least squares fitting to model parameters. The model was used to compensate for motion in both projection and backprojection operations in the subsequent image reconstruction iterations. These updated images were then used to update the motion model, and the two steps were alternated between. The final output is a volumetric reference image and a motion model that can be used to generate images at any other time point from surrogate measurements. A retrospective patient dataset consisting of eight lung cancer patients was used to evaluate the method. The absolute intensity differences in the lung regions compared to ground truth were 50.8±43.9HU in peak exhale phases (reference) and 80.8±74.0 in peak inhale phases (generated). The 50th percentile of voxel registration error of all voxels in the lung regions with >5mm amplitude was 1.3mm. The MC-SART was also applied to measured patient cone-beam projections acquired with a linac-mounted CBCT system. Results from this patient data demonstrate the feasibility of MC-SART and showed qualitative image quality improvements compared to other state-of-the-art algorithms. We have developed a simultaneous image reconstruction and motion model estimation method that uses Cone-beam computed tomography (CBCT) projections and respiratory surrogate measurements to reconstruct a high-quality reference image and motion model of a patient in treatment position. The method provided superior performance in both HU accuracy and positional accuracy compared to other existing methods. The resultant reference image and motion model can be combined with respiratory surrogate measurements to generate volumetric images representing patient anatomy at arbitrary time points.

McSART: an iterative model-based, motion-compensated SART algorithm for CBCT reconstruction

4D cone beam computed tomography (CBCT) images of the thorax and abdomen can have reduced quality due to the limited number of projections per respiratory bin used in gated image reconstruction. In this work, we present a new algorithm to reconstruct high quality CBCT images by simultaneously reconstructing images and generating an associated respiratory motion model. This is done by updating model parameters to compensate for motion during the iterative image reconstruction process. CBCT image acquisition was simulated using the digital eXternal CArdiac Torso (XCAT) phantom, simulating breathing motion using four patient breathing traces. 4DCBCT images were reconstructed using the simultaneous algebraic reconstruction technique (SART), and compared to the proposed motion-compensated SART (McSART) algorithm. McSART used a motion model that describes tissue position as a function of diaphragm amplitude and velocity. The McSART algorithm alternately updated the motion model and image reconstruction, increasing the number of projections used for image reconstruction with every iteration. The model was able to interpolate and extrapolate deformations according to the magnitude of the surrogate signal. Without noise, the final iteration McSART images had HU errors at 31%, 34%, and 44% of their SART-reconstructed counterparts compared to ground truth XCAT images, with corresponding root-mean-square (RMS) motion model errors of 0.75 mm, 1.08 mm, and 1.17 mm respectively. With added image noise, McSART’s HU error was 31% of the SART-reconstructed 4DCBCT error, with a 1.43 mm RMS motion model error. Qualitatively, blurring and streaking artifacts were reduced in all the reconstructed images compared to 3D or SART-reconstructed 4DCBCT. The output of the algorithm was a high quality reference image and a corresponding motion model, that could be used to deform the reference image to any other point in a breathing cycle.

Variational Models for Joint Subsampling and Reconstruction of Turbulence-Degraded Images

Turbulence-degraded image frames are distorted by both turbulent deformations and space–time-varying blurs. To suppress these effects, we propose a multi-frame reconstruction scheme to recover a latent image from the observed distorted image sequence. Recent approaches are commonly based on registering each frame to a reference image, by which geometric turbulent deformations can be estimated and a sharp image can be restored. A major challenge is that a fine reference image is usually unavailable, as every turbulence-degraded frame is distorted. A high-quality reference image is crucial for the accurate estimation of geometric deformations and fusion of frames. Besides, it is unlikely that all frames from the image sequence are useful, and thus frame selection is necessary and highly beneficial. In this work, we propose a variational model for joint subsampling of frames and extraction of a clear image. A fine image and a suitable choice of subsample are simultaneously obtained by iteratively reducing an energy functional. The energy consists of a fidelity term measuring the discrepancy between the extracted image and the subsampled frames, as well as regularization terms on the extracted image and the subsample. Different choices of fidelity and regularization terms are explored. By carefully selecting suitable frames and extracting the image, the quality of the reconstructed image can be significantly improved. Extensive experiments have been carried out, which demonstrate the efficacy of our proposed model. In addition, the extracted subsamples and images can be put in existing algorithms to produce improved results.

No-Reference Utility Estimation with a Convolutional Neural Network

Traditional quality estimators evaluate an image's resemblance to a reference image. However, quality estimators are not well suited to the similar but somewhat different task of utility estimation, where an image is judged instead by how useful it would be in terms of extracting information about image content. While full-reference image utility metrics have been developed which outperform quality estimators for the utility-prediction task, assuming the existence of a high-quality reference image is not always realistic. The Oxford Visual Geometry Group's (VGG) deep convolutional neural network (CNN) [1], designed for object recognition, is modified and adapted to the task of utility estimation. This network achieves no-reference utility estimation performance near the full-reference state of the art, with a Pearson correlation of 0.946 with subjective utility scores of the CU-Nantes database and root mean square error of 12.3. Other noreference techniques adapted from the quality domain yield inferior performance. The CNN also generalizes better to distortion types outside of the training set, and is easily updated to include new types of distortion. Early stages of the network apply transformations similar to those of previously developed full-reference utility estimation algorithms.

Removing Turbulence Effect via Hybrid Total Variation and Deformation-Guided Kernel Regression

It remains a challenge to simultaneously remove geometric distortion and space-time-varying blur in frames captured through a turbulent atmospheric medium. To solve, or at least reduce these effects, we propose a new scheme to recover a latent image from observed frames by integrating a new hybrid total variation model and deformation-guided spatial-temporal kernel regression. The proposed scheme first constructs a high-quality reference image from the observed frames using low-rank decomposition. Then, to generate an improved registered sequence, the reference image is iteratively optimized using a variational model containing the combined regularization of local and non-local total variations. The proposed optimization algorithm efficiently solves this model with convergence guarantee. Next, to reduce blur variation, deformation-guided spatial-temporal kernel regression is carried out to fuse the registered sequence into one image by introducing the concept of the near-stationary patch. Applying a blind deconvolution algorithm to the fused image produces the final output. Extensive experimental testing shows, both qualitatively and quantitatively, that the proposed method can effectively alleviate distortion, and blur and recover details of the original scene compared to the state-of-the-art methods.