Image Fidelity Metrics Research Articles

Multi-spectral multi-image super-resolution of Sentinel-2 with radiometric consistency losses and its effect on building delineation

High resolution remote sensing imagery is used in a broad range of tasks, including detection and classification of objects. High-resolution imagery is however expensive to obtain, while lower resolution imagery is often freely available and can be used for a range of social good applications. To that end, we curate a multi-spectral multi-image dataset for super-resolution of satellite images. We use PlanetScope imagery from the SpaceNet-7 challenge as the high resolution reference and multiple Sentinel-2 revisits of the same location as the low-resolution imagery. We present the first results of applying multi-image super-resolution (MISR) to multi-spectral remote sensing imagery. We, additionally, introduce a radiometric-consistency module into the MISR model to preserve the high radiometric resolution and quality of the Sentinel-2 sensor. We show that MISR is superior to single-image super-resolution (SISR) and other baselines on a range of image fidelity metrics. Furthermore, we present the first assessment of the utility of multi-image super-resolution on a semantic and instance segmentation – common remote sensing tasks – showing that utilizing multiple images results in better performance in these downstream tasks, but MISR pre-processing is non-essential.

Deep Learning-Based Compression for Phase-Only Hologram

With the aid of rapid development in optical and display devices, Holography has become an emerging technology to represent 3D objects providing the most natural and realistic depth illusion to the users. To reproduce a light field of objects, a hologram records the interference patterns between the object and reference wavefields. Especially for computer-generated holograms, they contain vast amount of data and must be efficiently compressed for storage or transmission. Although several coding solutions, including standardized compression tools and their variations, have been tried in the literature, the coding efficiency is not satisfactory because holograms have completely different unique properties from those observed in natural images, whereas the legacy coding solutions are optimized for the latter. In this work, we propose a deep-learning based image compression network for phase-only holograms. The proposed network is trained in an end-to-end manner with the help of data augmentation technique and aims at minimizing the entropy of the input hologram. In comprehensive experiments, we demonstrate that our compression network shows the state-of-the-art coding performance that even surpasses the latest compression standard Versatile Video Coding by up to 39% in terms of BD-rate gain. Furthermore, we provide in-depth analysis for experimental results in both holographic and numerical reconstruction domains, such as individual performances of the test codecs, a performance comparison with different objective image fidelity metrics, and subjective quality evaluation results.

Development of an algorithm to automatically compress a CT image to visually lossless threshold

BackgroundTo develop an algorithm to predict the visually lossless thresholds (VLTs) of CT images solely using the original images by exploiting the image features and DICOM header information for JPEG2000 compression and to evaluate the algorithm in comparison with pre-existing image fidelity metrics.MethodsFive radiologists independently determined the VLT for 206 body CT images for JPEG2000 compression using QUEST procedure. The images were divided into training (n = 103) and testing (n = 103) sets. Using the training set, a multiple linear regression (MLR) model was constructed regarding the image features and DICOM header information as independent variables and regarding the VLTs determined with median value of the radiologists’ responses (VLTrad) as dependent variable, after determining an optimal subset of independent variables by backward stepwise selection in a cross-validation scheme.The performance was evaluated on the testing set by measuring absolute differences and intra-class correlation (ICC) coefficient between the VLTrad and the VLTs predicted by the model (VLTmodel). The performance of the model was also compared two metrics, peak signal-to-noise ratio (PSNR) and high-dynamic range visual difference predictor (HDRVDP). The time for computing VLTs between MLR model, PSNR, and HDRVDP were compared using the repeated ANOVA with a post-hoc analysis. P < 0.05 was considered to indicate a statistically significant difference.ResultsThe means of absolute differences with the VLTrad were 0.58 (95% CI, 0.48, 0.67), 0.73 (0.61, 0.85), and 0.68 (0.58, 0.79), for the MLR model, PSNR, and HDRVDP, respectively, showing significant difference between them (p < 0.01). The ICC coefficients of MLR model, PSNR, and HDRVDP were 0.88 (95% CI, 0.81, 0.95), 0.85 (0.79, 0.91), and 0.84 (0.77, 0.91). The computing times for calculating VLT per image were 1.5 ± 0.1 s, 3.9 ± 0.3 s, and 68.2 ± 1.4 s, for MLR metric, PSNR, and HDRVDP, respectively.ConclusionsThe proposed MLR model directly predicting the VLT of a given CT image showed competitive performance to those of image fidelity metrics with less computational expenses. The model would be promising to be used for adaptive compression of CT images.

Bridging the Gap Between Imaging Performance and Image Quality Measures

Imaging system performance measures and Image Quality Metrics (IQM) are reviewed from a systems engineering perspective, focusing on spatial quality of still image capture systems. We classify IQMs broadly as: Computational IQMs (CP-IQM), Multivariate Formalism IQMs (MF-IQM), Image Fidelity Metrics (IF-IQM), and Signal Transfer Visual IQMs (STV-IQM). Comparison of each genre finds STV-IQMs well suited for capture system quality evaluation: they incorporate performance measures relevant to optical systems design, such as Modulation Transfer Function (MTF) and Noise-Power Spectrum (NPS); their bottom-up, modular approach enables system components to be optimized separately. We suggest that correlation between STV-IQMs and observer quality scores is limited by three factors: current MTF and NPS measures do not characterize scene-dependent performance introduced by imaging system non-linearities; contrast sensitivity models employed do not account for contextual masking effects; cognitive factors are not considered. We hypothesize that implementation of scene and process-dependent MTF (SPD-MTF) and NPS (SPD-NPS) measures should mitigate errors originating from scene dependent system performance. Further, we propose implementation of contextual contrast detection and discrimination models to better represent low-level visual performance in image quality analysis. Finally, we discuss image quality optimization functions that may potentially close the gap between contrast detection/discrimination and quality.

Metric for Estimating Congruity between Quantum Images

An enhanced quantum-based image fidelity metric, the QIFM metric, is proposed as a tool to assess the “congruity” between two or more quantum images. The often confounding contrariety that distinguishes between classical and quantum information processing makes the widely accepted peak-signal-to-noise-ratio (PSNR) ill-suited for use in the quantum computing framework, whereas the prohibitive cost of the probability-based similarity score makes it imprudent for use as an effective image quality metric. Unlike the aforementioned image quality measures, the proposed QIFM metric is calibrated as a pixel difference-based image quality measure that is sensitive to the intricacies inherent to quantum image processing (QIP). As proposed, the QIFM is configured with in-built non-destructive measurement units that preserve the coherence necessary for quantum computation. This design moderates the cost of executing the QIFM in order to estimate congruity between two or more quantum images. A statistical analysis also shows that our proposed QIFM metric has a better correlation with digital expectation of likeness between images than other available quantum image quality measures. Therefore, the QIFM offers a competent substitute for the PSNR as an image quality measure in the quantum computing framework thereby providing a tool to effectively assess fidelity between images in quantum watermarking, quantum movie aggregation and other applications in QIP.

High power-saving and fidelity-conserving object-based local dimming for LCD systems with LED-based backlight units

A local dimming technique which is enabled on LED-based LCD systems is expected to give more power saving than a global one. However, prior local dimming techniques have not considered human visual system-awareness much as done in an advanced global dimming technique. In this paper, we propose a novel local dimming technique using an object-based approach for both good human visuality and high power saving. It utilizes prevalent colors of individual objects in a given image to do initial dimming, and then enhances the image using a proper fidelity threshold to reduce visible artifacts. Experimental results show that the proposed technique achieves power saving up to 13 and 12 times higher than human visual system-aware global dimming and prior well-designed local dimming techniques, respectively, with a similar human visual system-aware image quality level in a uniform backlight optical profile. It achieves power saving up to 15 and 6 times higher in a Gaussian-based backlight optical profile. Due to a good use of a simpler image fidelity metric, the proposed technique also can make the complexity go lower while keeping a satisfactory image quality.

VPSNR: a visualization-aware image fidelity metric tailored for diagnostic imaging

Often, the large amounts of data generated in diagnostic imaging cause overload problems for IT systems and radiologists. This entails a need of effective use of data reduction beyond lossless levels, which, in turn, underlines the need to measure and control the image fidelity. Existing image fidelity metrics, however, fail to fully support important requirements from a modern clinical context: support for high-dimensional data, visualization awareness, and independence from the original data. We propose an image fidelity metric, called the visual peak signal-to-noise ratio (vPSNR), fulfilling the three main requirements. A series of image fidelity tests on CT data sets is employed. The impact of visualization transform (grayscale window) on diagnostic quality of irreversibly compressed data sets is evaluated through an observer-based study. In addition, several tests were performed demonstrating the benefits, limitations, and characteristics of vPSNR in different data reduction scenarios. The visualization transform has a significant impact on diagnostic quality, and the vPSNR is capable of representing this effect. Moreover, the tests establish that the vPSNR is broadly applicable. vPSNR fills a gap not served by existing image fidelity metrics, relevant for the clinical context. While vPSNR alone cannot fulfill all image fidelity needs, it can be a useful complement in a wide range of scenarios.

Advantage in image fidelity and additional computing time of JPEG2000 3D in comparison to JPEG2000 in compressing abdomen CT image datasets of different section thicknesses

This study aimed to assess the advantage of the Joint Photographic Experts Group 2000 (JPEG2000) 3D (part 2) over JPEG2000 in compressing abdomen computed tomography (CT) image data sets of different section thicknesses (STs). Twenty CT scans were reconstructed with six STs (0.67, 1, 2, 3, 4, and 5 mm) and were then compressed to seven compression ratios (CRs) (reversible, 6:1, 8:1, 10:1, 12:1, 14:1, and 16:1) using JPEG2000 and JPEG2000 3D algorithms. Computing (encoding and decoding) times were measured. The image fidelity of the compressed images was quantitatively measured with two computerized image fidelity metrics, peak signal-to-noise ratio (PSNR) and multiscale structural similarity (MS-SSIM). For 120 selected case-relevant images (20 patients x one image per patient x 6 STs), five radiologists independently compared original and compressed images and assessed the fidelity of the compressed images on a four-grade scale. Wilcoxon signed-rank tests and Friedman tests with post hoc Dunn tests were used for the comparisons between the two compressions and among the six STs, respectively For each combination of the ST and irreversible CR, JPEG2000 3D showed higher image fidelity than JPEG2000 in terms of PSNR (p < 0.0001), MS-SSIM (p < 0.0001), and five radiologists' grading (p-values ranged from <0.0001 to 0.003). At each CR, the advantage of JPEG2000 3D in image fidelity, measured as the differences in the two computerized image fidelity metrics (PSNR and MS-SSIM), significantly increased as the ST increased from 0.67 to 2 mm, and then slowly decreased as the ST increased from 2 to 5 mm. Similar trends were observed in visual analyses of 120 selected images by five radiologists. At each CR, the 3D-to-2D encoding-time ratio significantly decreased (p < 0.001) as the ST increased from 0.67 to 2 mm, and then slowly increased (p < 0.001) as the ST increased from 2 to 5 mm. The 3D-to-2D decoding-time ratio at each CR did not show a notable biphasic trend across the ST. In compressing abdomen CT image data sets of different STs, the advantage of JPEG2000 3D over JPEG2000 increases as the ST increases from 0.67 to 2 mm, and then slowly decreases as the ST increases from 2 to 5 mm. The practical advantage of JPEG2000 3D is limited for a submillimeter ST due to its greater computing time with only a marginal improvement in image fidelity.

A Comparison of Three Image Fidelity Metrics of Different Computational Principles for JPEG2000 Compressed Abdomen CT Images

This study aimed to evaluate three image fidelity metrics of different computational principles--peak signal-to-noise ratio (PSNR), high-dynamic range visual difference predictor (HDR-VDP), and multiscale structural similarity (MS-SSIM)--in measuring the fidelity of JPEG2000 compressed abdomen computed tomography images from a viewpoint of visually lossless compression. Three hundred images with 0.67- or 5-mm section thickness were compressed to one of five compression ratios ranging from reversible compression to 15:1. The fidelity of each compressed image was measured by five radiologists' visual analyses (distinguishable or indistinguishable from the original) and the three metrics. The Spearman rank correlation coefficients of the PSNR, HDR-VDP, and MS-SSIM values with the number of readers responding as indistinguishable were 0.86, 0.94, and 0.86, respectively. Using the pooled readers' responses as the reference standard, the area under the receiver-operating-characteristic curve for the HDR-VDP (0.99) was significantly greater than that for the PSNR (0.95) (p < 0.001) and for the MS-SSIM (0.96) (p = 0.003), and there was no significant difference between the PSNR and MS-SSIM (p = 0.70). In measuring the image fidelity, the HDR-VDP outperforms the PSNR and MS-SSIM, and the MS-SSIM and PSNR are comparable.

A perceptual approach to trimming and tuning unstructured lumigraphs

We present a novel perceptual method to reduce the visual redundancy of unstructured lumigraphs, an image based representation designed for interactive rendering. We combine features of the unstructured lumigraph algorithm and image fidelity metrics to efficiently rank the perceptual impact of the removal of subregions of input views ( subviews ). We use a greedy approach to estimate the order in which subviews should be pruned to minimize perceptual degradation at each step. Renderings using varying numbers of subviews can then be easily visualized with confidence that the retained subviews are well chosen, thus facilitating the choice of how many to retain. The regions of the input views that are left are repacked into a texture atlas. Our method takes advantage of any scene geometry information available but only requires a very coarse approximation. We perform a user study to validate its behaviour, as well as investigate the impact of the choice of image fidelity metric as well as that of user parameters. The three metrics considered fall in the physical, statistical and perceptual categories. The overall benefit of our method is the semiautomation of the view selection process, resulting in unstructured lumigraphs that are thriftier in texture memory use and faster to render. Using the same framework, we adjust the parameters of the unstructured lumigraph algorithm to optimise it on a scene by scene basis.

A new universal colour image fidelity metric

We extend a recently introduced universal grayscale image quality index to a newly developed perceptually decorrelated colour space. The resulting colour image fidelity metric quantifies the distortion of a processed colour image relative to its original version. We evaluated the new colour image fidelity metric through observer experiments in which subjects ranked images according to perceived distortion. The metric correlates strongly with human perception and can therefore be used to assess the performance of colour image coding and compression schemes, colour image enhancement algorithms, synthetic colour image generators, and colour image fusion schemes.

Color image fidelity metrics evaluated using image distortion maps

Several color image fidelity metrics are evaluated by comparing the metric predictions to empirical measurements. Subjects examined image pairs consisting of an original and a reproduction. They marked locations on the reproduction that differed detectably from the original. We refer to the distribution of error marks by the subjects as image distortion maps. The empirically obtained image distortion maps are compared to the predicited visible difference calculated using (1) the widely used root mean square error (point-by-point RMS) computed in uncalibrated RGB values, (2) the point-by-point CIELAB Δ E 94 values (CIE, 1994), and (3) S-CIELAB Δ E 94, a spatial extension of CIELAB Δ E metric. The uncalibrated RMS metric did not predict the perceptual image distortion data well. The point-by-point CIELAB Δ E 94 metric provided better predictions, and the S-CIELAB metric, which incorporated the spatial color sensitivity of the eye, gave the most accurate predictions. None of the metrics provided an excellent fit to the data. Image areas with poor predictions were concentrated in regions containing large negative local contrast. When these areas were excluded from our data analysis, both S-CIELAB and CIELAB predictions had much better agreement with the perceptual data. This suggests that the next step in improving color image fidelity metrics is to redefine color difference formula such as CIELAB Δ E 94 in terms of local contrast.