Light Field Image Quality Research Articles

Feature-level contrastive learning for full-reference light field image quality assessment

Light field (LF), as a new visual information representation for immersive experiences in virtual reality and augmented reality, has attracted extensive attention. It is of significance to establish the light field image quality assessment (LF-IQA) that conforms to the characteristics of human visual perception. Due to the spatial–angular inter-twined imaging mechanism, the quality evaluation for LF images faces more challenges than ordinary 2D/3D IQA. This paper proposes a feature-level contrastive learning based full-reference LF-IQA with a divide-and-conquer learning strategy. It first employs the cross-attention mechanism to learn the local contrast features related to the reference features from three subsets of distorted sub-aperture images (SAIs). Then a Transformer-based feature fusion is used to predict the overall perceptual quality. The evaluation experiments are carried out on three commonly used LF-IQA datasets: Win5-LID, NBU-LF1.0, and SHU. The experimental results demonstrate that our model outperforms the existing state-of-the-art LF-IQA methods.

Multi-exposure fused light field image quality assessment for dynamic scenes: Benchmark dataset and objective metric

The luminance dynamic range in natural scenes is exceedingly wide. However, capturing the wide dynamic range information of natural scenes through a single exposure is challenging for commercial light field cameras. One solution to expand the luminance dynamic range of captured light field images is to employ multi-exposure imaging techniques. Nevertheless, this approach may introduce various distortions during the image fusion procedure, especially for dynamic scenes. To evaluate the visual quality of multi-exposure fused light field images (MEFLFIs) and compare the performance of different multi-exposure light field fusion methods, we initially conduct a subjective quality assessment on MEFLFIs for dynamic scenes and establish the first MEFLFI benchmark dataset, which comprises 480 fused light field images generated by eleven state-of-the-art multi-exposure fusion algorithms and two tone-mapping algorithms, along with their corresponding subjective scores. Subsequently, a novel specially designed objective quality assessment metric for MEFLFIs is developed, which incorporates both local–global joint features and angular quality-aware features. The key contribution of this work lies in establishing a pioneering benchmark dataset for MEFLFIs along with a novel tailored objective quality metric. Experimental results conducted over our established MEFLFI database demonstrate that the proposed metric achieves superior performance, with Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SROCC) values of 0.8895 and 0.8658 respectively, representing a significant improvement of approximately 5.7 % and 4.9 % compared to the second-best specifically-designed LFI quality metric available. Furthermore, ablation experiments have confirmed that the integration of local–global joint features and angular features significantly enhances the performance of our objective metric, resulting in a better alignment with human visual perception.

Transformer-Based Light Field Geometry Learning for No-Reference Light Field Image Quality Assessment

Transformer-Based Light Field Geometry Learning for No-Reference Light Field Image Quality Assessment

Splitting the backbone: A novel hierarchical method for assessing light field image quality

The rising popularity of light field imaging underscores the pivotal role of image quality in user experience. However, evaluating the quality of light field images presents significant challenges owing to their high-dimensional nature. Current quality assessment methods for light field images predominantly rely on machine learning or statistical analysis, often overlooking the interdependence among pixels. To overcome this limitation, we propose an innovative approach that employs a universal backbone network and introduces a dual-task framework for feature extraction. Specifically, we integrate a staged “primary-secondary” hierarchical evaluation mode into the universal backbone networks, enabling accurate quality score inference while preserving the intrinsic information of the original image. Our proposed approach reduces inference time by over 75% compared to existing methods, simultaneously achieving state-of-the-art results in terms of evaluation metrics. By harnessing the efficiency of neural networks, our framework offers an effective solution for the quality assessment of light field images, providing superior accuracy and speed compared to current methodologies.

Light Field Image Quality Assessment Using Natural Scene Statistics and Texture Degradation

Light Field Image Quality Assessment Using Natural Scene Statistics and Texture Degradation

Pseudo Light Field Image and 4D Wavelet-Transform-Based Reduced-Reference Light Field Image Quality Assessment

Reduced-reference light field image (LFI) quality assessment (RR LFIQA) automatically assesses image quality with only partial information about the reference LFI is available. Existing RR LFIQA has difficulty extracting effective RR information and perceptual features to represent the LFI quality. In this paper, we propose an RR LFIQA model based on pseudo LFI (PLFI) and four-dimensional (4D) wavelet transform. To extract RR information related to LFI perceptual quality, a PLFI is created as the RR information of the LFI using a view synthesis algorithm. Considering that the high-dimensional characteristics of the PLFI, 4D wavelet transform is used to decompose the original and distorted PLFIs. The 4D wavelet transform essentially performs a continuous 1D wavelet transform for the 4D signal to enable the local 4D structure of the PLFIs to be characterized effectively in the 4D wavelet domain. A novel spatial-angular weighting strategy is proposed to describe the importance of each location for quality evaluation, to further improve the performance of the proposed method. Experimental results on four benchmark datasets show that the proposed model performs better than the representative 2DIQA and LFIQA models. Our code will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JianjunXiang/RRLFIQA_4DDWT</uri> .

Blind Quality Evaluator of Light Field Images by Group-Based Representations and Multiple Plane-Oriented Perceptual Characteristics

Due to the emergency of multi-view cameras and commercial Light Field (LF) cameras, the demand of high-performance LF quality evaluator is of great significance for guiding LF acquisition, processing and application and further promoting the visual perceived quality of LF visualizations. However, LF Images (LFIs), as high-dimensional data, suffer from various quality degradations not only in the spatial domain but also in the angular domain. Therefore, it is of great challenge to predict LF quality accurately. An effective LF evaluator should be able to represent these heterogeneous artifacts. In this paper, we provide a novel No-Reference LF Quality Assessment Evaluator (NR LF-QAE) to tackle this problem. Firstly, to measure angular consistency among viewports, we utilize group-based representations to character information similarity of aligned view stacks. Secondly, to better describe the texture information of LFIs, unifying spatial-angular texture statistic measurement is performed via Local Binary Patterns from Three Orthogonal Planes (LBP-TOP). Thirdly, we design 3D Log-Gabor filters to extract LF global structure information in Sub-Aperture Images (SAIs) as spatial feature characterizations and 2D Log-Gabor filters are adopted to characterize ray direction/depth information in Epipolar Plane Images (EPIs) as angular feature characterizations. By comprehensive LF information analyses in angular consistency and spatial-angular feature extraction with texture and structure descriptors, experimental results demonstrate the superiority of the proposed NR LF-QAE over the state-of-the-art comparative models in predicting the quality of LFIs on three available benchmark databases. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zerosola/NR-LF-QAE</uri> .

MAFBLiF: Multi-Scale Attention Feature Fusion-Based Blind Light Field Image Quality Assessment

MAFBLiF: Multi-Scale Attention Feature Fusion-Based Blind Light Field Image Quality Assessment

Stitched Wide Field of View Light Field Image Quality Assessment: Benchmark Database and Objective Metric

Stitched Wide Field of View Light Field Image Quality Assessment: Benchmark Database and Objective Metric

A New Benchmark Database and Objective Metric for Light Field Image Quality Evaluation

A New Benchmark Database and Objective Metric for Light Field Image Quality Evaluation

Blind light field image quality assessment based on deep meta-learning.

In recent years, the use of deep convolutional neural networks (DCNNs) for light field image quality assessment (LFIQA) has gained significant attention. Despite their notable successes, it is widely accepted that training DCNNs heavily depends on a large amount of annotated data. Additionally, convolutional network-based LFIQA methods show a limitation in capturing long-range dependencies. Unfortunately, LFIQA is essentially a typical small-sample problem, leading to existing DCNN-based LFIQA metrics requiring data augmentation but with unsatisfactory performance. To address these issues, this study proposes utilizing the self-attention capability of the Swin Transformer to efficiently capture spatial-angular information while employing meta-learning for small-sample learning in the LFIQA task. Specifically, a collection of LFIQA tasks is gathered, representing different distortions. Then, meta-learning is employed to acquire shared prior knowledge across diverse distortions. Finally, the quality prior model is fine-tuned on a target LFIQA task to obtain the final LFIQA model quickly. Experimental results show that the proposed LFIQA metric achieves high consistency with subjective scores, and outperforms several state-of-the-art LFIQA approaches.

PVBLiF: A Pseudo Video-Based Blind Quality Assessment Metric for Light Field Image

Going beyond traditional 2D imaging is not only an emerging trend of imaging technology, but also the key to a more immersive user experience. Light Field Image (LFI) is a typical high-dimensional imaging format, and the quality evaluation of which is very challenging but necessary. In this paper, we propose a novel Pseudo Video-based Blind quality assessment metric for Light Field image (PVBLiF). In contrast to most previous Light Field Image Quality Assessment (LF-IQA) metrics, in which different types of 2D representations derived from LFI are used for quality assessment indirectly, our metric exploits a more intuitive 3D representation, named Pseudo Video Block Sequence (PVBS), to evaluate the perceptual quality of LFI. For this purpose, we first divide the LFI into a massive number of non-overlapping PVBSs, which simultaneously contain spatial and angular information of LFI. Then, we propose a novel network (named PVBSNet) based on Convolutional Neural Networks (CNNs) to extract the spatio-angular features of PVBS and further evaluate the PVBS quality. The proposed PVBSNet consists of four stages: multi-information division, intra-feature extraction, cross-feature fusion, and quality regression. Finally, a Saliency- and Variance-guided Pooling (SVPooling) method is presented to integrate all the PVBS quality into the overall quality of LFI. The proposed PVBLiF metric has been extensively evaluated on three widely-used LFI datasets: Win5-LID, NBU-LF1.0, and SHU. Experimental results demonstrate that our proposed PVBLiF metric outperforms state-of-the-art metrics and is capable of highly approximating the performance of human observers. The source code of PVBLiF is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhengyuZhang96/PVBLiF</uri> .

LFACon: Introducing Anglewise Attention to No-Reference Quality Assessment in Light Field Space.

Light field imaging can capture both the intensity information and the direction information of light rays. It naturally enables a six-degrees-of-freedom viewing experience and deep user engagement in virtual reality. Compared to 2D image assessment, light field image quality assessment (LFIQA) needs to consider not only the image quality in the spatial domain but also the quality consistency in the angular domain. However, there is a lack of metrics to effectively reflect the angular consistency and thus the angular quality of a light field image (LFI). Furthermore, the existing LFIQA metrics suffer from high computational costs due to the excessive data volume of LFIs. In this paper, we propose a novel concept of "anglewise attention" by introducing a multihead self-attention mechanism to the angular domain of an LFI. This mechanism better reflects the LFI quality. In particular, we propose three new attention kernels, including anglewise self-attention, anglewise grid attention, and anglewise central attention. These attention kernels can realize angular self-attention, extract multiangled features globally or selectively, and reduce the computational cost of feature extraction. By effectively incorporating the proposed kernels, we further propose our light field attentional convolutional neural network (LFACon) as an LFIQA metric. Our experimental results show that the proposed LFACon metric significantly outperforms the state-of-the-art LFIQA metrics. For the majority of distortion types, LFACon attains the best performance with lower complexity and less computational time.

No-Reference Light Field Image Quality Assessment Using Four-Dimensional Sparse Transform

Light field imaging can simultaneously capture the intensity and direction information of light rays in the real world. Light field image (LFI) with four-dimensional (4D) data suffers from quality degradation in the process of compression, reconstruction and processing. How to evaluate the visual quality of LFI is thought-provoking. This paper proposes a no-reference LFI quality assessment metric based on high-dimensional sparse transform. Firstly, LFI's sub-aperture gradient image array (SAGIA), which is still a 4D signal, is generated by high-pass filtering between adjacent SAIs. Then, SAGIA is transformed with 4D discrete cosine transform (4D-DCT). 4D-DCT coefficients of SAGIA can characterize the angular and spatial information of LFI. And the logarithmic amplitudes of the coefficients at the same position of SAGIA?s transformed 4D blocks are averaged as the coefficient energy. Subsequently, the 4D-DCT coefficients of SAGIA are divided into the spatial-angular frequency bands and spatial-angular orientation bands, and the corresponding energy features are extracted by converging the coefficient energy of the same band. In addition, the coefficients' amplitudes at the same position of blocks are fitted by the Weibull distribution. Then, the fitted parameters of each position are concatenated, and cropped with principal component analysis to obtain the compact features. Finally, the extracted features are pooled to predict the visual quality of the distorted LFIs. The experimental results demonstrate that the proposed method is more consistent with the subjective evaluation on three LFI databases, compared with the state-of-the-art image quality assessment methods and LFI quality assessment methods.

Blind light field image quality assessment combining disparity compensation with 3D data processing

ä¸Žä¼ ç»Ÿçš„2Då›¾åƒç›¸æ¯”ï¼Œå ‰åœºå›¾åƒè®°å½•äº†åœºæ™¯ä¸­å ‰çº¿çš„å¼ºåº¦å’Œæ–¹å‘ä¿¡æ¯ï¼Œåœ¨å¤šåª’ä½“æŠ€æœ¯åº”ç”¨é¢†åŸŸä¸­å æ®ç€é‡è¦çš„åœ°ä½ã€‚ä½†åœ¨å ‰åœºå›¾åƒçš„äº§ç”Ÿã€ä¼ è¾“ç­‰å¤„ç†è¿‡ç¨‹ä¸­ä¼šä¸å¯é¿å åœ°å¼•å ¥å¤±çœŸï¼Œå½±å“ç”¨æˆ·è§†è§‰ä½“éªŒï¼Œå› è€Œéœ€æž„å»ºæœ‰æ•ˆã€å‡†ç¡®çš„å ‰åœºå›¾åƒè´¨é‡è¯„ä»·æ–¹æ³•å¯¹å ¶è¿›è¡Œè¯„ä¼°ã€‚æœ¬æ–‡åŸºäºŽå ‰åœºå›¾åƒçš„ä¼ªè§†é¢‘å’Œæžå¹³é¢å›¾åƒæå‡ºäº†ç»“åˆè§†å·®è¡¥å¿ä¸Ž3Dæ•°æ®å¤„ç†çš„ç›²å ‰åœºå›¾åƒè´¨é‡è¯„ä»·æ–¹æ³•ã€‚é¦–å ˆï¼Œé‡‡ç”¨è§†å·®è¡¥å¿æ¨¡å—å¤„ç†å ‰åœºå›¾åƒä¼ªè§†é¢‘åºåˆ—ï¼Œç„¶åŽå¯¹ç»è¿‡è§†å·®è¡¥å¿åŽçš„ä¼ªè§†é¢‘åºåˆ—è¿›è¡Œä¸‰ç»´ç¦»æ•£å°æ³¢å˜æ¢ä»¥åŠä¸‰ç»´åŽ»å‡å€¼å¯¹æ¯”åº¦å½’ä¸€åŒ–å¤„ç†ï¼Œå¹¶æå–é¢‘åŸŸå’Œç©ºåŸŸç‰¹å¾ã€‚åŒæ—¶ï¼Œåœ¨æžå¹³é¢å›¾åƒä¸Šæå–æ–¹å‘æ¢¯åº¦ç›´æ–¹å›¾ç‰¹å¾è¡¨å¾å ¶è§’åº¦ä¿¡æ¯çš„å¤±çœŸã€‚æœ€åŽï¼Œè¿ç”¨æ”¯æŒå‘é‡æœºå»ºç«‹å›¾åƒè´¨é‡è¯„ä»·æŒ‡æ ‡åˆ°ä¸»è§‚è´¨é‡å¾—åˆ†çš„å›žå½’æ¨¡åž‹ã€‚æ‰€æç®—æ³•åœ¨å ¬å¼€æ•°æ®åº“NBU-LF1.0，Win5-LID和SHU上的PLCC分别达到0.886 1，0.928 7和0.976 9ï¼Œå®žéªŒç»“æžœè¡¨æ˜Žï¼Œä¸Žç»å ¸çš„äºŒç»´å›¾åƒè´¨é‡è¯„ä»·æ–¹æ³•ä»¥åŠå ˆè¿›çš„å ‰åœºå›¾åƒè´¨é‡è¯„ä»·æ–¹æ³•ç›¸æ¯”ï¼Œæœ¬æ–‡æ–¹æ³•ä¸Žä¸»è§‚è´¨é‡è¯„ä»·ç»“æžœçš„ä¸€è‡´æ€§æ›´é«˜ã€‚

Light‐field fundus imaging under astigmatism – An eye model study

AbstractPurposeOne strong risk factor for glaucoma is a suspicious optic nerve head appearance and an increase in depth. With the upcoming light‐field (LF) technology one‐shot 3D retinal imaging and measuring is possible. Due to optical aberrations such as astigmatism, the imaged depth can be distorted. The study presented here investigated the influence of astigmatism on LF image quality and depth estimation using an eye model.MethodsWe developed a new eye model that added astigmatism by integrating an additional cylindrical lens between the cornea and lens. We used cylindrical lenses from −4 to 0 dpt in 1 dpt steps in vertical and horizontal orientations. The fundus was modelled as a half hollow sphere with a radius of 10 mm and a central papilla with 0.8 mm depth. The fundus was coated with red acrylic lacquer, and vessels were drawn with a thin brush with black acrylic lacquer. LF fundus images (15 for each cylindrical lens and orientation) were taken with a fundus camera (FF450, Carl Zeiss Meditec AG, Germany) equipped with an industrial LF camera (RX12, Raytrix GmbH, Germany). Depth measurements were performed with the Raytrix LF software (RxLive 5.0.046.0) as difference from a vertical vessel to a horizontal orientated vessel. The same measuring point was used for all images.ResultsThe LF images taken with a cylindrical lens show a visually poorer image quality. Depth estimates were only possible in the central image area. The fundus images show increasing depth differences between vertical and horizontal vessels with increasing astigmatism up to 0.37 mm at −4 dpt in contrast to the flat imaged fundus without cylindrical lens. The sign of the depth differences changes when the orientation of the inserted cylindrical lens is changed.ConclusionsThe imaged LF of a fundus of an astigmatic eye model is measurably distorted and affects vertical and horizontal structures differently. This must be taken into account when evaluating and interpreting LF depth measurements.

Blind Light Field Image Quality Assessment with Tensor Color Domain and 3d Shearlet Transform

Blind Light Field Image Quality Assessment with Tensor Color Domain and 3d Shearlet Transform

Objective Quality Assessment of Lenslet Light Field Image Based on Focus Stack

The large amount of complex scene information recorded by light field imaging has the potential for immersive media applications. Compression and reconstruction algorithms are crucial for the transmission, storage, and display of such massive data. Most of the existing quality evaluation indexes do not effectively account for light field characteristics. To accurately evaluate the distortions caused by compression and reconstruction algorithms, it is necessary to construct an image evaluation index that reflects the angular-spatial characteristics of the light field. This work proposes a full-reference light field image quality evaluation index that attempts to extract less information from the focus stack to accurately evaluate the entire light field quality. The proposed framework includes three specific steps. First, we construct a key refocused image extraction framework by the maximal spatial information contrast and the minimal angular information variation. Specifically, the gradient and phase congruency operators are used in the extraction framework. Second, a novel light field quality evaluation index is built based on the angular-spatial characteristics of the key refocused images. In detail, the features used in the key refocused image extraction framework and the chrominance feature are combined to construct the union feature. Third, the similarity of the union feature is pooled by the relevant visual saliency map to obtain the predicted score. Finally, the overall quality of the light field is measured by applying the proposed index to the key refocused images. The high efficiency and precision of the proposed method are shown by extensive comparison experiments.

Light Field Image Quality Assessment With Auxiliary Learning Based on Depthwise and Anglewise Separable Convolutions

In multimedia broadcasting, no-reference image quality assessment (NR-IQA) is used to indicate the user-perceived quality of experience (QoE) and to support intelligent data transmission while optimizing user experience. This paper proposes an improved no-reference light field image quality assessment (NR-LFIQA) metric for future immersive media broadcasting services. First, we extend the concept of depthwise separable convolution (DSC) to the spatial domain of light field image (LFI) and introduce “light field depthwise separable convolution (LF-DSC)”, which can extract the LFI’s spatial features efficiently. Second, we further theoretically extend the LF-DSC to the angular space of LFI and introduce the novel concept of “light field anglewise separable convolution (LF-ASC)”, which is capable of extracting both the spatial and angular features for comprehensive quality assessment with low complexity. Third, we define the spatial and angular feature estimations as auxiliary tasks in aiding the primary NR-LFIQA task by providing spatial and angular quality features as hints. To the best of our knowledge, this work is the first exploration of deep auxiliary learning with spatial-angular hints on NR-LFIQA. Experiments were conducted in mainstream LFI datasets such as Win5-LID and SMART with comparisons to the mainstream full reference IQA metrics as well as the state-of-the-art NR-LFIQA methods. The experimental results show that the proposed metric yields overall 42.86% and 45.95% smaller prediction errors than the second-best benchmarking metric in Win5-LID and SMART, respectively. In some challenging cases with particular distortion types, the proposed metric can reduce the errors significantly by more than 60%.

Pseudo Video and Refocused Images-Based Blind Light Field Image Quality Assessment

The commercial light field camera is able to capture four-dimensional Light Field Image (LFI), which can be visualized to LFI contents on 2D displays by means of the Pseudo Video (PV) or the Refocused Images (RIs) generated with the refocusing function of LFI. However, the quality degradation of LFI will affect user’s visual experience of LFI contents. Hence, it is crucial to develop an effective LFI quality assessment method to monitor the LFI quality. Most existing subjective databases of LFI use PV and RIs visualization techniques to assess the quality of LFI. Therefore, as the way of presenting LFI on 2D display, PV and RIs are closely related to the subjective perception of LFI by human eyes. Based on these two visualization techniques, this article proposes a novel PV and RIs based blind LFI quality assessment method, in which the feature extraction is divided into two parts. In the first part, the PV’s structure, motion and disparity information are extracted with multi-scale and multi-directional Shearlet transform. In the other part, the spatial structure, depth and semantic information of the RIs are obtained. Finally, support vector regression is used to nonlinear map the perceptual features to quality score of LFI. The experimental results on four LFI databases show that the proposed method has better correlation with human visual perception, compared with the classical 2D image quality assessment methods as well as the state-of-the-art LFI quality assessment methods.