Laboratory For Image And Video Engineering Research Articles

A Study of Subjective and Objective Quality Assessment of HDR Videos.

As compared to standard dynamic range (SDR) videos, high dynamic range (HDR) content is able to represent and display much wider and more accurate ranges of brightness and color, leading to more engaging and enjoyable visual experiences. HDR also implies increases in data volume, further challenging existing limits on bandwidth consumption and on the quality of delivered content. Perceptual quality models are used to monitor and control the compression of streamed SDR content. A similar strategy should be useful for HDR content, yet there has been limited work on building HDR video quality assessment (VQA) algorithms. One reason for this is a scarcity of high-quality HDR VQA databases representative of contemporary HDR standards. Towards filling this gap, we created the first publicly available HDR VQA database dedicated to HDR10 videos, called the Laboratory for Image and Video Engineering (LIVE) HDR Database. It comprises 310 videos from 31 distinct source sequences processed by ten different compression and resolution combinations, simulating bitrate ladders used by the streaming industry. We used this data to conduct a subjective quality study, gathering more than 20,000 human quality judgments under two different illumination conditions. To demonstrate the usefulness of this new psychometric data resource, we also designed a new framework for creating HDR quality sensitive features, using a nonlinear transform to emphasize distortions occurring in spatial portions of videos that are enhanced by HDR, e.g., having darker blacks and brighter whites. We apply this new method, which we call HDRMAX, to modify the widely-deployed Video Multimethod Assessment Fusion (VMAF) model. We show that VMAF+HDRMAX provides significantly elevated performance on both HDR and SDR videos, exceeding prior state-of-the-art model performance. The database is now accessible at: https://live.ece.utexas.edu/research/LIVEHDR/LIVEHDR_index.html. The model will be made available at a later date at: https://live.ece.utexas.edu//research/Quality/index_algorithms.htm.

No-Reference Blur Assessment Based on Re-Blurring Using Markov Basis

Blur is produced in a digital image due to low pass filtering, moving objects or defocus of the camera lens during capture. Image viewers are annoyed by blur artefact and the image's perceived quality suffers as a result. The high-quality input is relevant to communication service providers and imaging product makers because it may help them improve their processes. Human-based blur assessment is time-consuming, expensive and must adhere to subjective evaluation standards. This paper presents a revolutionary no-reference blur assessment algorithm based on re-blurring blurred images using a special mask developed with a Markov basis and Laplace filter. The final blur score of blurred images has been calculated from the local variation in horizontal and vertical pixel intensity of blurred and re-blurred images. The objective scores are generated by applying proposed algorithm on the two image databases i.e., Laboratory for image and video engineering (LIVE) database and Tampere image database (TID 2013). Finally, on the basis of objective and subjective scores performance analysis is done in terms of Pearson linear correlation coefficient (PLCC), Spearman rank-order correlation coefficient (SROCC), Mean absolute error (MAE), Root mean square error (RMSE) and Outliers ratio (OR). The existing no-reference blur assessment algorithms have been used various methods for the evaluation of blur from no-reference image such as Just noticeable blur (JNB), Cumulative Probability Distribution of Blur Detection (CPBD) and Edge Model based Blur Metric (EMBM). The results illustrate that the proposed method was successful in predicting high blur scores with high accuracy as compared to existing no-reference blur assessment algorithms such as JNB, CPBD and EMBM algorithms.

Study of the Subjective and Objective Quality of High Motion Live Streaming Videos.

Video livestreaming is gaining prevalence among video streaming service s, especially for the delivery of live, high motion content such as sport ing events. The quality of the se livestreaming videos can be adversely affected by any of a wide variety of events, including capture artifacts, and distortions incurred during coding and transmission. High motion content can cause or exacerbate many kinds of distortion, such as motion blur and stutter. Because of this, the development of objective Video Quality Assessment (VQA) algorithms that can predict the perceptual quality of high motion, live streamed videos is greatly desired. Important resources for developing these algorithms are appropriate databases that exemplify the kinds of live streaming video distortions encountered in practice. Towards making progress in this direction, we built a video quality database specifically designed for live streaming VQA research. The new video database is called the Laboratory for Image and Video Engineering (LIVE) Livestream Database. The LIVE Livestream Database includes 315 videos of 45 source sequences from 33 original contents impaired by 6 types of distortions. We also performed a subjective quality study using the new database, whereby more than 12,000 human opinions were gathered from 40 subjects. We demonstrate the usefulness of the new resource by performing a holistic evaluation of the performance of current state-of-the-art (SOTA) VQA models. We envision that researchers will find the dataset to be useful for the development, testing, and comparison of future VQA models. The LIVE Livestream database is being made publicly available for these purposes at https://live.ece. utexas.edu/research/LIVE_APV_Study/apv_index.html.

Fusing deep learning and statistical visual features for no-reference image quality assessment

Fusing the pertinence of natural scene statistics-based methods and the ubiquity of convolutional neural network-based methods, a no-reference image quality assessment (IQA) method fusing deep learning and statistical visual features for no-reference image quality assessment (FDSVIQA) is proposed. For the statistical visual features, a local normalized luminance map and a local normalized local binary pattern (LBP) map of the image are constructed, and the local normalized luminance features and the gradient-weighted local normalized LBP features are extracted on the two maps, respectively. These two kinds of features are concatenated to build the image statistical visual features. For deep learning, the local normalized luminance block and the localized normalized LBP block are input into a double-path deep learning network, and the statistical visual features are input into the deep learning network to be integrated with the depth features. After learning and training, IQA is achieved. The performance of the proposed FDSVIQA algorithm is tested on the Laboratory for Image and Video Engineering (LIVE), LIVE Multiply Distorted Image Quality Database, and Multiply Distortion Optics Remote Sensing Image databases. Experimental results show that the FDSVIQA algorithm has excellent subjective and objective consistency and good robustness for both distorted natural images and distorted remote sensing images. In addition, the FDSVIQA has database independence.

Spatial and Temporal Feature-Based Reduced Reference Quality Assessment for Rate-Varying Videos in Wireless Networks

For the impact of the bitrate change of video streaming services according to the available bandwidth on user satisfaction, in this paper, we propose a spatial and temporal feature-based reduced reference (RR) quality assessment for rate-varying videos in wireless networks called STRQAW. First, simulating the orientation selectivity mechanism of the human visual system (HVS), the histogram of the orientation selectivity-based visual pattern in each frame is extracted as the spatial feature. The histogram similarity between the rate-varying video and the original video is computed as the spatial metric. Second, we extract the temporal variation of the DCT coefficients of the consecutive frame differences as the temporal feature. The temporal variation similarity between the rate-varying video and the original video is calculated as the temporal metric. Finally, we take into account the recency effect and assess the overall quality by combining the temporal and spatial metric. The experimental results using the Laboratory for Image and Video Engineering (LIVE) mobile video quality assessment (VQA) database show that STRQAW is consistent with the subjective assessment results, which means it reflects human subjective feelings well and it provides an evaluation for adjusting compression-coding rates in real time. STRQAW can be used to guide video application providers and network operators working towards satisfying end-user experiences.

No-Reference Objective Video Quality Measure for Frame Freezing Degradation.

In this paper we present a novel no-reference video quality measure, NR-FFM (no-reference frame–freezing measure), designed to estimate quality degradations caused by frame freezing of streamed video. The performance of the measure was evaluated using 40 degraded video sequences from the laboratory for image and video engineering (LIVE) mobile database. Proposed quality measure can be used in different scenarios such as mobile video transmission by itself or in combination with other quality measures. These two types of applications were presented and studied together with considerations on relevant normalization issues. The results showed promising correlation values between the user assigned quality and the estimated quality scores.

On Hypothesis Testing for Comparing Image Quality Assessment Metrics [Tips & Tricks

In developing novel image quality assessment (IQA) metrics, researchers should compare their proposed metrics with state-of-the-art metrics. A commonly adopted approach is by comparing two residuals between the nonlinearly mapped scores of two IQA metrics and the difference mean opinion score, which are assumed from Gaussian distributions with zero means. An F-test is then used to test the equality of variances of the two sets of residuals. If the variances are significantly different, then we conclude that the residuals are from different Gaussian distributions and that the two IQA metrics are significantly different. The F-test assumes that the two sets of residuals are independent. However, given that the IQA metrics are calculated on the same database, the two sets of residuals are paired and may be correlated. We note this improper usage of the F-test by practitioners, which can result in misleading comparison results of two IQA metrics. To solve this practical problem, we introduce the Pitman test to investigate the equality of variances for two sets of correlated residuals. Experiments on the Laboratory for Image and Video Engineering (LIVE) database show that the two tests can provide different conclusions.

Deep Activation Pooling for Blind Image Quality Assessment

Driven by the rapid development of digital imaging and network technologies, the opinion-unaware blind image quality assessment (BIQA) method has become an important yet very challenging task. In this paper, we design an effective novel scheme for opinion-unaware BIQA. We first utilize the convolutional maps to select high-contrast patches, and then we utilize these selected patches of pristine images to train a pristine multivariate Gaussian (PMVG) model. In the test stage, each high-contrast patch is fitted by a test MVG (TMVG) model, and the local quality score is obtained by comparing with the PMVG. Finally, we propose the deep activation pooling (DAP) to automatically emphasize the more important scores and suppress the less important ones so as to obtain the overall image quality score. We verify the proposed method on two widely used databases, that is, the computational and subjective image quality (CSIQ) and the laboratory for image and video engineering (LIVE) databases, and the experimental results demonstrate that the proposed method achieves better results than the state-of-the-art methods.

A Perception-Based Hybrid Model for Video Quality Assessment

It is known that the human visual system (HVS) employs independent processes (distortion detection and artifact perception—also often referred to as near-threshold and suprathreshold distortion perception) to assess video quality for various distortion levels. Visual masking effects also play an important role in video distortion perception, especially within spatial and temporal textures. In this paper, a novel perception-based hybrid model for video quality assessment is presented. This simulates the HVS perception process by adaptively combining noticeable distortion and blurring artifacts using an enhanced nonlinear model. Noticeable distortion is defined by thresholding absolute differences using spatial and temporal tolerance maps that characterize texture masking effects, and this makes a significant contribution to quality assessment when the quality of the distorted video is similar to that of the original video. Characterization of blurring artifacts, estimated by computing high frequency energy variations and weighted with motion speed, is found to further improve metric performance. This is especially true for low quality cases. All stages of our model exploit the orientation selectivity and shift invariance properties of the dual-tree complex wavelet transform. This not only helps to improve the performance but also offers the potential for new low complexity in-loop application. Our approach is evaluated on both the Video Quality Experts Group (VQEG) full reference television Phase I and the Laboratory for Image and Video Engineering (LIVE) video databases. The resulting overall performance is superior to the existing metrics, exhibiting statistically better or equivalent performance with significantly lower complexity.

Reduced-reference image quality assessment using moment method

ABSTRACTReduced-reference image quality assessment (RR IQA) aims to evaluate the perceptual quality of a distorted image through partial information of the corresponding reference image. In this paper, a novel RR IQA metric is proposed by using the moment method. We claim that the first and second moments of wavelet coefficients of natural images can have approximate and regular change that are disturbed by different types of distortions, and that this disturbance can be relevant to human perceptions of quality. We measure the difference of these statistical parameters between reference and distorted image to predict the visual quality degradation. The introduced IQA metric is suitable for implementation and has relatively low computational complexity. The experimental results on Laboratory for Image and Video Engineering (LIVE) and Tampere Image Database (TID) image databases indicate that the proposed metric has a good predictive performance.

Reduced reference blockiness and blurriness meter for image quality assessment

This research work demonstrates an image quality assessment meter (IQAM) in reduced reference (RR) mode for discrete cosine transform (DCT) based compressed images. The DCT based image compression technique is lossy. As a result, it introduces blockiness and blurriness artefacts in the coded image. To send the image in image transmission systems, the system bandwidth is limited, and image data need to be adapted accordingly. To determine the quality of image at the recipient end, fewer image parameters are sent rather than the complete reference image. The aim of this paper is to suggest a method for the receiver to evaluate the objective received image quality. The presented RR quality meter operates in the frequency domain for the estimation of artefacts in the compressed image. The quality meter takes the image, performs edge detection, transforms it into the frequency domain and estimates the harmonics. The estimated strength of harmonics in the compressed image is compared with the received RR parameter for blockiness and blurriness distortions. For improved quality judgment, the combined effect of both these artefacts is measured as well. Peak signal to noise ratio (PSNR) is another most familiar method of image quality assessment. The received image quality is also observed by measuring the PSNR. The objective values of the artefacts obtained by the designed RR IQAM algorithm are correlated with the subjective differential mean opinion scores (DMOS) blockiness and blurriness values of Laboratory for Image and Video Engineering (LIVE) image database2. The results obtained are well matched and give 93–94% correlation with the LIVE database values, whereas the correlation obtained by PSNR values with LIVE DMOS values is only around 70%. This proves the practical application of the designed IQAM in RR.

Multifocus Image Fusion based on Human Visual Perception

The process by which two or more images are merged into a single image is called image fusion, where important characteristics from each of the original image are revived. As images are acquired from different instrument modalities, in order to combine all the capture techniques fusion of image forms a fundamental process. Multifocus image fusion constructs an combined image from multiple source images having focus on different objects from same scene. To achieve this, a spatial domain algorithm is proposed which divides each source image into blocks of sizes varying adaptively. Edge information is extracted from the image by using edge detection techniques. The quality metrics will be obtained for each block, based on human visual perception instead of simple metrics like MSE and PSNR. For the purpose of testing the proposed work, a readily available database of Laboratory for Image and Video Engineering (LIVE) will be used. To demonstrate the quality of the final fused image, evaluation will be done based on the concepts of human visual perception.

Temporal Video Quality Model Accounting for Variable Frame Delay Distortions

We announce a new video quality model (VQM) that accounts for the perceptual impact of variable frame delays (VFD) in videos with demonstrated top performance on the laboratory for image and video engineering (LIVE) mobile video quality assessment (VQA) database. This model, called VQM_VFD, uses perceptual features extracted from spatial-temporal blocks spanning fixed angular extents and a long edge detection filter. VQM_VFD predicts video quality by measuring multiple frame delays using perception based parameters to track subjective quality over time. In the performance analysis of VQM_VFD, we evaluated its efficacy at predicting human opinions of visual quality. A detailed correlation analysis and statistical hypothesis testing show that VQM_VFD accurately predicts human subjective judgments and substantially outperforms top-performing image quality assessment and VQA models previously tested on the LIVE mobile VQA database. VQM_VFD achieved the best performance on the mobile and tablet studies of the LIVE mobile VQA database for simulated compression, wireless packet-loss, and rate adaptation, but not for temporal dynamics. These results validate the new model and warrant a hard release of the VQM_VFD algorithm. It is freely available for any purpose, commercial, or noncommercial at http://www.its.bldrdoc.gov/vqm/ .

Image quality assessment based on multi-scale representation of structure

This paper presents an image quality assessment algorithm using representation of image structures in scale space. It is based on the finding that difference-of-Gaussian (DoG) can capture the structures of an image with flexibility and it is sensitive to image degradations. A set of DoG signals are first computed to represent the image structures at different octaves and scales. A coarse quality score is calculated by comparing the DoG signals between the reference image and test image at each octave and scale. Information content weighting is then incorporated to generate the final quality score. Experimental results on five singly-distorted databases and the Laboratory for Image and Video Engineering (LIVE) multiply-distorted database show the effectiveness of the proposed method.

A SURVEY ON EFFICIENT NO-REFERENCE BLUR ESTIMATION METHODS

Blur estimation in image processing has come to be of great importance in assessing the quality of ima ges. This work presents a survey of no-reference blur estimation methods. A n o-reference method is particularly useful, when the input image used for blur estimation does not have an available corresponding reference image. This paper provides a comparison of the methodologies of four no-reference blur estimation methods. The first met hod applies a scale adaptive technique of blur esti mation to get better accuracy in the results. The second blur metric involves findin g the energy using second order derivatives of an i mage using derivative pair of quadrature filters. The third blur metric is based on the kurtosis measurement in the discrete dyadic wavelet transform (DDWT) of the images. The fourth method of blur estimation is obt ained by finding the ratio of sum of the edge width s of all the detected edges to the total number of edges. The results provided are use ful in comparing the methods based on metrics like Spearman correlation coefficient. The results are obtained by evaluation on images from the Laboratory for Image and Video Engineering (LIVE) database. The various methods are evaluated on the images by adding varying content of noise. The performance is evaluated for 3 different categories namely Gaussian blur, motion blur and al so JPEG2000 compressed images. Blur estimation fin ds its application in quality assessment, image fusion and auto-focusing in image s. The sharpness of an image can also be found from the blur metric as sharpness is inversely proportional to blur. Sharpness metric s can also be combined with other metrics to measur e overall quality of the image.

Study of Subjective and Objective Quality Assessment of Video

We present the results of a recent large-scale subjective study of video quality on a collection of videos distorted by a variety of application-relevant processes. Methods to assess the visual quality of digital videos as perceived by human observers are becoming increasingly important, due to the large number of applications that target humans as the end users of video. Owing to the many approaches to video quality assessment (VQA) that are being developed, there is a need for a diverse independent public database of distorted videos and subjective scores that is freely available. The resulting Laboratory for Image and Video Engineering (LIVE) Video Quality Database contains 150 distorted videos (obtained from ten uncompressed reference videos of natural scenes) that were created using four different commonly encountered distortion types. Each video was assessed by 38 human subjects, and the difference mean opinion scores (DMOS) were recorded. We also evaluated the performance of several state-of-the-art, publicly available full-reference VQA algorithms on the new database. A statistical evaluation of the relative performance of these algorithms is also presented. The database has a dedicated web presence that will be maintained as long as it remains relevant and the data is available online.

No‐Reference Image Quality Evaluation Model for JPEG and JPEG2000 Images

AbstractIn this paper, we present a new no‐reference (NR) image quality evaluation model for Joint Photographic Experts Group (JPEG) and JPEG2000 coded images. The proposed model is based on the blockiness around the block boundary, average absolute difference between adjacent pixels within the block, and zero‐crossing (ZC) rate within the block of the image. Subjective experimental results of the Laboratory for Image and Video Engineering (LIVE) Image Quality Assessment Database were used to train and test the model, which achieved sufficient quality prediction performance. Copyright © 2008 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.