Subjective Assessment Methodology Research Articles

Environmental noise annoyance trends In selected locations In Bratislava

The rapid development in traffic density and the economic transformation since 1989 has brought new problems concerning road traffic noise in Slovakia. The aim is to follow the time trends of noise annoyance in monitored localities in Bratislava at time intervals of 10, 20, and 30 years. We used the validated methodology for subjective assessment of noise annoyance in young and healthy individuals, as well as a method of objectification by direct measurement of sound levels. Respondents (n=3,465) were university students, living in the exposed and control dormitories representing a homogenous sample. The sharp increase in traffic noise burden in the exposed area was found at the first 10-year interval (LAeq=67.5 dB). A slight decrease occurred after 2014, and in 2019 up to LAeq=63.9 dB. A sharp increase in road traffic noise annoyance was observed in the first 10-year interval (ORMH=2.56 (95 % CI=1.93-3.42) vs 6.01 (95% CI=4.97-7.95) with a slightly decreasing trend in 2019. An increase in noise annoyance from entertainment facilities was observed as well. Despite a slightly declining trend, road traffic noise annoyance is still an important issue and there is a need for preventive measures to reduce such exposure in residential areas.

Stimuli Generation for Quality-of-Experience Evaluation of Mobile Applications

The assessment of Quality-of-Experience (QoE) corresponding to a given set of Quality-of-Service (QoS) parameters is of high importance for any mobile network operator. Methods for subjective QoE assessment such as MUlti Stimulus test with Hidden Reference and Anchors (MUSHRA) or Subjective Assessment Methodology for Video Quality (SAMVIQ) are based on collecting user feedback for one or several stimuli, i.e. differently processed versions of a source. 
 Generating these stimuli corresponding to a given set of QoS parameters for a QoE assessment of a mobile application is a time-consuming, non-trivial task. Therefore, in this paper we propose a novel publicly available open-source framework for stimuli generation for QoE assessment of mobile applications. It is based on the combination of network emulation and automatic user-interface control of a real mobile device, whose behavior is recorded and post-processed to generate suitable stimuli. 
 The article presents the basic concept of the framework, describes its open-source implementation, and concludes with an initial evaluation of the framework based on typical types of mobile applications.

Comparison of Subjective Methods for Quality Assessment of 3D Graphics in Virtual Reality

Numerous methodologies for subjective quality assessment exist in the field of image processing. In particular, the Absolute Category Rating with Hidden Reference (ACR-HR), the Double Stimulus Impairment Scale (DSIS), and the Subjective Assessment Methodology for Video Quality (SAMVIQ) are considered three of the most prominent methods for assessing the visual quality of 2D images and videos. Are these methods valid/accurate to evaluate the perceived quality of 3D graphics data? Is the presence of an explicit reference necessary, due to the lack of human prior knowledge on 3D graphics data compared to natural images/videos? To answer these questions, we compare these three subjective methods (ACR-HR, DSIS, and SAMVIQ) on a dataset of high-quality colored 3D models, impaired with various distortions. These subjective experiments were conducted in a virtual reality environment. Our results show differences in the performance of the methods depending on the 3D contents and the types of distortions. We show that DSIS and SAMVIQ outperform ACR-HR in terms of accuracy and point out a stable performance. In regard to the time-effort, DSIS achieves the highest accuracy in the shortest assessment time. Results also yield interesting conclusions on the importance of a reference for judging the quality of 3D graphics. We finally provide recommendations regarding the influence of the number of observers on the accuracy.

AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience

User expectations have a crucial impact on the levels of quality of experience (QoE) that they consider acceptable or satisfying. Measuring acceptability and annoyance has mainly been performed in separate or multi-step experiments without any control over participants’ expectations. This paper introduces a simple methodology to obtain the information about both of the entities in a single step and compares several data processing strategies useful for results interpretation. A specifically designed subjective experiment, conducted on compressed videos, has shown that the multi-step procedures could be replaced by our proposed single-step approach, regardless of the viewing conditions, while the novel approach is significantly preferred by observers for its low time requirements and higher intuitiveness. The test has simultaneously proven that user expectations can be altered by the instructions and it is, therefore, possible to simulate different user profiles regardless of the participants’ real habits. The acceptability/annoyance experimental results are also used to benchmark the state-of-the-art objective video quality metrics in predicting acceptability/annoyance of QoE. A case study on the determination of the threshold of acceptability/annoyance for objective quality metrics is conducted, which can be served as a guideline for video streaming service providers.

Optimal inter-view rate allocation for multi-view video plus depth over MPEG-DASH using QoE measures and paired comparison

Rapid advances in compression and transmission technologies enabled multi-view video streaming over the Internet through state-of-the-art MPEG standards such as dynamic adaptive streaming over HTTP (DASH) and high-efficiency video coding. Yet, standards for subjective and objective measurement of quality of experience (QoE) are lagging in the area. The MPEG-DASH system requires constant bitrate (CBR) coding and an efficient rate control strategy in order to increase QoE. Multi-view video plus depth format is preferable due to high compression gain but quality of intermediate view is crucial, strictly depending on performance of inter/intra-view rate allocation as well as rendering algorithms. We propose to find optimal ratio for inter-view rate allocation in CBR using our previously developed objective QoE measures using depth maps and structural similarities. To assess small differences and discriminate similar quality levels, a new subjective assessment methodology is also proposed as a multiple stimuli plus simultaneous presentation for paired comparison. The results show that the unequal rate allocation is superior to the equal rate allocation in terms of objective QoE as 0.012–0.04 (which translates into a bit rate saving of 15–24%) and subjective QoE as 0.3–0.5. Furthermore, a very high correlation is achieved between the new QoE measures and paired comparison results.

Performance analysis of a video quality ruler methodology for subjective quality assessment

Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.

Quality of experience measurement of compressed multi-view video

Quality of Experience (QoE) measurement in stereoscopic 3D is a challenging topic and existing standards such as ITU-R BT.500 may no longer be adequate for reliability and efficiency. Yet, objective measurement of QoE is not well researched for 3D images and videos. We present a new subjective assessment methodology providing better time-efficiency and more accurate scores at the same or even less number of assessors, using new objective metrics for QoE measurement of compressed multi-view video. Our research focus is twofold. First, the novel subjective testing methodology is developed, and QoE of different multi-view video compression technologies is evaluated. A time-efficient solution is proposed as a multiple stimuli plus simultaneous presentation extension of the conventional Double Stimulus Continuous Quality Scale (DSCQS) method. Second, objective quality metrics are proposed based on structural similarity and depth maps. Experiments are conducted on an auto-stereoscopic display while H.264/MPEG-4 Advanced Video Coding (AVC)- and High Efficiency Video Coding (HEVC)-based encoders are used for compression. The results demonstrate that the new methodology is superior to the DSCQS method in terms of both evaluation time and accuracy. Furthermore, high correlation is achieved with the new objective quality metrics.

Quality of Experience of adaptive video streaming: Investigation in service parameters and subjective quality assessment methodology

The usage of HTTP adaptive streaming (HAS) has become widely spread in multimedia services. Because it allows the service providers to improve the network resource utilization and user׳s Quality of Experience (QoE). Using this technology, the video playback interruption is reduced since the network and server status in addition to capability of user device, all are taken into account by HAS client to adapt the quality to the current condition. Adaptation can be done using different strategies. In order to provide optimal QoE, the perceptual impact of adaptation strategies from point of view of the user should be studied. However, the time-varying video quality due to the adaptation which usually takes place in a long interval introduces a new type of impairment making the subjective evaluation of adaptive streaming system challenging. The contribution of this paper is two-fold: first, it investigates the testing methodology to evaluate HAS QoE by comparing the subjective experimental outcomes obtained from ACR standardized method and a semi-continuous method developed to evaluate the long sequences. In addition, influence of using audiovisual stimuli to evaluate the video-related impairment is inquired. Second, impact of some of the adaptation technical factors including the quality switching amplitude and chunk size in combination with high range of commercial content type is investigated. The results of this study provide a good insight toward achieving appropriate testing method to evaluate HAS QoE, in addition to designing switching strategies with optimal visual quality.

Impact of Accuracy and Latency on Mean Opinion Scores for Speech Recognition Solutions

Speech recognition is no longer a technology of the future and is now broadly adopted in many products. Some solutions use low power, always on keyword spotting techniques to wake up the device before engaging the large vocabulary continuous speech recognition engine. This staged approach decreases power consumption and increases noise robustness. This paper presents a study that tested the effects of accuracy and latency on subjective ratings for a keyword triggered speech solution. A specialized software framework was developed in which the accuracy and latency of tasks were systematically controlled to understand the impact on user experience. A user interface was developed, based on existing industry solutions to simulate realistic use case scenarios. A within-subjects design was employed, in which data from a total of 47 participants was collected. Participants were asked to rate their experience on a five point scale (5 = excellent, 4 = good, 3 = fair, 2 = poor, 1 = bad). The experimental design followed an ITU MOS methodology for subjective assessment. There were three different latencies and four different accuracy levels for a total of 12 combinations. A two-way repeated measures analysis of variance indicated that the mean subjective ratings were a strong function of accuracy, and a weak function of latency. The relationship between mean perceptual ratings, accuracy, and latency were uncovered. With minimal degradation to accuracy, participants had high tolerance for latencies with average experience ratings in the ‘good’ range even for latencies up to 4seconds. Participants had a low tolerance when accuracy dropped to 70% or below, with average experience ratings below the ‘good’ range. In addition, thresholds were established for the upper acceptability bound of latency using the time until repeat of a command. These metrics and methods provide key insights to set user-centric design targets and inform architectural optimizations.

Mobile Network QoE-QoS Decision Making Tool for Performance Optimization in Critical Web Service

Regardless of the type of service that a company offers the customer satisfaction is a factor for success, if these services are in a highly competitive environment. This situation encourages companies to develop strategies to improve the Quality of the Experience (QoE) of their users. Strategies include improving their processes, or infrastructure for provisioning the services. Take these kind of decisions is very difficult because they ignore how the Key Performance Indicators (KPI) services are correlated with the information about user experience. This problem is approached from the perspective of mobile telecom operators, who have addressed this challenge through the Quality of Service (QoS) concept. Unfortunately, the QoS is only characterized by technical aspects, the user’s criteria are not included. Into a highly competitive environment, the user’s loyalty is a key component to be considered in the operator’s development plan. Nowadays, the mobile telecom operators focus their efforts to ensure not only the QoS but also the QoE. The aim of this paper was the develop a decision making tool that allows the mobile telco operators support their determinations about the maintenance of network infrastructure, as well as the expansion of the same, specifically for their critical web services; based in a correlated information between QoS and QoE. This tool was developed on the basis of the Pseudo Subjective Quality Assessment (PSQA) methodology.

Identifying Prospective Biomarkers for Cognitive Impairments during Pregnancy ? Review of Current Status and Some Preliminary Results

Pregnancy, childbirth, and early motherhood physiologically and psychologically affect a woman’s cognitive parameters. Hormonal alterations during early pregnancy, enlargement of the fetus during late pregnancy, and newborn with random sleep-wake patterns all contribute to cognitive parameters. Cognitive impairments are frequent in pregnancy and it is coupled with a range of undesirable outcomes, including maternal death. Recent studies indicate that pregnant women with cognitive impairment, mild to severe, are at increased risk for spontaneous abortion, preterm labor and for having a growth-retarded baby. The majority of older studies are cross-sectional designs, with comparisons to age-matched healthy controls. Many studies suffer from small sample sizes, lack statistical power for consistent interpretation and replication of findings about cognitive impairments during pregnancy. More recent studies are longitudinal designs to describe changes in cognitive parameters during the course of pregnancy. However, women’s baseline, pre pregnancy, psychiatric, neurological, and genetically inherited parameters are not considered. With very few published reports of cognitive complexities during pregnancy, it is very difficult to understand the implications of altered cognitive parameters patterns on postpartum period and women’s health outcomes. In prenatal period, a woman is in frequent contact with health professionals and it provides an opportunity to predict and prevent pregnancy complications. Early identification, appropriate referral to a specialist for pregnancy complications is key requirements to manage pregnancy outcome successfully. This requires the coordinated participation of professionals in primary care, maternity services and mental health services to work closely with women and their families. The purpose of this study is to review the current status of research being done in this nescient area and propose prospective biomarkers for cognitive impairments during pregnancy with reference to psychiatric parameters during pregnancy. The comprehensive model of accepted interrelationships between psycho - physiological, subjective, and objective assessment methodologies are presented. We identified a need for developing potential diagnostic instruments to screen pregnancy complications with more objective based methods and we present here some of the preliminary results of our work that has been done in this nascent area.

Study of Rating Scales for Subjective Quality Assessment of High-Definition Video

With the constant evolution of video technology and the deployment of new video services, content providers and broadcasters always face the challenge of delivering an adequate video quality which meets end-users expectations. The development of reliable quality testing and quality monitoring tools that can be used by broadcasters ultimately requires reliable objective video quality metrics. In turn, the validation of these objective models requires reliable subjective assessment, the most accurate representation of the quality perceived by end-users. Many different subjective assessment methodologies exist, and each has its advantages and drawbacks. One important element in a subjective testing methodology is the choice of the rating scale. In this paper, we make a direct comparison between four scales, which are either included in existing international standards or proposed to be used in future standardization activities related to video quality. We examine the subjective data from the points of view of response behavior from participants, similarity and variability of subjective scores. We discuss these results within the context of the subjective quality assessment of high-definition video compressed and transmitted over error-prone networks. Our experimental data show no overall statistical differences between the different scales. Results also show that the single-stimulus presentation provides highly repeatable results even if different scales or groups of participants are used.

Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments

The ever growing bandwidth in access networks, in combination with IPTV and video on demand (VoD) offerings, opens up unlimited possibilities to the users. The operators can no longer compete solely on the number of channels or content and increasingly make high definition channels and quality of experience (QoE) a service differentiator. Currently, the most reliable way of assessing and measuring QoE is conducting subjective experiments, where human observers evaluate a series of short video sequences, using one of the international standardized subjective quality assessment methodologies. Unfortunately, since these subjective experiments need to be conducted in controlled environments and pose limitations on the sequences and overall experiment duration they cannot be used for real-life QoE assessment of IPTV and VoD services. In this article, we propose a novel subjective quality assessment methodology based on full-length movies. Our methodology enables audiovisual quality assessment in the same environments and under the same conditions users typically watch television. Using our new methodology we conducted subjective experiments and compared the outcome with the results from a subjective test conducted using a standardized method. Our findings indicate significant differences in terms of impairment visibility and tolerance and highlight the importance of real-life QoE assessment.

Quality Assessment of Stereoscopic Images

Several metrics have been proposed in literature to assess the perceptual quality of two-dimensional images. However, no similar effort has been devoted to quality assessment of stereoscopic images. Therefore, in this paper, we review the different issues related to 3D visualization, and we propose a quality metric for the assessment of stereopairs using the fusion of 2D quality metrics and of the depth information. The proposed metric is evaluated using the SAMVIQ methodology for subjective assessment. Specifically, distortions deriving from coding are taken into account and the quality degradation of the stereopair is estimated by means of subjective tests.

The impact of varying frame rates and bit rates on perceived quality of low/high motion sequences with smooth/complex texture

There are evidences to indicate that a user's perception is affected by varying frame rate and bit rate with respect to different video content characteristics. We attempt to understand the differences of motion and texture characteristics by adopting both the objective assessment and the subjective assessment methodologies. In this paper, the peak signal-to-noise ratio (PSNR) is used as the objective measurement, while the double stimulus continuous quality scale (DSCQS) is used as the subjective measurement. We find that the low motion characteristic sequences suit to low frame rates while the high motion characteristic sequences suit to high frame rates under the limited bandwidths. The perceived quality of the complex texture characteristic sequences has significant difference at different bit rates. The results indicate that a multimedia scheduling system for transmitting the low motion and smooth texture (Type 1) sequences and the low motion and complex texture (Type 2) sequences should reduce frame rate to increase the bits of frame under the insufficient bandwidth, whereas the high motion and smooth texture (Type 3) sequences and the high motion and complex texture (Type 4) sequences should keep frame rate to decrease the bits of frame. The perceived quality of Type 1 and Type 3 sequences can be improved by increasing bit rate more than Type 2 and Type 4 under the limited bandwidth.

Subjective Multimedia Quality Assessment

The Video Quality Experts Group (VQEG) is preparing a programme of subjective multimedia quality tests. The results from these tests will be used to evaluate the performance of competing objective multimedia quality metrics. The reliability of the subjective test data is of great importance for VQEG's task. This paper provides an overview of VQEG's multimedia ad-hoc group. The work of this group will require subjective tests to be performed by laboratories located in Europe, Asia and North America. For VQEG's multimedia work to be successful, the subjective assessment methodology must be precisely defined and produce reliable and repeatable subjective quality data. Although international standards covering multimedia quality assessment methods are in force, there remains some uncertainty regarding the most effective approach to assessing the subjective quality of multimedia. A review of existing methods is provided. Two experiments are presented investigating the suitability of alternative subjective assessment methods (single-stimulus ACR and SAMVIQ). The results of these experiments are discussed within the context of the VQEG multimedia testing programme.

Subjective Quality Assessment of the H.264/AVC In-Loop De-Blocking Filter

Next generation codecs, benchmarked by the H.264/AVC standard, are providing substantial compression efficiency for the coding and transmission of video. Coupled with technologies offering larger transmission bandwidths over DSL, wireless and satellite networks, the capability of delivering high quality video services to the home is now a reality. The perceptual quality of the content delivered over communications networks will be crucial in ensuring a first-class customer experience. It is therefore important to assess the advantages and disadvantages of the optional features offered by next generation codecs. This paper describes a subjective assessment that was carried out to investigate the perceptual effects of switching the in loop de-blocking filter within the H.264/ AVC CODEC on or off. Although the filter is believed to substantially improve the perceptual quality of video, it has been suggested that in some cases negative perceptual effects can be produced. The H.264/AVC architecture allows de-blocking to be switched off in cases where there are limited processing resources or it is considered a negative perceptual effect may be introduced. This paper describes a study that examined the perceptual effects of de-blocking by employing a standardised subjective assessment methodology. The Absolute Category Rating (ACR) method was used to capture Difference Mean Opinion Scores (DMOS) for a range of video. Content was selected to span a wide and representative range of coding complexity. This content was then encoded at a variety of bit-rates to represent high, medium and low qualities. Results were used to examine the end-user perception of video quality when the de-blocking filter is switched on or off. The experimental design allowed the overall effects of the de-blocking filter to be examined and additionally the relationship between content and quality on the filter performance. The experiment found that the performance of the de-blocking filter was content-dependent. Results were used to discuss the advantages and disadvantages of in-loop de-blocking and there is an examination of content properties (e.g. spatial and temporal complexity) that influence the performance of de-blocking.

SAMVIQ—A New EBU Methodology for Video Quality Evaluations in Multimedia

This paper proposes a novel methodology, SAMVIQ, to be used primarily for formal subjective evaluations of video contents in the multimedia domain. SAMVIQ stands for Subjective Assessment Methodology for Video Quality. First, the conventional methods used to assess the quality of moving pictures in television are briefly described. Then, by pointing out some principal differences between conventional television and multimedia video, the details of the SAMVIQ methodology, which has been designed for the multimedia environments, are introduced. The main advantages of SAMVIQ are highlighted and some examples of how it can be efficiently used for the evaluation of video codecs in both clear and error-prone environments are shown.

The Validity of the EASE Expert System for Inhalation Exposures

Estimation and Assessment of Substance Exposure (EASE) is a computerized expert system developed by the UK Health and Safety Executive to facilitate exposure assessments in the absence of exposure measurements. The system uses a number of rules to predict a range of likely exposures or an 'end-point' for a given work situation. The purpose of this study was to identify a number of inhalation exposure measurements covering a wide range of end-points in the EASE system to compare with the predicted exposures. Occupational exposure data sets were identified from previous research projects or from consultancy work. Available information for each set of measurements was retrieved from archive storage and reviewed to ensure that it was adequate to enable EASE (version 2) predictions to be obtained. Exposure measurements and other relevant contextual data were abstracted and entered into a computer spreadsheet. EASE predictions were then obtained for each task or job and entered into the spreadsheet. In addition, we generated a random exposure range for each data set for comparison with the EASE predictions. Finally, we produced exposure assessments for a subset of the data using a structured subjective assessment method. We were able to identify approximately 4000 inhalation exposure measurements covering 52 different scenarios and 28 EASE end-points. The data included measurements of solvent vapours, non-fibrous dusts and fibres. In 62% of the end-points the EASE predictions were generally greater than the exposure measurements and in 30% of the end-points the EASE estimates were comparable with the measurements. The random allocation of exposure ranges was, as expected, less reliable than EASE, although there were still about one-third of the cases where the randomly generated exposure ranges generally agreed with the measurements. The structured subjective assessments undertaken by a human expert produced exposure estimates in better agreement with the measurements with about two-thirds of the end-points derived from these assessments in good agreement with the data. We argue that the inhalation exposure estimates from EASE could be improved by incorporating some of the parameters included in the structured subjective assessment methodology.

NTSC in-band data broadcasting using orthogonal frequency division multiplexing

This paper presents a new data broadcasting technique for transmitting data using the orthogonal frequency division multiplexing modulation scheme, within the NTSC television signal. The data signal is inserted in the vestigial sideband of the video signal. The advantages of this implementation are that the shape of the data spectrum minimizes the degradation of the picture quality and increases the noise margin of the data signal. Two approaches are taken to evaluate the performance of this system. In the first one, computer simulations are used to determine the available bit rate by varying the number of carriers. Bit error rate measurements are also done to evaluate the performance of the system when the data signal is combined with the NTSC television signal. In the second approach, the perceptual impact is determined, using accepted subjective assessment methodology, when varying levels of OFDM carriers are injected into the NTSC signal. The results show that the proposed system has the potential to offer higher data rate throughput than existing data broadcasting systems, with negligible degradation effects on the NTSC picture quality.