Subjective Quality Judgments Research Articles

Predicting image quality of forensic footwear impressions

The ability of a footwear examiner to confidently discern features of importance in a forensic examination is directly related to impression quality. As a result, quality directly impacts the strength an examiner can ascribe to any opinion of source attribution. Despite the importance of image quality during both the analysis and comparison phases of an examination, there is limited research on the estimation, variation, and prediction of footwear impression quality. In response, this study aims to develop a methodology for evaluating footwear impression quality by regressing image features against subjective judgments of quality. Using a dataset of more than 450 impressions evaluated by more than 40 participants, estimates of intra- and inter-rater consistency were computed. After identifying reliable raters, matrix completion was performed, thereby permitting data imputation. This was based on an approximate 90:10 split between training and testing data, ultimately resulting in 6,000 quality predictions, which exhibited a mean agreement with ground truth > 95%. Each image in the dataset was then reduced to a 10-dimensional feature vector describing image attributes such as complexity, contrast, sharpness, and noise. These features were used to train multiple ordinal and multinomial regression models aimed at predicting image quality. Model comparison resulted in an overall optimal accuracy of approximately 75% when comparing predicted/modeled quality against the ground truth of subjective rater opinions. The resulting semi-automated, reference-free and numerical prediction tool exhibits reasonable success for impression quality prediction when presented with the types of challenging and domain-specific images often encountered in forensic footwear comparisons. The outcomes and challenges associated with this investigation provide a foundation upon which future studies of quality can be modified and built, with the downstream goal of an increased understanding of how quality impacts weight of evidence.

A Subjective and Objective Study of Space-Time Subsampled Video Quality.

Video dimensions are continuously increasing to provide more realistic and immersive experiences to global streaming and social media viewers. However, increments in video parameters such as spatial resolution and frame rate are inevitably associated with larger data volumes. Transmitting increasingly voluminous videos through limited bandwidth networks in a perceptually optimal way is a current challenge affecting billions of viewers. One recent practice adopted by video service providers is space-time resolution adaptation in conjunction with video compression. Consequently, it is important to understand how different levels of space-time subsampling and compression affect the perceptual quality of videos. Towards making progress in this direction, we constructed a large new resource, called the ETRI-LIVE Space-Time Subsampled Video Quality (ETRI-LIVE STSVQ) database, containing 437 videos generated by applying various levels of combined space-time subsampling and video compression on 15 diverse video contents. We also conducted a large-scale human study on the new dataset, collecting about 15,000 subjective judgments of video quality. We provide a rate-distortion analysis of the collected subjective scores, enabling us to investigate the perceptual impact of space-time subsampling at different bit rates. We also evaluated and compare the performance of leading video quality models on the new database. The new ETRI-LIVE STSVQ database is being made freely available at (https://live.ece.utexas.edu/research/ETRI-LIVE_STSVQ/index.html).

On the use of the i-vector speech representation for instrumental quality measurement

The i-vector framework has been widely used to summarize speaker-dependent information present in a speech signal. Considered the state-of-the-art in speaker verification for many years, its potential to estimate speech recording distortion/quality has been overlooked. This paper is an attempt to fill this gap. We conduct a detailed analysis of how distortions are captured in the total variability space. We then propose a full-reference speech quality model based on i-vector similarities and three no-reference approaches. The first no-reference approach makes use of a single reference i-vector based on the average of i-vectors extracted from clean signals. A second approach relies on a vector quantizer codebook of representative clean speech i-vectors. Lastly, i-vectors and subjective ratings were used to train a no-reference deep neural network model for speech quality assessment. Four experiments have shown that the proposed methods, based on the i-vector speech representation, are well-suited for assessing speech quality. Results show correlations with subjective quality judgments similar to those achieved with standardized instrumental algorithms, particularly for degradations caused by noise and reverberation.ϖ

Knowledge of Potential Harms and Benefits of Tamoxifen among Women Considering Breast Cancer Preventive Therapy.

Tamoxifen reduces breast cancer incidence in women at increased risk, but may cause side effects. We examined women's knowledge of tamoxifen's potential harms and benefits, and the extent to which knowledge reflects subjective judgments of awareness and decision quality. After a hospital appointment, 408 (55.7%) women at increased risk of breast cancer completed a survey assessing objective knowledge about the potential benefit (risk reduction) and harms (endometrial cancer, thromboembolic events, and menopausal side effects) of tamoxifen, and subjective tamoxifen knowledge and decisional quality. Two hundred fifty-eight (63.2%) completed a 3-month follow-up survey. Sixteen percent (15.7%) of participants recognized the potential benefit and three major harms of using tamoxifen. These women were more likely to have degree-level education [vs. below degree level; OR, 2.24; 95% confidence interval (CI), 1.11-4.55] and good numeracy (vs. poor numeracy; OR, 5.91; 95% CI, 1.33-26.19). Tamoxifen uptake was higher in women who recognized all harms and benefits (vs. not recognizing; OR, 2.47; 95% CI, 0.94-6.54). Sixty-six percent (65.8%) of tamoxifen users were unaware of its potential benefit and harms. Most (87.1%) women reported feeling informed about tamoxifen, and subjective decisional quality was high [Mean (SD), 17.03 (1.87), out of 18]. Knowledge regarding the potential harms and benefit of tamoxifen is low in women considering prevention therapy, and they may need additional support to make informed decisions about tamoxifen preventive therapy.

Investigation and Evaluation of Winter Indoor Air Quality of Primary Schools in Severe Cold Weather Areas of China

The indoor air quality (IAQ) in classrooms has attracted more and more attention. Unfortunately, there is limited information relating to IAQ in the primary schools in severe cold weather areas of China. In this study, a field investigation on the IAQ of a primary school of Shenyang in northeast China was carried out by physical measurements and questionnaire surveys. The carbon dioxide (CO2) concentration in selected classrooms was continuously measured for a week, and the corresponding ventilation rate was calculated. Meanwhile, the perceptions of the IAQ, the purpose and the comfort degree of window opening have also been recorded from 106 pupils, aged 9–12. The results indicate the ventilation rate is considerably inadequate in about 99% of the class time due to the low frequency of window opening. The average daily CO2 concentration in these classrooms is 1510–3863 ppm, which is far higher than the recommended value of 1000 ppm. Most pupils understand that the purpose of opening windows in winter is to improve air quality. However, there are big differences between the measurement results and subjective judgments of indoor air quality. Contrary to the high measured CO2 concentration, around 70% pupils consider the air fresh, and only 3.7% pupils are dissatisfied and even very dissatisfied with IAQ in their classroom. It is necessary to change the existing manual window opening mode, because the pupils’ subjective judgment affects the window opening behavior.

The Role of Subjective Quality Judgements in User Preferences for Mobile Learning Apps

This study investigated whether subjective quality judgements on sound and picture quality across three devices (iPhone, iPad, and iPad mini) affected user preferences for learning applications. We tested 20 native Finnish-speaking users trialing generic audio clips, video clips, and two kinds of learning apps that were heavily reliant on sound. It was found that there was a main effect of the device on perceived sound quality, replicating earlier findings. However, these judgements did not impact on the users’ preferences for different devices nor on their preferences for different applications. The results are interpreted as indicating that perceived quality and affordances are less important for users in these contexts than other considerations (e.g., convenience, mobility, etc.).

Modeling the Perceptual Quality of Immersive Images Rendered on Head Mounted Displays: Resolution and Compression.

We develop a model that expresses the joint impact of spatial resolution s and JPEG compression quality factor qf on immersive image quality. The model is expressed as the product of optimized exponential functions of these factors. The model is tested on a subjective database of immersive image contents rendered on a head mounted display (HMD). High Pearson correlation and Spearman correlation (> 0.95) and small relative root mean squared error (< 5.6%) are achieved between the model predictions and the subjective quality judgements. The immersive ground-truth images along with the rest of the database are made available for future research and comparisons.

Evaluation of Language Feedback Methods for Student Videos of American Sign Language

This research investigates how to best present video-based feedback information to students learning American Sign Language (ASL); these results are relevant not only for the design of a software tool for providing automatic feedback to students but also in the context of how ASL instructors could convey feedback on students’ submitted work. It is known that deaf children benefit from early exposure to language, and higher levels of written language literacy have been measured in deaf adults who were raised in homes using ASL. In addition, prior work has established that new parents of deaf children benefit from technologies to support learning ASL. As part of a long-term project to design a tool to automatically analyze a video of a students’ signing and provide immediate feedback about fluent and non-fluent aspects of their movements, we conducted a study to compare multiple methods of conveying feedback to ASL students, using videos of their signing. Through two user studies, with a Wizard-of-Oz design, we compared multiple types of feedback in regard to users’ subjective judgments of system quality and the degree students’ signing improved (as judged by an ASL instructor who analyzed recordings of students’ signing before and after they viewed each type of feedback). The initial study revealed that displaying videos to students of their signing, augmented with feedback messages about their errors or correct ASL usage, yielded higher subjective scores and greater signing improvement. Students gave higher subjective scores to a version in which time-synchronized pop-up messages appeared overlaid on the student's video to indicate errors or correct ASL usage. In a subsequent study, we found that providing images of correct ASL face and hand movements when providing feedback yielded even higher subjective evaluation scores from ASL students using the system.

Investigation of the usefulness of zaleplon at two doses to induce afternoon-sleep under noise interference and its effects on psychomotor performance and vestibular function.

BackgroundMilitary operation personnel often suffer from sleep difficulty because of their work requirements. In this study, we investigated the efficacy of zaleplon at two doses to induce afternoon-sleep under noise interference and its effects on psychomotor performance and vestibular function; we subsequently established the optimal dosage regimen for military operation personnel.MethodsTwenty-two healthy young male volunteers were recruited for the study. Eight subjects took 10 mg or 15 mg of zaleplon and placebo alternately and then were exposed to noise. Changes in polysomnography (PSG) indices, including sleep latency (SL), sleep efficiency (SE) and sleep structure, were recorded after drug administration. After awakening, the volunteers’ subjective judgments of sleep quality and sleepiness were measured. Eight volunteers underwent 3 psychomotor performance tests at a one-week interval, and the psychomotor performance tests were conducted before and after taking zaleplon and placebo. Six volunteers participated in the vestibular function test session, and parameters, including optokinetic nystagmus (OKN), vestibular ocular reflex (VOR), visual-vestibular ocular reflex (VVOR) and vestibular ocular reflex fixation suppression (VOR-Fix), were detected by the same experimental design as described above. The data of sleep observations were subjected to one-way variance analysis.ResultsCompared with the placebo group, SL was shortened significantly, and the scores of subjective sleep quality and sleep depth were clearly increased in the zaleplon 10 mg group (P < 0.05). Moreover, the SE and the percent of REM (rapid eye movement) sleep were increased remarkably in the zaleplon 15 mg group (P < 0.01). Furthermore, the SE, percent of REM sleep and scores of subjective sleep depth in the zaleplon 15 mg group were significantly higher than in the zaleplon 10 mg group (P < 0.05). The psychomotor performance did not change significantly after ingestion of 10 mg or 15 mg of zaleplon, whereas the OKN and VOR gains were lower in the two dose groups of zaleplon (P < 0.05) and restored to normal 3 h after drug ingestion.ConclusionZaleplon is an ideal hypnotic for military personnel, and its hypnotic efficiency is dose-related under noise interference; a 15 mg dose of zaleplon could provide significantly better sleep than a 10 mg dose of zaleplon.

Gender and parenting: Effects of parenting failures on evaluations of mothers and fathers

Stereotypes may function as standards, such that individuals are judged relative to within-category expectations. Subjective judgments may mask stereotyping effects, whereas objective judgments may reveal stereotype-consistent patterns. We examined whether gender stereotypes about parenting lead judges to rate women and men as equally “good” parents while objective judgments favor women and whether parenting performance moderates this pattern. Participants evaluated a mother or father who successfully or unsuccessfully performed a parenting task. Subjective judgments of parent quality (“s/he is a good parent”) revealed no parent gender effects, but objective estimates of parenting performance favored mothers. In a hypothetical divorce scenario, participants also favored mothers in custody decisions. However, this pro-mother bias decreased when the mother failed at the parenting task (through her own fault). Performance did not affect custody decisions for fathers. We suggest parenting quality matters more for evaluations of mothers than for fathers because negative performance violates stereotyped expectations.

Voice assessment: updates on perceptual, acoustic, aerodynamic, and endoscopic imaging methods.

This paper describes recent advances in perceptual, acoustic, aerodynamic, and endoscopic imaging methods for assessing voice function. We review advances from four major areas. PERCEPTUAL ASSESSMENT: Speech-language pathologists are being encouraged to use the new consensus auditory-perceptual evaluation of voice inventory for auditory-perceptual assessment of voice quality, and recent studies have provided new insights into listener reliability issues that have plagued subjective perceptual judgments of voice quality. ACOUSTIC ASSESSMENT: Progress is being made on the development of algorithms that are more robust for analyzing disordered voices, including the capability to extract voice quality-related measures from running speech segments. AERODYNAMIC ASSESSMENT: New devices for measuring phonation threshold air pressures and air flows have the potential to serve as sensitive indices of glottal phonatory conditions, and recent developments in aeroacoustic theory may provide new insights into laryngeal sound production mechanisms. ENDOSCOPIC IMAGING: The increased light sensitivity of new ultra high-speed color digital video processors is enabling high-quality endoscopic imaging of vocal fold tissue motion at unprecedented image capture rates, which promises to provide new insights into the mechanisms of normal and disordered voice production. Some of the recent research advances in voice function assessment could be more readily adopted into clinical practice, whereas others will require further development.

Quality assessment of false-colored fused displays

Abstract— The problem of assessing the quality of fused images (composites created from inputs of differing modalities, such as infrared and visible light radiation) is an important and growing area of research. Recent work has shown that the process of assessing fused images should not rely entirely on subjective quality methods, with objective tasks and computational metrics having important contributions to the assessment procedure. The current paper extends previous findings, applying a psychophysical selection task, metric evaluation, and subjective quality judgments to a range of fused surveillance images. Fusion schemes included the contrast pyramid and shift invariant discrete wavelet transform (Experiment 1), the complex wavelet transform (Experiments 1 & 2), and two false-coloring methods (Experiment 2). In addition, JPEG2000 compression was applied at two levels, as well as an uncompressed control. Reaction time results showed the contrast pyramid to lead to slowest performance in the objective task, whilst the presence of color greatly reduced reaction times. These results differed from both the subjective and metric results. The findings support the view that subjective quality ratings should be used with caution, especially if not accompanied by some task.

Evaluating spoken dialogue systems according to de-facto standards: A case study

In the present paper, we investigate the validity and reliability of de-facto evaluation standards, defined for measuring or predicting the quality of the interaction with spoken dialogue systems. Two experiments have been carried out with a dialogue system for controlling domestic devices. During these experiments, subjective judgments of quality have been collected by two questionnaire methods (ITU-T Rec. P.851 and SASSI), and parameters describing the interaction have been logged and annotated. Both metrics served the derivation of prediction models according to the PARADISE approach. Although the limited database allows only tentative conclusions to be drawn, the results suggest that both questionnaire methods provide valid measurements of a large number of different quality aspects; most of the perceptive dimensions underlying the subjective judgments can also be measured with a high reliability. The extracted parameters mainly describe quality aspects which are directly linked to the system, environmental and task characteristics. Used as an input to prediction models, the parameters provide helpful information for system design and optimization, but not general predictions of system usability and acceptability.

Efficacy of a Cochlear Implant Simultaneous Analog Stimulation Strategy Coupled with a Monopolar Electrode Configuration

The present study was performed to evaluate the efficacy and clinical feasibility of using monopolar stimulation with the Clarion Simultaneous Analog Stimulation (SAS) strategy in patients with cochlear implants. Speech recognition by 10 Clarion cochlear implant users was evaluated by means of 4 different speech processing strategy/electrode configuration combinations; ie, SAS and Continuous Interleaved Sampling (CIS) strategies were each used with monopolar (MP) and bipolar (BP) electrode configurations. The test measures included consonants, vowels, consonant-nucleus-consonant words, and Hearing in Noise Test sentences with a +10 dB signal-to-noise ratio. Additionally, subjective judgments of sound quality were obtained for each strategy/configuration combination. All subjects but 1 demonstrated open-set speech recognition with the SAS/MP combination. The group mean Hearing in Noise Test sentence score for the SAS/MP combination was 31.6% (range, 0% to 92%) correct, as compared to 25.0%, 46.7%, and 37.8% correct for the CIS/BP, CIS/MP, and SAS/BP combinations, respectively. Intersubject variability was high, and there were no significant differences in mean speech recognition scores or mean preference ratings among the 4 strategy/configuration combinations tested. Individually, the best speech recognition performance was with the subject's everyday strategy/configuration combination in 72% of the applicable cases. If the everyday strategy was excluded from the analysis, the subjects performed best with the SAS/MP combination in 37.5% of the remaining cases. The SAS processing strategy with an MP electrode configuration gave reasonable speech recognition in most subjects, even though subjects had minimal previous experience with this strategy/configuration combination. The SAS/MP combination might be particularly appropriate for patients for whom a full dynamic range of electrical hearing could not be achieved with a BP configuration.

Sensitivity and Specificity of Averaged Electrode Voltage Measures in Cochlear Implant Recipients

The primary purpose of this study was to evaluate the diagnostic accuracy of averaged electrode voltages (AEVs) for identifying malfunctioning electrodes in cochlear implant users. AEVs were measured using common ground (N = 33 ears) and monopolar stimulation modes (N = 40 ears) in adult subjects implanted with the Nucleus 24M or Nucleus 24R(CS) cochlear implant. AEV measures were then compared with common ground electrode impedance measures as well as to subjective judgments of electrode sound quality to determine the sensitivity and specificity of AEVs. Results showed excellent specificity for both modes of AEV testing: 97.9% for common ground and 99.8% for monopolar. Sensitivity for common ground AEV testing was 91.7%; however the sensitivity of monopolar AEVs was very poor, at 7.7%. Monopolar AEVs accurately identified open circuits but missed all short circuits. These results are consistent with the sensitivity of impedance measures made with each of the two stimulation modes. Common ground AEVs are more reliable than monopolar AEVs for identifying malfunctioning electrodes.

Image quality evaluation based on recognition times for fast image browsing applications

Mean squared error (MSE) and peak signal-to-noise-ratio (PSNR) are the most common methods for measuring the quality of compressed images, despite the fact that their inadequacies have long been recognized. Quality for compressed still images is sometimes evaluated using human observers who provide subjective ratings of the images. Both SNR and subjective quality judgments, however, may be inappropriate for evaluating progressive compression methods which are to be used for fast browsing applications. In this paper, we present a novel experimental and statistical framework for comparing progressive coders. The comparisons use response time studies in which human observers view a series of progressive transmissions, and respond to questions about the images as they become recognizable. We describe the framework and use it to compare several well-known algorithms (JPEG, set partitioning in hierarchical trees (SPIHT), and embedded zerotree wavelet (EZW)), and to show that a multiresolution decoding is recognized faster than a single large-scale decoding. Our experiments also show that, for the particular algorithms used, at the same PSNR, global blurriness slows down recognition more than do localized "splotch" artifacts.

Features of stimulation affecting tonal-speech perception: implications for cochlear prostheses.

Tone languages differ from English in that the pitch pattern of a single-syllable word conveys lexical meaning. In the present study, dependence of tonal-speech perception on features of the stimulation was examined using an acoustic simulation of a CIS-type speech-processing strategy for cochlear prostheses. Contributions of spectral features of the speech signals were assessed by varying the number of filter bands, while contributions of temporal envelope features were assessed by varying the low-pass cutoff frequency used for extracting the amplitude envelopes. Ten normal-hearing native Mandarin Chinese speakers were tested. When the low-pass cutoff frequency was fixed at 512 Hz, consonant, vowel, and sentence recognition improved as a function of the number of channels and reached plateau at 4 to 6 channels. Subjective judgments of sound quality continued to improve as the number of channels increased to 12, the highest number tested. Tone recognition, i.e., recognition of the four Mandarin tone patterns, depended on both the number of channels and the low-pass cutoff frequency. The trade-off between the temporal and spectral cues for tone recognition indicates that temporal cues can compensate for diminished spectral cues for tone recognition and vice versa. An additional tone recognition experiment using syllables of equal duration showed a marked decrease in performance, indicating that duration cues contribute to tone recognition. A third experiment showed that recognition of processed FM patterns that mimic Mandarin tone patterns was poor when temporal envelope and duration cues were removed.

Effects of electrode configuration and place of stimulation on speech perception with cochlear prostheses.

Recent research and clinical experience with cochlear implants suggest that subjects' speech recognition with monopolar or broad bipolar stimulation might be equal to or better than that obtained with narrow bipolar stimulation or other spatially restricted electrode configurations. Furthermore, subjects often prefer the monopolar configurations. The mechanisms underlying these effects are not clear. Two hypotheses are (a) that broader configurations excite more neurons resulting in a more detailed and robust neural representation of the signal and (b) that broader configurations achieve a better spatial distribution of the excited neurons. In this study we compared the effects of electrode configuration and the effects of longitudinal placement and spacing of the active electrodes on speech recognition in human subjects. We used experimental processor maps consisting of 11 active electrodes in a 22-electrode scala tympani array. Narrow bipolar (BP), wide bipolar (BP + 6), and monopolar (MP2) configurations were tested with various locations of active electrodes. We tested basal, centered, and apical locations (with adjacent active electrodes) and spatially distributed locations (with every other electrode active) with electrode configuration held constant. Ten postlingually deafened adult human subjects with Nucleus prostheses were tested using the SPEAK processing strategy. The effects of electrode configuration and longitudinal place of stimulation on recognition of CNC phonemes and words in quiet and CUNY sentences in noise (+10 dB S/N) were similar. Both independent variables had large effects on speech recognition and there were interactions between these variables. These results suggest that the effects of electrode configuration on speech recognition might be due, in part, to differences among the various configurations in the spatial location of stimulation. Correlations of subjective judgments of sound quality with speech-recognition ability were moderate, suggesting that the mechanisms contributing to subjective quality and speech-recognition ability do not completely overlap.

Image Quality of Privacy Filters for Displays

This paper presents an image quality evaluation of privacy filters for CRT display workstations. A photometric procedure was developed to evaluate the optical quality of privacy filters across horizontal display viewing angles. Then, the procedure was applied to two commercially-available privacy filter products. The results of the optical evaluation were compared with subjective image quality judgments of the privacy-filtered CRTs viewed under various ambient illumination, screen contrast polarity, and viewing angle conditions. The findings establish a human factors basis and procedure for objectively characterizing the image quality of privacy filters used on CRT displays.

Information theory and variance estimation techniques in the analysis of category rating and paired comparisons

Speech clarity judgments of bandpass filtered sentences were measured by category rating and paired comparisons in five age groups—4, 5, 6, 7, and 8 years old, and adults [data for children were provided from Eisenberg and Dirks, J. Speech Hearing Res. 38, 1157–1167 (1995)]. The judgment data within each age group were subjected to information theory and variance estimation analytical techniques in order to investigate developmental differences in task performance capability. All response variability was converted into equivalent signal detection ‘‘error’’ components, representing the ability of subjects in a given age group to perform a given psychophysical procedure. Preliminary results revealed a different developmental trend for each psychophysical procedure. For category rating, children 4 to 6 years of age exhibited higher error components in their responses, suggesting reduced capability to extract information compared with adults. In contrast, children as young as 5 years of age were approaching adultlike low error components in the paired comparisons. In this particular study, paired comparisons may provide a greater amount of information with less variance than category rating when subjective judgments of sound quality are measured. [Work supported by NIH.]