A consistency measure for psychometric measurements.

  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract
Translate article icon Translate Article Star icon
Take notes icon Take Notes

Adaptive tracking procedures in psychophysics may produce erroneous, "untypical" results and non-converging tracks due to, e.g., inattention of the test subject or external disturbances. This paper presents a multi-state psychometric model, which is used to rate the outcome of psychometric measurement procedures with a consistency measure. The consistency measure may be used for a post hoc, automated consistency estimation for any psychometric measurement procedure that can be modeled with a sigmoid psychometric function. The model calculates the log likelihood difference between single and two interleaved psychometric functions, potentially underlying a recorded adaptive track. A binary classifier was tested with a range of candidates for consistency measures with simulated, inconsistent tracks, and expert ratings of empirical tracks. The proposed consistency measure was identified as the best candidate to classify inconsistent tracks, while expert ratings were best predicted with the spectrum of the stimulus level, which is shown to be a suboptimal predictor of consistency. A threshold of the proposed measure for the German matrix sentence test is 10 to test for inconsistency, with a sensitivity of 60% and a specificity of 80%.

ReferencesShowing 10 of 23 papers
  • Open Access Icon
  • Cite Count Icon 345
  • 10.1109/mic.2013.20
Quality Control in Crowdsourcing Systems: Issues and Directions
  • Mar 1, 2013
  • IEEE Internet Computing
  • M Allahbakhsh + 5 more

  • Open Access Icon
  • Cite Count Icon 753
  • 10.3758/bf03194543
Adaptive procedures in psychophysical research.
  • Nov 1, 2001
  • Perception & Psychophysics
  • Marjorie R Leek

  • Open Access Icon
  • Cite Count Icon 27228
  • 10.1038/s41592-019-0686-2
SciPy 1.0: fundamental algorithms for scientific computing in Python
  • Feb 3, 2020
  • Nature Methods
  • Pauli Virtanen + 99 more

  • Open Access Icon
  • Cite Count Icon 15
  • 10.1177/2331216520970011
Speech Audiometry at Home: Automated Listening Tests via Smart Speakers With Normal-Hearing and Hearing-Impaired Listeners
  • Jan 1, 2020
  • Trends in Hearing
  • Jasper Ooster + 5 more

  • Open Access Icon
  • Cite Count Icon 28
  • 10.1177/2331216520938929
Individual Aided Speech-Recognition Performance and Predictions of Benefit for Listeners With Impaired Hearing Employing FADE.
  • Jan 1, 2020
  • Trends in Hearing
  • Marc R Schädler + 3 more

  • Cite Count Icon 313
  • 10.3109/00206090109073110
ICRA Noises: Artificial Noise Signals with Speech-like Spectral and Temporal Properties for Hearing Instrument Assessment: Ruidos ICRA: Señates de ruido artificial con espectro similar al habla y propiedades temporales para pruebas de instrumentos auditivos
  • Jan 1, 2001
  • International Journal of Audiology
  • Wouter A Dreschler + 3 more

  • Open Access Icon
  • PDF Download Icon
  • Cite Count Icon 6
  • 10.1121/10.0014898
The one-up one-down adaptive (staircase) procedure in speech-in-noise testing: Standard error of measurement and fluctuations in the track.
  • Oct 1, 2022
  • The Journal of the Acoustical Society of America
  • Cas Smits + 4 more

  • Cite Count Icon 1
  • 10.1080/14992027.2022.2142683
Development and evaluation of the Cantonese matrix sentence test
  • Nov 22, 2022
  • International Journal of Audiology
  • Hongmei Hu + 5 more

  • Cite Count Icon 61
  • 10.1080/14992020500243851
The role of silent intervals for sentence intelligibility in fluctuating noise in hearing-impaired listeners
  • Jan 1, 2006
  • International Journal of Audiology
  • Kirsten Carola Wagener + 2 more

  • Cite Count Icon 58
  • 10.3109/14992027.2015.1061708
Matrix sentence intelligibility prediction using an automatic speech recognition system
  • May 1, 2015
  • International Journal of Audiology
  • Marc René Schädler + 3 more

Similar Papers
  • Research Article
  • Cite Count Icon 80
  • 10.3758/bf03212251
Estimation of psychometric functions from adaptive tracking procedures.
  • May 1, 1992
  • Perception & psychophysics
  • Marjorie R Leek + 2 more

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections of the true generating functions when at least 200 trials were included in the tracks. However, for 50- and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, if both threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three- or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.

  • Research Article
  • Cite Count Icon 1
  • 10.1121/1.4782439
Absolute-detection results with a Bayesian adaptive tracking procedure using simulated and human subjects
  • May 1, 2007
  • The Journal of the Acoustical Society of America
  • Georg Stiefenhofer + 2 more

Adaptive tracking procedures are widely used in psychophysics. The common, fixed step-size, up-down procedures are known to be biased and yield convergence dependent upon the step-size used [e.g., B. W. Edwards and G. H. Wakefield, J. Acoust. Soc. Am. 83, S17 (1988)], provide an estimate of only one parameter of the psychometric function, and in general make nonoptimal use of information (a priori and a posteriori) concerning a given subject’s psychometric function. Recently introduced Bayesian adaptive procedures [e.g., L. L. Kontsevich and C. W. Tyler, Vision Res. 39, 2729–2737 (1999)] make optimal use of all information in both the estimation of parameters of the psychometric function, as well as in stimulus placement via a minimum-entropy rule. No direct comparison of the up-down and Bayes procedures has been carried out, however. We present results of Monte Carlo simulations that demonstrate the Bayes procedure is as efficient, and often more efficient, than common up-down procedures, even when estimating more parameters than the up-down procedures. We will also present results from absolute-detection experiments with human subjects (with and without hearing loss) that demonstrate the Bayes procedures behavior outside the simulation environment. These experiments used a priori information derived from an audiogram database.

  • Research Article
  • Cite Count Icon 16
  • 10.1121/1.390766
Identification and discrimination of rise time: is it categorical or noncategorical?
  • Apr 1, 1984
  • The Journal of the Acoustical Society of America
  • Diane Kewley‐Port + 1 more

Previous studies have reported that rise time of sawtooth waveforms may be discriminated in either a categorical-like manner under some experimental conditions or according to Weber's law under other conditions. In the present experiments, rise time discrimination was examined with two experimental procedures: the traditional labeling and ABX tasks used in speech perception studies and an adaptive tracking procedure used in psychophysical studies. Rise time varied from 0 to 80 ms in 10-ms intervals for sawtooth signals of 1-s duration. Discrimination functions for subjects who simply discriminated the signals on any basis whatsoever as well as functions for subjects who practiced labeling the endpoint stimuli as " pluck " and "bow" before ABX discrimination were not categorical in the ABX task. In the adaptive tracking procedure, the Weber fraction obtained from the jnds of rise time was found to be a constant above 20-ms rise time. The results from the two discrimination paradigms were then compared by predicting a jnd for rise time from the ABX discrimination data by reference to the underlying psychometric function. Using this method of analysis, discrimination results from previous studies were shown to be quite similar to the discrimination results observed in this study. Taken together the results demonstrate clearly that rise time discrimination of sawtooth signals follows predictions derived from Weber's law.

  • Research Article
  • Cite Count Icon 3
  • 10.1121/1.2019355
Discrimination of rise time in nonspeech signals: Is it categorical or noncategorical?
  • Apr 1, 1982
  • The Journal of the Acoustical Society of America
  • Diane Kewley-Port + 1 more

Previous studies have reported that rise time at the onset of a sawtooth waveform may be discriminated in a categorical‐like manner under some experimental conditions or according to Weber's law under other conditions. In the present experiments, rise time discrimination was examined with both the traditional ABX paradigm and an adaptive tracking procedure. Rise time varied from 0 to 80 ms in 10‐ms intervals at the onset of a one second duration sawtooth signal. Discrimination functions were not categorical in the ABX task. This result was true even for subjects who practiced labeling the endpoint stimuli as “pluck” and “bow” before discrimination. In the adaptive tracking procedure, the Weber fraction obtained from the JND's of rise time was constant above 20‐ms rise time. The results from the two paradigms were then compared by predicting a JND for rise time from ABX discrimination by means of the underlying psychometric function. Using this analysis, the apparently categorical discrimination results from previous studies were shown to be quite similar to the results observed in this study. Our results demonstrate clearly that rise time discrimination follows Weber's law. [Supported by NINCDS grant NS‐12179.]

  • Research Article
  • Cite Count Icon 1
  • 10.2466/pms.1990.70.1.243
An Adaptive (Tracking) Procedure for Measuring Visual Search
  • Feb 1, 1990
  • Perceptual and Motor Skills
  • Ethel Matin + 1 more

PEST, an adaptive (tracking) procedure originally developed for sensory research, was modified for cognitive studies with the multiple sequential frames paradigm and a letter-search task. The program which controlled stimulus presentation was designed to track the frame duration required for successful search on 75% of trials (the duration threshold). For each of the three subjects used, the threshold increased as a linear function of the number of sequentially presented frames, which varied between 1 and 10. Each tracking run was immediately followed by 40 trials with frame-exposure time fixed at the computer-selected threshold. Performance on these fixed level trials was close to the expected 75% correct. The Discussion addresses issues related to the use of the threshold measure in basic and applied cognitive research, explores the present findings about performance on fixed level trials at the computer-selected threshold, and examines the increase in duration threshold with the number of sequentially presented frames.

  • Research Article
  • Cite Count Icon 1
  • 10.1121/1.4783498
Effect of temporal uncertainty on psychometric functions for increment detection.
  • Apr 1, 2009
  • The Journal of the Acoustical Society of America
  • Walt Jesteadt + 2 more

Increased temporal uncertainty might be expected to produce increased internal noise, resulting in poorer detection and shallower psychometric functions. An alternative model in the literature predicts steeper psychometric functions with increased uncertainty. As a test of these two alternatives, detection of a 20‐ms intensity increment was assessed for eight subjects in a two‐interval, forced‐choice adaptive procedure where the increment was presented 100, 150, 200, 250, or 300 ms after the onset of a 400‐ms, 4‐kHz tone presented at 70 dB SPL. In one condition, the temporal position varied randomly from trial to trial. In the other, the temporal position was fixed for an entire 50‐trial block. Detection thresholds were higher and psychometric functions obtained by analyzing the trial‐by‐trial data from the adaptive tracks were shallower in the higher temporal uncertainty condition. Thresholds in both conditions were higher for temporal positions more distant from the center of the 400‐ms tone. Results support a model in which temporal uncertainty contributes to internal noise resulting in shallower psychometric functions. [Work supported by R01 DC006648 and T32 DC000013.]

  • Research Article
  • Cite Count Icon 189
  • 10.1016/j.neuron.2005.05.012
Visibility Reflects Dynamic Changes of Effective Connectivity between V1 and Fusiform Cortex
  • Jun 1, 2005
  • Neuron
  • John-Dylan Haynes + 2 more

Visibility Reflects Dynamic Changes of Effective Connectivity between V1 and Fusiform Cortex

  • Research Article
  • 10.1121/10.0037580
Changes in psychometric functions on two out of three auditory tasks during adolescence
  • Apr 1, 2025
  • The Journal of the Acoustical Society of America
  • Serena A Sereki + 3 more

Adolescents continue to develop on some psychoacoustic tasks, but most studies use adaptive tracking procedures to estimate thresholds; therefore, the effect of prolonged maturation on psychometric functions is relatively unknown. Here, we tested adolescents (age 10–18) and young adults (age 19–23) on three conditions: temporal interval discrimination (100 ms standard), gap detection in bandpass noise, and frequency discrimination (1000 Hz standard). Performance was measured using the method of constant stimuli and a 3AFC task. Data were analyzed by measuring threshold (66.7% correct; proxy for sensitivity), the slope of the psychometric function (a proxy for internal noise), and the proportion of listeners who met criteria for outliers (more than 2 SD from the mean), had poorly fit psychometric functions, or were unable to complete the task. Preliminary results suggest that temporal interval discrimination thresholds, slopes, and proportion of excluded data did not change with age. Gap detection thresholds improved and slopes became steeper with development, and the proportion of excluded data decreased with increasing age. Frequency discrimination thresholds did not improve with age, but slopes became steeper and excluded data decreased with maturation. These data suggest that internal noise may be decreasing during adolescence, even when thresholds stay the same. [Funded by NIDCD.]

  • Research Article
  • Cite Count Icon 2
  • 10.1121/1.419064
Factors in the detection of environmental sounds
  • May 1, 1997
  • The Journal of the Acoustical Society of America
  • Brian Gygi + 2 more

In separate experiments, detection and identification thresholds were obtained for a set of 25 common environmental sounds (e.g., dog barking, car starting). In the detection task, threshold values of the event-to-noise ratio (Ev/N) were established using an adaptive tracking procedure. The identification experiment was part of a larger individual differences study reported earlier [Watson et al., J. Acoust. Soc. Am. 99, 2516(A) (1996)]. A 3AFC identification task was used, with sounds presented at eight levels of Ev/N. Thresholds were estimated from the psychometric functions derived from the data. The correlation between thresholds in the two tasks indicates that the identifiability of these sounds is only weakly related to their detectability. Several models were evaluated to identify the properties that determine the detectability of environmental sounds. Leaky integrator and cross-correlation models were used to predict detection data; identification data were evaluated using a multidimensional scaling algorithm.

  • Conference Article
  • Cite Count Icon 1
  • 10.1109/acoustics.2018.8502357
The Wald Sequential Test as an Alternative Statistics in Staircase Adaptive Procedures
  • Sep 1, 2018
  • Jan Zera

In adaptive procedures used in psychoacoustics, the Wald sequential statistical test has been employed only in the PEST procedure. A specificity of the Wald test is that it allows selecting the target threshold value at any point on the psychometric function. This feature is useful when there are special test requirements, such as selecting a point on a psychometric function corresponding to a given detectability index value. Adaptive, staircase, up-down procedures are based on rules of signal level setting that are simple to implement but, in contrast to the Wald test, they allow to estimate the threshold level only for a few points on the psychometric function. In this study the feasibility of an adaptive, up-down staircase procedure which uses a decision rule based on the Wald sequential statistical test was examined. The staircase procedures, combined with the Wald test for selecting signal level, were tested as to their ability to converge at various threshold levels, their accuracy, and bias. The study was aimed to determine whether the Wald test may serve as a proper replacement for standard rules of signal level setting in adaptive up-down procedures.

  • Research Article
  • Cite Count Icon 23
  • 10.1007/s13349-018-0306-3
Automated long-term dynamic monitoring using hierarchical clustering and adaptive modal tracking: validation and applications
  • Sep 19, 2018
  • Journal of Civil Structural Health Monitoring
  • Giacomo Zonno + 3 more

Historical buildings demand constant surveying because anthropogenic (e.g., use, pollution or traffic vibration) and natural or environmental hazards (e.g., environmental changes or earthquakes) can endanger their existence and safety. Particularly, in the Andean region of South America, earthen historical constructions require special attention and investigation due to the high seismic hazard of the area next to the Pacific coast. Structural Health Monitoring (SHM) can provide useful, real-time information on the condition of these buildings. In SHM, the implementation of automatic tools for feature extraction of modal parameters is a crucial step. This paper proposes a methodology for the automatic identification of the structural modal parameters. An innovative and multi-stage approach for the automatic dynamic monitoring is presented. This approach uses the Data-Driven Stochastic Subspace Identification method complemented by hierarchical clustering for automatic detection of the modal parameters, as well as an adaptive modal tracking procedure for providing a clear visualization of long-term monitoring results. The proposed methodology is first validated in data acquired in an emblematic sixteenth century historical building: the monastery of Jeronimos in Portugal. After proving its efficiency, the algorithm is used to process almost 5000 events containing data acquired in the church of Andahuaylillas, a sixteenth century adobe building located in Cusco, Peru. The results in these cases demonstrate that accurate estimation of predominant modal parameters is possible in those complex structures even if relatively few sensors are installed.

  • Research Article
  • Cite Count Icon 26
  • 10.1007/s12541-017-0022-7
Adaptive tracking control of a quadrotor unmanned vehicle
  • Feb 1, 2017
  • International Journal of Precision Engineering and Manufacturing
  • To Xuan Dinh + 1 more

The design of a flight control system for an under-actuated quadrotor aircraft in presence of parametric uncertainties and external disturbance is quite challenging. In this study, we propose an adaptive trajectory tracking control base on sliding mode approach, an adaptive command filtered backstepping technique to stabilize the attitude of the quadrotor and using online estimators to estimate unknown aerodynamic parameters and external disturbances. First of all, the mathematical model of a quadrotor unmanned aerial vehicle (UAV) is presented. The adaptive tracking trajectory position and attitude control scheme are then formulated and the perturbations in quadrotor system are compensated by employing special Lyapunov functions. Simulation results are given to demonstrate the validity and effectiveness of the proposed algorithm on a quadrotor model under different conditions.

  • Research Article
  • Cite Count Icon 1
  • 10.1016/s1874-1029(08)60102-0
Adaptive Tracking and Disturbance Rejection for a Class of Distributed Systems
  • Aug 1, 2009
  • Acta Automatica Sinica
  • Xiao-Zheng Jin + 1 more

Adaptive Tracking and Disturbance Rejection for a Class of Distributed Systems

  • Research Article
  • Cite Count Icon 1
  • 10.1121/1.2021322
Frequency discrimination for conditions of roving standard and random interstimulus interval
  • May 1, 1984
  • The Journal of the Acoustical Society of America
  • Gerald Kidd + 1 more

A two‐interval forced‐choice adaptive tracking procedure was used to estimate the frequency difference limen (DL). In one condition, the frequency of the standard tone was constant for every trial with the frequency of the comparison tone varied according to the adaptive procedure. The interval between standard and comparison stimuli (ISI) was fixed. In contrast to this “fixed‐standard fixed‐ISI” condition, other conditions were tested in which the standard frequency and/or the ISI were chosen on each trial according to pseudo‐random algorithms. The results indicate that the frequency DL may increase by as much as a factor of 2 for some roving‐standard and random ISI conditions. In addition, a trial‐by‐trial record of each run was used to assess discrimination performance as a function of frequency and ISI. The decay of the sensory trace with increasing ISI, and uncertainty about signal frequency and timing appear to account for the changes in discrimination. [Supported by NIH.]

  • Research Article
  • Cite Count Icon 16
  • 10.1121/1.415865
Adaptive psychophysical procedures and imbalance in the psychometric function.
  • Jul 1, 1996
  • The Journal of the Acoustical Society of America
  • Kourosh Saberi + 1 more

One class of adaptive psychophysical procedures was studied, using simulated and human observers. These procedures are those which require an increase in stimulus intensity after an incorrect response, and a decrease after k successive correct responses. This paper analyzes how step size and the value of k affect the mean and standard deviation of threshold estimates based on a k-down 1-up adaptive procedure. Computer simulations are used to study the bias in threshold estimates, which are most evident when larger step size and small values of k are used. The adaptive procedure can be characterized by a function called the imbalance of the track, the relative probability of adjusting the stimulus either up or down at equal stimulus distances from the equilibrium point. These imbalance functions can be used to understand the threshold biases obtained in the computer simulations. The computer simulations also show that the average number of reversals obtained per trial is dependent on different values of k, but are largely independent of step size. The standard error of the threshold estimates, however, varies systematically with step size, but are nearly independent of k. Finally, we compare the stability of threshold estimates for human listeners using two very different sets of parameters: a very large step size (approximately half the range of the psychometric function) with k = 4, and the conventional k = 3 with an initial 4-dB and a final 2-dB step size.

More from: The Journal of the Acoustical Society of America
  • New
  • Research Article
  • 10.1121/10.0039806
Phase-preserving nonreciprocal dynamics in coupled nonlinear oscillatorsa).
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • Ali Kogani + 1 more

  • New
  • Research Article
  • 10.1121/10.0039708
Experimental assessment of passive radiator parameters using normal-incidence sound transmission measurements.
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • Brian E Anderson + 4 more

  • New
  • Research Article
  • 10.1121/10.0039808
A high-robustness algorithm for synthetic array using a single moving vector hydrophone.
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • Rongxin Zhu + 5 more

  • New
  • Research Article
  • 10.1121/10.0039669
North Atlantic right whale detection range performance quantification on a bottom-mounted linear hydrophone array using a calibrated acoustic source.
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • Vincent E Premus + 5 more

  • New
  • Research Article
  • 10.1121/10.0039807
Singing in virtual versus real rooms: Is it the same?
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • Pasquale Bottalico + 5 more

  • New
  • Research Article
  • 10.1121/10.0039583
Unsupervised stroke clustering for developing common stroke set for percussion in Indian art music.
  • Nov 1, 2025
  • The Journal of the Acoustical Society of America
  • R Gowriprasad + 4 more

  • Research Article
  • 10.1121/10.0039665
Analytical and numerical modeling of non-collinear wave mixing due to material and interface nonlinearity.
  • Oct 1, 2025
  • The Journal of the Acoustical Society of America
  • Junzhen Wang + 4 more

  • Research Article
  • 10.1121/10.0039573
Automotive sound field reproduction using deep optimization with spatial domain constraint.
  • Oct 1, 2025
  • The Journal of the Acoustical Society of America
  • Yufan Qian + 2 more

  • Research Article
  • 10.1121/10.0037228
Observations and modeling of range-dependent propagation in low-frequency tones emitted by a container shipa).
  • Oct 1, 2025
  • The Journal of the Acoustical Society of America
  • Peter H Dahl + 1 more

  • Addendum
  • 10.1121/10.0039663
Erratum: Dual boundary element method for solving the two-dimensional Helmholtz equation for the damped wave equation [J. Acoust. Soc. Am. 158(3), 2358-2376 (2025)
  • Oct 1, 2025
  • The Journal of the Acoustical Society of America
  • Kue-Hong Chen + 2 more

Save Icon
Up Arrow
Open/Close
  • Ask R Discovery Star icon
  • Chat PDF Star icon

AI summaries and top papers from 250M+ research sources.

Search IconWhat is the difference between bacteria and viruses?
Open In New Tab Icon
Search IconWhat is the function of the immune system?
Open In New Tab Icon
Search IconCan diabetes be passed down from one generation to the next?
Open In New Tab Icon