Adaptive Testing Methods Research Articles

Uncertainty estimates of psychoacoustic thresholds obtained from group test

Adaptive psychoacoustic test methods, in which the next signal level depends on the response to the previous signal, are the most efficient for determining psychoacoustic thresholds of individual subjects. In many tests conducted in the NASA psychoacoustic labs, the goal is to determine thresholds representative of the general population. To do this economically, non-adaptive testing methods are used in which three or four subjects are tested at the same time with predetermined signal levels. This approach requires us to identify techniques for assessing the uncertainty in resulting group-average psychoacoustic thresholds. In this presentation we examine the Delta Method of frequentist statistics, the Generalized Linear Model (GLM), the Nonparametric Bootstrap, a frequentist method, and Markov Chain Monte Carlo Posterior Estimation and a Bayesian approach. Each technique is exercised on a manufactured, theoretical dataset and then on datasets from two psychoacoustics facilities at NASA. The Delta Method is the simplest to implement and accurate for the cases studied. The GLM is found to be the least robust, and the Bootstrap takes the longest to calculate. The Bayesian Posterior Estimate is the most versatile technique examined because it allows the inclusion of prior information.

Development of an IRT-Based Short Form to Assess Applied Cognitive Function in Outpatient Rehabilitation.

Item response theory and computerized adaptive testing methods allow the development of a large calibrated item bank from which different subsets of questions can be selected for administration and scored on a common scale. The objective of this study was to develop an outpatient rehabilitation self-report short form for the Activity Measure for Post-Acute Care Applied Cognition item bank. Using data from a convenience sample of 235 rehabilitation outpatients, item content and item response theory-based test information function parameters were used in item selection. Internal consistency reliability, intraclass correlation coefficient (ICC), and percentage at the lowest (floor) and highest (ceiling) scores were evaluated for the short form and full item bank. A 15-item short form was developed. The internal consistency of the short form was 0.86. The ICC3,1 for the short form and item bank was 0.97 (95% confidence interval, 0.94-0.98). No floor effects were noted, and ceiling effects were 27.66% (short form) and 26.38% (full item bank). The Applied Cognition outpatient short form demonstrated acceptable psychometric properties and provides a bridge to item response theory-based measurement for settings where point-of-care computing is not available.

소프트웨어 컴포넌트 기반 로봇 시스템을 위한 입출력 연관관계 기반 적응형 조합 테스팅 기법

In the testing of a software component-based robot system, generating test cases for the system is a time-consuming and difficult task that requires the combining of test data. This paper proposes an adaptive combinatorial testing method which is based on the input/output relationship among components and which automatically generates the test cases for the system. The proposed algorithm first generates an input/output relationship graph in order to analyze the input/output relationship of the system. It then generates the reduced set of test cases according to the analyzed type of input/output relationship. To validate the proposed algorithm some comparisons are given in terms of the time complexity and the number of test cases.

A Novel Student Ranking Algorithm for Off-line Examinations

Computerization of the examination process has facilitated fast and unbiased large-scale evaluation, by using the popular format of multiple choice based question papers. The primary drawback of this format is the possibility of guessing by the students. This has been largely remedied by online Computerized Adaptive Testing (CAT) methods where the next question presented to a student depends upon his/her responses to earlier questions of known difficulty levels. The student capabilities are evaluated in real time. However, the case where the questions’ difficulty levels are not apriori known, but in fact estimated aposteriori with respect to aggregate student performance, has been less analyzed. This paper addresses the problem of offline estimation of the student capability levels using a maximum likelihood estimation method and proposes to use these estimated capability levels rather than raw-marks for the purpose of ranking of students. We focus on the case when guessing is ruled out, for example, due to exact numerical answer possibilities instead of multiple choices. With respect to suitable performance objectives, like low errors in rank-allotment, the estimated-capability based ranking emerges more reliable than the traditional marks based ranking of students.

Measurement precision for Oswestry Back Pain Disability Questionnaire versus a web-based computer adaptive testing for measuring back pain.

Relative precision is to denote the ratio of the error variance of two sample designs. The aim of the present study was to compare measurement precisions of the Oswestry Back Pain Disability Questionnaire and a computer adaptive testing (CAT) method for measuring disability resulting from back pain. A cross-sectional study was carried out with two independent convenient samples from two out-patient rehabilitation clinics for back pain (N = 42) and non-back pain groups (N = 42). Participants were asked to fill out the ODQ and CAT of International Classification of Functioning, Disability and Health – Activity Measure (ICF-AM). A series of Rasch analyses were performed to calculate person ability measures. The CAT measures had greater precision in discriminating the groups than did the ODQ measure in comparisons of the relative precision. The CAT measure appears to be more effective than the ODQ measure in terms of measurement precision. By administering test items calibrated in a way, CAT measures using item response theory may promise a means with measurement precision as well as efficiency.

Adaptive Gestures for GUI Testing on Smartphones

Is the variety of smartphones really a blessing to users? How can software developers ensure the quality of their products within only 18 weeks of development? Is it possible to test a large number of device models within a short time period? Conventional methods extract the coordinates of touch events, and cannot easily adapt to different device models. In this paper, we propose a fast, automated, adaptive gesture testing method for a wide variety of smartphones.

A Comparative Study of Oswestry Back Pain Disability Questionnaire Versus Computer Adaptive Testing for Measuring Back Pain

The aim of the present study was to compare measurement precisions of the Oswestry Back Pain Disability Questionnaire (ODQ) and a computer adaptive testing (CAT) method. The ODQ has been regarded as one of the most reliable condition-specific measure for back pain for decades. Cross-sectional study was carried out with two independent convenient samples from two out-patient rehabilitation clinics for back pain (n1=42) and non-back pain group (n2=42). Participants were asked to fill out the ODQ and CAT of International Classification of Functioning, Disability and Health-Activity Measure (ICF-AM). A series of Rasch analyses were performed to calculate person ability measures. The CAT measures had greater relative precision in discriminating the groups than did the ODQ measure in comparisons of the relative precision. The CAT measure appears to be more effective than did the ODQ measure in terms of measurement precision. By administering test items calibrated in a way, CAT measures using item response theory may promise a means with measurement precision as well as efficiency.

A Comparative Study of Item Difficulty Hierarchy of Self-Reported Activity Measure Versus Metabolic Equivalent of Tasks

The purposes of this study were: 1) to show the item difficulty hierarchy of walking/moving construct of the International Classification of Functioning, Disability and Health-Activity Measure (ICF-AM), 2) to evaluate the item-level psychometrics for model fit, 3) to describe the relevant physical activity defined by level of activity intensity expressed as Metabolic Equivalent of Tasks (MET), and 4) to explore what extent the empirical activity hierarchy of the ICF-AM is linked to the conceptual model based on the level of energy expenditure described as MET. One hundred and eight participants with lower extremity impairments were examined for the present study. A newly created activity measure, the ICF-AM using an item response theory (IRT) model and computer adaptive testing (CAT) method, has a construct on walking/moving construct. Based on the ICF category of walking and moving, the instrument comprised items corresponding to: walking short distances, walking long distances, walking on different surfaces, walking around objects, climbing, and running. The item difficulty hierarchy was created using Winstep software for 20 items. The Rasch analyses (1-parameter IRT model) were performed on participants with lower extremity injuries who completed the paper and pencil version of walking/moving construct of the ICF-AM. The classification of physical activity can also be performed by the use of METs that is often preferred to determine the level of physical activity. The empirical item hierarchy of walking, climbing, running activities of the ICF-AM instrument was similar to the conceptual activity hierarchy based on the METs. The empirically derived item difficulty hierarchy of the ICF-AM may be useful in developing MET-based activity measure questionnaires. In addition to convenience of applying items to questionnaires, implications of the finding could lead to the use of CAT method without sacrificing the objectivity of physiologic measures.

Adaptive Radio Frequency (RF) Identification Based on Frequency Domain Analysis for Open Non-Gaussian Background Environment

With the development of electronic technology, electronic equipments are suffering more electromagnetic interferences, so it is necessary to detect and evaluate their electromagnetic interference degree. The most commonly used method is the anechoic chamber test method that conducted in closed electromagnetic wave environment. However, there are some disadvantages, such as low detection efficiency and high testing cost. In order to avoid these advantages, a new method based on frequency domain analysis is presented to adaptively identify RF band of a detected device working in a non-Gaussian background environment. In this method, a non-parametric hypothesis testing is used to solve the interference of background noise on the detection result in an open environment. Further, an adaptive threshold hypothesis test method is also used to obtain the radio frequency spectrum of the detected signals for low SNR in order to overcome the frequency domain time-varying characteristics. The simulations show that the adaptive RF identification based on frequency domain analysis can improve the detection accuracy in the non-Gaussian background environment.

Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis

Abstract. Proper specification of model parameters is critical to the performance of land surface models (LSMs). Due to high dimensionality and parameter interaction, estimating parameters of an LSM is a challenging task. Sensitivity analysis (SA) is a tool that can screen out the most influential parameters on model outputs. In this study, we conducted parameter screening for six output fluxes for the Common Land Model: sensible heat, latent heat, upward longwave radiation, net radiation, soil temperature and soil moisture. A total of 40 adjustable parameters were considered. Five qualitative SA methods, including local, sum-of-trees, multivariate adaptive regression splines, delta test and Morris methods, were compared. The proper sampling design and sufficient sample size necessary to effectively screen out the sensitive parameters were examined. We found that there are 2–8 sensitive parameters, depending on the output type, and about 400 samples are adequate to reliably identify the most sensitive parameters. We also employed a revised Sobol' sensitivity method to quantify the importance of all parameters. The total effects of the parameters were used to assess the contribution of each parameter to the total variances of the model outputs. The results confirmed that global SA methods can generally identify the most sensitive parameters effectively, while local SA methods result in type I errors (i.e., sensitive parameters labeled as insensitive) or type II errors (i.e., insensitive parameters labeled as sensitive). Finally, we evaluated and confirmed the screening results for their consistency with the physical interpretation of the model parameters.

The Psi-marginal adaptive method (or: how to give nuisance parameters the attention they deserve, no more and no less).

Adaptive testing methods serve to maximize the information gained regarding the values of the parameters of a psychometric function (PF). Such methods typically target only one or two (‘threshold’ and ‘slope’) of the PFs four parameters while assuming fixed values for the ‘nuisance’ parameters (‘guess rate’ and ‘lapse rate’). However, due to some redundancy among the parameters of the PF (e.g., Prins, J. Vis. 12(6):25, 2012) bias in parameter estimates results when the assumed values for the nuisance parameters do not match the generating values. This bias may be alleviated somewhat when observations are fitted in a subsequent procedure in which the nuisance parameters are allowed to vary. When this is done however, the uncertainty regarding the values of the nuisance parameters (not addressed by the adaptive method) contributes to the uncertainty in the values of the parameters of interest. Here I propose the Psi-marginal adaptive method which addresses this issue specifically. The method is based on Kontsevich & Tyler’s (Vision Res. 39(16):2729-37, 1999) Psi-method. However, in the proposed method a posterior distribution defined across all parameters of unknown value is maintained. Critically, selection of stimulus intensities is based on the expected information gain in the marginal posterior distribution defined across the parameters of interest only. The appeal of this method is that it will target nuisance parameters but only when doing so maximizes the expected information gain regarding the values of the parameters of interest. The method is extremely flexible: any combination of the PF’s four parameters may be included in the posterior and any combination of those included may be treated as nuisance parameters. Simulations indicate that use of the Psi-marginal method results in smaller bias and higher precision in threshold and slope estimates compared to the original Psi method. Meeting abstract presented at VSS 2013

The psi-marginal adaptive method: How to give nuisance parameters the attention they deserve (no more, no less)

Adaptive testing methods serve to maximize the information gained regarding the values of the parameters of a psychometric function (PF). Such methods typically target only one or two ("threshold" and "slope") of the PF's four parameters while assuming fixed values for the "nuisance" parameters ("guess rate" and "lapse rate"). Here I propose the "psi-marginal" adaptive method, which can target nuisance parameters but only when this is the most optimal manner in which to reduce uncertainty in the value of the parameters of primary interest. The method is based on Kontsevich and Tyler's (1999) psi-method. However, in the proposed method a posterior distribution defined across all parameters of unknown value is maintained. Each of the parameters is specified either as a parameter of primary interest whose estimation should be optimized or as a nuisance parameter whose estimation should be subservient to the estimation of the parameters of primary interest. Critically, selection of stimulus intensities is based on the expected information gain in the marginal posterior distribution defined across the parameters of interest only. The appeal of this method is that it will target nuisance parameters adaptively and only when doing so maximizes the expected information gain regarding the values of the parameters of interest. Simulations indicate that treating the lapse rate as a nuisance parameter in the psi-marginal method results in smaller bias and higher precision in threshold and slope estimates compared to the original psi method. The method is highly flexible and various other uses are discussed.

Advances in Patient-Reported Outcomes: The NIH PROMIS(®) Measures.

Patient-reported outcomes (PRO) are questionnaire measures of patients’ symptoms, functioning, and health-related quality of life. They are designed to provide important clinical information that generally cannot be captured with objective medical testing. In 2004, the National Institutes of Health launched a research initiative to improve the clinical research enterprise by developing state-of-the-art PROs. The NIH Patient-Reported Outcomes Measurement System (PROMIS) and Assessment Center are the products of that initiative. Adult, pediatric, and parent-proxy item banks have been developed by using contemporary psychometric methods, yielding rapid, accurate measurements. PROMIS currently provides tools for assessing physical, mental, and social health using short-form and computer-adaptive testing methods. The PROMIS tools are being adopted for use in clinical trials and translational research. They are also being introduced in clinical medicine to assess a broad range of disease outcomes. Recent legislative developments in the United States support greater efforts to include patients’ reports of health experience in order to evaluate treatment outcomes, engage in shared decision-making, and prioritize the focus of treatment. PROs have garnered increased attention by the Food and Drug Administration (FDA) for evaluating drugs and medical devices. Recent calls for comparative effectiveness research favor inclusion of PROs. PROs could also potentially improve quality of care and disease outcomes, provide patient-centered assessment for comparative effectiveness research, and enable a common metric for tracking outcomes across providers and medical systems.

Computerized Adaptive Testing in Poland / Komputerowe Testowanie Adaptacyjne W Polsce

Abstract New trends relating to computer-based testing of learners’ achievements are presented in the paper. It describes adaptive testing methods and results of studies in this problem area. Essential questions connected with the Item Response Theory (IRT) were also discussed. The presented data indicate that computer-based adaptive testing should be popularized in Poland to its fullest extent.

Computer Adaptive Testing Method for Measuring Disability in Patients With Back Pain

Most conventional instruments measuring disability rely on total score by simply adding individual item responses, which is dependent on the items chosen to represent the underlying construct (test-dependent) and a test statistic, such as coefficient alpha for the estimate of reliability, varying from sample to sample (sample-dependent). By contrast, item response theory (IRT) method focuses on the psychometric properties of the test items instead of the instrument as a whole. By estimating probability that a respondent will select a particular rating for an item, item difficulty and person ability (or disability) can be placed on same linear continuum. These estimates are invariant regardless of the item used (test-free measurement) and the ability of sample applied (sample-free measurement). These advantages of IRT allow the creation of invariantly calibrated large item banks that precisely discriminate the disability levels of individuals. Computer adaptive testing (CAT) method often requiring a testing algorithm promise a means for administering items in a way that is both efficient and precise. This method permits selectively administering items that are closely matched to the ability level of individuals (measurement precision) and measuring the ability without the loss of precision provided by the full item bank (measurement efficiency). These measurement properties can reasonably be achieved using IRT and CAT method. This article aims to investigate comprehensive overview of the existing disability instrument for back pain and to inform physical therapists of an alternative innovative way overcoming the shortcomings of conventional disability instruments. An understanding of IRT and CAT method will equip physical therapist with skills in interpreting the measurement properties of disability instruments developed using the methods.

An optimal test design to evaluate the ability of an examinee by using the stress–strength model

Although the item response theory (IRT) has been widely used to test systems such as the Test of English as a Foreign Language (TOEFL), it is not well known to teachers of mathematics in universities and colleges. The superiority of the IRT over the classical test method is also valid in mathematics in universities and high schools. Since the number of problems in mathematics to be tested in a short time period is strictly limited, particularly in universities, an accurate and efficient method to evaluate the abilities of students is strongly required. In this paper, we propose to use the stress–strength model to estimate the students’ abilities. To perform the optimal test, the up-and-down method by Dixon and Mood [W.J. Dixon and A.M. Mood, A method for obtaining and analyzing sensitivity data, J. Am. Statist. Assoc. 54 (1948), pp. 109–126] is incorporated. Then, the proposed model becomes a generalized test model that includes the stratified adaptive test method by Weiss and the up-and-down method [D.J. Weiss, The stratified adaptive computerized ability test, Research Report 73-3, Psychometric Methods Program, Department of Psychology, University of Minnesota, 1973].

Development of computerized adaptive testing (CAT) for the EORTC QLQ-C30 physical functioning dimension

Computerized adaptive test (CAT) methods, based on item response theory (IRT), enable a patient-reported outcome instrument to be adapted to the individual patient while maintaining direct comparability of scores. The EORTC Quality of Life Group is developing a CAT version of the widely used EORTC QLQ-C30. We present the development and psychometric validation of the item pool for the first of the scales, physical functioning (PF). Initial developments (including literature search and patient and expert evaluations) resulted in 56 candidate items. Responses to these items were collected from 1,176 patients with cancer from Denmark, France, Germany, Italy, Taiwan, and the United Kingdom. The items were evaluated with regard to psychometric properties. Evaluations showed that 31 of the items could be included in a unidimensional IRT model with acceptable fit and good content coverage, although the pool may lack items at the upper extreme (good PF). There were several findings of significant differential item functioning (DIF). However, the DIF findings appeared to have little impact on the PF estimation. We have established an item pool for CAT measurement of PF and believe that this CAT instrument will clearly improve the EORTC measurement of PF.

Speech Perception with Cochlear Implants as Measured Using a Roving-Level Adaptive Test Method

Aims: This paper uses a new method of speech testing where the hearing aid or cochlear implant (CI) users are tested in a more realistic listening situation. Methods: Groups of 11 subjects matched for performance with 5 different CI systems, for a total of 55 subjects, were tested with an adaptive test regime where the presentation level of the speech signal roved by ±10 or ±15 dB. Results: Speech reception thresholds varied widely between –4.8 and 17.3 dB with the ±15 dB roving condition being more difficult than the ±10 dB roving condition. We also found significant differences in speech reception threshold between groups using different devices. Conclusion: The test method used in our study, which attempts to test CI users in a more realistic listening situation, is sensitive to the effects of various subject-specific and technical parameters on everyday speech perception with CIs.

Functional Adaptive Sequential Testing

The study of cognition, perception, and behavior often requires the estimation of thresholds as a function of continuous independent variables (e.g., contrast threshold as a function of spatial frequency, subjective value as a function of reward delay, tracking speed as a function of the number of objects tracked). Unidimensional adaptive testing methods make estimation of single threshold values faster and more efficient, but substantial efficiency can be further gained by taking into account the relationship between thresholds at different values of an independent variable. Here we present a generic method--1functional adaptive sequential testing (FAST)--for estimating thresholds as a function of another variable. This method allows efficient estimation of parameters relating an independent variable (e.g., stimulus spatial frequency; or reward delay) to the measured threshold along a stimulus strength dimension (e.g., contrast; or present monetary value). We formally describe the FAST algorithm and introduce a Matlab toolbox implementation thereof; we then evaluate several possible sampling and estimation algorithms for such two-dimensional functions. Our results demonstrate that efficiency can be substantially increased by considering the functional relationship between thresholds at different values of the independent variable of interest.

Clinical Outcomes for Patients Classified by Fear-Avoidance Beliefs and Centralization Phenomenon

Werneke MW, Hart DL, George SZ, Stratford PW, Matheson JW, Reyes A. Clinical outcomes for patients classified by fear-avoidance beliefs and centralization phenomenon. Objectives To (1) determine the prevalence of pain pattern classification subgroups (centralization, noncentralization, and not classified) observed during the initial evaluation of patients experiencing high versus low Fear-Avoidance Beliefs Questionnaire for physical activity (FABQ-PA) scores, (2) examine the association between discharge pain intensity and functional status (FS) outcomes based on FABQ-PA and pain pattern subgroups, and (3) compare minimal clinically important improvement for FS and pain intensity for FABQ-PA and pain pattern classification subgroups. Design Observational cohort design. Setting Suburban hospital-based outpatient rehabilitation clinic. Participants Consecutive patients with low back syndromes (N=238, mean ± SD, 59.1±17.0y; minimum=20, maximum=91). Interventions Interventions were designed to match patient classification by fear-avoidance level and pain pattern. Main Outcome Measures Two outcome measures were assessed: patient self-reported FS and pain intensity. FS was assessed by using computerized adaptive testing methods. Maximal pain intensity was assessed by using an 11-point numeric pain scale: 0 (no pain) to 10 (worst imaginable pain). Results There were no differences (χ 2 2=3.7, P=.16) in proportion of patients classified by pain pattern experiencing high or low fear-avoidance beliefs. After controlling for the effect of available risk-adjustment variables, only dual-level classification subgroups, symptom acuity, payer type, and intake FS or pain intensity were associated with discharge FS or pain outcomes. The highest proportion of patients achieving minimal clinically important improvement in pain and FS were reported by the following patient subgroup: centralization and low fear. Conclusions Pain pattern and FABQ-PA characteristics impacted rehabilitation outcomes. We recommend that both factors be considered when managing patients with low back pain in an effort to optimize rehabilitation outcomes.