Dichotomous Scoring Research Articles

Analysis of Latent Dimensionality of Dichotomously and Polytomously Scored Test Data

Three closely related latent space dimensionality assessment procedures are surveyed. They are DIMTEST (called POLY-DIMTEST in the polytomous case), HCA/PROX, and DETECT. Each procedure works for both dichotomous and polytomous scoring. These procedures form a conceptual unity because they all estimate item pair conditional covariances given latent ability. As such, they are nonparametric and weak local independence based procedures. DIMTEST assesses for dichotomously scored items whether a test’s latent space is unidimensional or not. POLY-DIMTEST does the same for polytomously scored items. HCA/PROX does a hierarchical cluster analysis of items based on the Roussos dimensionality-sensitive proximity measure, thereby searching for item clusters that are dimensionally homogeneous within cluster and heterogeneous between clusters. When such approximate simple structure holds, DIMTEST is used in conjunction with HCA/PROX to identify which level of the HCA/PROX cluster partition hierarchy best describes the multidimensional simple structure driving the data. DETECT, with the aide of Zhang’s genetic algorithm based optimization procedure, counts the number of dominant dimensions, measures the size of the departure from unidimensionality, and searches for dimensionally homogeneous item clusters when approximate simple structure holds. All three procedures are described and judged effective based on a blend of simulation studies and real data analyses, using studies in the literature as well as some studies reported herein. Finally, an overview of some work in progress is given. This includes enhancements of the above procedures as well as some new procedures.

Increasing Alpha Reliabilities of Multiple-Choice Tests with Linear Polychotomous Scoring

Little research has been conducted on the use of linear polychotomous scoring of multiple-choice test items. Therefore, several tests were analyzed using both dichotomous and polychotomous scoring of test items to assess how the alpha reliabilities of the tests change based on the type of scoring used. In each case, the alpha reliabilities of the tests increased, with the same or fewer number of items in each test, when polychotomous (vs dichotomous) scoring of multiple-choice test items was used.

Volume detection threshold: quantitative comparison of computed radiography and screen-film radiography in detection of pneumothoraces in an animal model that simulates the neonate.

To quantitatively compare computed radiography (CR) and screen-film radiography (SFR) in the detection of pneumothorax in an animal model that simulates the neonate. Three rabbits underwent placement of 5-F catheters in the right pleural space. Eight CR and eight SFR images were obtained. Volume-controlled pneumothorax was induced by incrementally injecting 4 cm3 of air (24 cm3 total) and four CR and four SFR images were obtained after each injection. Four radiologists blindly viewed the images twice (eight readings per image, 1,600 total observations). A dichotomous yes-no score for the presence of pneumothorax was used to calculate the volume detection threshold (VDT), defined as the air volume at which 50% of the pneumothoraces can be detected. The authors plotted the likelihood of a pneumothorax against the air volume by using the confidence indicator. Intraobserver VDT values for CR and SFR were not significantly different. CR is as accurate as SFR in the detection of pneumothorax in this quantitative animal model.

The development of a Clinician-Administered PTSD Scale

Several interviews are available for assessing PTSD. These interviews vary in merit when compared on stringent psychometric and utility standards. Of all the interviews, the Clinician-Administered PTSD Scale (CAPS-1) appears to satisfy these standards most uniformly. The CAPS-1 is a structured interview for assessing core and associated symptoms of PTSD. It assesses the frequency and intensity of each symptom using standard prompt questions and explicit, behaviorally-anchored rating scales. The CAPS-1 yields both continuous and dichotomous scores for current and lifetime PTSD symptoms. Intended for use by experienced clinicians, it also can be administered by appropriately trained paraprofessionals. Data from a large scale psychometric study of the CAPS-1 have provided impressive evidence of its reliability and validity as a PTSD interview.

The development of a clinician‐administered PTSD scale

AbstractSeveral interviews are available for assessing PTSD. These interviews vary in merit when compared on stringent psychometric and utility standards. Of all the interviews, the Clinician‐Administered PTSD Scale (CAPS‐1) appears to satisfy these standards most uniformly. The CAPS‐1 is a structured interview for assessing core and associated symptoms of PTSD. It assesses the frequency and intensity of each symptom using standard prompt questions and explicit, behaviorally‐anchored rating scales. The CAPS‐1 yields both continuous and dichotomous scores for current and lifetime PTSD symptoms. Intended for use by experienced clinicians, it also can be administered by appropriately trained paraprofessionals. Data from a large scale psychometric study of the CAPS‐1 have provided impressive evidence of its reliability and validity as a PTSD interview.

Scoring and Standard Setting with Standardized Patients

The purpose of this study was to compare the continuous method of scoring a performance test composed of standardized patients with a derivative method that assigned each examinee a dichotomous score and to explore the use of Ang off 's method at the case level with these two scoring methods. Both scoring systems produced reasonable means and distributions of scores. The continuous scores were somewhat more reproducible than the dichotomous scores although neither was very relay?ducible. Pass rates for both scoring systems were appropriately very high and thus the reproducibility of the pass/fail decisions was also high. Regardless of the scoring system, the application of Angoff's method reported in this article has the advantages of efficiency and enhanced credibility to the experts.

Conditional Association, Essential Independence and Monotone Unidimensional Item Response Models

We consider two recent approaches to characterizing the manifest probabilities of a strictly unidimensional latent variable representation (one satisfying local independence and response curve monotonicity with respect to a unidimensional latent variable) for binary response variables, such as those arising from the dichotomous scoring of items on standardized achievement and aptitude tests. Holland and Rosenbaum showed that conditional association is a necessary condition for strict unidimensionality; and Stout treated the class of essentially unidimensional models, in which the latent variable may be consistently estimated as the length of the response sequence grows using the proportion of positive responses. Of particular concern are strictly unidimensional representations that are minimally useful in the sense that: (1) the latent variable can be consistently estimated from the responses; (2) the regression of proportion of positive responses on the latent variable is monotone; and (3) the latent variable is not constant in the population. We introduce two new conditions, a negative association condition and a natural monotonicity condition on the empirical response curves, that help link strict unidimensionality with the conditional association and essential unidimensionality approaches. These conditions are illustrated with a partial characterization of useful, strictly unidimensional representations.

A phase model of psychotherapy outcome: causal mediation of change.

A 3-phase model of psychotherapy outcome is proposed that entails progressive improvement of subjectively experienced well-being, reduction in symptomatology, and enhancement of life functioning. The model also predicts that movement into a later phase of treatment depends on whether progress has been made in an earlier phase. Thus, clinical improvement in subjective well-being potentiates symptomatic improvement, and clinical reduction in symptomatic distress potentiates life-functioning improvement. A large sample of psychotherapy patients provided self-reports of subjective well-being, symptomatic distress, and life functioning before beginning individual psychotherapy and after Sessions 2, 4, and 17 when possible. Changes in well-being, symptomatic distress, and life functioning means over this period were consistent with the 3-phase model. Measures of patient status on these 3 variables were converted into dichotomous improvement-nonimprovement scores between intake and each of Sessions 2, 4, and 17. An analysis of 2 x 2 cross-classification tables generated from these dichotomous measures suggested that improvement in well-being precedes and is a probabilistically necessary condition for reduction in symptomatic distress and that symptomatic improvement precedes and is a probabilistically necessary condition for improvement in life functioning.

The Relationship Between the Distribution of Item Difficulties and Test Reliability

Measurement authorities propose that the reliability of multiple-choice tests will be enhanced if the distribution of item difficulties is concentrated at approximately S O . This recommendation is reinforced and extended in this article by viewing the 0/1 item scoring as a dichotomization of an underlying normally distributed ability score. If guessing is not a consideration, the dichotomous item scoring leads to maximum item and test reliability when all items have a difficulty of S O . When guessing is possible, an upward shift leads to an optimal difficulty that lies between .57 and .67. It is further shown, however, that the effect on test reliability is surprisingly small, even when the distribution of item p values is spread between .27 and .79 or centers around a mean difficulty as high as .73.

Mokken Scale Analysis: Theoretical Considerations and an Application to Transitivity Tasks

This study provides a discussion and an application of Mokken scale analysis. Mokken scale analysis can be characterized as a nonparametric item response theory approach. The Mokken approach to scaling consists of two different item response models, the model of monotone homogeneity and the more restrictive model of double monotonicity. Methods for empirical data analysis using the two Mokken model versions are discussed. Both dichotomous and polytomous item scores can be analyzed by means of Mokken scale analysis. Three empirical data sets pertaining to transitive inference items were analyzed using the Mokken approach. The results are compared with the results obtained from a Rasch analysis.

Development of a novel EEG rating scale for head injury using dichotomous variables

We developed a new EEG rating scale for electrographic assessment of head injured patients. Phenomena present in posttraumatic EEG were scored as dichotomous variables (present or absent). These phenomena included background activity (alpha, beta, theta, delta), sleep spindles, focal abnormalities, reactivity and variability, epileptiform activity, and specific comatose patterns. Each variable was weighted according to its perceived prognostic value: i.e., normal alpha 10, flat EEG - 10, spindles 4, etc. Combinations of possible scores ranged from + 23 to - 10. Fifty-seven EEGs from different head injured patients were independently and retrospectively analyzed by two investigators. There was a high correlation for intra- ( r = 0.95) and inter- ( r = 0.85) observer rating using the dichotomous test. When patients with scores over 15 (i. e., with reactive alpha) and patients with scores of - 10 (i.e., ECI records) were excluded, the intra-rater and inter-rater correlations were still high (0.81 and 0.76, respectively). There was a high correlation between Glasgow outcome score at discharge and the dichotomous EEG score. This EEG scale scores most major categories of EEG activity, utilizes a multipoint scale for correlation purposes, and allows data to be analyzed in sub-categories (i.e., spindles in coma). The separate weighting score allows for refinement of the scale after data collection (i.e., to fit prospective outcome). We feel that this scale is reproducible and valid, and may be applicable to other patient groups with severely altered EEGs.

The spheres of control scale: 10 yr of research

The Spheres of Control Scale comprises three subscales: Personal, Interpersonal, and Socio-Political Control. The instrument has now been available for almost 10 yr and has been used in a wide variety of studies. To assist researchers using this instrument, we have assembled psychometric data on a wide variety of samples. These data include norms, correlates and reliabilities for the full scale and its three subscales. Statistics are reported for both the 7-point and dichotomous scoring formats. Several weak items on the personal control scale have been replaced. They improve the internal consistency of the Personal Control scale but do not affect the scale norms. Other items have been simplified. The items for the updated version (SOC-3) are provided.

An irt-based model for dichotomous longitudinal data

The LLRA (linear logistic model with relaxed assumptions; Fischer, 1974, 1977a, 1977b, 1983a) was developed, within the framework of generalized Rasch models, for assessing change in dichotomous item score matrices between two points in time; it allows to quantify change on latent trait dimensions and to explain change in terms of treatment effects, treatment interactions, and a trend effect. A remarkable feature of the model is that unidimensionality of the item set is not required. The present paper extends this model to designs with any number of time points and even with different sets of items presented on different occasions, provided that one unidimensional subscale is available per latent trait. Thus unidimensionality assumptions within subscales are combined with multidimensionality of the item set. Conditional maximum likelihood methods for parameter estimation and hypothesis testing are developed, and a necessary and sufficient condition for unique identification of the model, given the data, is derived. Finally, a sample application is presented.

Developmental relationships among metamemory, elaborative strategy use, and associative memory

Two experiments examined the relationships among metamemory knowledge, the use of associative memory strategies like elaboration, and cued-recall memory. In Experiment 1, 5th and 12th graders verbalized their strategies while studying noun pairs; their cued-recall memory of those pairs was then tested. Metamemory knowledge was assessed before or after this task. ANOVAs revealed developmental increases in associative strategies, cued recall, and metamemory. Multiple regression indicated the increase in associative strategies predicted all but 4% of the increase in cued recall; metamemory development in turn predicted all but 4% of the increase in associative strategies. Metamemory was a better predictor than a nonverbal measure of intelligence. The strong relationships were due partly to methodological refinements, including a counterbalanced design, a direct strategy measure, a reliable composite rather than single-item metamemory measure, ordinal rather than dichotomous scoring of metamemory items, and regression rather than χ 2 analyses. The results of Experiment 1 were replicated in Experiment 2. Metamemory development also predicted most of the grade difference in cued recall that remained after study strategies were equated by elaboration instructions, suggesting it may explain improvements in retrieval strategies as well.

Assessing Partial Knowledge in Vocabulary

This study reports an attempt to assess partial knowledge in vocabulary. Fifty multiple‐choice vocabulary items were constructed so that the incorrect choices followed the stages of vocabulary acquisition defined by O'Connor (1940). Ability estimates based on Rasch dichotomous and polychotomous models were compared to determine if there were any gains in validity or reliability as a result of using the polychotomous scoring model rather than the dichotomous scoring model. An attempt was also made to determine the appropriateness of O'Connor's stage theory of vocabulary acquisition for predicting the type of errors that examinees of differing ability would make on the test items. The results indicate that the reliability and concurrent validity of the polychotomous scoring of a subset of items that fit the polychotomous scoring model were significantly higher than those for dichotomous scoring of the same subset of items. The results also indicate moderate support for O'Connor's theory of vocabulary acquisition.

Optimal Detection of Certain Forms of Inappropriate Test Scores

Optimal appropriateness indices, recently introduced by Levine and Drasgow (1984), provide the highest rates of detection of aberrant response patterns that can be obtained from item responses. In this article they are used to study three important problems in ap propriateness measurement. First, the maximum detec tion rates of two particular forms of aberrance are de termined for a long unidimensional test. These detection rates are shown to be moderately high. Sec ond, two versions of the standardized l0 appropriate ness index are compared to optimal indices. At low false alarm rates, one standardized l0 index has detec tion rates that are about 65% as large as optimal for spuriously high (cheating) test scores. However, for the spuriously low scores expected from persons with ill-advised testing strategies or reading problems, both standardized l0 indices are far from optimal. Finally, detection rates for polychotomous and dichotomous scorings of the item responses are compared. It is shown that dichotomous scoring causes serious de creases in the detectability of some aberrant response patterns. Consequently, appropriateness measurement constitutes one practical testing problem in which sig nificant gains result from the use of a polychotomous item response model.

Optimal scoring of amnesia on the Harvard Group Scale of Hypnotic Susceptibility, Form A.

Abstract Response to the suggestion for posthypnotic amnesia on the Harvard Group Scale of Hypnotic Susceptibility, Form A (Shor & E. Orne, 1962) was studied in a sample of 1351 Ss. Reversibility proved to be a somewhat better criterion of response than initial amnesia, as indexed by both item difficulty and item-to-total correlations. Continuous scoring of recall, however, was no better than dichotomous scoring. A joint dichotomous criterion, consisting of recall = s 3 on the initial amnesia test followed by recall 2 2 additional items on the reversibility test, is offered as both empirically defensible and conceptually preferable.

Note on the Reliability and Scoring of the Chapin Social Insight Test

The Chapin Social Insight Test ( 1) was developed to measure a person's ability to recognize the psychodynamics in a particular situation and to resolve problematic situations. Although this test has been available for many years, the only assessments of its properties are those reported by Gough (2, 3). The present study was undertaken to update the information on the reliability of the Chapin cest and to assess two methods of scoring. Students in lower level psychology classes were the subjects. There were 53 males and 112 females. The subjects' ages ranged from 18 to 67 yr. (M = 23.32 yr., SD = 7.61). The Insight test was administered individually to 30 males and 65 females and to two groups of 23 males and 47 females who took the test during class periods. Insight scores were computed using the recommended item weights and by scoring the items dichotomously. Using the first procedure, scores ranged from 11 to 34 (M = 19.90, SD = 5.12). Using the dichotomous method, scores ranged from 7 to 21 (M = 11.87, SD = 2.86). Those who took the cest individually did not differ from those who took it in groups for both the recommended scoring (M = 19.80 and SD = 5.70 vs M = 20.04 and SD = 4.22) and the dichotomous scoring (M = 11.79 and SD = 3.11 vs M = 11.97 and SD = 2.48). Total scores by both procedures were highly correlated (s = 95, df = 163). The average item-total correlation for the weighted-scoring technique was .25, resulting in a reliability estimate of .62. For the dichotomous method, the average item-total correlation was .26 and the reliability estimate was .65. These are similar to the item-total correlations and reliabilities reported by Gough (3). The results of this study support the use of the Chapin Social Insight Test as an adequately reliable measure of the construct. However, since the dichotomous scoring procedure was as effective as the prescribed weighted-scoring method and the totals derived by each method were highly correlated, the use of the weighted scoring does not seem to be necessary (4).

Risk taking through the adult life span.

Research on aging in general and on cautiousness in adulthood in particular has been plagued by several problems including (a) drawing of samples from the extremes of the adult age range; (b) relying on single criterion variables; and (c) paucity of data on the psychometric properties of scales. In the present study, three measures of risk taking were administered to a sample ranging from 18-78 years old. Reliability estimates indicated that the Choice Dilemmas Questionnaire had low reliability, both absolutely and in comparison to the vocabulary Risk Test and the Risk Taking Questionnaire. Convergent validity analyses indicated that the Choice Dilemma Questionnaire was substantially correlated with the Risk Taking Questionnaire but not with the Vocabulary Risk Test. Age differences were found only on the Choice Dilemmas Questionnaire using dichotomous scoring. Results suggest that properties of the scale may, in part, account for observed age differences in risk taking scores on the Choice Dilemma Questionnaire. The currently popular notion that people become more cautious as they grow older was challenged and directions were provided for future research.

Biserial Estimators in the Presence of Guessing

As estimators of correlation of a criterion variable with the continuous latent variable that underlies a dichotomous test item score, both the biserial correlation coefficient, rb, and Brogden’s coefficient of selective efficiency, S*, are severely depressed (negatively biased) by guessing, regardless of sample size or true latent correlation. Formulas and charts are given for computing better estimates, r̂ and Ŝ*, free of guessing bias, based on observed proportions of right, wrong, and omitted answers. In Monte Carlo studies both r̂ and Ŝ* had smaller mean square errors in the presence of guessing than rb and S*, and Ŝ* stayed on target even when the latent and criterion variable were given (the same) rectangular, bimodal, or chi square distribution.