Mental Test Theory Research Articles

A scoping review of Rasch analysis and item response theory in otolaryngology: Implications and future possibilities.

Item response theory (IRT) is a methodological approach to studying the psychometric performance of outcome measures. This study aims to determine and summarize the use of IRT in otolaryngological scientific literature. A systematic search of the Medline, Embase, and the Cochrane Library databases was performed for original English-language published studies indexed up to January 28, 2023, per the following search strategy: ("item response theory" OR "irt" OR "rasch" OR "latent trait theory" OR "modern mental test theory") AND ("ent" OR "otorhinolaryngology" OR "ear" OR "nose" OR "throat" OR "otology" OR "audiology" OR "rhinology" OR "laryngology" OR "neurotology" OR "facial plastic surgery"). Fifty-five studies were included in this review. IRT was used across all subspecialties in otolaryngology, and most studies utilizing IRT methodology were published within the last decade. Most studies analyzed polytomous response data, and the most commonly used IRT models were the partial credit and the rating scale model. There was considerable heterogeneity in reporting the main assumptions and results of IRT. IRT is increasingly being used in the otolaryngological scientific literature. In the otolaryngology literature, IRT is most frequently used in the study of patient-reported outcome measures and many different IRT-based methods have been used. Future IRT-based outcome studies, using standardized reporting guidelines, might improve otolaryngology-outcome research sustainably by improving response rates and reducing patient response burden. 2.

A dialectic on validity: Explanation-focused and the many ways of being human

In line with the journal volume’s theme, this essay considers lessons from the past and visions for the future of test validity. In the first part of the essay, a description of historical trends in test validity since the early 1900s leads to the natural question of whether the discipline has progressed in its definition and description of test validity. There is no single agreed-upon definition of test validity; however, there is a marked coalescing of explanation-centered views at the meta-level. The second part of the essay focuses on the author's development of an explanation-focused view of validity theory with aligned validation methods. The confluence of ideas that motivated and influenced the development of a coherent view of test validity as the explanation for the test score variation and validation is the process of developing and testing the explanation guided by abductive methods and inference to the best explanation. This description also includes a new re-interpretation of true scores in classical test theory afforded by the author’s measure-theoretic mental test theory development—for a particular test-taker, the variation in observed test-taker scores includes measurement error and variation attributable to the different ecological testing settings, which aligns with the explanation-focused view wherein item and test performance are the object of explanatory analyses. The final main section of the essay describes several methodological innovations in explanation-focused validity that are in response to the tensions and changes in assessment in the last 25 years.

The Determination of Sample Size in a Bayesian Estimation of Population Proportions: How and Why to Do It in a Regression Framework

Principally to reduce the cost by reducing the sample size required to conduct survey research, this article presents and illustrates the use of a method to determine the sample sizes required to obtain Bayesian estimates of population proportions with specified margins of error. The development proceeds within a regression framework derived from mental test theory. Specifically, building on prior work, the development presented here enables a researcher to conduct a survey to obtain pure Bayes estimates of the proportion of all members of a defined population choosing each one of a number of mutually exclusive and exhaustive options or falling into each one of a number of mutually exclusive and exhaustive categories, including two. The regression framework not only provides useful insight into Bayesian and classical statistics but also enables the development to proceed without explicit reference to the differing parent distributions of the sample and population proportions, both being asymptotically normal. In addition to the sample-size advantage, which is substantial, this article identifies other practical advantages that Bayesian has over classical estimation of population proportions and, in a somewhat in-depth comparison of the two, discusses other reasons a Bayesian method may be a powerful substitute for the classical method of estimating population proportions via independent random sampling.

Измерение критического мышления взрослых: методология и опыт разработки

<p>Critical thinking (next CT) is one of the key competencies in the modern world. Despite the variety of assessment systems for secondary and higher education, there is a lack of tests for assessing this competence for adults. The importance of adult assessment is emphasized primarily by the need for employers to have employees with developed CT in various work contexts. The article describes the procedure for creating a measurement test for students of the training course «Critical Thinking» of the «Yandex.Practice» platform and the results of assessing its psychometric quality. When creating this test, we relied on the Evidence Centered Design (ECD) approach. The test is performed in computer form and consists of tasks aimed at identifying the relevant observed behavior of the subject. The validity of the developed test, including its compliance with theoretical expectations about the factor structure, is confirmed by evidence of constructive, criterion, convergent and content validity. The limitations of this study and possible directions for future developments are considered. The result of the work is a validated test for measuring critical thinking for adults, the quality of the tasks of which is confirmed by psychometric analysis in modern mental test theory (IRT — Item Response Theory).</p>

On the complementarity of classical test theory and item response models: item difficulty estimates and computerized adaptive testing

This study aims to provide statistical evidence of the complementarity between classical test theory and item response models for certain educational assessment purposes. Such complementarity might support, at a reduced cost, future development of innovative procedures for item calibration in adaptive testing. Classical test theory and the generalized partial credit model are applied to tests comprising multiple choice, short answer, completion, and open response items scored partially. Datasets are derived from the tests administered to the Portuguese population of students enrolled in the 4th and 6th grades. The results show a very strong association between the estimates of difficulty obtained from classical test theory and item response models, corroborating the statistical theory of mental testing.

On Issues of Validity and Especially on the Misery of Convergent Validity

On Issues of Validity and Especially on the Misery of Convergent Validity

Applying IRT to Estimate Learning Ability and K-means Clustering in Web based Learning

E-learning provides a convenient and efficient way for learning and enriching people’s lives. But there is no appropriate way to estimate and diagnose people when they learning with e-learning environment/system. For learning ability estimation issue, Item Response Theory which plays an important role in modern mental test theory is applied. Besides, K-means clustering method is also applied to cluster learner’s ability for remedying courses or enhancing courses. We integrate these two theories and propose a combination methodology to solve the estimation and diagnostic issues in e-learning environment. A web-based assist system is provided as well. Experimental data is collected with forty sophomore students studying “Business Data Communication” class at Dept. of Information Management in Chung Hua University in Taiwan. We illustrated the method to observe and estimate the variation of learner’s ability. This methodology and system could make some valuable contribution in e-learning related study and society.

A Note on Team-Specific Home Advantage in the NBA

Recently it was reported that in the NBA as a whole, two thirds of the home advantage which teams enjoy when playing at home is accumulated in the first quarter. Home advantage can also be determined for individual teams, and there is good reason for doing so. For example, the relation of home advantage to team statistics such as assists, rebounds, and turnovers can be studied team-specifically but not in the league as a whole. Before any such project is undertaken, however, a major technical problem must be addressed. Formally, team-specific home advantage is a difference score between positively correlated variables (games won at home minus games won away), and difference scores are notoriously unreliable. This unreliability, moreover, is not just an empirical generalization. There is a formal basis for it in the theory of mental tests. This study reports that over a four-year period in the NBA the estimated reliability of team-specific home advantage was 0.284, even though the estimated reliabilities of games won at home and games won away were 0.772 and 0.833 respectively. The implications for research on home advantage are discussed.

THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES*

THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES*

The theory of mental testing and the correlation between physical noise level and annoyance

Applying the ‘‘theory of psychological testing,’’ physical noise measurement procedures are regarded as ‘‘tests’’ variables and the related annoyance as the criterion variable. The concepts of ‘‘reliability,’’ ‘‘validity’’ and ‘‘equivalence’’ then can be used to assess the precision of different noise measurements. Referring to data of an investigation on aircraft noise in the vicinity of Munich Airport in 1969, in which different physical as well as annoyance data were sampled from the same subjects, it turns out, that the measurements called Leq1, Ls, Leq3, Leq4, NNI and FB1 meet the equivalence criterions of test theory, that is, they cannot statistically be distinguished, and their coefficients of reliability are close to one. Other measurements like D10, H81, log N and Leq10, equivalent to the former six. However, even the former six are only of moderate validity (about 0.5). Regarding psychological testing, therefore, physical noise is only a poor measure when used to predict individual annoyance, but it suffices when used to predict group averages. Moreover, it can be concluded from the theory that the attempt to enhance validity by modification of the highly reliable physical measuring procedures cannot be successful. It would be more effective to enlarge the reliability of the psychological measurement procedures of annoyance.

Comprehensive Latent Trait Theory

Latent trait theory, or item response theory, started as a mental test theory, or as a social attitude measurement theory, dealing solely with the dichotomous response level. It has been developed tremendously in the past generation both in theory and in practice, however, to include the graded response level, the nominal response level and the continuous response level, both in the unidimensional and in the multidimensional latent spaces. The present paper attempts to give the reader some perspectives of this comprehensive theory, without intending to be a fair review of all the published papers in this area. Some promising applications, including the computerized adaptive testing and the Rorschach diagnosis, are also given.

Models of Decisionmaking Processes for Multiple‐Choice Test Items: An Analysis of Spatial Ability

Latent class models of decisionmaking processes related to multiple‐choice test items are extremely important and useful in mental test theory. However, building realistic models or studying the robustness of existing models is very difficult. One problem is that there are a limited number of empirical studies that address this issue. The purpose of this paper is to describe and illustrate how latent class models, in conjunction with the answer‐until‐correct format, can be used to examine the strategies used by examinees for a specific type of task. In particular, suppose an examinee responds to a multiple‐choice test item designed to measure spatial ability, and the examinee gets the item wrong. This paper empirically investigates various latent class models of the strategies that might be used to arrive at an incorrect response. The simplest model is a random guessing model, but the results reported here strongly suggest that this model is unsatisfactory. Models for the second attempt of an item, under an answer‐until‐correct scoring procedure, are proposed and found to give a good fit to data in most situations. Some results on strategies used to arrive at the first choice are also discussed

Test theory and psychometrika: The past twenty-five years

On the occasion ofPsychometrika's fiftieth anniversary, the past twenty-five years' developments in mental test theory are reviewed, with special emphasis on the topics receiving attention in the pages of this journal. (Analogous reviews forPsychometrika's first quarter century were given by Gulliksen and Guilford in 1961.) Much of the recent progress in test theory (and in other branches of psychometrics as well) has been made by treating the problems in this field as being essentially ones of statistical inference. It is concluded that (a) research in test theory is in a healthy state and (b)Psychometrika is an important source of information about that research.

Advances in Item Response Theory and Applications: An Introduction

Test theories can be divided roughly into two categories. The first is classical test theory, which dates back to Spearman’s conception of the observed test score as a composite of true and error components, and which was introduced to psychologists at the beginning of this century. Important milestones in its long and venerable tradition are Gulliksen’s Theory of Mental Tests (1950) and Lord and Novick’s Statistical Theories of Mental Test Scores (1968). The second is item response theory, or latent trait theory, as it has been called until recently. At the present time, item response theory (IRT) is having a major impact on the field of testing. Models derived from IRT are being used to develop tests, to equate scores from nonparallel tests, to investigate item bias, and to report scores, as well as to address many other pressing measurement problems (see, e.g., Hambleton, 1983; Lord, 1980). IRT differs from classical test theory in that it assumes a different relation of the test score to the variable measured by the test. Although there are parallels between models from IRT and psychophysical models formulated around the turn of the century, only in the last 10 years has IRT had any impact on psychometricians and test users. Work by Rasch (1980/1960), Fischer (1974), 9 Birnbaum (1968), ivrighi and Panchapakesan (1969), Bock (1972), and Lord (1974) has been especially influential in this turnabout; and Lazarsfeld’s pioneering work on latent structure analysis in sociology (Lazarsfeld, 1950; Lazarsfeld & Henry, 1968) has also provided impetus. One objective of this introduction is to review the conceptual differences between classical test theory and IRT. A second objective is to introduce the goals of this special issue on item response theory and the seven papers. Some basic problems with classical test theory are reviewed in the next section. Then, IRT approaches to educational and psychological measurement are presented and compared to classical test theory. The final two sections present the goals for this special issue and an outline of the seven invited papers.

A Review of the Beta-Binomial Model and its Extensions

Both the binomial and beta-binomial models are applied to various problems occurring in mental test theory. The paper reviews and critiques these models. The emphasis is on the extensions of the models that have been proposed in recent years, and that might not be familiar to many educators.

Konstruktion übergeordneter AMP-Skalen: „manisch-depressives“ und „schizophrenes Syndrom«6

A previous two-factor solution of the AMP system was confirmed by a factor analysis in 552 patients, mainly suffering from endogenous psychoses. Based on these results, two second-order scales ('manic-depressive' and 'schizophrenic syndrome') were constructed according to the classical theory of mental testing. The scales are orthogonal. Their reliability and validity is satisfactory. It is suggested to use the two new scales together with the known 9 AMP scales.

Estimating true score in the compound binomial error model

Several procedures have been proposed in the statistical literature for estimating simultaneously the mean of each ofk binomial populations. In terms of mental test theory, however, it is not clear that these procedures should be used when an item sampling model applies since the binomial error model is usually viewed as an oversimplification of the “true” situation. In this study we compare empirically several of these estimation techniques. Particular attention is given to situations where observations are generated according to a two-term approximation to the compound binomial distribution.

The measurement of differential deficit

DISTURBED schizophrenics show a generalized performance deficit. They perform less well than normal subjects on almost any task that requires a voluntary response. Therefore, a lower than normal score on any single task cannot be interpreted as indicating a deficit of special importance in schizophrenia. Recognizing this, most investigators of cognitive deficit in schizophrenia have been concerned with differential deficit, that is, a greater deficit in one ability than in at least one other ability. Differential deficit in ability is measured using two or more tasks. A differential deficit in ability is inferred customarily from a greater performance deficit on one task than on at least one other task. On previous occasions we have pointed out that a differential deficit in performance does not necessarily indicate a differential deficit in ability. 1.2 It may, instead, merely reflect the schizophrenics’ generalized performance deficit coupled with the fact that one of the two tasks measures generalized deficit better than the other. We have pointed out that the higher are the reliability and the variance of a task, the greater will be the difference in mean score between the more competent and the less competent subjects. Since schizophrenics usually perform less well than normal subjects, they will show a greater performance deficit on the one of the two tasks that yields the greater dispersion of scores. We have advocated that to measure differential deficit in ability, one must match tasks on several psychometric characteristics. We explore this problem again now in order to offer what we believe to be a clearer statement of the problem. In addition, we now propose a solution with slightly less cumbersome requirements for adequately matched tests. (We use the term ‘tests’ here in the broad sense to include any task that is used to measure cognitive deficit). The central problem is how to move from statements about differential deficit in performance on specific tests to statements about differential deficit in ability. We will present here some implications of classic mental test theory for the solution of this problem. Some principles of psychometric theory When parallel tests are given on successive occasions, any given subject will usually not earn identical scores on the two testings. He will score differently even when he does not change in his knowledge of the material and shows no practice effects as a result of his taking the earlier tests. These fluctuations in test score are attributable to changes in such variables as the subject’s motivation, fatigue, emotional state, alertness, luck in guessing, and to *Preparation of this article was supported by a research grant (MH-18354) from the National Institute of Mental Health, United States Public Health Service.

STATISTICAL PROBLEMS IN MENTAL TEST THEORY*

ABSTRACTSome unsolved statistical problems in mental test theory are outlined.

Empirical bayes point estimates of latent trait scores without knowledge of the trait distribution

In this paper, recent developments in empirical Bayes procedures are tied in with current work in mental test theory. Point estimators of “true scores” are derived for the binomial and Rasch test models. These estimators are shown to be asymptotically optimal. Smoothing and an empirical study of the behavior of empirical Bayes estimates are taken up in the final section.