IRT Scale Research Articles

VPBrowse: Genome-based representation of MS/MS spectra to quantify 10,000 bovine proteins.

SWATH is a data acquisition strategy acclaimed for generating quantitatively accurate and consistent measurements of proteins across multiple samples. Its utility for proteomics studies in nonlaboratory animals, however, is currently compromised by the lack of sufficiently comprehensive and reliable public libraries, either experimental or predicted, and relevant platforms that support their sharing and utilization in an intuitive manner. Here we describe the development of the Veterinary Proteome Browser, VPBrowse (http://browser.proteo.cloud/), an on-line platform for genome-based representation of the Bos taurus proteome, which is equipped with an interactive database and tools for searching, visualization, and building quantitative mass spectrometry assays. In its current version (VPBrowse 1.0), it contains high-quality fragmentation spectra acquired on QToF instrument for over 36,000 proteotypic peptides, the experimental evidence for over 10,000 proteins. Data can be downloaded in different formats to enable analysis using popular software packages for SWATH data processing whilst normalization to iRT scale ensures compatibility with diverse chromatography systems. When applied to published blood plasma dataset from the biomarker discovery study, the resource supported label-free quantification of additional proteins not reported by the authors previously including PSMA4, a tissue leakage protein and a promising candidate biomarker of animal's response to dehorning-related injury.

The growth of knowledge and self-perceived competence during long-term internships: comparing preparatory versus accompanying seminars in teacher education programs

Pedagogical field experiences offer valuable learning opportunities in teacher education programs. While most previous studies have exclusively addressed self-perceived competencies, less is known about the extent to which competence objectively changes during internships. With this in mind, this study aims to examine the development of knowledge on instruction, self-perceived competence, teacher self-efficacy, and their interrelationship during a long- term internship with accompanying or preparatory seminars in educational sciences. Therefore, 275 student teachers were surveyed and structural equation modelling, IRT scaling, and analyses of variances were used as analytical methods. Results of the study revealed a significant change in self- perceived competence and teacher self-efficacy, but only a minimal change in knowledge. The results also showed that accompanying seminars lead to a substantial increase in knowledge relative to preparatory seminars. Furthermore, it turned out that self-perceived competence and its change are less related to knowledge as an objective measure of competence than future teachers’ self- efficacy. Based on these findings, the results from previous studies on student teaching and its effectiveness need to be interpreted more cautiously.

Response Demands of Reading Comprehension Test Items: A Review of Item Difficulty Modeling Studies

ABSTRACT Item difficulty modeling studies involve (a) hypothesizing item features, or item response demands, that are likely to predict item difficulty with some degree of accuracy; and (b) entering the features as independent variables into a regression equation or other statistical model to predict difficulty. In this review, we report findings from 13 empirical item difficulty modeling studies of reading comprehension tests. We define reading comprehension item response demands as reading passage variables (e.g., length, complexity), passage-by-item variables (e.g., degree of correspondence between item and text, type of information requested), and item stem and response option variables. We report on response demand variables that are related to item difficulty and illustrate how they can be used to manage item difficulty in construct-relevant ways so that empirical item difficulties are within a targeted range (e.g., located within the Proficient or other proficiency level range on a test’s IRT scale, where intended).

The impact of the difficulty level of compromised common items on IRT scaling and equating under the common item equating design

The common item equating design requires two forms of a test which have a set of items in common in order to control for differences in examinee ability. The common set is subject to compromise when it is used repeatedly, which most likely becomes a serious threat to test fairness. If cheating occurs on common items, the equating process produces inaccurate results which might vary as a result of common item difficulty. This simulation study was conducted to evaluate the impact of the difficulty level of compromised common items on the equating process. The recovery of scaling coefficients and equated scores was assessed using bias and RMSE under various cheating conditions. The results indicated that cheating on higher-difficulty common items produced the most overestimation in the scaling coefficients; which, in turn, caused the most inflation in equating true scores for all test takers, whether they engage in cheating or not.

The Impact of the Difficulty Level of Compromised Common Items on IRT Scaling and Equating under the Common-Item Equating Design

The Impact of the Difficulty Level of Compromised Common Items on IRT Scaling and Equating under the Common-Item Equating Design

Parallel lines? Policy mood in a plurinational democracy

We usually think of democratic accountability in national terms – if the people do not approve of a government, they can replace it. However, in a plurinational democracy it is not obvious that such a single national public exists. We consider this problem in the case of Scotland, providing the first application of the macro-polity approach to a plurinational democracy. We provide a systematic study of how public opinion in Scotland changes over time compared to that in the rest of Great Britain, using recently developed Bayesian IRT scaling techniques, and ask whether Scottish and British public opinion move in parallel. To the extent that there is a separate Scottish public opinion with a separate party system, this forces us to rethink the way that democracy and accountability work in plurinational political systems.

Linking Large-Scale Reading Assessments: Measuring International Trends Over 40 Years

ABSTRACTSince the start of the new millennium, international comparative large-scale studies have become one of the most well-known areas in the field of education. However, the International Association for the Evaluation of Educational Achievement (IEA) has already been conducting international comparative studies for about half a century. The present study aims to demonstrate how to link recent and older studies onto the same scale in order to study long-term trends within and across countries. It discusses the comparability of the assessment material in the Progress in International Reading Literacy Study (PIRLS) and previous IEA studies on reading at the end of primary school. Thereafter, we use a concurrent calibration of all item parameters to link the studies onto a common IRT scale extending from 1970 until the present.

The Effect of Changing Content on IRT Scaling Methods

Equating test forms is an essential activity in standardized testing, with increased importance with the accountability systems in existence through the mandate of Adequate Yearly Progress. It is through equating that scores from different test forms become comparable, which allows for the tracking of changes in the performance of students from one year to the next. This study compares three different item response theory scaling methods (fixed common item parameter, Stocking & Lord, and Concurrent Calibration) with respect to examinee classification into performance categories, and estimation of the ability parameter, when the content of the test form changes slightly from year to year, and the examinee ability distribution changes. The results indicate that calibration methods, especially concurrent calibration, produced more stable results than the transformation method.

The Long-Term Sustainability of IRT Scaling Methods in Mixed-Format Tests

Due to recent research in equating methodologies indicating that some methods may be more susceptible to the accumulation of equating error over multiple administrations, the sustainability of several item response theory methods of equating over time was investigated. In particular, the paper is focused on two equating methodologies: fixed common item parameter scaling (with two variations, FCIP-1 and FCIP-2) and the Stocking and Lord characteristic curve scaling technique in the presence of nonequivalent groups. Results indicated that the improvements made to fixed common item parameter scaling in the FCIP-2 method were sustained over time. FCIP-2 and Stocking and Lord characteristic curve scaling performed similarly in many instances and produced more accurate results than FCIP-1. The relative performance of FCIP-2 and Stocking and Lord characteristic curve scaling depended on the nature of the change in the ability distribution: Stocking and Lord characteristic curve scaling captured the change in the distribution more accurately than FCIP-2 when the change was different across the ability distribution; FCIP-2 captured the changes more accurately when the change was consistent across the ability distribution.

Measurement of vocational competences: an analysis of the structure and reliability of current assessment practices in economic domains

Both fostering and measuring action competence remain central targets of vocational education and training research; adequate measurement approaches clearly are prerequisites for international, large-scale assessments. For the German Chamber of Commerce and Industry, competence assessments of industrial managers rely mainly on final examinations that attempt to measure not just knowledge but also action competence. To evaluate this test instrument, this article considers two questions: (1) Can the test assess action competence with validity, and (2) how reliable are the corresponding assessment results? The study relied on statistical procedures (e.g., IRT scaling), applied empirically to a sample of 1,768 final examinations. As a result the current examination appears neither adequate nor accurate as an instrument to capture action competence. We conclude that several improving steps have to be undertaken to improve the economic assessment.

Using IRT Trait Estimates Versus Summated Scores in Predicting Outcomes

It has been argued that item response theory trait estimates should be used in analyses rather than number right (NR) or summated scale (SS) scores. Thissen and Orlando postulated that IRT scaling tends to produce trait estimates that are linearly related to the underlying trait being measured. Therefore, IRT trait estimates can be more useful than summated scores when examining relationships between test scores and external variables. Also, when the model holds, IRT trait estimates possess an interval scale that is a property assumed for dependent variables by most statistical procedures used in educational research. The objective of this study was to use Monte Carlo methods to compare the performance of IRT trait estimates and SS scores in predicting outcome variables in the context of health and behavioral assessment. The use of scores based on the graded-response model versus summated scores was compared. Results indicated that IRT-based scores and summated scores are comparable when evaluating the relationships between test scores and outcome measures. Thus, applied researchers could use summated scores in predictive studies and circumvent evaluating the assumptions underlying use of IRT-based scores.

A Comment on Early Student Blunders on Computer-Based Adaptive Tests

This article refutes a recent claim that computer-based tests produce biased scores for very proficient test takers who make mistakes on one or two initial items and that the ‘‘bias’’ can be reduced by using a four-parameter IRT model. Because the same effect occurs with pattern scores on nonadaptive tests, the effect results from IRT scoring, not from adaptive testing. Because very proficient test takers rarely err on items of middle difficulty, the so-called bias is one of selective data analysis. Furthermore, the apparently large score penalty for one error on an otherwise perfect response pattern is shown to result from the relative stretching of the IRT scale at very high and very low proficiencies. The recommended use of a four-parameter IRT model is shown to have drawbacks.

Disaggregated Outcomes of Gender, Ethnicity, and Poverty on Fifth Grade Science Performance

This nationwide study examined the relationships among gender, ethnicity, and poverty with fifth graders’ (n = 8,741) science performance. Extant fifth grade data files (2003–2004), from the Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), were used. An ANOVA test revealed that males performed significantly better than females on science assessments, and this difference was maintained across ethnic groups. The science scores, in order of ethnicity from the highest to lowest scores, were White, Asian, Hispanic, and African American. Asian fifth graders showed the largest ethnic discrepancy between students above and below poverty. The unique features of this study were the depth of the disaggregation of the data and statistical analyses. Disaggregation of data by all three variables revealed compounding consequences for students at the extremes. White “at/above poverty” males had the highest mean science IRT scale score, while African American “below poverty” females had the lowest mean score.

Agreement of patient and physician ratings on mobility and self-care in neurological diseases

With a newly developed questionnaire, which integrates the patient and provider ratings on mobility and self-care on an IRT scale (MOSES-Combi) and thus ensures the same measurement scale for patients and providers, we examined the level of agreement between neurology patients and the physicians treating them, how agreement changes after rehabilitation, and what factors affect the extent of agreement. A total of 258 neurology patients from nine inpatient rehabilitation centers and the physicians treating them filled out the MOSES-Combi at the beginning and end of rehabilitation. Multilevel analyses were conducted. While the average value of the person parameters for patients and physicians is quite similar at the beginning of rehabilitation, there are highly significant differences after rehabilitation. The patient generally describes himself as more limited. Agreement at the individual level at the start of rehabilitation is moderate to substantial, with ICCs between .552 and .750. At the end of rehabilitation, agreement reduces. Factors that predict agreement are site and setting rehabilitation goals in the respective areas of mobility and self-care being observed. Our study indicates that general organizational conditions can have an influence on the extent of agreement.

Post-hoc IRT equating of previously administered English tests for comparison of test scores

Despite growing concerns about declining scholastic abilities of Japanese students throughout Japan prior to the implementation of the revised Courses of Study in 2002, little empirical evidence was available at that time to support this perceived decline in academic performance. This research describes post-hoc IRT equating of previously administered English tests for retrospective comparison of test scores. Eight forms of a prefecture-(state-)wide English test for high school students administered from 1995 to 2002 were analyzed through the two-parameter IRT model. Because there were no common items and no common examinees among the eight forms, seven composite tests were created for equating with a common-examinee design. Using the mean and sigma method to estimate equating constants, the ability parameter estimates of approximately 140,000 examinees became comparable on a common IRT scale. The results showed that the mean scores among first-year high school students were in a declining trend during the eight-year period. This is a first step toward demonstrating trends in English ability among Japanese students year by year based on IRT using previously administered large-scale English tests. The post-hoc equating method presented here is applicable to any similar circumstance in language testing research.

A Unified Approach to IRT Scale Linking and Scale Transformations

Abstract. This paper examines IRT scale transformations and IRT scale-linking methods used in the nonequivalent groups with anchor test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean-mean, mean-var linking, concurrent calibration, Stocking and Lord, and Haebara characteristic curves approaches, and fixed-item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then, a rewriting of the log-likelihood function together with an appropriately implemented maximization procedure of the log-likelihood function under linear (or nonlinear) restrictions will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models such as the generalized partial credit model.

Modelling mathematics problem solving item responses using a multidimensional IRT model

This research examined students’ responses to mathematics problem-solving tasks and applied a general multidimensional IRT model at the response category level. In doing so, cognitive processes were identified and modelled through item response modelling to extract more information than would be provided using conventional practices in scoring items. More specifically, the study consisted of two parts. The first part involved the development of a mathematics problem-solving framework that was theoretically grounded, drawing upon research in mathematics education and cognitive psychology. The framework was then used as the basis for item development. The second part of the research involved the analysis of the item response data. It was demonstrated that multidimensional IRT models were powerful tools for extracting information from a limited number of item responses. A problem-solving profile for each student could be constructed from the results of IRT scaling.

Retrospective Study of English Achievement at a High School Using an IRT Scale

Retrospective Study of English Achievement at a High School Using an IRT Scale

EXAMINING THE VALIDITY OF A COMPUTER‐BASED GENERATING‐EXPLANATIONS TEST IN AN OPERATIONAL SETTING

ABSTRACTGenerating Explanations (GE) is a computer‐delivered item type that presents a situation and asks the examinee to pose as many plausible reasons for it as possible. Previous research suggests that GE measures a divergent thinking ability largely independent of the convergent skills tapped by the GRE General Test. This study was conducted to determine if prior GE validity results generalized to the GRE candidate population, how population groups performed, what effects partial‐credit modeling might have for validity, and what problems were associated with operational administration. Validity results showed that earlier findings were generally supported: GE was found to be reliable but only marginally related to the General Test and to make significant (but small) independent contributions to the explanation of relevant criteria. With respect to population groups, GE produced smaller gender and ethnic group differences than did the General Test and showed the same relations to outside criteria across groups, suggesting it was measuring similar skills in each population. Attempts to model GE responses on a partial‐credit IRT scale succeeded but produced no improvement in relations with external criteria over those obtained by summing raw item scores. Finally, interviews conducted with examinees to detect potential delivery problems suggested that the directions needed to be shortened.

Scale Shrinkage and the Estimation of Latent Distribution Parameters

In this paper the phenomenon of scale shrinkage is examined. Specifically, emphasis is placed on the pattern of decreasing variances in IRT scale scores from fall to spring within a grade. It is concluded that certain situations exist in which scale shrinkage is predictable with unidimensional tests. It depends, to a large degree, on the match between item difficulties and the level of examinee ability. As the mismatch increases, so do distortions of scale because of systematic estimation errors (bias), measurement errors, and unobtainable ability estimates. These problems exist for all observed or estimated scores; however, it is shown in this paper that questions concerning the population distributions of true ability can potentially be addressed with empirical Bayes techniques.