Proficiency Estimates Research Articles

Integrating Cognitive Views Into Psychometric Models for Reading Comprehension Assessment

To demonstrate how methodologies for assessing reading comprehension can grow out of views of the construct suggested in the reading research literature, we constructed tasks and carried out psychometric analyses that were framed in accordance with 2 leading reading models. In estimating item difficulty and subsequently, examinee proficiency, an item response theory (IRT) model called the linear logistic test model was extended to incorporate reader as well as task attributes as covariates. A novel aspect of this modeling was reader effects—interest and prior knowledge—specific to text passages that the examinees read in the assessment. In the demonstration, the theory-motivated task and reader attributes were found to be significantly related to item difficulty. In particular, examinees' comprehension proficiency estimates positively affected within-person effects concerning the reader's familiarity and interest in a passage. This study suggests that it is both feasible and informative to incorporate variables for various comprehension components into the psychometric analysis.

Hybrid Measurement Models for Technology-Enhanced Assessments Through mIRT-bayes

Technology-enhanced assessments (TEAs) are rapidly emerging in educational measurement. In contexts such as simulation and gaming, a common challenge is handling complex streams of information, for which new statistical innovations are needed that can provide high quality proficiency estimates for the psychometrics of complex TEAs. Often in educational assessments with formal measurement models, latent variable models such as item response theory (IRT) are used to generate proficiency estimates from evidence elicited. Such robust techniques have become a foundation of educational assessment, when models fit. Another less common approach to compile evidence is through Bayesian networks, which represent a set of random variables and their conditional dependencies via a directed acyclic graph. Network approaches can be much more flexibly designed for complex assessment tasks and are often preferred by task developers, for technology-enhanced settings. However, the Bayesian network-based statistical models often are difficult to validate and to gauge the stability and accuracy, since the models make assumptions regarding conditional dependencies that are difficult to test. Here a new measurement model family, mIRT-bayes, is proposed to gain advantages of both latent variable models and network techniques combined through hybridization. Specifically, the technique described here embeds small Bayesian networks within an overarching multidimensional IRT model (mIRT), preserving the flexibility for task design while retaining the robust statistical properties of latent variable methods. Applied to simulation-based data from Harvard's Virtual Performance Assessments (VPA), the results of the new model show acceptable fit for the overarching mIRT model, along with reduction of the standard error of measurement through the embedded Bayesian networks, compared to use of mIRT alone. Overall for respondents, a finer grain-size of inference is made possible without additiona testing time or scoring resources, showing potentially promise for this family of new hybrid models.

Improved Student Reasoning About Carbon-Transforming Processes Through Inquiry-Based Learning Activities Derived from an Empirically Validated Learning Progression

This paper reports on our use of a fine-grained learning progression to assess secondary students’ reasoning through carbon-transforming processes (photosynthesis, respiration, biosynthesis). Based on previous studies, we developed a learning progression with four progress variables: explaining mass changes, explaining energy transformations, explaining subsystems, and explaining large-scale systems. For this study, we developed a 2-week teaching module integrating these progress variables. Students were assessed before and after instruction, with the learning progression framework driving data analysis. Our work revealed significant overall learning gains for all students, with the mean post-test person proficiency estimates higher by 0.6 logits than the pre-test proficiency estimates. Further, instructional effects were statistically similar across all grades included in the study (7th–12th) with students in the lowest third of initial proficiency evidencing the largest learning gains. Students showed significant gains in explaining the processes of photosynthesis and respiration and in explaining transformations of mass and energy, areas where prior research has shown that student misconceptions are prevalent. Student gains on items about large-scale systems were higher than with other variables (although absolute proficiency was still lower). Gains across each of the biological processes tested were similar, despite the different levels of emphasis each had in the teaching unit. Together, these results indicate that students can benefit from instruction addressing these processes more explicitly. This requires pedagogical design quite different from that usually practiced with students at this level.

A Comparison of IRT Vertical Scaling Methods in Determining the Increase in Science Achievement

This study is based on a vertical scaling implemented with reference to the Item Response Theory, and involves a comparison of vertical scaling results obtained through the application of proficiency estimation methods and calibration methods. The vertical scales thus developed were assessed with reference to the criteria of grade-to-grade growth, grade-to-grade variability, and the separation of grade distributions. The data used in the study pertains to a dataset composed of a total of 1500 students from twelve primary schools in the province of Ankara, characterized by different levels of socio-economic cultural development. The comparison of the findings pertaining to the first and second sub-problems reveals that the mean differences found through separate calibration were lower than those applicable to concurrent calibration, while the standard deviation found in the case of separate calibration were again lower than the values established through concurrent calibration. Furthermore, the scale of impact in the case of separate calibration was again lower than the values applicable to concurrent calibration. The results reached for all three criteria, using the concurrent calibration method were ranked in the order ML < MAP < EAP, with ML leading to the lowest value while EAP producing the highest one. In case of separate calibration, on the other hand, the ranking of results was found to vary with reference to the criteria applied.

On the validity of repeated assessments in the UMAT, a high-stakes admissions test.

The possibility that the validity of assessment is compromised by repeated sittings of highly competitive and high profile selection tests has been documented and is of concern to stake-holders. An illustrative example is the Undergraduate Medicine and Health Sciences Admission Test (UMAT) used by some medical and dental courses in Australia and New Zealand. The proficiencies of all applicants who sat the UMAT from one to four sittings between 2006 and 2012 were estimated on the same metric using the probabilistic Rasch model. A fit index characterising each profile's degree of conformity to the model was also calculated. Confirming expectations, mean proficiencies increased with repeated sittings on all three UMAT scales with the greatest difference (which was nevertheless relatively small) between the first two sittings. The fit index showed that the increases in proficiency estimates arose from additional easier items being answered correctly on repeated sittings rather than additional more difficult ones, suggesting that improvements are not on the substantive construct of the variable of assessment but in skills in answering the questions. Although strategies for dealing with the increase in proficiency estimates on repeated sittings could be canvassed, these results suggest that the validity of results on repeated sittings was not compromised. Accordingly, it might be concluded that although particular individuals might improve substantially between sittings, any validity is not likely to be compromised with the possibility that for some applicants, the second sitting might be the most valid.

P-11 Estimation of pilot proficiency during simulator trainings: An auditory steady-state response study

P-11 Estimation of pilot proficiency during simulator trainings: An auditory steady-state response study

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM–ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom they considered to be typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Out of a total of 3,310 examinees taking different trial versions of the placement test, 470 learner prototypes were identified. For this set of prototypes, Rasch model estimates of EFL proficiency served as input to a series of ROC analyses, one for each pair of adjacent proficiency levels. Cut scores were derived using the Youden index that maximizes the overall rate of correct classification and minimizes the overall rate of misclassification. Findings confirmed that this method allows for the setting of cut scores that show a high level of classification accuracy in terms of the correspondence with expert categorizations of examinee prototypes. In addition, the ROC-based cut scores were associated with higher classification accuracy than cut scores derived from a logistic regression analysis of the same data. Potential further uses and implications of the PGM–ROC approach in the context of language testing and assessment are discussed.

Assessment of Achievement and Growth by Vertical Scaling: Comparison of Vertical Scaling Methods

In this study, item response theory-based vertical scaling was conducted, and the vertical scaling results obtained by using calibration methods and proficiency estimation were compared. The obtained vertical scales were evaluated according to the criteria of grade-tograde growth, grade-to-grade variability, and the separation of grade distributions. For this study, the data was simulated by R program. According to the results, the mean differences in both the methods of concurrent calibration and separate calibration methods increased compared to the levels observed in 8th grade, and that the largest value was obtained through the EAP method. The mean differences obtained through separate calibration were lower than those obtained through concurrent calibration. When standard deviation values were compared, it was observed that the largest values were obtained through EAP in both calibration methods, and that the standard deviation values obtained in both methods were generally close to each other. When effect size values were examined, in both calibration methods, the effect size values increased toward the 8th grade. The effect size values obtained through separate calibrations were lower than those obtained through concurrent calibration. The results generated by all three proficiency estimation methods were similar to

The 15-item version of the Boston Naming Test as an index of English proficiency

Objective: The present study was designed to examine the potential of the Boston Naming Test – Short Form (BNT-15) to provide an objective estimate of English proficiency. A secondary goal was to examine the effect of limited English proficiency (LEP) on neuropsychological test performance.Method: A brief battery of neuropsychological tests was administered to 79 bilingual participants (40.5% male, MAge = 26.9, MEducation = 14.2). The majority (n = 56) were English dominant (EN), and the rest were Arabic dominant (AR). The BNT-15 was further reduced to 10 items that best discriminated between EN and AR (BNT-10). Participants were divided into low, intermediate, and high English proficiency subsamples based on BNT-10 scores (≤6, 7–8, and ≥9). Performance across groups was compared on neuropsychological tests with high and low verbal mediation.Results: The BNT-15 and BNT-10 respectively correctly identified 89 and 90% of EN and AR participants. Level of English proficiency had a large effect (partial η2 = .12–.34; Cohen’s d = .67–1.59) on tests with high verbal mediation (animal fluency, sentence comprehension, word reading), but no effect on tests with low verbal mediation (auditory consonant trigrams, clock drawing, digit-symbol substitution).Conclusions: The BNT-15 and BNT-10 can function as indices of English proficiency and predict the deleterious effect of LEP on neuropsychological tests with high verbal mediation. Interpreting low scores on such measures as evidence of impairment in examinees with LEP would likely overestimate deficits.

Investigating Robustness of Item Response Theory Proficiency Estimators to Atypical Response Behaviors Under Two‐Stage Multistage Testing

AbstractThe purpose of this study is to evaluate the extent to which item response theory (IRT) proficiency estimation methods are robust to the presence of aberrant responses under the GRE® General Test multistage adaptive testing (MST) design. To that end, a wide range of atypical response behaviors affecting as much as 10% of the test items were simulated using a generic GRE 2‐stage MST and the 2‐parameter logistic (2PL) IRT model. As expected, some differences were found among the 5 estimators in terms of the recovery of the true theta ability; for example, Bayesian estimators had lower error variance and their estimates were regressed to the mean. Once the IRT theta estimates were scaled onto a comparable reporting score scale, however, it was found that all the estimation methods investigated, including the one currently used to score GRE MSTs, were equally robust under the simulated conditions.

Logical conditions analysis method for diagnostic test results decoding applied to competence elements proficiency

Subject of Research.Representation features of education results for competence-based educational programs are analyzed. Solution importance of decoding and proficiency estimation for elements and components of discipline parts of competences is shown. The purpose and objectives of research are formulated. Methods. The paper deals with methods of mathematical logic, Boolean algebra, and parametrical analysis of complex diagnostic test results, that controls proficiency of some discipline competence elements. Results. The method of logical conditions analysis is created. It will give the possibility to formulate logical conditions for proficiency determination of each discipline competence element, controlled by complex diagnostic test. Normalized test result is divided into noncrossing zones; a logical condition about controlled elements proficiency is formulated for each of them. Summarized characteristics for test result zones are imposed. An example of logical conditions forming for diagnostic test with preset features is provided. Practical Relevance. The proposed method of logical conditions analysis is applied in the decoding algorithm of proficiency test diagnosis for discipline competence elements. It will give the possibility to automate the search procedure for elements with insufficient proficiency, and is also usable for estimation of education results of a discipline or a component of competence-based educational program.

The Rating System of the Rural School Pupils' Assessment of the Republic of Kazakhstan

Currently, comprehensive school teachers of Republic of Kazakhstan pay special attention to of pupils' knowledge based on personally oriented approach. In work A black box: what there inside? An of knowledge of pupils as a way of increase of efficiency of teaching and educational P. Blek and D. William note that Improvement of quality of at schools is one of important state priorities... initiatives for purpose of improvement of planning and management, statement are more whole, more careful and frequent inspections - all this... doesn't give desirable feedback in improvement of teaching and educational process at school conclusion arises: something in this scheme isn't enough... In our work we try to glance in a black box. We in detail consider such aspect of as the forming assessment which is a basis of effective training (1). Today became quite obvious that in modern Kazakhstan education traditional of estimation of proficiency of pupils was insolvent. The cult instead of a cult of knowledge dominates. Similar situation - result of Former Soviet approach to of estimation of pupils which assigned a duty to have an appreciation on all subjects to each pupil, and only three points from five are carried to that. In a 5-mark scale of estimation applied in Kazakhstan such points, as 2 and 1 - negative. The similar perception formally narrowed a 5-mark scale to 3-mark. As a result teachers of schools are compelled by points 3, 4, 5 and estimated judgments corresponding to them to estimate pupils of different level and different degree of readiness. This of estimation operates on pupils of both comprehensive schools, and gymnasiums, lyceums, establishments of correctional developing type. To distinguish these marks which are put down in documents on education as practice shows, it is simply impossible that is a serious contradiction. As a result we receive unauthenticated of an of quality of in general. Kazakhstan signed in 1999 Bologna convention, is on threshold of new educational reform - transition to 12 years' education. For this period different systems of estimation were considered: 10-mark (2), 12-mark (3), 100-mark (4), test and modular (5) and others. In this article we present results of pilot study on of rural schools in Republic of Kazakhstan, Aktobe region Kenkiyak village. Having chosen theme of experiment: Control and evaluation activities in conditions of transition to 12-year education system a working group of teachers-experimenters choses one of its directions for detailed study - rating evaluation system. At present in all territories of Kazakhstan a five-point evaluation has been introduced, which may not always estimate each child's individual development trajectory. The new evaluation is introduced in order to create optimal conditions for efficient tracking of individual pupils' achievements using a rating evaluation system. The object of research throughout work is learning process of Mathematics, Cognition of world, Chemistry, Russian and Kazakh languages in forms 3-8.Total in experiment 10 forms were involved. The experiment took place during three years from2009 to 2012.Results of conducted research were those changes in teaching process, which have caused qualitative characteristics such as increased digestible material and depth of its understanding, activity of pupils at lesson, their creative independence, change of self-criticism and self-evaluation, etc. Rating allows to create maximal comfort environment of learning and upbringing, allows to transfer learning activities of pupils in internal need. The particular importance is educational aspect of new evaluation for pupil - from self-discipline and to focus on his own interests and needs.

The Influence of Aerobic Fitness on the Relationship between Academic Performance and Motor Proficiency

Purpose: To determine whether physical fitness mediates the relationship between academic performance and motor proficiency in children. Methods: The academic achievement of 1864 students (F:926, M:938, age 11.91 (SD:0.34) from the Niagara Region of Ontario was derived from the average of provincial standardized tests for reading, writing, and mathematics. The Bruininks-Oseretsky Test of Motor Performance (short-form) determined motor proficiency. Fitness (peak oxygen uptake) was established with the 20-metre shuttle run multistage test. Results: Hierarchical linear modeling identified several significant predictors of academic performance. After controlling for age (p=0.7002), gender (p<0.0001), and parental education (p<0.0001), both motor proficiency (p<0.0001) and physical fitness (p=0.0004) remained significant. The addition of physical fitness to the model reduced the point estimate of motor proficiency from 0.004 to 0.003, a reduction of 25%. However, further examination of the 95% confidence intervals suggests that this reduction is not significant. Conclusions: These results suggest that aerobic fitness does not mediate the relationship between motor proficiency and academic performance. Both physical fitness and motor proficiency have independent roles in predicting academic performance.

Perceptual Estimates of Motor Skill Proficiency Are Constrained by the Stability of Coordination Patterns

ABSTRACTThis study demonstrated that motor skill proficiency ratings are constrained by the same order parameter dynamics that constrain action production and action perception processes. Participants produced rhythmic actions simulated by an animated stick figure of the human arm. The primary finding was that participants’ proficiency ratings covaried most with relative phase (φ) variability compared to mean relative phase. In-phase (φ = 0°) was produced with the least variability and received the highest proficiency rating, whereas the patterns φ = ±150° were attempted with the most variability and received the lowest proficiency ratings. A temporal delay in attempting to produce the animated pattern had a large impact on produced relative phase, yet had little impact on the proficiency ratings. Proprioceptive processes provide individuals information on motor skill proficiency. The lead or lag motion of the hand to forearm segment of the animated arm was identified consistently through visual processes and revealed asymmetries in the mapping of visual input to motor output. The results are consistent with concepts from the dynamic pattern theory of coordination and are discussed with regard to relative phase as an informational variable that constraints the perception-action system across many levels.

Barthel Index of activities of daily living: item response theory analysis of ratings for long-term care residents.

The Barthel Index (BI) is a frequently used measure of independence in the activities of daily living (ADLs). Item functioning of various versions of the BI have been examined using Rasch analysis. Item response theory (IRT) models for ordered polytomous responses may provide more insight into item functioning across levels of independence in ADLs. To compare the fit and appropriateness of the one-parameter logistic model (1PL), the partial credit model (PCM), and the extension of the generalized partial credit model implemented in ConQuest (GPCM-CQ) for the 15-item BI. This article is a secondary analysis of baseline BI data obtained from four randomized controlled trials for 788 residents from multiple long-term care facilities. Parameters of three different IRT-based models (1PL Rasch model, PCM, and GPCM-CQ) were estimated. Fit of items and response vectors was assessed. Overall fit was compared across the three models. Item difficulties were similar for all three models. Most of the 15 items were located at a moderate level of functional independence. In all three models, "don brace" was the easiest ADL but had poor discrimination; "climbing stairs" was the most difficult ADL. Multiple items showed misfit in both 1PL and PCM. Item parameters and person proficiency estimates were highly correlated for the PCM and GPCM-CQ models. The difference in deviance between the PCM and GPCM-CQ was significant. In the GPCM-CQ, most items showed good discrimination, but several had negative or very low discrimination parameters. GPCM-CQ results suggested that further revision of the BI may be warranted. Because some items showed poor discrimination, caution should be used when measuring ADL independence with the BI item set.

Nested multiple imputation in large-scale assessments

In order to measure the proficiency of person populations in various domains, large-scale assessments often use marginal maximum likelihood IRT models where person proficiency is modelled as a random variable. Thus, the model does not provide proficiency estimates for any single person. A popular approach to derive these proficiency estimates is the multiple imputation of plausible values (PV) to enable subsequent analyses on complete data sets. The main drawback is that all variables that are to be analyzed later have to be included in the imputation model to allow the distribution of plausible values to be conditional on these variables. These background variables (e.g., sex, age) have to be fully observed which is highly unlikely in practice. In several current large-scale assessment programs missing observations on background variables are dummy coded, and subsequently, dummy codes are used additionally in the PV imputation model. However, this approach is only appropriate for small proportions of missing data. Otherwise the resulting population scores may be biased. Alternatively, single imputation or multiple imputation methods can be used to account for missing values on background variables. With both imputation methods, the result is a two-step procedure in which the PV imputation is nested within the background variable imputation. In the single+multiple-imputation (SMI), each missing value on background variables is replaced by one value. In the multiple+multiple-imputation (MMI), each missing value is replaced by a set of imputed values. MMI is expected to outperform SMI as SMI ignores the uncertainty due to missing values in the background data. In a simulation study, both methods yielded unbiased population estimates under most conditions. Still, the recovery proportion was slightly higher for the MMI method. The advantages of the MMI method are apparent for fairly high proportions of missing values in combination with fairly high dependency between the latent trait and the probability of missing data on background variables.

The incremental contribution of TOEIC® Listening, Reading, Speaking, and Writing tests to predicting performance on real-life English language tasks

Typically, English language proficiency tests yield multiple scores – usually for each of the four traditional language domains. In order to maximize the usefulness of test scores, they may need to be accompanied by information concerning how they complement one another. Using self-assessments by some 2300 TOEIC test takers, this study aimed to show that language performance in a particular domain (speaking, for instance) can be predicted by using a test that corresponds specifically to that domain (i.e., the TOEIC Speaking test), but also that it can be even better predicted by supplementing domain-related scores with scores (e.g., TOEIC Listening scores) that do not correspond directly to the target domain. The results of a hierarchical linear regression analysis revealed that adding test scores that did not correspond directly to the target performance domain made statistically and practically significant contributions to prediction in the target domain. Thus, more precise estimates of English language proficiency in a specific domain are possible by assessing skills not only in that domain but in other related domains as well.

Nitrogen resorption efficiency in mature trees and seedlings of four tree species co-occurring in a Mediterranean environment

Many studies have revealed marked changes in the patterns of nitrogen use along ontogeny in tree species. However, there are no reports of the ontogenetic changes in resorption efficiency, despite the fact that nitrogen resorption from senescing leaves is one of the most important plant mechanisms to conserve and optimize the use of this nutrient. Our objective is to analyze the differences in nitrogen resorption between seedlings and adults and to elucidate which of the leaf traits changing along ontogeny determine these possible differences. Green and senescent leaves were sampled from five seedlings and mature specimens of four tree species. N resorption efficiencies were calculated from the differences between N contents in green and shed leaves collected from the same specimens. Minimum N concentrations in leaf litter were used as an estimation of resorption proficiency. N concentration in green leaves and N resorption efficiency were higher in adult specimens for all species. However, with the exception of Pinus pinea, there were no significant differences in proficiency. Contrary to our expectations, the species studied seem to have a lower potential for N resorption during the initial stages of their life cycle. The higher N concentrations seen in adult leaves could explain the higher percentage they are able to withdraw during senescence, probably because they have higher amounts of readily translocatable soluble molecules.

Person Proficiency Estimates in the Dichotomous Rasch Model When Random Guessing Is Removed From Difficulty Estimates of Multiple Choice Items

Andrich, Marais, and Humphry showed formally that Waller’s procedure that removes responses to multiple choice (MC) items that are likely to be guessed eliminates the bias in the Rasch model (RM) estimates of difficult items and makes them more difficult. The former did not study any consequences on the person proficiency estimates. This article shows that when the procedure is applied, the more proficient persons who are least likely to guess benefit by a greater amount than the less proficient, who are most likely to guess. This surprising result is explained by appreciating that the more proficient persons answer difficult items correctly at a greater rate than do the less proficient, even when the latter guess some items correctly. As a consequence, increasing the difficulty of the difficult items benefits them more than the less proficient persons. Analyses of a simulated and real example are shown illustratively. To not disadvantage the more proficient persons, it is suggested that Waller’s procedure be used when the RM is used to analyze MC items.

Efficiency of Item Selection Method in Variable-length Computerized Adaptive Testing for the Testlet Response Model: Constraint-weighted A-stratification Method

Most Computerized adaptive tests (CATs) are constructed on the foundation of standard item response theories (IRT). This model is a powerful psychometric paradigm and thus it has been used by several testing programs. However, the model, by its assumption, ignores dependence structure of items even though tests are made up of test lets (sets of items). When traditional IRT models are applied to tests composed of test let, violations of independence assumptions as required by traditional IRT models will result in imprecise ability estimates. Hence, the test let response theory (TRT) model is recommended for estimating examinee's proficiency estimates when tests composed of test let were used. The purpose of this study is to explore the accuracy of the TRT model when it is applied to the variable-length CATS. This article employed a variable-length item selection method adapted from the constraint-weighted a-stratification method to control items exposure and contents balancing in item selection processes. The Monte Carlo method was employed to compare the Pearson product-moment correlation, bias, mean squared error (MSE) of the TRT model in estimating examinees. In this study we compared the results based on three test let selection methods: the maximization information method, the randomization method, the constraint-weighted a-stratification method. The results indicate that the constraint-weighted a-stratification method outperforms the other adapted variable-length CAT based on TRT model.