Parallel Test Forms Research Articles

Test Equating Using Linear Equating Method

The study aimed to equate two test forms using the linear equating method and estimate the amount of equating errors associated with the method. The linear equating method was used to adjust for the difficulties of the test Form A and Form B on students’ Mathematics test scores. A 20-item mathematics instrument was used for data collection. The bootstrap method was used in the determination of the amount of equating errors that come with this equating method. Data on participants’ test scores on these two congeneric parallel test forms were used for this study. All the requirements for equating the two test forms were met with a reliability coefficient of .71. The scores of a new Form B of the test were then equated to its referenced Form A using a linear equating method. Findings from the study revealed that the linear equating method produced scores that were beyond the required possible observed scores, especially at the lower end of the scores. Again, the amount of error associated with the linear equating method seems to decrease consistently from the low ability scores to the high ability scores.

Validating Parallel-Forms Tests for Assessing Anesthesia Resident Knowledge.

We created a serious game to teach first year anesthesiology (CA-1) residents to perform general anesthesia for cesarean delivery. We aimed to investigate resident knowledge gains after playing the game and having received one of 2 modalities of debriefing. We report on the development and validation of scores from parallel test forms for criterion-referenced interpretations of resident knowledge. The test forms were intended for use as pre- and posttests for the experiment. Validation of instruments measuring the study's primary outcome was considered essential for adding rigor to the planned experiment, to be able to trust the study's results. Parallel, multiple-choice test forms development steps included: (1) assessment purpose and population specification; (2) content domain specification and writing/selection of items; (3) content validation by experts of paired items by topic and cognitive level; and (4) empirical validation of scores from the parallel test forms using Classical Test Theory (CTT) techniques. Field testing involved online administration of 52 shuffled items from both test forms to 24 CA-1's, 21 second-year anesthesiology (CA-2) residents, 2 fellows, 1 attending anesthesiologist, and 1 of unknown rank at 3 US institutions. Items from each form yielded near-normal score distributions, with similar medians, ranges, and standard deviations. Evaluations of CTT item difficulty (item p values) and discrimination (D) indices indicated that most items met assumptions of criterion-referenced test design, separating experienced from novice residents. Experienced residents performed better on overall domain scores than novices (P < .05). Kuder-Richardson Formula 20 (KR-20) reliability estimates of both test forms were above the acceptability cut of .70, and parallel forms reliability estimate was high at .86, indicating results were consistent with theoretical expectations. Total scores of parallel test forms demonstrated item-level validity, strong internal consistency and parallel forms reliability, suggesting sufficient robustness for knowledge outcomes assessments of CA-1 residents.

Reconceptualization of Coefficient Alpha Reliability for Test Summed and Scaled Scores

AbstractCoefficient alpha reliability persists as the most common reliability coefficient reported in research. The assumptions for its use are, however, not well‐understood. The current paper challenges the commonly used expressions of coefficient alpha and argues that while these expressions are correct when estimating reliability for summed scores, they are not appropriate to extend coefficient alpha to correctly estimate the reliability for nonlinearly transformed scaled scores such as percentile ranks and stanines. The current paper reconceptualizes coefficient alpha as a complement of the ratio of two unbiased estimates of the summed score variance. These include conditional summed score variance assuming uncorrelated item scores (gives the error score variance) and unconditional summed score variance incorporating intercorrelated item scores (gives the observed score variance). Using this reconceptualization, a new equation of coefficient generalized alpha is introduced for scaled scores. Coefficient alpha is a special case of this new equation since the latter reduces to coefficinet alpha if the scaled scores are the summed scores themselves. Two applications (cognitive and psychological assessments) are used to compare the performance (estimation and bootstrap confidence interval) of the reliability coefficients for different scaled scores. Results support the new equation of coefficient generalized alpha and compare it to coefficient generalized beta for parallel test forms. Coefficient generalized alpha produced different reliability values, which were larger than coefficient generalized beta for different scaled scores.

Comparability of difficulty levels of translation tasks in CET-6 parallel test forms: evidence from product and process-based data

ABSTRACT This study investigates the comparability of three parallel translation tasks selected from a College English Test Band-6 (CET-6) and explores the major linguistic features contributing to translation difficulty. Data obtained from the participants’ subjective rating, eye-tracking, and performance evaluation were triangulated to measure the comparability of difficulty levels of parallel translation tasks. Data of word translation entropy, translation errors, and participants’ retrospective reports were correlated to examine the difficulty triggers. The results show that: (i) the text comparability was evidenced by eye-tracking indicators and performance measurements, but not supported by subjective ratings; (ii) the domain content words (DCWs) were reported by the participants as the major cause of translation difficulties and the unequal number of DCWs among the three tasks led to inconsistent ratings for the task difficulty. Our findings suggest that test-takers’ subjective perception and their cognitive skills deserve serious consideration by test designers, as these two factors can better demonstrate difficulty levels among parallel tasks. Our study postulates a new direction to establish a relationship between task characteristics and test validity, and provides suggestions for the CET-6 committee and other examination boards with practical methods to be able to compare the difficulty levels of parallel translation tasks.

A Survey on Problem Formulations and (Meta)Heuristic-Based Solutions in Automated Assembly of Parallel Test Forms

Parallel test forms are ubiquitous in the educational, aptitude, achievement, and licensure testing. The problem of automated assembling of parallel test forms has been extensively explored for almost 40 years. Many different mathematical models of the problem formulations and a plenty of different solutions have emerged over the last two decades, indicating that the problem has matured. However, its investigation is still challenging, especially today, when the importance of distance learning and remote knowledge testing is rapidly growing. The diversity of proposed approaches originated notably from the variety of scientific fields involved such as psychometrics, applied mathematics, operations research, and artificial intelligence. Majority of solutions of the problem are (meta)heuristics-based, since they consider the problem as a combinatorial optimization problem which is NP-hard. In this paper, a comprehensive review of this research field, referring to related works since 1985, is conducted. Problem formulations and solutions of the problem are separately classified. Possible avenues of future research are pointed out.

Equivalence of Parallel Tests in a Basic Statistics Course in Higher Education Using Classical Measurement Theory

Developing and administering parallel test forms to students in higher education offsets the cost of having assessment scores that have low validity. This research demonstrated the validity and equivalence of parallel tests in a Basic Statistics course. Among other things, the study: (1) established and compared the item specifications of the items on the different test forms developed, and (2) determined the extent of parallelism of the alternate test forms. Three carefully designed alternate forms of achievement tests (using item specification and test specification table) were administered to 504 second-year students. In addition, academic resilience scale was administered to the same students to help ascertain the criterion validity of the alternate forms. The study revealed some level of similarities in the statistical specifications of the alternate test forms. Further analysis showed that the three alternate test forms developed were congeneric forms of parallelism. The authors concluded that developing classical parallel forms of the test is not feasible, but having congeneric parallel test forms offset the cost of having less valid scores which do not represent students’ attainment levels. Faculty members are encouraged to make use of parallel test forms in assessing students in higher education.

Introducing a Computerized Figural Memory Test Based on Automatic Item Generation: An Analysis With the Rasch Poisson Counts Model.

An automatic item generator for figural memory test items called figumem was developed. It is available in R. A cognitive model allowed the generation of hypothetically parallel items within three difficulty levels determined by visual information load. In a pilot study, participants solved three items for each level of visual load. Within an item response theory approach, the Rasch Poisson counts model and modifications of it were fitted to the data. Results showed overall satisfying fit. Visual information load explained most of the variance in item difficulty. Differences in difficulty between items of the same family were comparatively low, displaying the utility of the item generator for the creation of parallel test forms. Implications, limitations, and suggestions for the use and extensions of figumem are discussed.

Automated Test Assembly Using SAS Operations Research Software in a Medical Licensing Examination

Mathematical programming has been widely used by professionals in testing agencies as a tool to automatically construct equivalent test forms. This study introduces the linear programming capabilities (modeling language plus solvers) of SAS Operations Research as a platform to rigorously engineer tests on specifications in an automated manner. To that end, real items from a medical licensing test are used to demonstrate the simultaneous assembly of multiple parallel test forms under two separate linear programming scenarios: (a) constraint satisfaction (one problem) and (b) combinatorial optimization (three problems). In the four problems from the two scenarios, the forms are assembled subjected to various content and psychometric constraints. Assembled forms are next assessed using psychometric methods to ensure equivalence about all test specifications. Results from this study support SAS as a reliable and easy-to-implement platform for form assembly. Annotated codes are provided to promote further research and operational work in this area.

The American College of Surgeons Entering Resident Readiness Assessment Program: Development and National Pilot Testing Results.

To assess the readiness of entering residents for clinical responsibilities, the American College of Surgeons (ACS) Division of Education developed the "Entering Resident Readiness Assessment" (ACS-ERRA) Program. ACS-ERRA is an online formative assessment that uses a key features approach to measure clinical decision-making skills and focuses on cases encountered at the beginning of residency. Results can be used to develop learning plans to address areas that may need reinforcement. A national panel of 16 content experts, 3 medical educators, and a psychometrician developed 98 short, key features cases. Each case required medical knowledge to be applied appropriately at challenging decision points during case management. Four pilot testing studies were conducted sequentially to gather validity evidence. Residents from programs across the United States participated in the studies (n = 58, 20, 87, 154, respectively). Results from the pilot studies enabled improvements after each pilot test. For the psychometric pilot (final pilot test), 2 parallel test forms of the ACS-ERRA were administered, each containing 40 cases, resulting in overall mean testing time of 2 hours 2 minutes (SD = 43 min). The mean test score was 61% (SD = 9%) and the G-coefficient reliability was 0.90. Results can be used to identify strengths and weaknesses in residents' decision-making skills and yield valuable information to create individualized learning plans. The data can also support efforts directed at the transition into residency training and inform discussions about levels of supervision. In addition, surgery program directors can use the aggregate test results to make curricular changes.

Multidimensional Test Assembly Using Mixed-Integer Linear Programming: An Application of Kullback-Leibler Information.

Many educational testing programs require different test forms with minimal or no item overlap. At the same time, the test forms should be parallel in terms of their statistical and content-related properties. A well-established method to assemble parallel test forms is to apply combinatorial optimization using mixed-integer linear programming (MILP). Using this approach, in the unidimensional case, Fisher information (FI) is commonly used as the statistical target to obtain parallelism. In the multidimensional case, however, FI is a multidimensional matrix, which complicates its use as a statistical target. Previous research addressing this problem focused on item selection criteria for multidimensional computerized adaptive testing (MCAT). Yet these selection criteria are not directly transferable to the assembly of linear parallel test forms. To bridge this gap the authors derive different statistical targets, based on either FI or the Kullback-Leibler (KL) divergence, that can be applied in MILP models to assemble multidimensional parallel test forms. Using simulated item pools and an item pool based on empirical items, the proposed statistical targets are compared and evaluated. Promising results with respect to the KL-based statistical targets are presented and discussed.

Differential long-term medication impact on executive function and delay aversion in ADHD

ABSTRACTThe objective was to compare long-term effects of methylphenidate (MPH) and atomoxetine (ATX) on executive functions (EF) and delay aversion (DAv) in ADHD. A randomized controlled trial was conducted. A comprehensive neuropsychological battery was administered at three moments (Naive, Post-1, Post-2). ADHD participants who showed deficits in the naive evaluation (n = 26) were randomized to receive either MPH (n = 13; Mage = 11 ± 1, MIQ 96 ± 8) or ATX (n = 13; Mage = 10 ± 1, MIQ 106 ± 16.5) optimal dosages. Parallel test forms were administered after three (Post-1) and six months (Post-2) of treatment. A control group (n = 19; Mage = 11 ± 1; MIQ = 106 ± 16.5) performed the neuropsychological battery similarly. Both MPH and ATX significantly improved scores in verbal working memory (vWM) (Naive: p < 0.0001, d = 0.75; Post-1: p = 0.71, d = 0.12), spatial working memory (sWM) (Naive p < 0.0001, d = 0.63; Post-2 p = 0.44; d = 0.03), planning (Naive p < 0.0001, d = 0.54; Post-2 p = 0.6, d = 0.18), decision making (Naive p < 0.001 d = 0.28; Post-1 = 0.06 d = 0.12) and inhibition (Naive <0.0001, d = 0.66; Post-2 p = 0.08, d = 0.00), reaching an improved treatment response after three months of treatment in vWM and after six months in sWM, planning, and inhibition. No beneficial effect on DAv and risk taking was found with MPH and neither with ATX. Long-term treatment in range of optimal clinical dosages with either MPH or ATX improves EF, but not DAv in children with ADHD.

Parallel short forms for the assessment of activities of daily living in cardiovascular rehabilitation patients (PADL-cardio): development and validation

Purpose: To develop and validate parallel short forms for the assessment of activities of daily living in cardiac rehabilitation patients (PADL-cardio I & II).Method: PADL-cardio I & II were developed based on a sample of 106 patients [mean age = 57.6; standard deviation (SD) = 11.1; 72.6% males] using Rasch analysis and validated with a sample of 81 patients (mean age = 59.1; SD = 11.1; 88.9% males). All patients answered PADL-cardio and the Short Form 12 Health Survey.Results: Both versions of PADL-cardio are composed of 10 items. The fit to the Rasch model was given documented by a non-significant Item-trait interaction score (PADL-cardio I: χ2 = 31.08, df = 30, p = 0.41; PADL-cardio II: χ2 = 45.6, df = 40, p = 0.25). The two versions were free of differential item functioning. Person-separation reliability was 0.72/0.78 and unidimensionality was given. The two versions correlated with r = 0.98 and the correlation between PADL-cardio and the underlying item bank was 0.99 for both versions. Concurrent validity is indicated through correlations with the Short Form 12 Health Survey (r = −0.37 to −0.40).Conclusion: PADL-cardio provides a short and psychometrically sound option for the assessment of activities of daily living in cardiovascular rehabilitation patients. The two versions of PADL-cardio are equivalent. Hence, they can be used to reduce practice and retest effects in repeated measurement, facilitating the longitudinal assessment of activities of daily living.Implications for RehabilitationNew parallel test forms for the assessment of activities of daily living in cardiac rehabilitation (PADL-cardio I & PADL-cardio II) are available.PADL-cardio I & II consist of 10 items and are therefore especially timesaving. Concurrent validity is given through correlations with the Short Form Health Survey 12.Therapeutic success could be determined more precisely by the parallel forms reducing practice and retest effects.

Detecting differential item functioning in behavioral indicators across parallel forms.

Despite the crucial importance of the notion of parallel forms within Classical Test Theory, the degree of parallelism between two forms of a test cannot be directly verified due to the unobservable nature of true scores. We intend to overcome some of the limitations of traditional approaches to analyzing parallelism by using the Differential Item Functioning framework. We change the focus on comparison from total test scores to each of the items developed during test construction. We analyze the performance of a single group of individuals on parallel items designed to measure the same behavioral criterion by several DIF techniques. The proposed approach is illustrated with a dataset of 527 participants that responded to the two parallel forms of the Attention Deficit-Hyperactivity Disorder Scale (Caterino, G&oacute;mez-Benito, Balluerka, Amador-Campos, &amp; Stock, 2009). 12 of the 18 items (66.6%) show probability values associated with the Mantel &#967; 2 statistic of less than .01. The standardization procedure shows that half of DIF items favoured Form A and the other half Form B. The &ldquo;differential functioning of behavioral indicators&rdquo; (DFBI) can provide unique information on parallelism between pairs of items to complement traditional analysis of equivalence between parallel test forms based on total scores.

Investigating Test Equating Methods in Small Samples Through Various Factors

(ProQuest: ... denotes formulae omitted)In education and psychology, tests and scales are widely used for monitoring the learning levels of examinees, placing them within upper-instructional levels, selecting staff members, conducting guidance, and performing clinical services. Important decisions are made about examinees by taking into account their test or scale scores. However, the scores obtained from tests and scales have to be accurate in order to make a fair decision about examinees.Sometimes tests and scales are administrated at different times; due to security reasons, different parallel test forms can be used at the same time to overcome this problem. However, this situation causes some problems. Even if test developers construct test forms whose content and statistical characteristics are the same, the forms can have different difficulty levels. While some test forms consist of easy items, others may have difficult items that can cause examinees' scores to differentiate. To overcome this problem, test forms that are constructed at different times should be arranged as different forms of the same test. This case, however, leads to concerns about the simplicity or difficulty of the test forms. The scores of the examinees who are tested at different times cannot be directly compared. When two different forms measuring the same construct are administrated to different groups of examinees, the difficulty of the items on the test forms may not be equal. Test equating is used to overcome these difficulties and can interchangeably interpret the scores obtained from test forms (Kolen & Brennan, 2004; vonDavier, Holland, & Thayer, 2004).Kolen and Brennan (2004) defined test equating as the statistical process used for adjusting scores obtained from test forms so that these scores can be used interchangeably. Crocker and Algina (1986) have defined test equating as a process that establishes equivalent scores from two different measurement instruments; they pointed out that when the percentiles corresponding with the X and Y scores obtained from different tests that have equal reliability and measure the same construct are equal, the tests that these X and Y scores were obtained from are equal. Angoff (1984) defined test equating as the process of converting the system of units from one test form to the system of units of another test form, pointing out that scores obtained from different forms are equated after the scores are transformed. Consequently, test equating emerged due to the fact that two or more tests forms which measure the same content and construct can produce different scores for the same examinees.Certain requirements must be satisfied to equate two test forms. There are many different views related to these requirements in the literature. Hambleton and Swaminathan (1985) listed these requirements as the properties of symmetry, same specifications, equity, and group. With symmetry, when transforming the scores from Form X to Form Y, the inverse of this transformation process should also be valid (Kolen & Brennan, 2004). According to the property of same specifications, the test forms to be equated are required to have the same content and statistical properties. The scores obtained from an equation which ignores these statistical properties cannot be used interchangeably (Kolen & Brennan, 2004). In the property of equity of equating, as proposed by Lord (1980), indifference towards whether Form X or Form Y is administered to the examinees must be claimed. However, this property holds for when test forms are identical. When identical forms are constructed, it is not necessary to equate forms (Crocker & Algina, 1986; Kolen & Brennan, 2004). When claiming the property of group invariance, equating test forms will be independent of examinee group; it does not matter which group is chosen for calculating the equating function between the scores from Form X and Form Y (Kolen & Brennan, 2004; Ozturk & Anil, 2012). …

Students’ Energy Concepts at the Transition Between Primary and Secondary School

Energy is considered both a core idea and a crosscutting concept in science education. A thorough understanding of the energy concept is thought to help students learn about other (related) concepts within and across science subjects, thereby fostering scientific literacy. This study investigates students’ progression in understanding the energy concept in biological contexts at the transition from primary to lower secondary school by employing a quantitative, cross-sectional study in grades 3–6 (N = 540) using complex multiple-choice items. Based on a model developed in a previous study, energy concepts were assessed along four aspects of energy: (1) forms and sources of energy, (2) transfer and transformation, (3) degradation and dissipation, and (4) energy conservation. Two parallel test forms (A and B) indicated energy concept scores to increase significantly by a factor of 2.3 (A)/1.7 (B) from grade 3 to grade 6. Students were observed to progress in their understanding of all four aspects of the concept and scored highest on items for energy forms. The lowest scores and the smallest gain across grades were found for energy conservation. Based on our results, we argue that despite numerous learning opportunities, students lack a more integrated understanding of energy at this stage, underlining the requirement of a more explicit approach to teaching energy to young learners. Likewise, more interdisciplinary links for energy learning between relevant contexts in each science discipline may enable older students to more efficiently use energy as a tool and crosscutting concept with which to analyze complex content.

Item Selection for the Development of Parallel Forms From an IRT-Based Seed Test Using a Sampling and Classification Approach

Two sampling-and-classification–based procedures were developed for automated test assembly: the Cell Only and the Cell and Cube methods. A simulation study based on a 540-item bank was conducted to compare the performance of the procedures with the performance of a mixed-integer programming (MIP) method for assembling multiple parallel test forms. The study investigated the statistical equivalence of the forms generated by the three test assembly methods (Cell Only, Cell and Cube, and MIP) in terms of test information functions, test characteristic curves, mean square deviations, and practical constraints, such as content balancing and nonoverlap among forms. The results indicated that the 13-point MIP method outperformed the other two methods in terms of the “closeness” test information functions between the reference form and the assembled parallel tests. Regarding test characteristic curves, the Cell Only and Cell and Cube methods yielded more similar test characteristic curves than the MIP method. Constraining test information functions apparently does not guarantee that the assembled forms will yield similar test characteristic curves. Overall, the Cell Only and Cell and Cube methods have the potential to provide results similar to the optimization approach.

Age-dependent effects of carotid endarterectomy or stenting on cognitive performance

Although evidence is accumulating that age modifies the risk of carotid angioplasty and stenting (CAS) versus endarterectomy (CEA) for patients with significant carotid stenosis, the impact of age on cognition after either CEA or CAS remains unclear. In this study, we analyzed the effects of age on cognitive performance after either CEA or CAS using a comprehensive neuropsychological test battery with parallel test forms and a control group to exclude a learning effect. The neuropsychological outcomes after revascularization were determined in 19 CAS and 27 CEA patients with severe carotid stenosis. The patients were subdivided according to their median age (<68 years and ≥68 years); 27 healthy subjects served as a control group. In all patients clinical examinations, MRI scans and a neuropsychological test battery that assessed four major cognitive domains were performed immediately before, within 72 h, and 3 months after CEA or CAS. While patients <68 years of age showed no significant cognitive alteration after either CEA or CAS, a significant cognitive decline was observed in patients ≥68 years in both treatment groups (p = 0.001). Notably, this cognitive deterioration persisted in patients after CEA, whereas it was only transient in patients treated with CAS. These results demonstrate an age-dependent effect of CEA and CAS on cognitive functions. In contrast to the recently observed increased clinical complication rates in older subjects after CAS compared with CEA, CEA appears to be associated with a greater, persistent decline in cognitive performance than CAS in this subgroup of patients.

The validation of parallel test forms: ‘Mountain’ and ‘beach’ picture series for assessment of language skills

Pictures are widely used to elicit expressive language skills, and pictures must be established as parallel before changes in ability can be demonstrated by assessment using pictures prompts. Why parallel prompts are required and what it is necessary to do to ensure that prompts are in fact parallel is not widely known. To date, evidence of equivalence has been documented for only a handful of picture instruments. The present study aims to demonstrate that two picture series, one describing hiking on a mountain, the other a picnic at a beach, designed to elicit narrative writing from children, are statistically parallel. To verify their being parallel, a random group design and a covariate were used. MANCOVA and confirmatory factor analysis confirmed that the two alternate picture forms were parallel in terms of means, variances, and factor structures with regard to multiple writing components. This study makes a unique contribution to picture-based assessment, first by publishing this parallel picture set and making it available as a tool for assessing children’s productive skills using longitudinal designs, and second by demonstrating a process by which any other test forms, pictorial or otherwise, can be validated as parallel.

Standard Error Estimation of 3PL IRT True Score Equating With an MCMC Method

A Markov chain Monte Carlo (MCMC) method and a bootstrap method were compared in the estimation of standard errors of item response theory (IRT) true score equating. Three test form relationships were examined: parallel, tau-equivalent, and congeneric. Data were simulated based on Reading Comprehension and Vocabulary tests of the Iowa Tests of Basic Skills®. For parallel and congeneric test forms within valid IRT true score ranges, the pattern and magnitude of standard errors of IRT true score equating estimated by the MCMC method were very close to those estimated by the bootstrap method. For tau-equivalent test forms, the pattern of standard errors estimated by the two methods was also similar. Bias and mean square errors of equating produced by the MCMC method were smaller than those produced by the bootstrap method; however, standard errors were larger. In educational testing, the MCMC method may be used as an additional or alternative procedure to the bootstrap method when evaluating the precision of equating results.

Creating IRT-Based Parallel Test Forms Using the Genetic Algorithm Method

In educational measurement, the construction of parallel test forms is often a combinatorial optimization problem that involves the time-consuming selection of items to construct tests having approximately the same test information functions (TIFs) and constraints. This article proposes a novel method, genetic algorithm (GA), to construct parallel test forms effectively. The sum of squared errors of the generated TIFs produced by GA were compared with those of the Swanson and Stocking method, and the Wang and Ackerman method. Experimental results show that tests constructed using GA yielded lower error, and an average improvement ratio above 90%.