Alternate Test Forms Research Articles

Equivalence of Parallel Tests in a Basic Statistics Course in Higher Education Using Classical Measurement Theory

Developing and administering parallel test forms to students in higher education offsets the cost of having assessment scores that have low validity. This research demonstrated the validity and equivalence of parallel tests in a Basic Statistics course. Among other things, the study: (1) established and compared the item specifications of the items on the different test forms developed, and (2) determined the extent of parallelism of the alternate test forms. Three carefully designed alternate forms of achievement tests (using item specification and test specification table) were administered to 504 second-year students. In addition, academic resilience scale was administered to the same students to help ascertain the criterion validity of the alternate forms. The study revealed some level of similarities in the statistical specifications of the alternate test forms. Further analysis showed that the three alternate test forms developed were congeneric forms of parallelism. The authors concluded that developing classical parallel forms of the test is not feasible, but having congeneric parallel test forms offset the cost of having less valid scores which do not represent students’ attainment levels. Faculty members are encouraged to make use of parallel test forms in assessing students in higher education.

Validity of Teleneuropsychological Assessment in Older Patients with Cognitive Disorders.

The feasibility and reliability of neuropsychological assessment at a distance have been demonstrated, but the validity of this testing medium has not been adequately demonstrated. The purpose of this study was to determine the ability of video teleconferencing administration of neuropsychological measures (teleneuropsychology) in discriminating cognitively impaired from non-impaired groups of older adults. It was predicted that measures administered via video teleconference would distinguish groups and that the magnitude of differences between impaired and non-impaired groups would be similar to group differences achieved in traditional administration. The sample consisted of 197 older subjects, separated into two groups, with and without cognitive impairment. The cognitive impairment group included 78 individuals with clinical diagnoses of mild cognitive impairment or Alzheimer's disease. All participants completed counterbalanced neuropsychological testing using alternate test forms in both a teleneuropsychology and a traditional face-to-face (FTF) administration condition. Tests were selected based upon their common use in dementia evaluations, brevity, and assessment of multiple cognitive domains. Results from FTF and teleneuropsychology test conditions were compared using individual repeated measures ANCOVA, controlling for age, education, gender, and depression scores. All ANCOVA models revealed significant main effects of group and a non-significant interaction between group and administration condition. All ANCOVA models revealed non-significant main effects for administration condition, except category fluency. Results derived from teleneuropsychologically administered tests can distinguish between cognitively impaired and non-impaired individuals similar to traditional FTF assessment. This adds to the growing teleneuropsychology literature by supporting the validity of remote assessments in aging populations.

Comparison of the Computer and Hand Administered Versions of the Brown Location Test (BLT).

The Brown Location Test (BLT) was developed to remedy some of the problems in existing visual-based memory tests. The hand version has demonstrated good psychometric properties, the ability to provide lateralization information for mesial temporal lobe epilepsy patients, and has normative data. The purpose of this study was to compare the hand administered format to the more recently developed computer administered format. We used Generalizability Theory analyses to assess the degree of variability in scores across the hand and computer versions of the test, and across alternate test forms, A and B. We also compared the means and standard deviations for the different versions and forms using paired t-tests, and Pearson correlation coefficients. There was minimal variability and high levels of score similarity across the various test administration formats and forms. The high degree of comparability between versions allows one to apply the validity findings and normative data collected using the hand administered version to the computer version of the BLT.

Equating in small-scale language testing programs

Language programs need multiple test forms for secure administrations and effective placement decisions, but can they have confidence that scores on alternate test forms have the same meaning? In large-scale testing programs, various equating methods are available to ensure the comparability of forms. The choice of equating method is informed by estimates of quality, namely the method with the least error as defined by random error, systematic error, and total error. This study compared seven different equating methods to no equating – mean, linear Levine, linear Tucker, chained equipercentile, circle-arc, nominal weights mean, and synthetic. A non-equivalent groups anchor test (NEAT) design was used to compare two listening and reading test forms based on small samples (one with 173 test takers the other, 88) at a university’s English for Academic Purposes (EAP) program. The equating methods were evaluated based on the amount of error they introduced and their practical effects on placement decisions. It was found that two types of error (systematic and total) could not be reliably computed owing to the lack of an adequate criterion; consequently, only random error was compared. Among the seven methods, the circle-arc method introduced the least random error as estimated by the standard error of equating (SEE). Classification decisions made using the seven methods differed from no equating; all methods indicated that fewer students were ready for university placement. Although interpretations regarding the best equating method could not be made, circle-arc equating reduced the amount of random error in scores, had reportedly low bias in other studies, accounted for form and person differences, and was relatively easy to compute. It was chosen as the method to pilot in an operational setting.

Practice Effects and Stability of Neuropsychological and UHDRS Tests Over Short Retest Intervals in Huntington Disease.

In Huntington disease (HD), cognitive changes due to disease-progression or treatment are potentially confounded by "practice effects" (PE)--performance improvement from prior exposure to test materials. A practice run-in ("dual baseline") was used in an HD cognitive trial to determine if PE could be minimized and evaluate performance trajectories over multiple visits. Non-depressed adults (N = 36) with mild to moderate HD-related cognitive deficits participated in a clinical trial to examine the efficacy of citalopram to enhance cognition. Cognitive tests were administered at three visits (2 weeks separating each visit), before active treatment randomization. Some tests were also administered at screening. Therefore 3-4 pre-treatment repetitions were available. We examined test improvement using repeated-measures ANOVAs with planned pairwise comparisons. Despite the practice run-in and use of alternate test forms, results indicated ongoing improvements over at least three test sessions on all three UHDRS cognitive tests. Trails A and B showed improvements between the third and fourth session, which suggests that one pre-baseline visit may not be effective in reducing practice on this important and commonly used test. Overall, 7 out of 13 variables showed some degree of short-term PE, even after multiple sessions and alternate forms. Tests assessing processing speed and memory may be particularly confounded by ongoing PE across at least 2-3 sessions. Practice run-in periods and alternate forms may help minimize the impact of such effects in HD clinical trials but awareness of which tests are most susceptible to PE is important in clinical trial design.

Three-Element Item Selection Procedures for Multiple Forms Assembly: An Item Matching Approach.

In educational measurement, building multiple alternate test forms is usually highly desirable in practice. Test assembly problems are classified as NP-hard (Non-deterministic Polynomial-time hard) problems, and their computation time increases exponentially as the problem size grows. The purpose of this study was to construct parallel forms item-by-item based on a seed test, using two proposed item selection heuristic methods (random item sequence assigned to maximum distance form with content balance [R-MD-B] and random item sequence assigned to random form with content balancing [R-R-B]) incorporating a flexible content balancing method. The results were compared with two modified versions of the Armstrong, Jones, and Wu heuristics (random item sequence assigned to maximum distance form with Armstrong's replacement [R-MD-R] and random item sequence assigned to random form with Armstrong's replacement [R-R-R]) and two models incorporating mixed integer programming (MIP) methods. The results showed that the R-MD-B and R-R-B methods yielded results comparable with the two MIP methods with less computation time in terms of the item information functions, test characteristic curves, and content coverage. The results also showed that R-MD-B and R-R-B can be extended to handle test assembly constraints in practical settings.

Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment.

The use of videoconference technology to deliver health care diagnostics and treatment continues to grow at a rapid pace. Telepsychiatry and telepsychology applications are well-accepted by patients and providers, and both diagnostic and treatment outcomes have generally been similar to traditional face-to-face interactions. Preliminary applications of videoconference-based neuropsychological assessment (teleneuropsychology) have yielded promising results in the feasibility and reliability of several standard tests, although large-scale studies are lacking. This investigation was conducted to determine the reliability of video teleconference (VTC) - based neuropsychological assessment using a brief battery of standard neuropsychological tests commonly used in the evaluation of known or suspected dementia. Tests included the Mini-Mental State Examination (MMSE), Hopkins Verbal Learning Test-Revised, Digit Span forward and backward, short form Boston Naming Test, Letter and Category Fluency, and Clock Drawing. Tests were administered via VTC and in-person to subjects, counterbalanced using alternate test forms and standard instructions. Two hundred two adult subjects were tested in both rural and urban settings, including 83 with cognitive impairment and 119 healthy controls. We found highly similar results across VTC and in-person conditions, with significant intraclass correlations (mean=.74; range: 0.55-0.91) between test scores. Findings remained consistent in subjects with or without cognitive impairment and in persons with MMSE scores as low as 15. VTC-based neuropsychological testing is a valid and reliable alternative to traditional face-to-face assessment using selected measures. More VTC-based studies using additional tests in different populations are needed to fully explore the utility of this new testing medium.

Single-word intelligibility testing in aphasia: Alternate forms reliability, phonetic complexity and word frequency

Background: We recently constructed a single-word speech intelligibility test intended to quantify magnitude of speech sound impairment in individuals with aphasia. To minimise listener learning and strategising, the test included a large set of possible (alternate) test forms and was constructed with high and diverse phonologic similarity among candidate words. Although the corpus was limited in phonetic complexity to single syllables, and although criteria for minimal word frequency were applied, it is possible that more fine-grained differences in phonetic complexity and/or word frequency may introduce varying levels of difficulty across alternate test forms and that reliability, therefore, may be compromised.Aims: The dual purposes of this study were to evaluate alternate forms reliability for the new intelligibility test and to determine whether word frequency and/or phonetic complexity affected word identification scores.Methods & Procedures: All words in the 600-word test corpus were coded for overall phonetic complexity and for frequency of occurrence in spoken English. Fifty-one versions of the target 50-word test were generated from this corpus by following designated pseudo-random selection procedures. Speech samples were collected from 13 speakers with aphasia, who each repeated three or four of these 50-word sets. Ten normal-hearing listeners were asked to indicate the words they thought the speakers were trying to say. Per cent accuracy was computed for each speaker, test form and target word.Outcomes & Results: The intra-class correlation within speakers was 0.97, indicating that the scores across alternate forms were highly reproducible. Analyses at the word level showed that both phonetic complexity and word frequency affected identification accuracy. The effects were seen in almost all participants whose intelligibility scores were in the impaired range.Conclusions: High frequency of occurrence and low phonetic complexity increase repetition accuracy for individuals with moderate to severe sound production difficulties and aphasia. However, the pseudo-random word selection in the examined word intelligibility test was sufficient to ensure strong alternate forms reliability. Further constraints on the word selection process for this test are not warranted.

Using a Linear Regression Method to Detect Outliers in IRT Common Item Equating

Common test items play an important role in equating alternate test forms under the common item nonequivalent groups design. When the item response theory (IRT) method is applied in equating, inconsistent item parameter estimates among common items can lead to large bias in equated scores. It is prudent to evaluate inconsistency in parameter estimates of common items before conducting IRT equating. The evaluation of inconsistency in parameter estimates is typically achieved through detecting outliers in the common item set. In this study, a linear regression method is proposed as a detection method. The newly proposed method was compared with a traditional method in various conditions. The results of this study confirmed the necessity of detecting and removing outlying common items. The results also show that the newly proposed method performed better than did the traditional method in most conditions.

Quantitative differences in retest effects across different methods used to construct alternate test forms

Allowing respondents to retake a cognitive ability test has shown to increase their test scores. Several theoretical models have been proposed to explain this effect, which make distinct assumptions regarding the measurement invariance of psychometric tests across test administration sessions with regard to narrower cognitive abilities and general mental ability. We modeled retest effects in four psychometric tests as a function of specific retest form and general mental ability in order to compare the validity of these models and their generalizability across three different kinds of retest forms. To do so automatic item generation was used to construct two kinds of alternate retest: (1) isomorphic retests and (2) psychometrically matched retests. A total of N=358 respondents completed all four measures twice, receiving either identical retest forms, isomorphic retest forms or psychometrically matched retest forms at the second test administration session. Item response theory modeling supported strict measurement invariance across all test forms and time-points of measurement but indicated variation in respondents' retest score gains due to individual differences in general mental ability and the kind of retest form used. In general, retest effects were more pronounced for high-g respondents, identical retests and isomorphic retest forms and for mental rotation and algebra word problems. Latent mean and covariance structure analyses indicated that retesting did not affect the g-factor saturation of the four cognitive ability tests but revealed that retest score gains were hollow with respect to psychometric g.

SOURCES OF SCORE SCALE INCONSISTENCY

ABSTRACTFor testing programs that administer multiple forms within a year and across years, score equating is used to ensure that scores can be used interchangeably. In an ideal world, samples sizes are large and representative of populations that hardly change over time, and very reliable alternate test forms are built with nearly identical psychometric properties. Under these conditions, most equating methods produce score conversions close to the identity function. Unfortunately, equating is sometimes performed on small non‐representative samples with variable distributions of ability, and administered tests are built to vague specifications. Here, different equating methods produce different results because they are based on different assumptions. In the nearly ideal case, there are smaller deviations from the identity function because great effort is taken to control variation. Even when equating is conducted under these desirable conditions, the random variation in form‐to‐form equating, when concatenated over time, can produce substantial shifts in score conversions, that is, scale drift. In this paper, we make distinctions among different sources of variation that may contribute to score‐scale inconsistency, and identify practices that are likely to contribute to it.

The Potential Impact of Not Being Able to Create Parallel Tests on Expected Classification Accuracy

In many practical testing situations, alternate test forms from the same testing program are not strictly parallel to each other and instead the test forms exhibit small psychometric differences. This article investigates the potential practical impact that these small psychometric differences can have on expected classification accuracy. Ten different sets of tests were assembled by minimizing the differences in test information at five θ locations. The impact of the psychometric differences between the assembled test forms was quantified for two different groups of simulated examinees across a range of possible cut scores. Results indicated that using sequential or simultaneous test assembly is preferred to random test assembly. Analyses also implied that the small differences in the psychometric properties between tests produced differences in overall classification accuracy that were less than 1.5%. The biggest differences in classification accuracy were found when the test information functions were not as well matched in regions where there were more examinees. Although these differences were fairly small, they may have the potential to have a practically significant impact on decision making. It is suggested that classification accuracy critically depends on the differences in test information, the location of the cut score, and the groups of examinees considered.

Detecting Answer Copying Using Alternate Test Forms and Seat Locations in Small‐Scale Examinations

Two types of answer‐copying statistics for detecting copiers in small‐scale examinations are proposed. One statistic identifies the “copier‐source” pair, and the other in addition suggests who is copier and who is source. Both types of statistics can be used when the examination has alternate test forms. A simulation study shows that the statistics do not depend on the total‐test score. Another simulation study compares the statistics with two known statistics, and shows that they have substantial power. The new statistics are applied to data from a small‐scale examination (N= 230) with two alternate test forms. Auxiliary information on the seat location of the examinees and the test scores of the examinees was used to determine whether or not examinees could be suspected.

Evaluation of Practice Effects in Language and Spatial Processing Test Performance

Recent research demonstrates that practice effects are attenuated through the administration of alternate-form memory tests. However, little is known about the degree of practice that can be expected when alternate forms of nonmemory tests are administered repeatedly. Two groups of healthy older adults were assigned to either a same- or alternate-forms condition. Participants completing the same forms of a confrontation naming task improved significantly over 4 testing sessions. On verbal fluency, participant performance significantly improved when completing only alternate forms. No significant practice effects were observed on tests of spatial processing. Practice effects caused by item-specific practice may be reduced via alternate test forms. However, similar reductions will be less apparent when practice effects are the result of test-specific practice.

Evaluation of Practice Effects in Language and Spatial Processing Test Performance

Recent research demonstrates that practice effects are attenuated through the administration of alternate-form memory tests. However, little is known about the degree of practice that can be expected when alternate forms of nonmemory tests are administered repeatedly. Two groups of healthy older adults were assigned to either a same- or alternate-forms condition. Participants completing the same forms of a confrontation naming task improved significantly over 4 testing sessions. On verbal fluency, participant performance significantly improved when completing only alternate forms. No significant practice effects were observed on tests of spatial processing. Practice effects caused by item-specific practice may be reduced via alternate test forms. However, similar reductions will be less apparent when practice effects are the result of test-specific practice.

Developing a Computer‐Based TTCT: Promises and Problems*

ABSTRACTThe purposes of the present study were to examine the equivalence of paper‐pencil and computer‐based versions of the Torrance Tests of Creative Thinking (TTCT), Figural Forms A and B, and to investigate the patterns of responses involved in the computer‐based TTCT. The participants were 34 fifth‐ and sixth‐grade students currently enrolled in two public schools in southern Texas. The computer‐based TTCT was developed to present all the features of the paper‐pencil TTCT with the additional benefits offered by computer‐based testing. Means, variances, test‐retest reliability, and attitudinal differences were analyzed to investigate equivalence between the paper‐pencil and computer‐based TTCT. Response patterns involved in the figural tasks were explored based on data collected from the computer‐based test (e.g., latency, tool using patterns, correction activities). The findings of the study indicated that the results obtained from the computer‐based TTCT were not equivalent with the paper‐pencil version. However, additional information regarding response patterns obtained from the computer‐based TTCT showed clear indication of different response tendencies among different groups of students and between the two alternate test forms (A and B).

PARALLEL TEST FORM DEVELOPMENT: A PROCEDURE FOR ALTERNATE PREDICTORS AND AN EXAMPLE

A procedure for developing alternate test forms that are parallel in the sense that scores on the different forms have similar means, standard deviations, and factor structures is described and applied to a bio‐data inventory and a situational judgment test. Careful consideration of item‐by‐item parallelism during development resulted in alternate forms that were parallel at the item level. Further, comparison with a biodata test form comprised of items randomly selected from a pool of biodata items revealed that for the types of measures described here it may be necessary to produce parallel forms of each item to create alternate forms that are parallel in the way in which Cronbach (1947) originally defined parallelism.

Factorially Equivalent Test Batteries

Parallel and alternate test forms have many uses. These tests are designed to be used interchangeably. The first criterion for interchangeable tests is that they measure the same constructs. Multiple forms of a widely used multiple aptitude test battery were investigated using the multiple groups analysis procedures of confirmatory factor analysis. Beginning with the model confirmed by Ree and Carretta (1994), 6 forms of the battery were tested. Total sample size was 15,120. Results show a consistent factor structure across the 6 forms. When compared to the Form 8a prototype, Forms 8b, 9a, and 9b were found to be highly similar, whereas the factor-variable parameter constraints for Forms 10a and 10b showed them to be less similar. Item-ordering effects did not produce inequality in Forms 8 and 9.

MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING

ABSTRACTThe basic equations of item response theory (IRT) provide a foundation for inferring examinees' abilities from responses to different test items. In practice, examinees do not generally provide a response to all items—for reasons that may or may not have been intended by the test administrator, and that may or may not be related to their ability. The mechanisms that produce missingness must be taken into account if correct inferences are to be drawn. Using concepts introduced by Rubin (1976), we discuss the implications for Bayesian and direct likelihood ability parameter estimation that are entailed by alternate test forms, targeted testing, adaptive testing, time limits, omitted responses, and examinee choice of tasks. Attention is focused on whether, in each case, the mechanism for missingness is “ignorable,” and, in those cases in which it is not, how it can be modeled.

The Effect of Altering the Position of Options in a Multiple-Choice Examination

Many testing programs rely on equating procedures to achieve comparability of scores on alternate test forms. One commonly accepted rule for developing equated examinations using the common-items nonequivalent groups (CINEG) design is that items common to the two examinations being equated should be identical. In test construction practice, this rule has been extended to include even the order in which options appear in the two examinations. The present study examined the performance of a common set of items on an examination in which the order of options for one test form was experimentally manipulated. The study sought to determine whether reordering multiple choice item options results in any significant effect on item difficulty. It was found that reordering options can have significant but unpredictable effects on item performance. A linkage is made to previous research and cautions are suggested regarding the effects of reordering options.