Alternate-form Reliability Research Articles

Reliability of the Assisting Hand Assessment in adolescents.

To investigate the interrater and test-retest reliability of the Assisting Hand Assessment in adolescents (Ad-AHA) with cerebral palsy (CP) and to evaluate the alternate-form reliability of different test activities. Participants were 112 adolescents with unilateral CP (60 males, 52 females; mean age 14y 5mo [standard deviation {SD} 2y 8mo], Manual Ability Classification System levels I-III). Reliability was evaluated using intraclass correlation coefficients (ICC), smallest detectable change (SDC), and Bland-Altman plots. ICCs for interrater (n=38) and test-retest reliability (n=31) were excellent: 0.97 (95% CI 0.94-0.98) and 0.99 (95% CI 0.98-0.99) respectively. The alternate-form reliability of different test activities was excellent for children (age 10-12y, n=30) performing the School-Kids AHA and Ad-AHA Board Game 0.99 (95% CI 0.98-0.99) and for adolescents (age 13-18y) performing the Ad-AHA Board Game compared to the Ad-AHA Present (n=28) 0.99 (95% CI 0.95-0.98), or the Ad-AHA Sandwich (n=29) 0.99 (95% CI 0.98-0.99) tasks. SDC for test-retest was 4.5 AHA-units. Ad-AHA scores are consistent across different raters and occasions. The good alternate-form reliability indicates that the different test activities can be used interchangeably in adolescents with unilateral CP. Differences greater than or equal to 5 AHA-units can be considered a change beyond measurement error. The use of logit based AHA-units makes change comparable for persons at different ability levels.

Reliability and Relationship to Retention of Assessing an Acquisition Rate for Sight Words With Kindergarten Students

Teaching children too many words during a lesson reduces retention. The amount of new information a student can successfully rehearse and recall later is called acquisition rate (AR), which has been reliably measured with students in first, third, and fifth grades. The purpose of this study was to examine the reliability of assessing AR for sight words with kindergarten students. A total of 32 kindergarten students from five classrooms across two elementary schools participated in the study. AR was measured twice over a 2-week period, and 1-day retention was measured for the first AR. The AR data resulted in a 2-week delayed alternate form reliability of r = .83, and there was also a strong correlation between AR and number of words retained 1 day later. The limitations, implications, and considerations for the name of the construct being assessed are discussed.

Progress Monitoring in Social Studies Using Vocabulary Matching Curriculum–Based Measurement

Two hundred and two (n = 202) sixth–grade students in social studies were administered a weekly vocabulary–matching curriculum–based measure (CBM) for 35 weeks. Students were also administered the Scholastic Reading Inventory (SRI), along with the annual state high–stakes test in Communication Arts. CBM scores were analyzed with respect to alternate form reliability, validity with criterion measures, and student growth over time. Results suggest that the vocabulary–matching CBM is reliable and valid with the SRI but not with the state test. Students showed an overall linear trend of growth, but this growth was flat in the middle of the semester. Implications for research and practice are discussed.

Development and Alternate Form Reliability of the Complex Task Performance Assessment (CTPA) for People With Mild Stroke.

Cognitive impairment is a common consequence of mild stroke. Current performance-based assessments for mild stroke can detect mild impairments in executive function but lack alternate forms to be used as outcome measures. This study aimed to develop an alternate form of the Complex Task Performance Assessment (CTPA-Alt), a performance-based assessment of executive function, and to establish the alternate form reliability of the CTPA-Alt. A repeated-measures study was conducted with 26 community participants. Participants were screened for eligibility and administered both forms of the CTPA; administration order was alternated. Overall performance was significantly correlated (rs = .44, p = .03), but pattern of scoring differed by CTPA form and order of administration. Our results indicate that the CTPA forms were similar but that the specific tasks in each form were different. The CTPA may be used as an ecologically valid outcome assessment with further considerations.

The Outcomes and Impact Scale - Revised: the psychometric properties of a scale assessing the impact of service provision.

The Outcomes and Impact Scale - Revised (O&IS-R) was developed to measure the impact of service provision on adults with disability. The aim of this research was to explore the psychometric properties of the scale following a recent revision. Adults with disability and/or their carers or support workers participated in the research. Participants were recruited across Australia using a range of strategies and completed the scale in relation to a service or support that they nominated. Two forms of the scale were developed (easy and standard English), which could be completed in hardcopy or via a bespoke website. The O&IS-R demonstrated good internal consistency and alternate form reliability. Factor analysis of the scale revealed one factor that clearly corresponded to the whole of life construct that the scale was designed to measure and, in general, participants with disability and staff understood the intent or purpose of the survey. All domains except the spiritual and political domain were considered to be relevant service outcomes. Some changes to wording were suggested to improve clarity. The O&IS-R is a psychometrically sound scale that can be used in service and research settings to measure the impact of services and supports.

A Curriculum-Based Measure of Language Comprehension for Preschoolers: Reliability and Validity of the Assessment of Story Comprehension

Despite research demonstrating the importance of language comprehension to later reading abilities, curriculum-based measures to assess language comprehension abilities in preschoolers remain lacking. The Assessment of Story Comprehension (ASC) features brief, child-relevant stories and a series of literal and inferential questions with a focus on causal and predictive inference skills surrounding the main story grammar components and a novel vocabulary word. Following an overview of the iterative development process and pilot studies, this article presents preliminary evidence of the fidelity of administration, reliability of scoring, alternate form reliability, and validity of the ASC. In all, 237 preschoolers, ages 3 to 5 years old, participated in this study. Fidelity of administration and scoring reliability averaged over 90%. Concurrent validity with two established language measures revealed correlations ranging from .67 to .81. Test–retest reliability and internal consistency also indicated high levels of reliability for this new tool; however, alternate form reliability results suggest further work is needed. Preliminary results indicate that the ASC holds promise as a viable curriculum-based measure that early childhood educators can use for monitoring preschoolers’ progress in language comprehension.

Accuracy of a Classical Test Theory-Based Procedure for Estimating the Reliability of a Multistage Test

The purpose of this simulation study was to assess the accuracy of a classical test theory (CTT)–based procedure for estimating the alternate-forms reliability of scores on a multistage test (MST) having 3 stages. We generated item difficulty and discrimination parameters for 10 parallel, nonoverlapping forms of the complete 3-stage test and ability parameters for a population of 30,000 simulated test takers. Using these parameters to generate item responses, we ran each of the 30,000 simulated test takers through each of the 10 simulated forms of the 3-stage test and computed the correlation of the scores on each pair of simulated test forms. We then computed the CTT estimate of the reliability, in the full population of 30,000 simulated test takers, of the total scaled scores resulting from the multistage testing procedure. We computed the estimate separately from the simulated responses to each of the 10 simulated forms of the 3-stage test. The reliability estimates from each simulated form of the MST differed by less than .005 from the average of the correlations between scores on that form and on the other 9 simulated forms of the MST.

Using an audit tool (MAPS Global) to assess the characteristics of the physical environment related to walking for transport in youth: reliability of Belgian data.

BackgroundThe aim was to examine inter-rater and alternate-form reliability of the Microscale Audit of Pedestrian Streetscapes (MAPS) Global tool to assess the physical environment along likely walking routes in Belgium.MethodsFor 65 children participating in the BEPAS-children study, routes between their individual homes and the nearest pre-defined destination were defined. Using MAPS Global, physical environmental characteristics of the routes were audited by 4 trained auditors (2 on-site, 2 online using Google Street View). Inter-rater reliability was studied for on-site and online ratings separately. Alternate-form reliability was examined by comparing on-site with online ratings.ResultsInter-rater reliability for on-site ratings was acceptable for 68% of items (kappa range 0.03–1.00) and for online ratings for 60% of items (kappa range −0.03 to 1.00). Acceptable alternate-form reliability was reported for 60% of items (kappa range −0.01 to 1.00/r range 0.31–1.00).ConclusionsMAPS Global can be used to assess the physical environment of potential walking routes. For areas where Google Street View imagery is widely covered and often updated, MAPS Global can be completed online.Electronic supplementary materialThe online version of this article (doi:10.1186/s12942-016-0069-1) contains supplementary material, which is available to authorized users.

Getting More From Your Maze: Examining Differences in Distractors

ABSTRACTThe present study examined the technical adequacy of maze-selection tasks constructed in 2 different ways: typical versus novel. We selected distractors for each measure systematically based on rules related to the content of the passage and the part of speech of the correct choice. Participants included 262 middle school students who were randomly assigned to 1 of the 2 maze formats. Scoring of the maze included both correct and correct-minus-incorrect scores. Students completed 3 criterion-reading tests: the Scholastic Reading Inventory, the AIMSweb R-Maze, and a high-stakes state assessment (the Missouri Assessment Program). Alternate-forms reliability was similar across maze formats; however, with regard to scoring procedure, reliability coefficients were consistently higher for correct than for correct-minus-incorrect scores. Validity coefficients were also similar across format with 1 exception: The coefficients for typical maze scores were stronger when compared with the Missouri Assessment Program scores than the coefficients for novel maze scores.

CLINICAL TRIALS IN THE PRECLINICAL TO MCI STAGES OF AD: CROSS-CULTURAL VALIDATION AND NORMATIVE STUDY OF THE TOMMORROW NEUROPSYCHOLOGICAL BATTERY

Clinical trials in preclinical Alzheimer’s disease (AD) require reliable and validated measures appropriate for the detection of subtle cognitive changes. In preparation for international site initiation in the TOMMORROW Study, a trial to delay the clinical onset of Mild Cognitive Impairment due to AD (MCI-AD), we conducted a multicultural and multilingual validation and normative study of a neurocognitive battery. The battery was culturally adapted for use in Italy, Switzerland (German language), and Russia to support the systematic application of operationalized criteria to detect MCI-AD in these countries. A total of 675 individuals comprised of 200 healthy controls [∼50/age strata/country; ages 65-88, including men and women with a range of low and high education] and 25 AD subjects from each country were recruited. Construct validity, test-retest, and alternate form reliability were assessed in healthy controls. Criterion and discriminant validity of the cognitive measures were tested in the entire sample (healthy controls and AD cases) using logistic regression, discriminant analysis, and Receiver Operating Characteristic analysis. Measure equivalence was examined in relationship to normative values from English-speaking populations in the United States. The translated and culturally adapted tests demonstrated high test-retest reliabilities as well as good construct and criterion validity. Tests performed comparably to the “gold standard” English versions with respect to direction, magnitude, and patterns of correlations and main effects. Several tests showed some cultural variability, underscoring the importance of country-specific norms. Gender influenced verbal episodic memory (women> men) and visual naming (men> women) performances in the German version. Education affected visual memory, naming, and attention measures in Italian; whereas, performance on a test of executive function (Trails B) required greater average time to completion in Russian when compared to the English language versions. The TOMMORROW neuropsychological battery has now been translated, culturally adapted, validated and normed for ages 65-88 in German, Italian, and Russian languages. The battery is in use currently in German (Switzerland & Germany) and English speaking sites participating in the trial, providing further opportunities for multilingual investigation of its utility in detecting MCI-AD and measuring change in preclinical AD.

Validity and reliability of the NAB Naming Test

Objective: Confrontation naming is commonly assessed in neuropsychological practice, but few standardized measures of naming exist and those that do are susceptible to the effects of education and culture. The Neuropsychological Assessment Battery (NAB) Naming Test is a 31-item measure used to assess confrontation naming. Despite adequate psychometric information provided by the test publisher, there has been limited independent validation of the test. Method: In this study, we investigated the convergent and discriminant validity, internal consistency, and alternate forms reliability of the NAB Naming Test in a sample of adults (Form 1: n = 247, Form 2: n = 151) clinically referred for neuropsychological evaluation. Results: Results indicate adequate-to-good internal consistency and alternate forms reliability. We also found strong convergent validity as demonstrated by relationships with other neurocognitive measures. We found preliminary evidence that the NAB Naming Test demonstrates a more pronounced ceiling effect than other commonly used measures of naming. Conclusions: To our knowledge, this represents the largest published independent validation study of the NAB Naming Test in a clinical sample. Our findings suggest that the NAB Naming Test demonstrates adequate validity and reliability and merits consideration in the test arsenal of clinical neuropsychologists.

Examining the Technical Adequacy of Progress Monitoring Using Early Writing Curriculum-Based Measures

The purpose of this study was to examine 4 early writing measures used to monitor the early writing progress of 1st-grade students. We administered the measures to 23 1st-grade students biweekly for a total of 16 weeks. We obtained 3-min samples and conducted analyses for each 1-min increment. We scored samples using 2 different methods: correct sequences and correct-minus-incorrect sequences. Delayed alternate-forms reliability was strong for the 3-min increment. We established concurrent criterion validity using the Test of Early Written Language–2 as well as teacher ratings of writing proficiency. We analyzed growth using hierarchical linear modeling, which suggested that all 4 measures were sensitive to biweekly growth. We discuss the results in terms of technical adequacy, utility of the measures, and the ability of the measures to serve as indicators of progress in early writing skills.

The Reliability of Randomly Generated Math Curriculum-Based Measurements

Curriculum-Based Measurement (CBM) is a direct method of academic assessment used to screen and evaluate students’ skills and monitor their responses to academic instruction and intervention. Interventioncentral.org offers a math worksheet generator at no cost that creates randomly generated math curriculum-based measures (M-CBMs). In this study, we examined the test–retest reliability and alternate-form reliability of four parallel, randomly generated M-CBMs designed to assess multiple arithmetic skills (addition, subtraction, multiplication, and division). The participants ( N = 283 sixth-grade students) completed each M-CBM worksheet twice during a semester. According to our results, these M-CBMs have moderate test–retest and alternate-form reliability. Applying the Spearman–Brown Prophecy Formula revealed that aggregating M-CBMs increases the reliability of these measures to acceptable levels for progress monitoring (i.e., above .80).

Validation of the Affective Word List as a measure of verbal learning and memory

Introduction: This study evaluated the Affective Word List (AWL), a measure designed to assess affective bias, as a measure of verbal learning and memory in the context of baseline concussion evaluations. The AWL was developed to assess affective bias in order to circumvent the tendency of some examinees to minimize self-report of depression symptoms. However, because it is designed as a traditional list-learning task, the cognitive indices of the AWL have the added potential to be used as measures of verbal learning and memory. It would be useful to have a performance-based measure that is sensitive to both the affective and cognitive consequences of concussion. Method: Participants from a university-based sports concussion program were used to evaluate the descriptive statistics and distribution of the AWL and its convergent and discriminant validity. A separate sample of undergraduate students, active in intramural or club athletics, served as participants for a delayed alternate-form reliability study. All reliability and validity results were compared with those of the Hopkins Verbal Learning Test–Revised (HVLT–R) and the Immediate Post-Concussion Assessment and Cognitive Testing Verbal Memory Composite (ImPACT-VM). Results: Results of this study showed that the cognitive indices of the AWL have normal distributions, and its four forms are equivalent. The AWL demonstrated moderate delayed alternate-form reliability, moderate convergent validity with other measures of verbal learning and memory, and strong discriminant validity with measures of processing speed and reaction time. Conclusions: Results of this study suggest that the AWL may have clinical utility as a measure of verbal learning and memory in concussion management and research.

An Independent Evaluation of the Technical Features of the Basic Reading Inventory

The present study investigated some psychometric properties of the Basic Reading Inventory (BRI), a widely used informal reading inventory. The BRI and Dynamic Indicators of Basic Early Literacy Skills (DIBELS) probes were administered to 149 third, fourth, and fifth graders. Test–retest and alternate forms reliability analyses indicated adequate levels of reliability for low-stakes decisions. Concurrent validity analyses indicated strong correlations with DIBELS, an established measure of reading fluency. These results are discussed in light of implications for use of the BRI for making instructional grouping, universal screening, and progress monitoring decisions.

Psychometric Evaluation of the Symptoms and Functioning Severity Scale (SFSS) Short Forms with Out-of-Home Care Youth.

There is a need for brief progress monitoring measures of behavioral and emotional symptoms for youth in out-of-home care. The Symptoms and Functioning Severity Scale (SFSS; Bickman et al., 2010) is one measure that has clinician and youth short forms (SFSS-SFs); however, the psychometric soundness of the SFSS-SFs with youth in out-of-home care has yet to be examined. The objective was to determine if the psychometric characteristics of the clinician and youth SFSS-SFs are viable for use in out-of-home care programs. The participants included 143 youth receiving residential treatment and 52 direct care residential staff. The current study assessed internal consistency and alternate forms reliability for SFSS-SFs for youth in a residential care setting. Further, a binary classification test was completed to determine if the SFSS-SFs similarly classified youth as the SFSS full version for low- and elevated-severity. The internal consistency for the clinician and youth SFSS-SFs was adequate (α = .75 to .82) as was the parallel forms reliability (r = .85 to .97). The sensitivity (0.80 to 0.95), specificity (0.88 to 0.97), and overall accuracy (0.89 to 0.93) for differentiating low and elevated symptom severity was acceptable. The clinician and youth SFSS-SFs have acceptable psychometrics and may be beneficial for progress monitoring and additional research should clarify their potential for progress monitoring of youth in out-of-home programs.

Web-Based Mathematics Progress Monitoring in Second Grade

We examined a web-based mathematics progress monitoring tool for second graders. The tool monitors the learning progress of two competences, number sense and computation. A total of 414 students from 19 classrooms in Germany were checked every 3 weeks from fall to spring. Correlational analyses indicate that alternate-form reliability was adequate for the chosen interval (.81 < r < .87). Results from latent growth curve modeling (LGCM) identified significant linear increases in students’ scores, and a significant variance in computation slopes was observed. Criterion validity coefficients, comparing the measures with student performance on standardized school achievement tests at the beginning and the end of the school year (DEMAT1+, DEMAT2+), were satisfactory (.59 < r < .76). Students conducted the tests independently, and assessment times were short. Implications for further research and classroom practice are discussed.

Assessing the inter-coder reliability of the Body Type Dictionary (BTD)

Abstract Computer-assisted content analysis has many advantages compared to a manual scoring system, provided that computerised dictionaries represent valid and reliable measures. This study aimed to assess the inter-coder reliability, alternate- form reliability and scoring consistency of the Body Type Dictionary (BTD) (Wilson 2006) based on Fisher and Cleveland’s (1956, 1958) manual body boundary scoring scheme. The results indicated an acceptable inter-coder agreement with barrier and penetration imagery in the sub-sample (N = 53) of manually coded Rorschach responses. Additionally manually coded scores showed an acceptable correlation with the computerised frequency counts, and thus indicating an alternate-form reliability. In the full data set (N = 526), barrier imagery in the Rorschach responses only correlated with the picture response test, showing low scoring consistency, which might disconfirm the notion of body boundary awareness representing a stable personality trait but instead it might be dependent on the level of cognitive dedifferentiation.

General Outcome Measures of Beginning Reading for Young Students in Taiwan: Technical Adequacy of Slopes

This study examined technical characteristics of slopes produced from two general outcome measures (GOMs)—the Naming Spelling and the Dictating Spelling tests. A total of 305 first-graders in Taiwan participated in the study and underwent weekly assessments for 6 weeks. Six alternate forms were developed for each of the two measures to monitor young students' learning progress of Zhu Yin Fu Hao (ZYFH). Technical characteristics of the two measures were examined in terms of reliability, sen- sitivity, and criterion validity for assessing student growth. Results showed that the alternate-form reliability coeffi- cients ranged from .88 to .94 (mean = .92) for the Naming Spelling test, and from .77 to .83 (mean = .80) for the Dictating Spelling test. Both tests demonstrated to be sensitive for monitoring student's learning progress by revealing a quadratic growth pattern and inter-individual differences in growth rates. The growth rates estimated from repeated Naming and Dictating Spelling tests were found to correlate with students' performance on the final exam of ZYFH for lower-performing students. Results support the use of these two GOMs tests as reliable, sen- sitive, and valid measures for monitoring first-graders' learning progress of ZYFH.

Assessing the Alternate-Form Reliability of Interview-Based and Web-Based Rorschach Responses Measuring Body Boundary Imagery and Regressive Imagery

This study assesses the alternate-form reliability of the Body Type Dictionary (BTD) for measuring body boundary imagery and primordial thought language in interview-based and web-based Rorschach responses. The intraclass correlation coefficient demonstrated fair to good agreement for barrier imagery, .72, and penetration imagery, .55, which indicates that the web-based administration of the Rorschach inkblot test represents an acceptable alternative to the traditional Rorschach interview assessment for measuring body boundary imagery. Primordial thought language had a fair level of agreement, .43, whereas conceptual thought language had poor agreement, .36. The results are discussed by relating empirical research outlining the mode-specific implications of psychometric test administration to the Rorschach inkblot test and its implications for body boundary awareness and regressive cognitive functioning, as well as by outlining the methodological and clinical limitations of web-based Rorschach application that could be addressed in future research.