Classical Item Analysis Research Articles

Measurement of semantic knowledge of object categories: Creating the Semantic Vanderbilt Expertise Test (SVET)

Poster presented at Vision Science Society Annual Meeting, May 2014, St. Pete Beach, FL A growing interest in individual differences in face and object recognition motivates the development of visual learning tests such as the Cambridge Face Memory Test (Duchaine & Nakayama, 2006) and the Vanderbilt Expertise Test (VET; McGugin et al., 2012). But experience with a category may also result in non-perceptual knowledge. We seek a reliable and valid measure of non-perceptual semantic knowledge in a standard format for a variety of object categories (rather than one category, Barton et al. 2009; Van Gulick & Gauthier, VSS2013) that applies to the full range of expertise in a domain. The Semantic Vanderbilt Expertise Test (SVET) focuses on one aspect of semantic knowledge that can be measured across categories: acquisition of relevant nomenclature. Each trial consists of a triplet of names, one real name of an object in that category and two foils. The SVET 1.0 includes 7 categories: birds, cars, dinosaurs, planes, shoes, Transformers, and trees. Through multiple iterations of data collected on Amazon Mechanical Turk test items were fine-tuned based on factor analysis, classical item analyses, and item response theory. In data from samples of 96-101 subjects per category, all tests showed good internal consistency (Cronbach’s alpha). We validated each SVET against subjects’ self-reports of their category knowledge, assessed with 7 domain-specific questions (Gauthier et al., submitted). Across all categories, subjects’ general rating of their category experience was most strongly correlated with SVET performance (r=.38) followed by their rating of how detailed an essay they could write about the category (r=.35). Interestingly, just like perceptual performance on the VET, the SVET correlation with age and gender differed across categories, suggesting a role for experience. The SVET can provide assessment of semantic knowledge to complement visual measures, and to help understand how performance is determined by the interaction of perceptual and cognitive abilities with experience. This work is supported by NSF (SBE-0542013), the Vanderbilt Vision Research Center (P30-EY008126), and the National Eye Institute (R01 EY013441).

Classical Item Analysis of the Greek State Certificate of English Language Proficiency "A" Level Exam

Classical Item Analysis of the Greek State Certificate of English Language Proficiency "A" Level Exam

Assessing cohesion in children’s writing: Development of a checklist

Abstract Cohesion in writing is achieved through the use of linguistic devices that tie ideas together across a text, and is an important element in the development of coherent writing. Research shows that inter- and intra-developmental differences may appear in how children learn to use these devices, but cohesion is commonly overlooked in the evaluation and instruction of writing. In this study, we developed a checklist to assess cohesion in the writing of children in Grades 4–7, with the purpose of informing instructional practices. Following the procedure outlined by Crocker and Algina (1986) , we developed and evaluated a checklist designed to assess the types of cohesive devices present in the writing of children. The checklist items showed fair to good discrimination between high and low scoring writers as demonstrated by a classical item analysis. We also found good interrater reliability, and evidence for discriminative validity. As internal consistency was weak, however, further research is needed to refine the instrument. Implications for the assessment of cohesion and future research are discussed.

Parental Human Papillomavirus Vaccine Survey (PHPVS): Nurse-Led Instrument Development and Psychometric Testing for Use in Research and Primary Care Screening

Human papillomavirus (HPV) vaccine was approved for girls aged 9-24 years in 2006 to prevent HPV infection and cervical cancer. The Parental Human Papillomavirus Survey (PHPVS) was framed on theoretical constructs of the health belief model (HBM) and developed to survey parents regarding their HPV knowledge, attitudes, and intent to vaccinate. We evaluated the psychometric properties of the PHPVS using classical item analysis and exploratory factor analysis (EFA) among a sample of 200 parents/caregivers. The EFA yielded a 4-factor unidimensional model that explained between 62% and 68% of the total variance depending on the extraction method used. The estimated Cronbach's alpha for the PHPVS was .96. The PHPVS is a reliable measure of HPV knowledge, attitudes, and intent to vaccinate.

Challenges of the Fennema-Sherman Test in the International Comparisons

The shortened version of Fennema-Sherman test is used to measure attitudes toward mathematics in several international testing settings like TIMSS and PISA. On the basis of Classical Item Analysis and Confirmatory Factor Analysis in the different achievement levels and sets of countries, it is suggested that there are two items on the Fennama-Sherman test which should be discarded due to cultural and achievement considerations. Items “Mathematics is more difficult for me than for many of my classmates” and “Mathematics is not one of my strengths” are too complicated for test takers who belong to the lowest quartile of achievement. These items also seem to carry culturally sensitive elements especially in East Asian countries where fairly good students answer illogically due to such negative wordings. Alternative possibilities for test items are recommended.

Assessing Recovery of People With Serious Mental Illness: Development of a New Scale

The authors describe the development of the Maryland Assessment of Recovery in People with Serious Mental Illness, or MARS, a 25-item self-report instrument that measures recovery of people with serious mental illness, and report a study of its psychometric properties. Doctoral-level scientists with expertise in serious mental illness drafted a set of survey items about the recovery process. Items reflected recovery domains outlined by the Substance Abuse and Mental Health Services Administration. After consultation with a panel of experts on recovery that included consumers and clinical scientists and with a small group of consumers, the instrument was narrowed to 67 items and administered to 166 individuals recruited from outpatient mental health clinics in two states. Item response theory and classical item analysis were used to select best-fitting items, reduce item redundancy, and improve the psychometric properties of the scale. Principal components analysis and confirmatory factor analysis were conducted to further examine dimensions of recovery measured by the scale. The MARS is quite practical for use with individuals with serious mental illness. It demonstrated excellent internal consistency (Cronbach's α=.95) and test-retest reliability (r=.898) and good face and content validity. The data provide initial support for use of the MARS to measure recovery of people with serious mental illness.

Advantages of the Rasch measurement model in analysing educational tests: an applicator's reflection

Educational achievement is a very important issue for parents, teachers, and the government. An accurate measurement plays a very important role in evaluating achievement fairly, and, therefore, analysis methods have been developed considerably in recent years. Education based on long-time learning processes forms a fruitful base for item tests, as it opens up possibilities to design a nearly unlimited number of items. There are several methods available to analyse such tests, but which method is the most useful and in which situation? The Rasch measurement from the Institute for Object Measurement (IOM) has pointed out to be the most useful in evaluating individual items and their functioning in a test. While the Rasch model gives a great deal of information about each single item on a test, the classical item analysis can provide additional information about response alternatives. Multiparametric item response theory (IRT) models can also provide some additional information about items, but those models require careful interpretation of results.

Classical Item Analysis Using Latent Variable Modeling: A Note on a Direct Evaluation Procedure

A directly applicable latent variable modeling procedure for classical item analysis is outlined. The method allows one to point and interval estimate item difficulty, item correlations, and item-total correlations for composites consisting of categorical items. The approach is readily employed in empirical research and as a by-product permits examining the latent structure of tentative versions of multiple-component measuring instruments. The discussed procedure is straightforwardly utilized with the increasingly popular latent variable modeling software Mplus, and is illustrated on a numerical example.

Classical item and test analysis with graphics: The ViSta-CITA program

Current advances in test development theory have mostly been influenced by item response theory. Notwithstanding this, classical test theory still plays a major part in the development of tests for applied educational and behavioral research. This article describes ViSta-CITA, a computer program that implements a set of classical item and test analysis methods that incorporate innovative graphics whose aim is to provide deeper insight into analysis results. Such an aim is achieved through the SpreadPlot, a graphical method designed to display multiple, simultaneous, interactive views of the analysis results. It behaves on a dynamic basis, so that users' changes (e.g., selecting a subset of items) are automatically updated in the graphical windows showing the analysis results. Moreover, ViSta-CITA is freely available, and its code is open to modifications or additions by the user. Features such as these constitute useful tools for research and teaching purposes related to test development.

The measurement of visuo–spatial and verbal–numerical working memory: Development of IRT-based scales

The objective of this study is to explore the potential for developing IRT-based working memory scales for assessing specific working memory components in children (8–13 years). These working memory scales should measure cognitive abilities reliably in the upper range of ability distribution as well as in the normal range, and provide a much-needed, reliable, and valid test for assessing high intellectual abilities in children. Six computer-assisted working memory tasks were administered to 172 children from regular schools and to 202 children from special schools and other institutions for the gifted. A factor analysis revealed a two-factor structure and the existence of a verbal–numerical and a visuo–spatial working memory scale. Classical item analysis and IRT analysis yielded good psychometric properties for both scales and revealed that the scales are appropriate for measuring high cognitive abilities. Both scales showed substantial and differential power for the explanation of variance in school achievement.

Refining Lodahl and Kejner’s Job Involvement Scale with a Convergent Evidence Approach: Applying Multiple Methods to Multiple Samples

Lodahl and Kejner’s Job Involvement (JI) measure has been and continues to be heavily used despite known measurement deficiencies. Using a convergent evidence approach, the authors examine the psychometric properties of that scale and offer a refined version that accurately taps the JI construct. Based on a combination of five methodologies (qualitative content analysis, classical item analyses, item response theory analyses, partial confirmatory factor analyses, and discriminant validity analysis) applied to five samples, results indicate that numerous items function inadequately as indicators of JI, whereas a core of items have superior item statistics and conceptually match the definition of JI. The advantages of using a convergent evidence approach are discussed.

Computer Program Exchange: CIA-D: An SAS Program for Classical Item Analysis of Dichotomously Scored Items

Computer Program Exchange: CIA-D: An SAS Program for Classical Item Analysis of Dichotomously Scored Items

Can the C-test be improved with classical item analysis?

The application of the rule-of ‘two’ for constructing C-tests produces two sorts of test items. Many items delineate acceptable facility and discrimination values, but a sizeable number of them are either extremely easy or extremely difficult to fill in. To investigate whether this defect can be avoided, a C-test with 5 texts and 126 items was constructed and tried with 146 Iranian English majors. On the basis of an item analysis, a tailored C-test with 100 items was developed and tried with 60 other subjects. The results of the study showed that no gains were made with the classical item analysis.

Item Function Characteristics and Dimensionality for Alternative Response Formats in Mathematics

The purpose of this study was to investigate the technical properties of stem-equivalent mathematics items differing only with respect to response format. Using socio- economic factors to define the strata, a proportional stratified random sample of 1,366 Connecticut sixth-grade students were administered one of three forms. Classical item analysis, dimensionality assessment, item response theory goodness-of-fit, and an item bias analysis were conducted. Analysis of variance and confirmatory factor analysis were used to examine the functioning of the items presented in the three different formats. It was found that, after equating forms, the constructed-response formats were somewhat more difficult than the multiple-choice format. However, there was no significant difference across formats with respect to item discrimination. A differential item functioning (DIF) analysis was conducted using both the Mantel-Haenszel procedure and the comparison of the item characteristic curves. The DIF analysis indicated that the presence of bias was not greatly affected by item format; that is, items biased in one format tended to be biased in a similar manner when presented in a different format, and unbiased items tended to remain so regardless of format.

Tailoring the Test to Fit the Students : Improvement of the C-Test through Classical Item Analysis

Tailoring the Test to Fit the Students : Improvement of the C-Test through Classical Item Analysis

Tailored cloze: improved with classical item analysis techniques

This study investigates the possibility that the reliability and validity of a cloze procedure can be improved by applying traditional item analysis and selection techniques. Students in a single level (n = 89) at the Guangzhou English Language Centre (People's Republic of China) were chosen for this study because previous experience and research had indicated that cloze tests gene rally produce low reliability and validity coefficients in samples wherein the range of ESL proficiencies is limited. This turned out to be the case when a 399-word every seventh word deletion cloze passage with 50 items was administered to this group. The study was designed such that 250 of the poten tial items in this passage could be piloted and item analyzed. The results of the item analysis were used to select the 'best' items on the basis of item facility and discrimination indices. The resulting 50 item 'tailored cloze' was then read ministered to the same group and the results for the original version of the cloze test were compared to those for the tailored version. These results indicate statistically significant and meaningful improvements in test quality due to the revision. The dispersion of scores, reliability and validity were all substantially improved by the item analysis and selection processes. The article concludes with discussion of the implications of these findings.

Comparative analyses of English as a Second Language reading comprehension data

Item responses on a multiple-choice reading comprehension test from 88 ESL subjects were submitted to classical test item analyses and the Rasch one-parameter latent trait model. The Rasch model identified more misfitting items than the classical item analyses and allowed the researchers to plot the reading comprehension items at their calibrated positions along the lines of the latent trait. The trait was defined at the easy end by items that assess literal comprehension and inference of explicit details and at the difficult end by items that assess multiple string inferences. The Rasch model also indicated that additional items were needed at both ends of the difficulty continuum, if subjects in those ranges are to be discriminated properly.

A psychometric investigation of the Multidimensional Health Locus of Control Questionnaire.

The validity of the alternate forms of the Multidimensional Health Locus of Control scale and the underlying dimensionality of the three test scales were investigated using 70 male and 77 female British university students. A classical item analysis, principal component and image component factor analysis were undertaken. The results indicated that the alternate forms could be considered parallel. However, a sex difference appeared in the factor structure of the test such that the male sample yielded only two factors while the female sample yielded the three scale factors as expected.

THE INTERNAL STRUCTURE OF THE MOTIVATION ANALYSIS TEST

Summary. Cattell's Motivation Analysis Test (MAT) is the only published instrument for assessing his comprehensive motivational theory. The homogeneity of the MAT was investigated by classical item analysis, and the 20 scales were found to have uniformly low item‐total correlations. This may explain why a number of attempts to factor analyse the test have given inconsistent results, and suggests that the MAT would require extensive revision before being applied to educational problems in Britain.

A Comparison of Classical and Latent Trait Item Analysis Procedures

Several methods of psychometric instrument construction have been suggested in the literature. Although there is considerable information describing theoretical and mathematical differences among these methods, there is a paucity of data regarding actual differences resulting from practical applications. Therefore, the present study was designed to compare classical item analysis procedure with the Rasch latent trait analysis in terms of several applied concerns. Respondents were 373 predominately white university undergraduates who completed the MacAndrew alcoholism scale and the Khavari Alcohol Test, an instrument for assessing quantity-frequency of alcohol consumption. Responses to the MacAndrew scale were analyzed using both classical and Rasch procedures and the resulting sub-scales were then compared in terms of specific items selected, relationship to the full scale, and assessment of actual alcohol consumption. Results showed substantial differences between the two procedures in terms of selected items. Several items were selected by the Rasch procedure but rejected by the classical procedure and vice versa. However, items selected by these procedures appear to make little difference in terms of correlations with the original full scale or with a criterion measure. Thus, due to its simplicity, classical item analysis may be indeed a procedure of choice if some type of item analysis is deemed necessary for clinical instruments.