Conduct Item Analysis Research Articles

Development of PE Metrics Elementary Assessments for National Physical Education Standard 1

This article describes how assessments in PE Metrics were developed following six steps: (a) determining test blueprint, (b) writing assessment tasks and scoring rubrics, (c) establishing content validity, (d) piloting assessments, (e) conducting item analysis, and (f) modifying the assessments based on analysis and expert opinion. A task force, composed of researchers, measurement and evaluation experts, teacher educators, K–12 physical education teachers, and education administrators, was formulated. The task force then determined a test blueprint for Grades K, 2, and 5 and developed corresponding assessments to assess the standards. The content validity evidence was established by a panel of experts examining the degree to which the content of the assessments matched the content of the national standards, specifically Standard 1. A total of 30 assessments (Kindergarten = 8, Grade 2 = 11, and Grade 5 = 11) were developed. They were piloted to a total of 773 students (Kindergarten = 232, Grade 2 = 268, and Grade 5 = 273). Descriptive statistics (e.g., M, SD, frequency) were computed for each assessment. More than 50% of the means were between 2.2 and 2.8 (on a 4-point scoring rubric). Assessment responses were well distributed; only 2.2% had an SD of 0, which indicates that assessments were well developed. With some editorial changes, the assessments were ready for the final calibration of PE Metrics construction.

Scales for assessing self-efficacy of nurses and assistants for preventing falls

This paper is a report of the development and testing of the Self-Efficacy for Preventing Falls Nurse and Assistant scales. Patient falls and fall-related injuries are traumatic ordeals for patients, family members and providers, and carry a toll for hospitals. Self-efficacy is an important factor in determining actions persons take and levels of performance they achieve. Performance of individual caregivers is linked to the overall performance of hospitals. Scales to assess nurses and certified nursing assistants' self-efficacy to prevent patients from falling would allow for targeting resources to increase SE, resulting in improved individual performance and ultimately decreased numbers of patient falls. Four phases of instrument development were carried out to (1) generate individual items from eight focus groups (four each nurse and assistant conducted in October 2007), (2) develop prototype scales, (3) determine content validity during a second series of four nurse and assistant focus groups (January 2008) and (4) conduct item analysis, paired t-tests, Student's t-tests and internal consistency reliability to refine and confirm the scales. Data were collected during February-December, 2008. The 11-item Self-Efficacy for Preventing Falls Nurse had an alpha of 0·89 with all items in the range criterion of 0·3-0·7 for item total correlation. The 8-item Self-Efficacy for Preventing Falls Assistant had an alpha of 0·74 and all items had item total correlations in the 0·3-0·7 range. The Self-Efficacy for Preventing Falls Nurse and Self-Efficacy for Preventing Falls Assistant scales demonstrated psychometric adequacy and are recommended to measure bedside staff's self-efficacy beliefs in preventing patient falls.

Reliability and Validity of the Perspectives of Support From God Scale

Existing spiritual support scales for use with cancer survivors focus on the support believed to come from a religious community, clergy, or health care providers. The objective of this study was to evaluate the reliability and validity of a new measure of spiritual support believed to come from God in older Christian African American cancer survivors. The Perceived Support From God Scale was administered to 317 African American cancer survivors aged 55-89 years. Psychometric evaluation involved identifying underlying factors, conducting item analysis and estimating reliability, and obtaining evidence on the relationship to other variables or the extent to which the Perceived Support From God Scale correlates with religious involvement and depression. The Perceived Support From God Scale consists of 15 items in two subscales (Support From God and God's Purpose for Me). The two subscales explained 59% of the variance. Cronbach's alpha coefficients were .94 and .86 for the Support From God and God's Purpose for Me subscales, respectively. Test-retest correlations were strong, supporting the temporal stability of the instrument. Pearson's correlations to an existing religious involvement and beliefs scale were moderate to strong. Subscale scores on Support From God were negatively correlated to depression. Initial support for reliability and validity was demonstrated for the Perceived Support From God Scale. The scale captures a facet of spirituality not emphasized in other measures. Further research is needed to evaluate the scale with persons of other racial/ethnic groups and to explore the relationship of spirituality to other outcome measures.

Development of a two-tier diagnostic test concerning genetics concepts: the study of validity and reliability

Development of a two-tier diagnostic test concerning genetics concepts: the study of validity and reliability

Study on the short version of general maladjustment scale and its reliability and validity

Objective To conduct item analysis on General Maladjustment Scale (Gm) and to develop a short-versioned scale.Methods Gm and Symptom Checklist 90 (SCL-90) were administered to 3767 recruits.difficulty and discrimination of each item of Gm were calculated,correlation between item score and total score was analyzed,short-versioned Gm was developed based on the results of item analysis.Cronbach's α coeffieient of the original scale and the short-versioned scale was calculated, criterion validity was analyzed using SCL-90 as the validity criterion.Results Difficulty of the original scale was from 0.03 to 0.72.21 items'difficultv was less than 0.20.Discrimination index was from -0.19 to 0.45.21 items'discrimination index was less than 0.30.Corre.lation coefficient of the item score and the total score is from -0.17 to 0.48.21 items'correlation coefficient was less than 0.30.The short-versioned scale was made up of the 16 items with better quality.Cronbach's α coeffi.cient of the original scale and the short-versioned scale was 0.713 and 0.700.respectively.Both the total score of the original scale and the short-versioned scale were significantly correlated with the scores of SCL-90,and all the correlation coefficient were above 0.40.The correlation coefficient between the score of the short-versioned scale and that of the original scale was 0.914.Conclusion Both the original scale and the short-versioned scale have a higher reliability and good validity. Key words: Military psychology; Maladjustment; Psychometrics; Reliability; Validity

Analytic assessment of multiple-choice tests

Background : Multiple choice tests (MCT),are widely known and applied as useful evaluation tests in the field of education especially in Medical Science. Items on a multiple-choice test consist of a stem, which is followed by a correct answer as well as three to jour distracters. Items on a well-written multiple-choice test will have stems that are precise and clear, one answer that is clearly correct or best, and distracters that are plausible. Purpose : The purpose of the present study is conducting item and test analysis to 24 MCTs given in first semester of 2000-2001 educational year in medical faculty of Mashad University of Medical Science. Methods : Data of this descriptive study were composed of 1496 MCQs gathered from 2092 answer sheets of 24 MCTs obtained from educational department of the medical faculty.A split-half method of reliability was employed to calculate reliability coefficient for MCTs. Items Difficulty and Discrimination index also were calculated for questions. Further studies should be undertaken for developments the methods for evaluation of validity, assessment of distracters and structural principles in MCTs . Results : Mean reliability coefficient of the exams was 0.72±0.13 and In more than 50% of cases, reliability coefficient was greater than 0.7. There was a significant difference between basic science exams and clinical clerkship exams in Reliability coefficient (P=0.001). Mean standard error a/measurement (SEM) was 3.51±1.11. In 52.2% of the cases, difficulty of MCQs was inappropriate and 49.3% of questions had inadequate discriminative power to discern between poor students and good students. Conclusion : Our finding indicate that only 33% of studied MCQs have desirable or acceptable item difficulty and discrimination indices both and 34.9% of those have no desirable or acceptable item difficulty neither acceptable discrimination index. Having subjects respond reliably on a measure is a great sta11, but there is another concept needed to gel down really well named validity. Keywords : multiple choice question, test analysis, reliability, item difficulty Discrimination index

SEMANTIC AND STRUCTURAL FACTORS AFFECTING THE PERFORMANCE OF MATCHED BLACK AND WHITE EXAMINEES ON ANALOGY ITEMS FROM THE SCHOLASTIC APTITUDE TEST

ABSTRACTAnalogy items from the Scholastic Aptitude Test (SAT) were evaluated for differential performance by Black and White examinees. Black and White examinees were first matched for overall SAT‐V scores prior to conducting item analyses. A content and psycholinguistic analysis of 220 disclosed SAT analogy items was performed. Regression analyses indicate that Black examinees consistently perform differentially better than matched White examinees on the hard analogy items! However, for easy items, particularly those that involve “science” content, White examinees appear consistently to perform differentially better than matched Black examinees. In addition, semantic relationships dealing with part/whole relationships in the item stem also contributed negatively to Black examinee percent correct responses. Three variables (item difficulty, science content, part/whole relationship) together account for 30% of the variance between the two ethnic (Black and White) groups. Of these three significant predictors, two are semantic (part/whole and science content) while the third (item difficulty) reflects a non‐semantic factor. Several hypotheses are advanced to explain these findings.