Item Analysis Procedures Research Articles

Medications Non-adherence Reasoning Scale (MedNARS): Development and psychometric properties appraisal.

Proper elucidation of medication non-adherence reasoning especially in older adults might pave the way for an auspicious therapeutic outcome. The main aim of this study was to develop and psychometrically test the Medications Non-adherence Reasoning (MedNARS) questionnaire for application in research and probably practice settings. A mixed methods design was utilized to develop the MedNARS. The item pool was mainly generated based on a qualitative query and literature review. The expert panel approved version of the MedNARS was psychometrically assessed on a convenience sample of 220 older patients with chronic disease. The internal consistency, test-retest reliability, content and face validity of the scale were appraised and its construct validity was assed using exploratory and confirmatory factor analyses. A nine-item version of the MedNARS was drafted based on the classical item analysis procedures and its estimated internal consistency measure of the Cronbach's alpha (0.85) and test-retest reliability (0.96) were in the vicinity of acceptable range. The exploratory factor analysis (EFA) output revealed a unidimensional structure for the MedNARS and the conducted confirmatory factor analysis (CFA) indicated an acceptable data fit for the extracted one-factor model. The goodness of fit indices were as the followings: χ2 /df=1.63(90% CI: 0.02 to 0.11), root mean squared error of approximation (RMSEA)=(0.07), comparative fit index (CFI)=0.95, Tucker-Lewis index (TLI)=0.93 and standardized root mean squared residual (SRMSR)=(0.05). The study findings were indicative of MedNARS's applicability and feasibility for use in assessment of medication non-adherence reasoning among the elderly patients with chronic diseases. The MedNARS as a brief and elder-friendly instrument can be applied both in research and practice settings to enhance efficiency, safety, and health outcomes of the therapeutic recommendations.

Exploring the Quality of Multiple-Choice Question Type of Test Items in Information and Communication Technology Using Item Analysis

The study explored the quality of multiple-choice question types of test items in Information and Communication Technology (ICT) using item analysis. The study was an exploratory type where secondary data was collected and analysed. The secondary data was made up of ten (10) Multiple Choice Questions (MCQ) items on an ICT test instrument. Thirty learners’ test instruments were sampled. The item analysis procedure included the item reliability coefficient, coefficient of variation, difficulty level, discrimination index, and distractor analysis. The study revealed that the test items had a low-reliability power and the class is homogenous. It was also found that ten per cent (10⁒) of the items were highly appropriate, fifty per cent (50⁒) needed revision, and forty per cent (40⁒) needed to be discarded. The study also found that some of the distractors were not appropriate and hence needed review while others need to be replaced. It was, therefore, recommended that item analysis should embrace race by teachers since it will foster effectiveness and efficiency carry out their duties in teaching and learning. Moreover, the Ministry of Education in Ghana must liaise with various heads of educational institutions to contract measurement and evaluation experts to organize workshops for their staff on how to do item analysis. Keywords: Reliability coefficient, Discrimination index, Difficulty level, Distractor analysis, Test items

Identifying Problematic Item Characteristics With Small Samples Using Mokken Scale Analysis.

Researchers frequently use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, when they have relatively small samples of examinees. Researchers have provided some guidance regarding the minimum sample size for applications of MSA under various conditions. However, these studies have not focused on item-level measurement problems, such as violations of monotonicity or invariant item ordering (IIO). Moreover, these studies have focused on problems that occur for a complete sample of examinees. The current study uses a simulation study to consider the sensitivity of MSA item analysis procedures to problematic item characteristics that occur within limited ranges of the latent variable. Results generally support the use of MSA with small samples (N around 100 examinees) as long as multiple indicators of item quality are considered.

Klasika: Program Analisis Item dan Tes dengan Pendekatan Klasik

This article introduces the Klasika software developed to run item and test analysis using the Classical Test Theory approach. Classical Test Theory is one of the specialized competencies and skills that undergraduate students of psychology must possess. Classical Test Theory becomes a mandatory course for all schools or departments of psychology in Indonesia. This article also provides a theoretical foundation of Classical Test Theory's essential concepts and statistical methods, specifically related to items and test statistics. The item analysis and test reliability procedures using Klasika, starting from the data preparation untill data interpretation, are explained with an empirical illustration. Finally, the analysis results using Klasika are compared with the results from Quest software to test the accuracy of the estimation results.

How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items.

Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven’s progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen’s ω and the canonical correlation RCC) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., N = 100 or N = 200). Both suggested effect size measures (Cohen’s ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen’s ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven’s progressive matrices items in a sample of N = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed.

Development of a test for measuring the knowledge level of farmers in maize cultivation

It has become imperative to develop an effective and holistic system of high production and tackling pests to make it more environmental friendly, economically viable and socially acceptable to farmers which can be achieved through adoption of technologies, so it is very important to assess the knowledge level of farmers in maize farming in order to identify gap in adoption. Hence, a study was designed to develop a standardized test using test construction methodology adopting item analysis procedure, pilot tested with 30 farmer members from non-sample area. The final test consisted of 7 questions having difficulty index value in the range of 30 to 80 and discrimination index value above 0.3 and above was retained and used to measure the knowledge of maize growing farmers. The reliability of the knowledge test was measured with the help of split-half method and the reliability co-efficient was found to be 0.8, which indicates that the knowledge test is reliable. Also, criterion validity was measured after establishing theoretical relationship between knowledge and adoption of recommended package of practice. Adoption levels of recommended package of practices for maize were calculated for 30 farmers. These scores were correlated with the knowledge scores. The ‘r’ value was found to be 0.85. Since the ‘r’ values were significant at 0.01 level of probability, the scale developed was considered as valid. The test was further administered to 150 respondents and it was found that majority of the respondents (44%) had medium level of knowledge about recommended package of practices for paddy cultivation.

A Standardised Knowledge Test to Measure the Extent of Knowledge of Agricultural Extension Personnel on M-Tools

A standardised teacher made test for assessing the extent of knowledge of agricultural extension on m-tools was developed using the item analysis procedure. For the purpose, a teacher made test consisting of 30 items were prepared and administered to 30 agricultural extension personnel. Based on the results of the study,14 items having difficulty index value ranging from 20 to 80 and discrimination index value above 0.10 were selected to construct the knowledge test. This standardised test can be used to measure knowledge level of the extension personnel on m-tools.

Item Analysis Methodology to Measure Knowledge of Farmer's on Eco-Friendly Farm Technology in Rice Cultivation

The increased pressure to maintain high level of rice output for consumption has resulted in increased use of pesticides on rice fields which leads destruction of rice ecosystem. It has become imperative to develop an effective and holistic system of tackling pests to make it more environmental friendly, economically viable and socially acceptable to farmers which can be achieved through eco-friendly technologies, so it is very important to assess the knowledge level of stakeholders in eco-friendly rice farming in order to identify gap in adoption. Hence a study was designed to develop a standardized test using test construction methodology adopting item analysis procedure, pilot tested with 26 farmer members from non-sample area. The final test consisted of 21 questions having difficulty index value in the range of 20 to 80 and discrimination index value above 0.10 was retained and used to measure the knowledge of stakeholders in eco-friendly rice farming.

Quale quantità e quale qualità per la comprensione delle esperienze educative

Which Quantity and Quality for Understanding Educational Experiences This paper illustrates the methodology adopted to design and calibrate a test for measuring the student’s linguistic abilities at the end of the third year of vocational training courses (10th grade). The work is part of a major ISFOL study also including tests on math and strategic abilities. The test was calibrated on a judgement sample of 1046 students. It measures listening, reading comprehension, mechanics and written composition through several tasks. This paper presents methodologies of two item analysis procedures the CTT («Classical Test Theory») and IRT («Item Response Theory»), and the procedures for designing derivate variables from different tasks. In particular, it presents the advantages of using the Rasch model for a better measure of the latent trait of ability and for calibrating the test items. The paper rounds off by illustrating the main results of the multivariate analysis of the study. These results should obviously be interpreted in the light of the characteristics of the target population, its social and cultural background, which is weak in this case, and the collection of its school failures.

The Structure of Voice Efficacy and its Role in the Formation Mechanism of Employee Voice Behavior

As one representative kind of extra-role and challenging-promotive discretionary behavior, employees’ voice behaviors are valuable for employees themselves and the organization as a whole, especially in the complex economic society in the modern times. Nowadays, there are multitudes of scholars who had paid more attention to this important organizational phenomenon, and explored the formation mechanisms under this behavior preliminarily, in varieties of perspectives. Although these findings were inspiring, the cognitive decision-making processes underlying the behavioral engagement had remained relatively unexplored. Since Chiaburu et al.(2008) had first mentioned about this topic, and assumed an initial framework explaining discretionary behaviors including voice in particular using cognitive processes model, there were few scholars met this. This study attempted to fill this gap. On the foundations of the Social Cognitive Theory (Bandura, 1986) and Conservation of Resources Theory(Hobfoll, 2002), after inducing and clarifying the definition of voice efficacy, this study explored the role that voice efficacy played in the cognitive mechanism of voice behavior. The research was consisted of two sub-studies which sampled from private-firm employees and joint-venture employees in Yangtze Delta area of China. Study 1, according to the advice of Kish-gephart et al.(2009), 26 random samples of employees were interviewed to collect critical incidents of voice efficacy, and 14 items were collected. Then a questionnaire was made up of based on this 14 items. After the data survey, 3 items were deleted by item analysis procedure. Then, exploratory factor analysis (EFA) was conducted with 188 samples, the results showed that voice efficacy was one-dimension construct containing 7 optimal items. The results of confirmatory factor analysis (CFA) with the other 188 samples further supported this construct. Study 2 probed the effects of voice efficacy in the formaiton mechanism of employee voice behavior. Based on 401 employee-supervisor matched samples, the results indicated that, general self-efficacy was positively related to voice behavior. In addition, voice efficacy mediated the positive relationship between general self-efficacy and voice behavior. Besides, the study also found that, under the condition of high job availability employee will conduct more voice behaviors. But job availability negatively moderated the positive relationship between voice efficacy and voice behavior. This outcome corroborated what conservation of Resources Theory purported, namely, resources are particularly salient in the context of loss (Hobfoll, 2002). The results of study 2 also supported the construct validity of what study 1 found, namely, voice efficacy. The implications, limitaions and future orientations of the study were discussed as well.

Psychometric Assessment of the PPDG: Utilizing Cronbach's Alpha as a Means of Reliability

Introduction: Since the development of the 10 item Purdue Pharmacist Directive Guidance (PPDG) Scale several studies of the psychometric properties of the PPDG have been conducted. Although Cronbach's alpha was calculated as a means of internal consistency reliability, a demonstration of the mean centering of the individual items from the instrument were not explored.
 Objectives: This study focused on investigating the mean stabilization of items within the PPDG as they pertain to Cronbach's reliability coefficient calculation.
 Methods: Using item analysis procedures in SPSS, the mean stability of items within the general factor of directive guidance and subscales of instruction and feedback and goal setting were examined for the PPDG.
 Results: Mean stability scores for entire PPDG scale and the subscales of instruction and feedback and goal setting were strong. Also, corrected item-total correlations and Cronbach's alphas following item deletion were good for the overall PPDG scale and the subscales.
 Conclusions: The results provide evidence to enhance understanding of the psychometric stability of the PPDG scale and its subscales.
 
 Type: Original Research

Teaching Efficacy, Interpersonal, Intrapersonal Skills and Teaching Performance in the Tertiary School

This study investigates the relationship between teachers’ interpersonal skills, teachers’ teaching efficacy, and teaching performance as perceived by the teacher, peers and students. The data on teaching efficacy, teachers’ interpersonal-intrapersonal skills were generated through the different scales which were developed and validated by the researcher. The self-report of teachers on their efficacy was utilized to know more the samples’ perception on teaching and their attitude in the teaching- learning process. The data on teaching performance were generated from the peer and student sample through the teaching performance instrument. Descriptive statistics, item analysis procedure, Pearson correlation coefficient, using the SPSSPC-DOS program were basic tools for data evaluation. T h e following results was derived from the data generated: Teachers’ interpersonal skill is directly related to the teaching efficacy scores. Teachers’ intrapersonal skill is significantly related to the teaching efficacy scores. Keywords - Teaching efficacy, interpersonal skills, intrapersonal skills, teaching performance

Development of the knowledge about memory loss and care (KAML-C) test.

The purpose of this study was to develop a test to assess the knowledge of family caregivers concerning memory loss, the early stages of Alzheimer's disease (AD), and related care issues. A total of 31 multiple-choice items were generated in three domains: medical information, caregiving, and legal/financial planning. The test was administered to experts in the field of dementia research and care, medical students, and family caregivers. Item analysis procedures were then used to reduce the test to 15 items. Results suggest that the Knowledge about Memory Loss and Care (KAML-C) test is a reliable and valid test for examining the knowledge level of family caregivers with a relative in the early stages of dementia.

The psychogeriatric and risk behavior assessment scale (PARBAS): a new measure for use with older adults living in the community

In the context of decreased access to inpatient mental health care, the assessment of risk behavior in older adults living in the community is a growing concern. The purpose of the study was to develop a risk behavior screening tool to help community practitioners plan early intervention for frail older adults with severe mental illness (SMI) referred to community health and social service centers. The initial instrument development involved discussion-based work among practitioners and researchers. It was followed by a content validation phase (involving nine experts), preliminary testing of inter-rater reliability (27 subjects, 12 pairs of raters), and item analysis procedures combining previous data. The psychogeriatric and risk behavior assessment scale (PARBAS) includes 34 items organized in 10 sections of risk behavior: (i) self neglect; (ii) non-compliance; (iii) substance abuse; (iv) risk toward oneself and others; (v) aggressive behavior; (vi) emotional distress; (vii) suicidal behavior; (viii) personal security; (ix) risk of victimization by others and (x) financial security. It can provide professionals with essential information about their clients’ risky behaviors, thereby contributing to timely and appropriate preventive interventions. Future research is needed to establish its psychometric properties.

Development of the key behaviors change inventory: A traumatic brain injury behavioral outcome assessment instrument

Kolitz BP, Vanderploeg RD, Curtiss G. Development of the Key Behaviors Change Inventory: a traumatic brain injury behavioral outcome assessment instrument. Arch Phys Med Rehabil 2003;84:277-84. Objective: To describe the development and initial validation of a neurobehavioral outcome measure, the Key Behaviors Change Inventory (KBCI), for individuals with traumatic brain injury (TBI). Design: Scale construction and development, and validity study. Setting: Large state university and postal survey. Participants: Seventy-five volunteer undergraduate students and 25 volunteer collateral informants of individuals with TBI participated in the item-analysis phase. Thirty members of the Brain Injury Association and 20 members of the National Multiple Sclerosis Society rated both an identified patient and an age- and gender-equated control in the validation phase. Interventions: Not applicable. Main Outcome Measures: Content validity was examined through expert panel item sorts. Scale internal consistencies were examined with the Cronbach [alpha ]. Construct validity was examined by comparing scale elevations between controls and 2 neurologic groups. Results: Item-analysis procedures resulted in 8 scales of 8 items each: inattention, impulsivity, unawareness of problems, apathy, interpersonal difficulties, communication problems, somatic difficulties, and emotional adjustment. Internal consistency reliability coefficients ranged from .82 to .91. Multivariate analysis of variance revealed significant (P[le ].001) differences in scale elevations among TBI, multiple sclerosis (MS), and control groups. The TBI and MS groups scored significantly higher than the control group on all scales; a subset of KBCI scales discriminated between the 2 neurologic groups. Conclusion: The KBCI was both sensitive and specific to typical behavioral changes after TBI, thus supporting its usefulness in rehabilitation settings. Cross-validation and development of a normative database are future steps necessary in its development. [copy ] 2003 by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation

Teacher Test Accountability.

Given the high stakes of teacher testing, there is no doubt that every teacher test should meet the industry guidelines set forth in the Standards for Educational and Psychological Testing. Unfortunately, however, there is no public or private business or governmental agency that serves to certify or in any other formal way declare that any teacher test does, in fact, meet the psychometric recommendations stipulated in the Standards. Consequently, there are no legislated penalties for faulty products (tests) nor are there opportunities for test takers simply to raise questions about a test and to have their questions taken seriously by an impartial panel. The purpose of this article is to highlight some of the psychometric results reported by National Evaluation Systems (NES) in their 1999 Massachusetts Educator Certification Test (MECT) Technical Report, and more specifically, to identify those technical characteristics of the MECT that are inconsistent with the Standards. A second purpose of this article is to call for the establishment of a standing test auditing organization with investigation and sanctioning power. The significance of the present analysis is twofold: a) psychometric results for the MECT are similar in nature to psychometric results presented as evidence of test development flaws in an Alabama class-action lawsuit dealing with teacher certification (an NES-designed testing system); and b) there was no impartial enforcement agency to whom complaints about the Alabama tests could be brought, other than the court, nor is there any such agency to whom complaints about the Massachusetts tests can be brought. I begin by reviewing NES's role in Allen v. Alabama State Board of Education, 81-697-N. Next I explain the purpose and interpretation of standard item analysis procedures and statistics. Finally, I present results taken directly from the 1999 MECT Technical Report and compare them to procedures, results, and consequences of procedures followed by NES in Alabama.

Het Toetsen van Diepe Woordkennis in een Vreemde Taal

In this article, we report on the construction of a test of deep word knowledge in French as a second language that can be used at advanced levels of proficiency. By 'deep word knowledge' we mean the knowledge of the semantic network of L2-words; our test focuses on the knowledge of the relations between L2-words and the words semantically related to each of these words; part of this is the knowledge of different meanings of polysemous words. In our test, each item consists of an L2-word, i.e. the item word, accompanied by six other L2-words that have or do not have a semantic relation with the item word. The testee has to mark the semantically related words. These relations may be paradigmatic (i.e. cases of synonymy, antonymy, or hyponymy) or syntagmatic (i.e. collocations, ?art-whole relations, or defining terms). The test was pretested on students of different university levels, i.e. advanced and near native students, and also on several reference groups of real native students. The results of the pretest showed the fairly good discriminatingpower of the test: differences between the mean scores of the groups of advanced, near native and native students proved to be significant. The test even discriminates between different levels of native proficiency (the french equivalents of ??, ?A and highschool students). The reliability (Cronbachs a) was estimated separately for the groups of advanced and near native students and proved satisfactory: .84 and .88 respectively. The data of the pretest administration was used in an item analysis procedure to modify and improve the test and to give it its final form. In the near future we will investigate the relationship between the deep word knowledge our test measures and scores on tests of broad, quantitative word knowledge; furthermore we will investigate the power of these two kinds of vocabulary scores to predict integrated language skills, in particular writing ability.

Predicting quality of life of students who have left special education high school programs.

A subsample (N = 505) was selected from a nationwide, stratified, probability sample of students with disabilities attending high school in 1985. In-school and out-of-school information about these students was obtained in 1987 and 1990 from school records, school personnel, and parents. Conventional item analysis procedures were used to construct a priori quality-of-life composites (social relationships, employment, and independence) from 17 questionnaire variables. These composites were related to 27 geopolitical, family, demographic, cognitive, disability, and school program variables using three multiple regression analyses, which indicated that the predictors accounted for 23.5% of the variance in social relationships; 25.6%, in independence; and 19.5%, in employment. Results suggest that although quality of life is multidimensional, competence appears to underlie many of its facets.

PREDICTING QUALITY OF LIFE OF YOUTHS AFTER THEY LEAVE SPECIAL EDUCATION HIGH SCHOOL PROGRAMS

A subsample of 713 students was selected from a nationwide stratified probability sample of 8,404 students with disabilities attending U.S. high schools in 1985. In-school and out-of-school information about these students was obtained in 1987 from their school records, school personnel, and parents. Conventional item analysis procedures were used to construct three quality-of-life composites—esteem, independence, and support—from 17 questionnaire variables. These composites were related to 28 geo-economic, family, demographic, cognitive, disability, and school program variables using a canonical correlation. The sum of three redundancy coefficients from the canonical analysis indicated that the predictors accounted for 36.9% of the variance in the three quality-of-life composites taken in concert, from which 5.3% was attributed uniquely to school program variables. The three canonical dimensions were labeled “general competence,” “sensory disability,” and “valued support” because of the patterns of quality-of-life variables and predictor variables with which they correlated. These analyses implied that quality of life for individuals with disabilities is difficult to disentangle from competence. However, the data set used for the present study included very few quality-of-life variables that are not related to competence—such as students' satisfaction with their lifestyles or students' freedom to make their own choices—making it difficult to detect these dimensions.

Reliability and validity of the Diabetes Family Behavior Scale (DFBS).

The Diabetes Family Behavior Scale (DFBS) was designed to measure diabetes-specific family support. The purposes of this study were to refine the scale and to assess reliability and criterion validity in terms of relationship to metabolic control. The DFBS was administered to 321 children and adolescents with insulin-dependent diabetes mellitus (IDDM). Blood was drawn for determination of glycosylated hemoglobin (HbA1c). Based on an item-analysis procedure, the DFBS was revised to include 47 items with two subscales, one to reflect guidance-control and one to reflect warmth-caring. Acceptable internal consistency was found for the DFBS total score (.86), and for the guidance-control (.81) and warmth-caring (.79) subscales. There was a statistically significant relationship in the expected direction between DFBS total score and HbA1c (r = -.12, P < .03), and between the guidance-control subscale and HbA1c (r = -.17, P < .002).