Distribution Of Item Difficulty Research Articles

Multimethods study to develop tools for competency-based assessments of implementation research training programmes in low and middle-income countries

IntroductionMethods and validated tools for evaluating the performance of competency-based implementation research (IR) training programmes in low–middle-income countries (LMICs) are lacking. In this study, we developed tools for assessing the...

Validation of the Chinese version of the exercise empowerment scale among college students (EES-C).

The escalating prevalence of physical inactivity among college students represents a significant public health challenge, particularly in light of its correlation with detrimental health outcomes. A growing body of evidence underscores the necessity of adhering to recommended levels of regular physical activity to thwart the onset of chronic diseases. One primary aim of school physical education initiatives is to elevate physical activity levels and bolster student motivation toward engaging in physical exercise. Engagement in sports activities has demonstrated efficacy in augmenting students' motor skills, elevating their self-efficacy, and enhancing cognitive competencies related to physical prowess, while also promoting sustained participation in physical activities. The Exercise Empowerment Scale (EES) has been formulated to quantitatively assess the degree of exercise empowerment. To date, no validated Chinese version of the EES has been reported in the literature. To address this, this present study developed and validated a Chinese version of the EES in a large sample of Chinese college students. A sample of Chinese college students (n = 885) was analyzed using Rasch analysis to examine the validity of the Exercise Empowerment Scale. In addition, physical activity and psychological resilience were used to investigate the potential associations with exercise empowerment. The EES-C was found to be a unidimensional scale, and the distribution of item difficulty was relatively even. The scale had good reliability (individual reliability of 0.87, and item reliability of 0.99). No differential item functioning (DIF) was observed across genders for any of the 13 EES-C items, suggesting equitable and unbiased applicability for both male and female respondents. The five-point scoring method of the EES scale was consistent with the overall distribution of the items and subjects. Exercise empowerment was positively correlated with autonomous physical activity and psychological resilience. The results of the study indicate that the EES-C possesses robust psychometric properties, rendering it suitable for application among Chinese college student populations. The adapted version of the EES-C provides a basis for further exploration of the predictive factors of physical activity in Chinese samples. The generalizability of our findings should be further verified in other populations in the future.

Rater Severity/Leniency and Bias in EFL Students' Composition Using Many-Facet Rasch Measurement (MFRM)

<p>The study aims to investigate the extent to which raters exhibit tendencies towards being overly severe, lenient, or even bias when evaluating students' writing compositions in Indonesia. Data were collected from 15 student essays and four raters with master's degrees in English education. The Many-facet Rasch measurement (MFRM), automatized by Minifac software, a program created for the Many-facet Rasch measurement, was used for data analysis. This was done by meticulously dissecting the assessment process into its distinct components—raters, essay items, and the specific traits or criteria being evaluated in the writing rubric. Each rater's level of severity or leniency, essentially how strict or lenient they are in assigning scores, is scrutinized. Likewise, the potential biases that raters might introduce into the grading process are carefully examined. The findings revealed that, while the raters used the rubric consistently when scoring all test takers, they varied in how lenient or severe they were. Scores of 70 were given more frequently than the other score. Based on the findings, composition raters may differ in how they rate students which potentially leading to student dissatisfaction, particularly when raters adopt severe scoring. The bias in scoring has highlighted that certain raters consistently tend to inaccurately score items, deviating from the established criteria (traits). Furthermore, the study also found that having more than four items/criteria (content, diction, structure, and mechanic) is essential to achieve a more diverse distribution of item difficulty and effectively measure students' writing abilities. These results are valuable for writing departments to improve the oversight of inter-rater reliability and rating consistency. To address this issue, implementing rater training is suggested as the most feasible method to ensure more dependable and consistent evaluations.</p>

Ensuring the “health” of a curricular program evaluation: Alignment and analytic quality of two instruments for use in evaluating the effectiveness of an interprofessional collaboration curriculum

To cultivate competencies in interprofessional collaboration (IPC) for patient-centered, team-based care, a multi-faceted training enhancement initiative was implemented at our academic primary care residency site. Evaluation of the activities from previously collected survey data occurred upon a 2-year review. First, the evaluation team scrutinized the instruments for alignment and appropriateness with planned IPC educational learning and behavior objectives. We found the two instruments were well supported by the literature and with appropriate evidence for validation, but were not well aligned to the objectives of this IPC training initiative, reducing appropriateness of potential inferences of the findings for this context. Second, the team assessed the analytic quality of survey results in item difficulty distribution and item fit to the requirements of a Rasch measurement model. This revealed low person separation due to high overall item agreement. Most residents agreed with most items, so the measures lacked the precision necessary to capture change in residents’ IPC competency. Our instrument review serves as a reminder of the need to gather validity evidence for the use of any existing tool within a new context, and offers a generalizable strategy to evaluate data sources for appropriateness and quality within a specific program.

The Norwegian CDI-III as an assessment tool for lexical and grammatical development in preschoolers.

Parental report instruments are a non-invasive way to assess children's language development and have proved to give both valid and reliable results when used with children under the age of 2;6 (and in some cases up to 3). In this study we examine the newly developed Norwegian edition of a language assessment tool for older preschoolers: MacArthur-Bates Communicative Development Inventory III (CDI-III), investigating whether this parental report tool can be used for assessing the language of monolingual Norwegian-speaking children between 2;6 and 4 years. NCDI-III results for 100 children between 2;6 and 4.0 are presented. All sections were significantly intercorrelated. All sections except Pronunciation showed growth with age. Internal consistency was measured both in terms of Cronbach's alpha and corrected item-scale correlation, and the results are discussed considering features of item difficulty distribution. Methodological considerations are discussed, as well as implications relevant both for possible later revisions and for CDI-III adaptations to new languages.

Rasch Calibration of the International Knee Documentation Committee Subjective Knee Form.

Among numerous knee-related patient-reported outcome measures, the International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) has been used across a wide variety of knee pathologies. However, traditional validation procedures (classical test theory) and existing studies (estimating item parameters) have limitations in establishing the measurement properties of the IKDC-SKF. Rasch analysis reveals a strong validation approach to improve IKDC-SKF clinical interpretation with larger samples. To assess psychometric properties, including differential item functioning, of the IKDC-SKF as a patient-reported measure of knee function. Cross-sectional study. Secondary data. Data were extracted from the cloud-based orthopedic and sports medicine global registry Surgical Outcome System (Arthrex). A total of 1725 individuals who underwent an arthroscopic knee procedure and completed all items on the IKDC-SKF. Rasch analysis including model-data fit, rating scale's function, item-person map (distribution of item difficulty and person ability), and differential item functioning (sex and age groups) was used to evaluate the psychometric properties of the IKDC-SKF. Ten misfit items were found and removed. The 5-point Likert scale of the 9-item IKDC-SKF worked well. Item difficulty ranged from 0.58 to 0.81 logits, and person's knee function had ranged from -5.56 to 4.86 logits, with a wide distribution. The IKDC-SKF was found to function similarly for sex (male vs female) and age. Rasch analysis identified a unidimensional structure retaining 9 of the original IKDC-SKF items; however, a more comprehensive inventory is necessary to assess a wider range of knee function and improve measurement validity.

Classroom assessment that tailor instruction and direct learning: A validation study

We report the validity of a test instrument that assesses the arithmetic ability of primary students by (a) describing the theoretical model of arithmetic ability assessment using Wilson’s (2004) four building blocks of constructing measures and (b) providing empirical evidence for the validation study. The instrument consists of 21 multiple-choice questions that hierarchically evaluate arithmetic intended learning outcomes (ILOs) on arithmetic ability, hierarchically, based on Bloom’s cognitive taxonomy for 138 primary three grade students. The theoretical model describes students’ arithmetic ability on three distinct levels: solid, developing, and basic. At each level, the model describes the characteristics of the tasks that the students can answer correctly. The analysis shows that the difficulty of the items followed the expected order in the theoretical construct map, where the difficulty of each designed item aligned with the cognitive level of the student, the item difficulty distribution aligned with the structure of the person construct map, and word problems required higher cognitive abilities than the calculation problems did. The findings, however, pointed out that more difficult items can be added to better differentiate students with different ability levels, and an item should be revised to enhance the reliability and validity of the research. We conclude that the conceptualizations of such formative assessments provide meaningful information for teachers to support learning and tailoring instruction.

Development and Rasch Analysis of the 18-Item Health Resilience Profile (HRP).

Existing resilience measures have psychometric shortcomings, and there is no current gold-standard resilience measure. Previous work indicates adults enrolled in a health coaching program may benefit from a resilience measure that is tailored and contextualized to this sample. This two-part study aimed to develop and evaluate a resilience instrument focused on health behavior change in adults in a health coaching program. Two studies were conducted to (1) create a resilience instrument (Health Resilience Profile; HRP) specific to adults attempting health behavior change (n = 427; female = 83.8%; age = 44.5 ± 11.9 years) and to (2) optimize the instrument performance using Rasch analysis (n = 493; female = 62.1%; age = 49.8 ± 12.5 years). Study 1 identified two issues: (1) four unacceptable misfit items and (2) inappropriate rating scale functioning. Study 2 evaluated an improved instrument based on the outcome of study 1 resulting in one more misfit item, and unidimensionality was supported. The new four-category rating scale functioned well. The item-person map indicated that item difficulty distribution was well matched to participants' resilience level, and items were free from measurement error. Finally, items did not show differential item functioning across age, sex, alcohol use, and obesity status. The 18-item HRP is optimized for adults in a health coaching program.

Item Analysis of Functional Measure Indicators in the National Health and Nutrition Examination Survey: A Cross-sectional Study.

This study aimed to develop a function, activities, and participation measurement scale using the questions on the National Health and Nutrition Examination Survey (NHANES) by applying item response theory approaches. This was a cross-sectional study. Survey data were collected from 2512 participants within the NHANES. Twenty questions from the NHANES physical functioning section were included. Exploratory and confirmatory factor analyses were conducted to evaluate the factorial structure and model fit of the NHANES instrument. A multidimensional Rasch modeling was used to model the response data after confirming its model fit. To assess validity across subgroups, differential item functioning test was conducted with respect to sex and race using the three-dimensional partial credit model. Item fit indices indicated a good fit (root-mean-square error of approximation = 0.049, standardized root-mean-square residual = 0.052, Tucker-Lewis index = 0.975). Rasch analysis indicated that "stoop/crouch/kneel," "stand for long periods," and "push/pull large objects" were the three most difficult items, whereas "walk between rooms," "use fork/knife," and "leisure at home" were the easiest. The measure demonstrated good internal consistency overall (Cronbach α = 0.90). The NHANES function, activities, and participation measure demonstrates sufficient evidence of reliability, internal consistency, and validity in noninstitutionalized community-dwelling population. Complete the self-assessment activity and evaluation online at http://www.physiatry.org/JournalCME. Upon completion of this article, the reader should be able to (1) Recognize the potential benefits of using validated patient-reported functional outcome measures within the publically available National Health and Nutrition Examination Survey (NHANES) data for rehabilitation research; (2) Describe the process of scale development and validation; (3) Identify item difficulty based on the item difficulty distribution; and (4) Identify the applicability of the measure to different population groups based on differential item functioning. Advanced. The Association of Academic Physiatrists is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.The Association of Academic Physiatrists designates this Journal-based CME activity for a maximum of 1.0 AMA PRA Category 1 Credit(s) ™. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Improving and validating a rating scale to assess student pharmacists’ perception of professional identity

BackgroundThere are limited data on the psychometric properties of instruments evaluating professional identity (PI) among student pharmacists. ObjectivesThis study analyzed the psychometric properties of an instrument to assess student pharmacists’ perception of their PI. MethodsData from a cross-sectional survey were used to establish validity and reliability evidence of student pharmacists’ perception of PI using Rasch analysis. Linear regression was conducted to evaluate the association of students’ characteristics with their perceived PI. Analyses were conducted in WINSTEPS (SWREG, Minnetonka, MN) and SAS (SAS, Cary, NC). ResultsThe analysis included 118 student pharmacists. Primary analysis suggested inadequate scale functionality; thus, some response categories were merged. With modified item anchors and the removal of the most misfitting item, the data fit the model as measured by student and item infit and outfit mean-squared values. The item-person map showed appropriate distribution of item difficulty, identifying medicine adviser and entrepreneur being the easiest and hardest items to agree with, respectively. Students’ abilities initially mismatched with item difficulty and exhibited a ceiling effect, requiring addition of items reflecting students’ higher perceptions of PIs. Although students’ reliability coefficient (RC) and separation index were 0.98 and 1.73, respectively, RC of the items was 0.94. Students who reported receiving formal PI training were 0.3 times more likely to have high perception of PI than those who did not report training. ConclusionThe instrument exhibited fair validity and reliability evidence for assessing the perception of PI among student pharmacists. Additional items to fill identified item gaps are needed to improve the instrument’s content validity.

Adjustment for guessing in a basic statistics test for Indonesian undergraduate psychology students using the rasch model

The purpose of this study comprises (1) calibrating the Basic Statistics Test for Indonesian undergraduate psychology students using the Rasch model, (2) testing the impact of adjustment for guessing on item parameters, person parameters, test reliability, and distribution of item difficulty and person ability, and (3) comparing person scores after adjustment for guessing with original analysis results. Adjustment for guessing was performed using the Rasch model-based method with a tailored analysis. Participants were 488 undergraduate psychology students (221 male, 267 female; mean age = 18.86, SD = 0.49) in Indonesia. The Rasch analysis results show that the assumptions of unidimensionality and local independence of the Basic Statistics Test are met. However, the two items do not fit the Rasch model on the original or tailored analysis. Comparing the original and tailored analyses shows that adjustment for guessing impacts the item parameters, person parameters, reliability, and distribution of item difficulties and person abilities. With the university’s minimum pass-levels (MPL) based on total scores (scores > 60), adjustment for guessing resulted in more respondents passing the MPL. Thus, adjustment for guessing is recommended for making pass-fail decisions based on a predetermined MPL. Implications and suggestions for future research are discussed.

Many-faceted rasch calibration of the foot function index-revised short form

BackgroundThe Foot Function Index Revised Short version (FFI-RS) is a foot- and ankle- patient-reported outcome measure (PROM), developed from the Foot Function Index (FFI). Previous studies, estimating item parameters and multidimensional properties, have limitations properly establishing the measurement properties of the FFI-RS. A multi-faceted Rasch analysis with a larger sample would allow for a more robust validation approach to improve the clinical interpretation of the FFI-RS using a multidimensional perspective. Therefore, the purpose of this study was to assess the psychometric properties of the FFI-RS as a PROM of foot function.MethodA total of 2184 patients with foot pathology who completed the FFI-RS were included in the data. Data were extracted from the cloud-based orthopedic and sports medicine global registry Surgical Outcome System (SOS). The psychometric properties of the FFI-RS were assessed using a many-faceted Rasch analysis that included model-data fit, rating scale function, item-person map (distribution of item difficulty and person ability), and item difficulty of the subscale.ResultsTwo misfit items were discovered and deleted; 32-items from the original FFI-RS were retained. The 4-item Likert scale functioned effectively and item difficulty (-0.58 to 1.48), subscale difficulty (-0.58 to 1.15), and person’s foot function (-6.62 to 6.24) had wide distributions.ConclusionsMany-faceted Rasch analysis revealed the FFI-RS had sound psychometric properties using the many-faceted Ranch analysis and retained 32 of the original items. Clinicians and researchers should consider weaknesses identified with items in the ‘Difficulty” subscale and future work should be conducted to modify or develop items that will more accurately evaluate a wide range of foot function levels.

A Comprehensive Simulation Study of Estimation Methods for the Rasch Model

The Rasch model is one of the most prominent item response models. In this article, different item parameter estimation methods for the Rasch model are systematically compared through a comprehensive simulation study: Different alternatives of joint maximum likelihood (JML) estimation, different alternatives of marginal maximum likelihood (MML) estimation, conditional maximum likelihood (CML) estimation, and several limited information methods (LIM). The type of ability distribution (i.e., nonnormality), the number of items, sample size, and the distribution of item difficulties were systematically varied. Across different simulation conditions, MML methods with flexible distributional specifications can be at least as efficient as CML. Moreover, in many situations (i.e., for long tests), penalized JML and JML with ε adjustment resulted in very efficient estimates and might be considered alternatives to JML implementations currently used in statistical software. Moreover, minimum chi-square (MINCHI) estimation was the best-performing LIM method. These findings demonstrate that JML estimation and LIM can still prove helpful in applied research.

Rasch모형을 적용한 운동지속의도 척도의 적합성 : 골프장 이용객을 대상으로

The purpose of this study is to verify the suitability of the exercise adherence intention scale for golf course user by applying the Rasch model. In order to achieve this, a sample of 327 golf course users was selected by using purposive sampling method from Gyeonggi and Chungcheong among which 27 data were removed and finally 300 data were used. Rasch model was applied to verify the fit of the scale, and Winsteps 3.65.0 program was used for analysis. The exercise adherence intention scale satisfies the one-dimensional assumption, and it was found that all of the response categories such as the 5-point Likert scale, the item fit, and the item difficulty were satisfied. However, it was found that the distribution of the response attribute score and question difficulty was not suitable. 5 Itemn of the exercise abherence intention scale for golf course users were appropriate. And if the item difficulty distribution is satisfied, it could increase the validity of the response results on exercise adherence intended for golf course users.

Fitting a Mixture Rasch Model to Visual Sequential Processing Memory Sub-dimension of ASIS: The Role of Covariates

Advanced Item Response Theory (IRT) practices serve well in understanding the nature of latent variables which have been subject to research in various disciplines. In the current study, 7-12 aged 2536 children’s responses to 20-item Visual Sequential Processing Memory (VSPM) sub-test of Anadolu-Sak Intelligence Scale (ASIS) were analyzed with Mixture Rasch Method (MRM). In the first phase of the study, concomitant (covariate) variables were not used. In the second phase, age and gender were added to the model, and then the two models were compared in terms of fit indices, the number of latent classes and the distribution of item difficulties in the latent classes. The results of the study suggested that there were three latent classes in both models; however, the latter model had greater fit compared to the former model. In addition, the latent classes in both models had similar characteristics, and the distributions of item difficulties in latent classes were also quite similar in both models while they had some differences in some aspects. The sizes of identical latent classes in both models varied between 15% and 30%. The results of the current study are expected to provide a deeper insight to researchers studying measurement theory and/or intelligence measurement.

Rasch 모형을 적용한 의도적 연습 척도의 문항 적합성 : 태권도 선수를 대상으로

Purpose: The purpose of this study is to validate the Deliberate Practice scale of Taekwondo athletes by applying the Rasch model.BR Method: In order to achieve this, a sample of 600 youth Taekwondo players (middle, high, university) was selected by using purposive sampling method from Seoul, Gyeonggi, Incheon, Gangwon, and Daegu area. Data acquired from 29 Taekwondo players were excluded due to inconsistency in their responses. Therefore, the data of 571 people were used for final analysis. For data processing, frequency analysis, confirmatory factor analysis, and Rasch RSM model were performed using amovi 1.0 and jMetrik 4.1.1 programBR Results: First, the real recognition scale of the Deliberate Practice satisfied one-dimensionality. Second, the scale of Deliberate Practice consisting of 5 points Likert was found to be inappropriate. Third, d6 questions of Deliberate Practice were found to be inappropriate, but all the other questions were appropriate. In terms of item difficulty, the d2 items had higher difficulty levels, and the d4 items generally had lower difficulty levels. Fourth, item difficulty through the Item-Person Map, it was found that the distribution of individual ability and the distribution of item difficulty were generally consistent in Deliberate Practice.BR Conclusion: The result of this study would provide basic data to promote the sports moral behavior of youth Taekwondo players.

Gottfries' Cognitive Scale for Staff Proxy Rating of Cognitive Function Among Nursing Home Residents.

For research purposes, there is a need for tools to assess an individual's level of cognitive function. For survey-based investigations in nursing home contexts, proxy ratings allow the assessment also of individuals with severe cognitive impairment. The aim of this study was to describe the feasibility and psychometric properties of Gottfries' cognitive scale when used in a nursing home context for proxy rating of cognitive function. The psychometric properties of Gottfries' cognitive scale were investigated in a sample of 8,492 nursing home residents in Västerbotten County, Sweden, using item response theory and classic scale theory-based approaches. Cognitive function could be scored in 97.1% of the assessed individuals. The scale had a negligible floor effect, it had items with a large spread in difficulties, it appeared linear, and it distributed the assessed individuals equally over the scale. Internal consistency (Cronbach's alpha) was 0.967, and an exploratory factor analysis revealed three factors of the scale - interpreted to represent orientation to time, to place, and to person. Gottfries' cognitive scale is a feasible tool for grading cognitive function among nursing home residents using staff proxy ratings. The scale has excellent psychometric properties with a very high internal consistency, a favorable distribution of item difficulties producing an almost rectangular distribution of scores, and a negligible floor effect. The scale thus can be recommended for use in survey-based investigations in nursing home contexts.

Evaluating Different Equating Setups in the Continuous Item Pool Calibration for Computerized Adaptive Testing.

The increasing digitalization in the field of psychological and educational testing opens up new opportunities to innovate assessments in many respects (e.g., new item formats, flexible test assembly, efficient data handling). In particular, computerized adaptive testing provides the opportunity to make tests more individualized and more efficient. The newly developed continuous calibration strategy (CCS) from Fink et al. (2018) makes it possible to construct computerized adaptive tests in application areas where separate calibration studies are not feasible. Due to the goal of reporting on a common metric across test cycles, the equating is crucial for the CCS. The quality of the equating depends on the common items selected and the scale transformation method applied. Given the novelty of the CCS, the aim of the study was to evaluate different equating setups in the CCS and to derive practical recommendations. The impact of different equating setups on the precision of item parameter estimates and on the quality of the equating was examined in a Monte Carlo simulation, based on a fully crossed design with the factors common item difficulty distribution (bimodal, normal, uniform), scale transformation method (mean/mean, mean/sigma, Haebara, Stocking-Lord), and sample size per test cycle (50, 100, 300). The quality of the equating was operationalized by three criteria (proportion of feasible equatings, proportion of drifted items, and error of transformation constants). The precision of the item parameter estimates increased with increasing sample size per test cycle, but no substantial difference was found with respect to the common item difficulty distribution and the scale transformation method. With regard to the feasibility of the equatings, no differences were found for the different scale transformation methods. However, when using the moment methods (mean/mean, mean/sigma), quite extreme levels of error for the transformation constants A and B occurred. Among the characteristic curve method the performance of the Stocking-Lord method was slightly better than for the Haebara method. Thus, while no clear recommendation can be made with regard to the common item difficulty distribution, the characteristic curve methods turned out to be the most favorable scale transformation methods within the CCS.

A Study on the Development of a Diagnostic Tool for the Assessment Professionality of Specialized Subject Teachers

최근 성취평가와 같은 역량 중심으로 평가의 패러다임이 변화함에 따라 학교 현장에서 교사의 평가 관련 역량 향상에 요구가 높아지고 있다. 이는 평가가 단순히 학생의 학업 성취를 평가하는 것 이상으로 교사의 수업 개선에 따른 학교교육 내실화 등 교육 전반에 영향을 미치기 때문이다. 하지만 현재 전문교과 교사의 평가와 관련된 연구는 상대적으로 적으며, 특히 전문교과 교사의 평가와 관련된 역량을 진단하고, 이를 기반으로 전문가로서의 교사 역량을 향상시키도록 정보를 제공할 수 있는 진단도구 개발과 관련된 연구는 거의 이루어지지지 않았다. 본 연구는 전문교과 교사 역량 중 ‘평가 전문성’에 기반을 두고 4가지 영역(평가 계획, 평가 수행, 평가 결과 활용, 평가 관련 윤리), 20개 문항으로 구성된 진단도구를 개발하였다. 그리고 타당도를 검증하기 위하여 2018년 12월 31일(월)부터 2019년 1월 15일(화)까지 전문교과 교사 75명을 대상으로 설문조사를 실시하고, MINISTEPS와 SPSS 24.0을 활용하여 그 결과를 분석하였다. 첫째, 본 연구에서 개발된 진단도구의 문항에 대한 차원성 검증 결과, 일차원성 가정을 충족하고, 각 문항에 대한 내적합도/외적합도 지수 뿐 아니라 문항 신뢰도가 적합한 것으로 나타나 해당 문항 모두 타당성과 신뢰성을 확보하였음을 확인하였다. 둘째 ‘문항×피험자 분포도’를 통해 문항 곤란도와 응답자 개인의 속성을 파악한 결과, 문항곤란도의 경우 평균을 중심으로 몰려있는 반면, 응답자 개인 속성의 경우 그 분포가 상대적으로 넓게 분포된 것으로 나타나 현재 ‘평가 전문성’ 관련 개인 속성이 매우 높거나 낮은 응답자를 진단할 수 있는 문항이 상대적으로 부족한 것으로 나타났다. 셋째, 5점 평정척도로써의 응답 범주의 적절성을 분석한 결과, 척도 점수에 대한 내/외적합도 지수가 타당하였고 단계조정값 역시 척도 점수와 비례하여 증가한 것으로 나타나 5점 평정척도를 활용한 설계가 적절함을 확인하였다. 또한 응답 확률 곡선에서 각 반응 범주의 곡선 간 겹쳐지는 지점인 경계모수가 뚜렷하게 구분된다는 점에서도 5점 평정 척도를 활용한 설계의 적절함을 확인할 수 있었다. 본 연구에서 개발된 ‘전문교과 교사의 평가 전문성 진단도구’는 특성화고등학교와 산업수요맞춤형 고등학교와 같은 직업계고등학교의 전문교과 교사의 자기계발 기회 제공 및 전문교과 교사의 평가 전문성 향상을 위한 기초 자료로 활용될 것으로 기대된다. 그리고 궁극적으로 직업계고등학교 학생의 학업 성취 뿐 아니라 교실 수업의 질 제고와 직업교육의 목적을 달성하는 데 긍정적인 기여를 할 것으로 기대된다.This study conducted a survey related to the Diagnostic Tool for the Evaluation of Specialized Subject Teachers, composed of four categories and 20 items, targeting total 75 specialized subject teachers. The results of analyzing the survey results by using the MINISTEPS and SPSS 24.O are as follows. First, in the results of verifying the dimensionality of the scale items developed by this study, on top of meeting the unidimensional assumption, all the results of internal/external goodness-of-fit indexes of each item and the reliability of items were also suitable, so that all the relevant items secured the validity and reliability to measure the assessment professionality of specialized subject teachers. Second, in the results of understanding the attributes of individual subjects and item difficulty through the item-subject distribution chart, the distribution of item difficulty was concentrated in the mean. Regarding the attributes of individual subjects, however, the distribution was relatively wide, which means the shortage of items suitable for the subjects who currently have very high or low personal attributes. Thus, it would be necessary to additionally develop the relevant items in the future. Third, in the results of analyzing the validity of response category as 5-point scale, the internal/external goodness-of-fit indexes of scale point were valid, and the phase adjusted value also increased in proportion to the scale point, so that the design using the 5-point scale was verified as suitable. As the threshold parameter which was the overlapped point of curves of each response category was clearly divided in the response probability curve, the validity of 5-point scale was verified.

Farklı Yetenek Dağılımlarında Madde Güçlük Dağılımı, Test Uzunluğu ve Örneklem Büyüklüğünün İncelenmesi

This is a post-hoc simulation study which investigates the effect of different item difficulty distributions, sample sizes, and test lengths on measurement precision while estimating the examinee parameters in right and left-skewed distributions. First of all, the examinee parameters were obtained from 20-item real test results for the right-skewed and left-skewed sample groups of 500, 1000, 2500, 5000, and 10000. In the second phase of the study, four different tests were formed according to the b parameter values: normal, uniform, left skewed and right skewed distributions. A total of 80 conditions were formed within the scope of this research by selecting 20-item and 30-item condition as the test length variable. In determining the measurement precision, the RMSE and AAD values were calculated. The results were evaluated in terms of the item difficulty distributions, sample sizes, and test lengths. As a result, in right-skewed examinee distribution, the highest measurement precision was obtained at the normal b distribution and the lowest measurement precision was obtained at the right skewed b distribution. A higher measurement precision was obtained in the 30-item test, however, it was observed that the change in the sample size didn’t affect the measurement precision significantly in right-skewed examinee distribution. In the left skewed distribution, the highest measurement precision was obtained at the normal b distribution and the lowest measurement precision was obtained at the left-skewed b distribution. Also it was observed that the change in the sample size and test length didn’t affect the measurement precision significantly in the left-skewed distribution.