Difficulty Index Research Articles

Development of standardized mathematics achievement test instruments for measuring senior secondary school students’ learning outcomes

This study developed a Standardized Mathematics Achievement Test (SMAT) instrument for measuring senior secondary school students’ learning outcomes in Dala education directorate, Kano State, Nigeria. The study adopted an instrumentation design. Three research questions were raised to guide the study. The population of the study comprised all 6,462 SS 3 students of the Dala education directorate. The sample size of the study was 132 SS 3 students drawn from the target population in 7 schools with a total number of 248 students using purposive sampling. Fifty multiple-choice items of the SMAT was developed by the researchers. The content validity of the SMAT was determined using a table of specifications. The reliability of the instrument was calculated using Kuder-Richardson formula 20 (KR-20). The result indicated that the content validity as presented in the table of specification of the SMAT items covered the six main sections of the Mathematics scheme. The result indicated that the internal consistency index of the SMAT was found to be a highly reliable instrument with a reliability coefficient of 0.79. The result also indicated that the difficulty index (DifI) of the SMAT items is moderately difficult, and the discriminatory power indicated that the test items distinguish between intelligent and weaker students. The study recommended that mathematics teachers and researchers should always determine the difficulty and discrimination indices of the test items before using such instruments as a tool for measuring students’ learning outcomes.

The effectiveness of a program based on differentiated instruction strategies to develop entrepreneurial skills among students

This study investigates the effectiveness of a program based on Differentiated Instruction (DI) in developing entrepreneurial skills among third-grade female students. A quasi-experimental design with experimental and control groups was used to achieve this objective. The study sample consisted of 60 randomly selected students: 30 students in the experimental group and 30 students in the control group. The study tools included a DI-based program of 15 sessions with three sessions dedicated to each skill (planning, decision-making, team management, innovation and responsibility) for the experimental group. In addition, an entrepreneurial skills test comprising 25 items distributed across five main skills was used. The study results show statistically significant differences at 0.05 between the mean scores of the students in the entrepreneurial skills test in favor of the experimental group after calculating the validity, reliability, difficulty, and discrimination indices. This result indicates the program's effectiveness in developing entrepreneurial skills among third-grade female students. The study recommends the inclusion of entrepreneurial skills in the curricula of the early grades in elementary education and the employment of DI strategies in the educational process. The findings highlight the potential for early introduction of entrepreneurial education and provide actionable recommendations for integrating such skills into elementary curricula.

Development and Validation of a Tool to assess the Retention of Clinical Anatomy Knowledge

Introduction: A thorough knowledge in anatomy is essential for the medical student and thereafter for the practicing doctor. Therefore, retention of knowledge acquired during formative years becomes valuable. Knowledge attrition may occur as students move from preclinical years towards clinical training. A tool to measure the level of retention becomes important to assess the level of attrition and apply remedial measures. Methodology: AKRA tool, a MCQ based tool was developed aligned to the learning objectives assessed and edited by opinions of clinicians. The AKRA tool was validated using expert opinion and statistical methods. For statistical assessment, it was administered to 100, 3rd year medical students and 20 recently passed out doctors and the scores calculated. Reliability was assessed using Cronbach alpha, and item discrimination using Pearson correlation. MCQs were graded into poor, marginal, good and excellent based on item discrimination and difficulty indices. Criterion validity was assessed by comparing the AKRA scores with those of an institutional examination. AKRA score was administered to students of two other institutions and scores were compared. Results: AKRA tool consisted of 42 MCQs. The reliability was high (Cronbach Alph = 0.81). The average score for AKRA was 61.78 (SD=12.3) %. The AKRA scores of the students correlated well with anatomy test scores achieved at the institutional examination (r=0.76). Scores between the other institutions did not show a significant difference. Conclusion: AKRA tool is a validated MCQ-based tool to assess the clinical anatomy knowledge in medical students and doctors.

Impacted lower third molar classification and difficulty index assessment: comparisons among dental students, general practitioners and deep learning model assistance

BackgroundAssessing the difficulty of impacted lower third molar (ILTM) surgical extraction is crucial for predicting postoperative complications and estimating procedure duration. The aim of this study was to evaluate the effectiveness of a convolutional neural network (CNN) in determining the angulation, position, classification and difficulty index (DI) of ILTM. Additionally, we compared these parameters and the time required for interpretation among deep learning (DL) models, sixth-year dental students (DSs), and general dental practitioners (GPs) with and without CNN assistance.Materials and MethodsThe dataset included cropped panoramic radiographs of 1200 ILTMs. The parameters examined were ILTM angulation, class, and position. The radiographs were randomly split into test datasets, while the remaining images were utilized for training and validation. Data augmentation techniques were applied. Another set of radiographs was used to compare the accuracy between human experts and the top-performing CNN. This dataset was also given to DSs and GPs. The participants were instructed to classify the parameters of the ILTMs both with and without the aid of the best-performing CNN model. The results, as well as the Pederson DI and time taken for both groups with and without CNN assistance, were statistically analyzed.ResultsAll the selected CNN models successfully classified ILTM angulation, class, and position. Within the DS and GP groups, the accuracy and kappa scores were significantly greater when CNN assistance was used. Among the groups, performance tests without CNN assistance revealed no significant differences in any category. However, compared with DSs, GPs took significantly less time for the class and total time, a trend that persisted when CNN assistance was used. With the CNN, the GPs achieved significantly higher accuracy and kappa scores for class classification than the DSs did (p = 0.035 and 0.010). Conversely, the DS group, with the CNN, exhibited higher accuracy and kappa scores for position classification than did the GP group (p < 0.001).ConclusionThe CNN can achieve accuracies ranging from 87 to 96% for ILTM classification. With the assistance of the CNN, both DSs and GPs exhibited significantly higher accuracy in ILTM classification. Additionally, compared with DSs with and without CNN assistance, GPs took significantly less time to inspect the class and overall.

Developing an Instrument to Assess Chemical Literacy for Prospective Chemistry Teachers

The lack of instruments designed to assess chemical literacy among prospective chemistry teachers highlights the need for robust tools to support science education. This study develops and validates an instrument to measure chemical literacy, addressing this gap and contributing to improved pedagogical practices. In this study, a chemical literacy instrument for general chemistry was developed, drawing on existing frameworks of chemical literacy. This research employs a development-based approach (RD), encompassing literature review, instrument development, and quality testing. The quality of the instrument was evaluated based on its reliability and validity. The validity was evaluated by expert panel review, difficulty index, discrimination index, and pearson correlation value. In this study, a multiple-choice chemical literacy test instrument was successfully developed using the PISA 2025 scientific literacy framework, resulting in 23 valid items. The PISA framework was chosen for its perceived dynamism compared to other frameworks. The reliability of the developed instrument reached 0.818, indicating that the questions are highly reliable. Future improvements to the instrument are necessary, with a focus on ensuring that all items fully meet the established validity criteria

Factors Associated With the Accuracy of Large Language Models in Basic Medical Science Examinations: Cross-Sectional Study.

Artificial intelligence (AI) has become widely applied across many fields, including medical education. Content validation and its answers are based on training datasets and the optimization of each model. The accuracy of large language model (LLMs) in basic medical examinations and factors related to their accuracy have also been explored. We evaluated factors associated with the accuracy of LLMs (GPT-3.5, GPT-4, Google Bard, and Microsoft Bing) in answering multiple-choice questions from basic medical science examinations. We used questions that were closely aligned with the content and topic distribution of Thailand's Step 1 National Medical Licensing Examination. Variables such as the difficulty index, discrimination index, and question characteristics were collected. These questions were then simultaneously input into ChatGPT (with GPT-3.5 and GPT-4), Microsoft Bing, and Google Bard, and their responses were recorded. The accuracy of these LLMs and the associated factors were analyzed using multivariable logistic regression. This analysis aimed to assess the effect of various factors on model accuracy, with results reported as odds ratios (ORs). The study revealed that GPT-4 was the top-performing model, with an overall accuracy of 89.07% (95% CI 84.76%-92.41%), significantly outperforming the others (P<.001). Microsoft Bing followed with an accuracy of 83.69% (95% CI 78.85%-87.80%), GPT-3.5 at 67.02% (95% CI 61.20%-72.48%), and Google Bard at 63.83% (95% CI 57.92%-69.44%). The multivariable logistic regression analysis showed a correlation between question difficulty and model performance, with GPT-4 demonstrating the strongest association. Interestingly, no significant correlation was found between model accuracy and question length, negative wording, clinical scenarios, or the discrimination index for most models, except for Google Bard, which showed varying correlations. The GPT-4 and Microsoft Bing models demonstrated equal and superior accuracy compared to GPT-3.5 and Google Bard in the domain of basic medical science. The accuracy of these models was significantly influenced by the item's difficulty index, indicating that the LLMs are more accurate when answering easier questions. This suggests that the more accurate models, such as GPT-4 and Bing, can be valuable tools for understanding and learning basic medical science concepts.

The impact of COVID-19 pre-university education on first-grade medical students. A performance study of students of a Department of Histology.

The recent coronavirus disease (COVID-19) forced pre-university professionals to modify the educational system. This work aimed to determine the effects of pandemic situation on students' access to medical studies by comparing the performance of medical students. We evaluated the performance of students enrolled in a subject taught in the first semester of the medical curriculum in two pre-pandemic academic years (PRE), two post-pandemic years (POST), and an intermediate year (INT) using the results of a final multiple-choice exam. Consistency analysis among periods was performed using the Cronbach alpha coefficient (α), the difficulty index with random effects correction (DI), and the point-biserial correlation index (PB). The five exams were homogeneous and had similar α, DI, and PB difficultness. Performance significantly decreased in POST students compared with PRE students, with a correlation between performance and the academic years (PRE-POST). A significant decrease in the percentage of correct answers was detected in the academic years, with POST students showing lower results than PRE students, but not in the percentage of questions answered incorrectly. Significantly higher percentages of unanswered questions were found among POST students. These results confirm the negative impact of the POST pre-university educational system on the performance of students accessing medical school and suggest that POST students could have a higher degree of uncertainty. Specific education programs should be implemented during the first years of the medical curriculum to tailor this effect and increase students' self-confidence and knowledge, which may be associated with confidence.

Osteoporosis Knowledge Assessment Tool – Tamil (OKAT-T) in Postmenopausal Women: A Validity and Reliability Study.

IntroductionOsteoporosis is a silent disease that is more prevalent among postmenopausal women (PMW) due to hormonal transition. Various toolkits, including the Osteoporosis Knowledge Assessment Tool (OKAT), were available for the knowledge assessment. The Osteoporosis-related knowledge is crucial for preventing osteoporosis, but there is no validated, reliable questionnaire in Tamil to measure this knowledge. AimTo validate the Tamil version of the OKAT (OKAT-T) and its psychometric properties to measure the knowledge of osteoporosis for use in postmenopausal women. MethodologyThis cross-sectional study was done in two phases, as translation of OKAT in Tamil and validation of OKAT-T among 430 postmenopausal women in both rural and urban regions. Reliability was examined by the Flesch reading ease (FKRE&G) and McNemar’s test, along with difficulty index, item discrimination and item-total correlations, inter-item consistency (Cronbach’s alpha coefficient). ResultThe overall mean age of the 430 PMW was 59.53 ± 9.83 years. The results showed a good and satisfactory face validity and FKRE&G score (55.9). The Cronbach’s alpha for the overall scale was calculated as 0.85 and considered to be good and satisfactory. As per the difficulty index, 19 items had a 0.3 to 0.7, implying that the questionnaire was easy to understand and satisfactory. Similarly, a test-retest was assessed, which was statistically significant for only six items out of 26, showing that the tool has stable reliability. ConclusionThe management of chronic disorders such as osteoporosis has become more challenging for patients especially among PMW and healthcare professionals due to the increasing life expectancy and urbanization. The use of an Osteoporosis Knowledge Assessment tool that has been tailored to people’s understanding and developed in the local language can raise awareness levels about osteoporosis, encourage the adoption of osteoprotective strategies, and provide guidance on treatment options.

A study of internal structure validity for the American board of surgery in training examination.

As principles of competency-based medical education are implemented into graduate medical education [GME] programs, it is imperative that the assessments employed are reliable and valid. Internal structure is a core component of validity evidence that has been under studied. In this study, we examined elements of the internal structure for the American Board of Surgery In-Training Examination [ABSITE]. This national retrospective cohort study utilized performance data for general surgery residents from 2018 to 2023 to determine aspects of the internal structure for the ABSITE. The item difficulty, item discrimination, internal consistency, and dimensionality of the exam were calculated. 55,986 unique test results were obtained in the study. The ABSITE exhibits strong reliability, (Cronbach's alpha >0.9), and appropriate item difficulty (0.66) and item discrimination index (>0.2) for a formative examination. Results of the exploratory factor analysis reveals that the ABSITE is unidimensional. The ABSITE is a unidimensional examination with strong reliability. Furthermore, the item difficulty and item discrimination levels of the exam are appropriate for a formative test. Future studies utilizing consensus methods could be employed to determine what specific construct is measured by the ABSITE.

Rasch Model Analysis to Develop Assessment Instruments for Student's Science Process Skills on Chemical Bonding Material

The assessment of student's science process skills (SPS) on chemical bonding material in schools does not yet meet the learning outcomes of the Merdeka curriculum, because the instruments used by teachers are more focused on knowledge aspects. The purpose of this research is to develop an assessment instrument for science process skills in the topic of chemical bonding. The type of this research is Research and Development (R&amp;D) using the Rasch model modified by Liu. This research produced 18 questions tailored to the number of learning objectives in the chemical bonding material and the indicators of scientific process skills in the Merdeka curriculum, which consist of 1) observing, 2) questioning and predicting, 3) planning and conducting investigations. 4) processing, analyzing data and information, 5) evaluating and reflecting, and 6) communicating results. The logical validity test was conducted by 5 validators consisting of 3 chemistry lecturers from FMIPA UNP and 2 chemistry teachers from State Senior High School (SMAN) 1 Banuhampu. The raw validation data were analyzed using Minifaced software, which showed that the data were valid according to the model. The raw trial data on 60 students from SMAN 1 Banuhampu were analyzed using Ministep, showing that all questions met the criteria of being valid, reliable, and having good difficulty and discrimination indices according to Rasch. These results indicate that the developed instrument is effective in measuring students' SPS on chemical bonding material and can be used to support learning based on a scientific approach.

THE DEVELOPMENT OF FORCE &amp; MOTION ACHIEVEMENT TEST (FMAT) FOR FORM TWO STUDENTS

Force and Motion (FM) is one of the most challenging science concepts for students. However, the instruments for assessing students’ understanding about this concept are limited. As such, this study focused on the development processes of an achievement test, specifically in this topic. Force and Motion Achievement Test (FMAT) was developed based on the Standard Curriculum of Secondary Schools (KSSM) for Science subject. It consists of 25 items, namely 22 objective items (Section A) and 3 subjective items (Section B). The FMAT was validated by 3 experts with more than 10 years experience. Consequently, 3 items were amended. A pilot study was conducted on 40 form two students, then followed by a reliability procedure through Internal Consistency with KR-20 for Section A and Split-Half method for Section B. The results from reliability analysis obtained an acceptable alpha coefficient value for both sections, which are 0.757 for Section A and 0.732 for Section B. In terms of difficulty index (p), there were 4 difficult items, 23 moderately difficult items, and 9 easy items. This findings indicate that the FMAT is valid and reliable to be utilised in any study involving the achievement test of Force and Motion topic. Future researchers are encouraged to perform item discrimination and distractor analysis to evaluate the items’ ability in distinguishing among high and low achiever students, and how well the incorrect options divert students away from the correct answer. The implication of this study lies in providing a scientifically validated and reliable instrument for measuring students' understanding of Force and Motion topic, that help teachers identify learning gaps and enhance teaching strategies, ultimately improving the quality of science education.

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2023, Korea: a psychometric study.

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2023, Korea: a psychometric study.

Methodological proposal for analyzing critical thinking test scores as a tool for development in higher education

The development of critical thinking (CT) enhances academic and professional opportunities. A review of literature reveals the use of fragmented analysis techniques, such as descriptive and correlational methods, among others, which hinder a deeper understanding of CT levels. This research aims to develop a methodology for analyzing Critical Thinking test scores, integrating five phases: exploratory, item analysis, scoring, gap analysis, and correlational. Using a quantitative approach, CT skills were analyzed with the Halpern Critical Thinking Assessment, which includes both open- and closed-ended questions to measure five skills: Verbal Reasoning (VR), Argument Analysis (AA), Hypothesis Testing (HT), Probability Use (PU), and Problem Solving (PS). The sample consisted of 214 students aged 18 and older. The item analysis phase categorized the items into quadrants: satisfactory, for review, or for elimination, based on difficulty and discrimination indices. The gap analysis revealed that Verbal Reasoning and open-ended formats were less satisfactory. The correlational phase, using heat maps, showed a stronger association between Verbal Reasoning and Probability Use. The methodological contributions include a variety of strategies that provide recommended procedures for analyzing tests or questionnaires in general. In today’s digital age, the development of critical thinking is not only a desirable skill but an essential necessity for the higher education system.

Analysing inflammatory responses after mandibular third molar extraction: a comparison of suture-less and multiple suture techniques.

This study aims to compare the outcomes of suture-less and multiple suture closure methods on postoperative pain, swelling, and trismus, and associated complications. This prospective, randomized clinical trial was conducted at the Department of Oral & Maxillofacial Surgery, Goa Dental College & Hospital. Inclusion criteria encompassed subjects aged between 17 and 55 years, classified as American Society of Anaesthesiologists (ASA) I, with asymptomatic impacted third molars falling within the moderately difficult impaction range (Pederson's difficulty index: 5-7). Study excluded individuals classified as ASA II, III, or IV, those with known or suspected allergies to the anaesthetic solution, immunocompromised patients, and individuals taking medications that could affect healing. Pregnant or lactating females and those with a history of bleeding disorders were also excluded. Wound closure methods were compared: Group A underwent primary closure with silk sutures, while Group B had suture-less closure. Subjects were allocated to the specific groups using a simple randomization method. This involved using a lottery-based random sequence to assign each participant to either Group A or Group B. Pain intensity, facial swelling, and trismus were key outcomes. Secondary outcomes included lingual nerve sensation and postoperative complications. Demographics factors, surgical details, radiographic and perioperative data, and physiological parameters were considered. Non-parametric tests and parametric test (repeated measure ANOVA) were employed. Statistical significance was set at P < 0.05. Among 101 participants, both closure techniques exhibited similar outcomes in pain, swelling, and lingual nerve function. However, suture-less closure resulted in significantly less trismus and fewer cases of delayed wound healing. Suture-less method after surgical removal of third molar may reduce trismus compared to multiple sutures. While pain, swelling, and lingual nerve function management were comparable.

Development and validation of a deep learning algorithm for the classification of the level of surgical difficulty in impacted mandibular third molar surgery

The aim of this study was to develop and validate a convolutional neural network (CNN) algorithm for the detection of impacted mandibular third molars in panoramic radiographs and the classification of the surgical extraction difficulty level. A dataset of 1730 panoramic radiographs was collected; 1300 images were allocated to training and 430 to testing. The performance of the model was evaluated using the confusion matrix for multiclass classification, and the actual scores were compared to those of two human experts. The area under the precision–recall curve of the YOLOv5 model ranged from 72% to 89% for the variables in the surgical difficulty index (level of occlusion, available retromolar space, impaction angle, number of roots, root curvature, and relationship to the second molar). The area under the receiver operating characteristic curve showed promising results of the YOLOv5 model for classifying third molars into three surgical difficulty levels (micro-average AUC 87%). Furthermore, the algorithm scores demonstrated good agreement with the human experts. In conclusion, the YOLOv5 model has the potential to accurately detect and classify the position of mandibular third molars, with high performance for every criterion in radiographic images. The proposed model could serve as an aid in improving clinician performance and could be integrated into a screening system.

Comparison of the effect of dexamethasone, ketoprofen and cold compress on postoperative quality of life following impacted lower third molar surgery: a randomized clinical trial.

To evaluate and compare the effect of dexamethasone, ketoprofen and cold compress on the quality of life (QoL) following surgical removal of impacted lower third molars (ILTMs). Eligible patients requiring ILTM extraction with a modified Pederson difficulty index score of 5-6 were recruited. The patients were randomly allocated into Groups A, B and C. Groups A and C received 100mg of ketoprofen and 8mg of dexamethasone per-oral respectively, preoperatively. Subjects in group B applied a pre-standardized ice pack over the angle of the mandible for 6h postoperatively. The QoL questionnaire was administered on postoperative days 1, 2 and 7. In total, seventy-eight subjects completed the study: 46 (59%) were male and had a mean age of 27.8 ± 4.9 years. The groups were similar sociodemographically. The overall QoL and appearance domain score were significantly better in patients on oral dexamethasone on postoperative day 1 than in the other groups. Oral dexamethasone demonstrates better improvement in postoperative QoL and appearance on day 1 following ILTM surgery compared to ice packs and ketoprofen. Although ice packs are readily available, can be used repeatedly and are a low-cost option, more research is necessary to determine their optimum therapeutic use in outpatient settings. Oral dexamethasone is superior to ice pack compress and ketoprofen in improving the postoperative QoL in ILTM surgery. PACTR202005593102009 at Pan African Clinical Trial Registry.

Analysis of practice math problems on statistics material for 8th grade students

To improve the quality of learning, learning evaluation has an important role. One of the learning evaluations that is often carried out is daily assessments to see the achievement of particular material. The aim of this research is to describe the quality of mathematics questions in statistics material for junior high school students. The method used in this research is quantitative descriptive research assisted by the AnBuSo program and R studio software. The data collection technique used documentation techniques, with data sources in the form of mathematics questions on statistics material. Question items analysis is applied to prove validity, estimate reliability, level of difficulty, discrimination, as well as analyzing the factors that cause the question items to be difficult and have poor discrimination. The respondents in this research were 200 students taken from three junior high schools in DIY. Meanwhile, qualitative data analysis is used to explain the difficulty index and discrimination of the test items. The results of this research show that the Aiken value is valid, and the reliability estimates are quite good. The results of analysis based on the CTT and IRT approaches obtained a good mean index of difficulty and discrimination. These findings imply that the validated and reliable question items can be used as a reference for developing higher-quality assessments to improve student learning outcomes.

Development of Persian Reading Comprehension Test and determination of its psychometric properties

This study aimed to develop a Persian Reading Comprehension Test (PRCT) and establish its psychometric properties in Persian-speaking students from fourth to sixth grades. 36 texts and 150 questions were created for the PRCT. The texts and questions were reviewed by 11 experts to ensure content validity. Then, the remaining texts and questions were tested on 5 students in Tehran schools to assess face validity. After this, the test’s difficulty and discrimination indexes were determined on 30 students. Finally, 12 texts and 40 questions were selected for the test. PRCT was conducted on 295 students with normal reading skills and 16 students with suspected reading disorders (SRD) to evaluate psychometric properties. The results indicated a significant difference in scores of the PRCT across all three educational levels. There was a significant difference between students with normal reading skills and those with SRD (U = 61.5, p < .001, r = .373). The correlation between the total score of the PRCT and the text comprehension subtest of the NEMA was calculated (r = .424, p = .039). The test-retest reliability and Cronbach’s alpha coefficient of the PRCT were 0.888 and 0.828, respectively. It appears that the PRCT is a standard test with suitable psychometric properties for evaluating reading comprehension.

Analysis of biology midterm exam items using a comparison of the classical theory test and the Rasch model

In biology learning, test instruments are essential for assessing students' understanding of complex concepts. A test instrument is a crucial factor in learning evaluation; however, its implementation remains minimal. This descriptive quantitative study aims to analyze the quality of test items using the classical approach in terms of validity, reliability, difficulty index, discrimination power, distractor effectiveness, and the Rasch model analysis. The data consists of 30 multiple-choice questions from a biology midterm exam administered to 40 students. Classical test data analysis uses Microsoft Excel, while Rasch model analysis uses Winsteps software. The validity test results from both approaches show 14 valid questions and 16 invalid ones. The reliability scores are 0.619 (adequate) for the classical approach's Cronbach's Alpha, 0.85 (good) for the Rasch model, and 0.65 (weak) for personal reliability. The classical test theory and the Rasch model categorize item difficulty into four levels. The classical approach produces five categories for item discrimination, while the Rasch model identifies three groups based on the item separation index (H=3.45) and two groups based on respondent ability (H=1.96). Distractor effectiveness shows 93.3% functional distractors in the classical test and 80% in the Rasch model. The Rasch model offers greater precision in measuring student ability and detecting bias. Both models should be integrated for comprehensive item analysis. Future tests should focus on improving invalid items and the quality of distractors.

Development and validation of situational judgement test for assessment of behavioural competencies required for effective medical practice in Nigeria

BackgroundThe personal qualities of health workers determine the way health services are provided to clients. Some key personal qualities (also called behavioural competencies) of physicians that contribute to quality healthcare delivery include ethical responsibility, empathy, patient-centeredness, diligence, good judgment, respectful, teamwork, team leadership/conflict management, ability to take correction and tolerance. In this study, we developed and validated clinical scenarios (dilemmas) for assessing priority behavioural competencies for medical practice in Nigeria.MethodsDrawing on prioritized competencies generated via a scoping review and nominal group technique (NGT) exercises in a previous study in the study series, Faculty members from the University of Nigeria Teaching Hospital were consulted to develop or adapt clinical scenarios that could be used to assess these competencies in a physician. The clinical scenarios and options were framed as situational judgement tests (SJTs) and these tests were administered to a random sample of 192 undergraduate and 111 postgraduate medical doctors in a tertiary hospital in Enugu State. Using Kane’s validity argument framework, we assessed scoring and generalization inferences of situational judgment tests (SJT) based on the developed scenarios.ResultScoring inference – difficulty and discrimination index – shows that most of the SJT items are good test items and can differentiate between high and low performers. The corrected point biserial correlations show positive correlation for most of the items. Generalization inference shows the items represent the domains of interest and are internally consistent. However, few items that show poor difficulty and discrimination index were subjected to re-evaluation and possible elimination.ConclusionThis study has produced a set of valid clinical scenarios that can be used to evaluate specific behavioural competencies among trainee medical doctors. It demonstrates that SJTs can be used to assess behavioural competencies for medical practice. However, further research is needed to establish the applicability of SJT beyond the immediate context, such as the medical school, in which it is developed.