Reliability Of Checklist Research Articles

Development, validation, and reliability testing of the standard treatment workflow checklist

Background: Standard Treatment Workflows (STWs) are systematically developed documents designed to enhance clinical care and patient outcomes. Validating these workflows is essential for their adoption across diverse clinical settings. This study aimed to develop and validate a checklist for evaluating STWs. Methods: The study was conducted in three phases. Initially, items were generated through a comprehensive literature review and expert consultations, followed by pre-testing (N=50). In the second phase, content validation (N=43) was performed using the Content Validity Index (CVI) and Content Validity Ratio (CVR) to assess the relevancy and essentiality of each item. Item-level (I-CVI) and scale-level (S-CVI) indexes were calculated. Items were accepted, modified, or rejected based on these scores. In the third phase, pilot testing (N=30) assessed the checklist's reliability and calculated Cronbach's alpha. Results: The final checklist comprised 33 close-ended and one open-ended items. Each item was deemed relevant and essential, with I-CVIs ranging from 0.96 to 1.00 and S-CVI/UA and S-CVI/Ave values of 0.83 and 0.99, respectively. CVR values ranged from 0.85 to 1.00, with an average CVR of 0.97. Reliability testing yielded Cronbach’s alpha values of 0.863 and 0.961 for the two parts of the checklist. Conclusions: The developed checklist is valid and reliable, with satisfactory CVI, CVR, and Cronbach’s alpha values. It can be used as a standard tool for assessing STW validation and effectively capturing essential aspects of STW evaluation.

Evaluation of the reliability and usability of CARE-Radiology: A descriptive-analytic study.

The study aimed to evaluate the reliability and usability of the CARE-Radiology checklist in assessing radiological case reports and provide a basis for its broader adoption and optimization. Ten randomly selected radiological case reports published in scientific journals in 2020 were evaluated using the CARE-Radiology checklist. Twenty-six experts from 10 countries were invited to independently assess all ten reports. The reliability of the checklist was measured using Fleiss' Kappa, and Cronbach's alpha coefficient. Usability was evaluated by recording the time taken to complete the assessments and requesting the evaluators to rate each item on a Likert scale for its easiness of use. The median time for evaluating one radiological case report was 15 min. The overall agreement among evaluators showed moderate reliability with a Kappa value of 0.47 and a Cronbach's alpha of 0.51. The mean compliance rate for the items of CARE-Radiology was 61.8%, with some items exceeding 90% compliance. Items related to abstracts and keywords had the lowest compliance rates. The evaluators found most items easy to understand, with a few exceptions. The CARE-Radiology checklist is relatively easy for researchers to use and understand. Continuous feedback is necessary for future revisions and updates, to enhance the effectiveness of the checklist, and to improve user experience.

Development and validation of an ovine cesarean surgery model and rubric

Clinical skills are learned through deliberate practice with specific feedback. However, it is not possible to provide such experience to each student in food animal reproductive procedures (e.g., cesarean surgery). Aims of this study were to create an ovine cesarean surgery model to facilitate skill acquisition, to create a scoring rubric to assess performance, and to gather validation evidence for the model and rubric. A commercially available ovine manikin was modified using poured silicone to create suturable skin, muscle, and uterus containing a commercially available fetal lamb manikin that could be manipulated and delivered. A scoring rubric consisting of a 16-item checklist and 5 global rating scores (GRS) was created. Model was tested by veterinarians (n = 14, experts) and veterinary students (n = 16, novices) in a validation study that evaluated evidence in content, internal structure, and relationship with other variables (level of training). Most experts (93%) felt that the model would be helpful to teach procedural steps; also experts (100%) agreed that the model would improve students’ skill in performing cesarean surgery. Novices scored lower than experts on the checklist and total GRS, and novices’ surgical times were longer than experts’ times. Content evidence and relationship with other variables evidence supported validation of the model and rubric. Checklist reliability (alpha = 0.58) could be improved by increasing the number of items and further refining them.

Developing evidenced-based quality assessment checklist for real practice in primary health care using standardized patients: a systematic review.

The aim of this review was to explore the quality assessment checklists development methods in previous researches using standardized patients (SPs), as well as to propose an evidence-based checklist development procedure for quality assessment of common conditions in primary health care (PHC) settings. We conducted a systematic review of studies that described checklist development method and extracted the methodology in terms of the developer, the basis and processes. Based on that, we formulated the development procedure according to the recommendations of the WHO Handbook for Guideline Development. We identified a total of 13 articles, and proposed the following five key steps: (I) forming a multidisciplinary team; (II) selecting and evaluating relevant references; (III) extracting medical information and forming the basic items; (IV) clinical expert consensus on the items; and (V) pre-testing the item pool and determining final items. SP has been proven to be an effective method to assess performance in practice. There are still some deficiencies in the developing of case-specific checklists using SPs. To ensure the validity and reliability of checklists, the development processes need to be more standardized and procedural.

Designing a Minimum Data Set of Triage Section in the Hospital Information System

Background: The triage process prioritizes emergency care patients as a first step, and distributes them to different levels of triage based on the severity of clinical conditions. Since this method depends on the memory of the triage performer, it may be affected by lack of time, complexity of triage, and bias, causing nurse confusion and ultimately affecting patient safety. One of the interventions that can improve the efficiency and quality of patientcare in this unit is the use of electronic medical records, so the present study aimed to design a minimum data set to collect data in a uniform and standard way in the triage unit. Methods: This is a descriptive-analytic study that was performed in hospitals affiliated to Qazvin University of Medical Sciences that had emergency medicine specialty in 2018. The study population consisted of 15 emergency medicine specialty and triage nursepractitioners who were selected by convenience sampling. To collect data from was used a researcher-made questionnaire designed that Validity and reliability of checklists were assessed by emergency medicine specialty and triage nurse in pre-hospital emergency with Cronbach's alpha of 0.89. The results were analyzed using SPSS 21 and statistical indices. Results: The survey results indicate that 6 data elements should be considered asmandatory components of MDS for the triage department. These elements include basic triage information, timing, evacuation, accidents, life-threatening conditions and counseling. Conclusion: The findings of the study indicated that minimal data design can lead to standardization and effective management of data by providing comprehensive data elements for triage, thus improving the quality of care, speed of emergency services andreducing costs. And most importantly increase patient satisfaction. It will also allow comparisons of information at different levels of effective decision-making and policy-making and prevention of future injuries.

The Managing Emergencies in Paediatric Anaesthesia global rating scale is a reliable tool for simulation‐based assessment in pediatric anesthesia crisis management

The use of simulation-based assessments for high-stakes physician examinations remains controversial. The Managing Emergencies in Paediatric Anaesthesia course uses simulation to teach evidence-based management of anesthesia crises to trainee anesthetists in the United Kingdom (UK) and Canada. In this study, we investigated the feasibility and reliability of custom-designed scenario-specific performance checklists and a global rating scale (GRS) assessing readiness for independent practice. After research ethics board approval, subjects were videoed managing simulated pediatric anesthesia crises in a single Canadian teaching hospital. Each subject was randomized to two of six different scenarios. All 60 scenarios were subsequently rated by four blinded raters (two in the UK, two in Canada) using the checklists and GRS. The actual and predicted reliability of the tools was calculated for different numbers of raters using the intraclass correlation coefficient (ICC) and the Spearman-Brown prophecy formula. Average measures ICCs ranged from 'substantial' to 'near perfect' (P≤0.001). The reliability of the checklists and the GRS was similar. Single measures ICCs showed more variability than average measures ICC. At least two raters would be required to achieve acceptable reliability. We have established the reliability of a GRS to assess the management of simulated crisis scenarios in pediatric anesthesia, and this tool is feasible within the setting of a research study. The global rating scale allows raters to make a judgement regarding a participant's readiness for independent practice. These tools may be used in the future research examining simulation-based assessment.

Feasibility of the Use of a Reliable and Valid Central Venous Catheter Blood Draw Bundle Checklist

The primary objective of this study was to test the feasibility of creating a central venous catheter blood draw bundle checklist to ensure adherence to the evidence-based blood draw procedure. This study included establishing checklist reliability and validity and periodic observations in 2 inpatient pediatric oncology units. The findings provided support for the reliability and validity of this checklist based on content validity, test-retest reliability, interrater agreement, and internal consistency and reinforced the need for periodic observations to ensure consistency in proper central venous catheter blood draw procedures.

Content validity and inter-rater reliability of a checklist to assess the ergonomic practice of computer professionals

Objectives: The objective of this study is to establish the Content validity and inter-rater reliability of an instrument (Checklist) to assess ergonomic practice of Computer professionals using a rigorous Judgment-quantification process and Intra-class correlation coefficient respectively. Methods and Measures: The Draft Checklist composed of 43 items related to Working Postures, Seating, Keyboard/Mouse, Monitor, Table and Accessories and finally Rest breaks and Exercises. A panel of 9 experts validated the Draft Ergonomic Practice Checklist. The Content Validity Index (CVI) was calculated after receiving all correspondence regarding content validity for each item of the checklist. Finally, a Focus group was held to evaluate the instrument for overall comprehensiveness. A total of 20 computer professionals both males and females of mean age of 33.8 years participated in the inter-rater reliability study. The raters of this study were 2 Physiotherapy faculties with mean clinical and teaching experience of 7.5 years. Both the raters assessed the ergonomic practice of the computer professionals using the checklist. Observations were recorded independently and not discussed among the raters. Analysis: The Content Validity Index (CVI) was calculated by tallying the results of the experts based on the degree to which the experts agree on the relevance and clarity of the items. Intraclass correlation coefficient was used to assess the inter-rater reliability of the ergonomic practice checklist based on the observations recorded independently by the raters. Results: Results from the panel of experts yielded a 0.98 overall Content validity index. Inter-rater reliability analysis revealed that r value was above 0.87 for all the sections of the checklist thus being in the excellent range. Conclusion: The process used to determine Content validity and the inter-rater reliability proved to offer structure and consistency to the instrument development. High CVI scores were generated for the items judged relevant to the content domain as well as for the overall instrument. The checklist demonstrated an excellent reliability. The results support the Content validity and the inter-rater reliability of this Checklist as a tool to assess the Ergonomic Practice of Computer Professionals.

Using Item Analysis to Assess Objectively the Quality of the Calgary-Cambridge OSCE Checklist

Background: The purpose of this study was to investigate the use of item analysis to assess objectively the quality of items on the Calgary-Cambridge Communications OSCE checklist.Methods: A total of 150 first year medical students were provided with extensive teaching on the use of the Calgary-Cambridge Guidelines for interviewing patients and participated in a final year end 20 minute communication OSCE station. Grouped into either the upper half (50%) or lower half (50%) communication skills performance groups, discrimination, difficulty and point biserial values were calculated for each checklist item.Results: The mean score on the 33 item communication checklist was 24.09 (SD = 4.46) and the internal reliability coefficient was ? = 0.77. Although most of the items were found to have moderate (k = 12, 36%) or excellent (k = 10, 30%) discrimination values, there were 6 (18%) identified as ‘fair’ and 3 (9%) as ‘poor’. A post-examination review focused on item analysis findings resulted in an increase in checklist reliability (? = 0.80).Conclusions: Item analysis has been used with MCQ exams extensively. In this study, it was also found to be an objective and practical approach to use in evaluating the quality of a standardized OSCE checklist.

Validity and reliability of a calcium checklist in early elementary-school children

To create, validate and assess the reliability of a checklist to measure calcium intake in children. Calcium intakes from a checklist and parent-assisted 24-h dietary recall were compared. Checklist reliability was assessed separately. After-school programmes in the United States. Forty-two children (18 males, 24 females, age = 8.0 +/- 0.9 years) participated in the validation analysis and 49 children (28 males, 21 females, age = 7.5 +/- 0.9 years) in the reliability analysis. No differences in mean calcium intakes were found by method or gender. The checklist correlated well with recall among girls (r = 0.65, P = 0.01) but not boys (r = -0.33, P = 0.19). Agreement over time was above 80% for most foods. The calcium checklist is useful for assessing calcium intake among groups of 6-10-year-old children in settings that preclude parental assistance. More research is needed to improve accuracy among boys.

The validity and reliability of the World Health Organization Mental Disorders Checklist for use in a telehealth clinic in Hong Kong

This research aimed to test the validity and reliability of the 'World Health Organization Mental Disorders Checklist' for use in a telehealth clinic in Hong Kong. The Checklist adopted four subscales: (i) depression; (ii) anxiety; (iii) alcohol use disorders; and (vi) functioning and disablement, and was translated from English into Chinese. It was validated by a panel of five experts to confirm its content validity (content validity index = 0.98) and cultural appropriateness in Hong Kong. The reliability of the checklist was supported by the findings of a test-retest procedure (Pearson correlation = 0.66-0.88, P < 0.01), internal consistency reliability (Cronbach's alpha = 0.54-0.83), and interrater reliability (Kendall's coefficient of concordance = 0.58-1.00, P < 0.01) involving a sample of 197 subjects from one telehealth clinic in Hong Kong.

Analytic global OSCE ratings are sensitive to level of training.

There are several reasons for using global ratings in addition to checklists for scoring objective structured clinical examination (OSCE) stations. However, there has been little evidence collected regarding the validity of these scales. This study assessed the construct validity of an analytic global rating with 4 component subscales: empathy, coherence, verbal and non-verbal expression. A total of 19 Year 3 and 38 Year 4 clinical clerks were scored on content checklists and these global ratings during a 10-station OSCE. T-tests were used to assess differences between groups for overall checklist and global scores, and for each of the 4 subscales. The mean global rating was significantly higher for senior clerks (75.5% versus 71.3%, t55 = 2.12, P < 0.05) and there were significant differences by level of training for the coherence (t55 = 3.33, P < 0.01) and verbal communication (t55 = 2.33, P < 0.05) subscales. Interstation reliability was 0.70 for the global rating and ranged from 0.58 to 0.65 for the subscales. Checklist reliability was 0.54. In this study, a summated analytic global rating demonstrated construct validity, as did 2 of the 4 scales measuring specific traits. In addition, the analytic global rating showed substantially higher internal consistency than did the checklists, a finding consistent with that seen in previous studies cited in the literature. Global ratings are an important element of OSCE measurement and can have good psychometric properties. However, OSCE researchers should clearly describe the type of global ratings they use. Further research is needed to define the most effective global rating scales.