Additional Reliability Testing Research Articles

Validity of the Rezum Simulator for Trainees

Background: Simulations have revolutionized surgical training and are an invaluable adjunct tool for augmenting the proficiency of surgeon and patient safety. Before being included in a practical assessment process, simulators need to be using various validity inference frameworks. Objective: We examine the construct validity and reliability of the Rezum simulator. Methods: Seventeen candidates of different professions voluntarily participated in the Rezum simulation workshop. The simulator provides a variety of variable metrics and challenges. Each candidate performed three cases of different difficulty levels with three trials of each case. Validity was measured statistically through a one-way analysis of variance (ANOVA) test, and a p-value of < 0.05 was considered significant. Additional reliability tests were provided, including intraclass correlation coefficients, a Cronbach test (0.7 is considered acceptable), and standard error of measurement. Results: The ANOVA of total scores among candidates was significant (p = 0.029). Senior registrars and consultants had the highest total scores. Procedure times did not differ significantly among candidates (p = 0.169). The reliability test for the total score was 0.899 (0.831–0.942), with a standard error value of 2.75, a standard deviation of 8.67, and a Cronbach alpha value of 0.915. Conclusion: We confer the primer evidence of Rezum simulation as a valid, reliable simulator of most of its metrics.

Validity of the Rezum Simulator for Trainees.

Simulations have revolutionized surgical training and are an invaluable adjunct tool for augmenting the proficiency of surgeon and patient safety. Before being included in a practical assessment process, simulators need to be using various validity inference frameworks. We examine the construct validity and reliability of the Rezum simulator. Seventeen candidates of different professions voluntarily participated in the Rezum simulation workshop. The simulator provides a variety of variable metrics and challenges. Each candidate performed three cases of different difficulty levels with three trials of each case. Validity was measured statistically through a one-way analysis of variance (ANOVA) test, and a p-value of < 0.05 was considered significant. Additional reliability tests were provided, including intraclass correlation coefficients, a Cronbach test (0.7 is considered acceptable), and standard error of measurement. The ANOVA of total scores among candidates was significant (p = 0.029). Senior registrars and consultants had the highest total scores. Procedure times did not differ significantly among candidates (p = 0.169). The reliability test for the total score was 0.899 (0.831-0.942), with a standard error value of 2.75, a standard deviation of 8.67, and a Cronbach alpha value of 0.915. We confer the primer evidence of Rezum simulation as a valid, reliable simulator of most of its metrics.

55 Development of a Novel Taxonomy for Adverse Events in the Emergency Department: Critical Review and Testing

To 1) perform a critical review of existing taxonomies that describe harm; 2) develop a usable event taxonomy for the emergency department; 3) perform a pilot-test of this taxonomy. Safety science argues that improvement efforts should focus on preventing and mitigating harm rather than on error, which is commonplace but infrequently leads to adverse events (AEs). Describing and categorizing AEs is important for quality and safety assessment and identifying areas for improvement. However, most taxonomies fail to describe harm experienced by patients (eg, hypoxia, hemorrhage), focusing instead on errors or using categorizations that are too broad to be useful (eg, “medication error”). A patient-centered, ED-specific common language and framework for describing AEs and near misses is needed to advance quality and safety in the acute care setting. We followed the PRISMA guidelines for critical reviews. Our search strategy included taxonomies from major patient safety organizations and ontology/taxonomy databases including: Linked Open Vocabularies, Basel Register of Thesauri, Ontologies & Classifications, Taxonomy Warehouse, BioPortal and SCOPUS 1823, using the concepts “patient safety,” “adverse event,” “adverse reaction,” and “taxonomy,” followed by a title and abstract review of the 50 most-cited manuscripts. Taxonomy Development and Testing: The taxonomy was empirically developed on ∼600 AEs identified in a companion project that used the NCC MERP Index to describe severity. Our iterative process balanced parsimony with specificity describing both harm and non-harm events. We collapsed rare events into higher level sub-categorizations and added modifiers to provide further description. Safety and quality experts at 5 medical centers trained on use of the taxonomy on a conference call. To estimate reliability, collaborators independently reviewed 25 AE scenarios. Inter-rater reliability was assessed using the Fleiss kappa statistic for multiple raters. The 50 most-cited articles (out of 165) and the taxonomies from 10 major patient safety organizations yielded 2 that included descriptions of harm: The Joint Commission’s Patient Safety Event Taxonomy and the taxonomy from Adventist Health System. We used the latter as starting point, defining 7 categories and 51 subcategories. We added up to 3 “modifiers” per event (from a total of 46 modifiers available) to better describe events. Twenty-one distinct category-subcategory dyads were represented in the 25 test scenarios. There was complete agreement between reviewers in 20/25 scenarios. Inter-rater reliability was excellent for both overall-category scoring (κ = 81%) as well as for dyad scoring (κ = 79%). We developed a taxonomy of adverse events and non-harm events for the ED modified from an existing framework. Use of the tool with minimal training yielded excellent performance with high interrater reliability. We will perform additional reliability testing to cover all possible dyads and once validated, plan to use this taxonomy in further work related to adverse event detection in the ED.

Mixed methods evaluation of a quality improvement and audit tool for nurse-to-nurse bedside clinical handover in ward settings

BackgroundNurse bedside handover quality is influenced by complex interactions related to the content, processes used and the work environment. Audit tools are seldom tested in ‘real’ settings. ObjectiveExamine the reliability, validity and usability of a quality improvement tool for audit of nurse bedside handover. DesignNaturalistic, descriptive, mixed-methods. SettingSix inpatient wards at a single large not-for-profit private health service in Victoria, Australia. ParticipantsFive nurse experts and 104 nurses involved in 199 change-of-shift bedside handovers. MethodsA focus group with experts and pilot test were used to examine content and face validity, and usability of the handover audit tool. The tool was examined for inter-rater reliability and usability using observation audits of handovers across six wards. Data were collected in 2013–2014. ResultsTwo independent observers for 72 audits demonstrated acceptable inter-observer agreement for 27 (77%) items. Reliability was weak for items examining the handover environment. Seventeen items were not observed reflecting gaps in practices. Across 199 observation audits, gaps in nurse bedside handover practice most often related to process and environment, rather than content items. Usability was impacted by high observer burden, familiarity and non-specific illustrative behaviours. ConclusionThe reliability and validity of most items to audit handover content was acceptable. Gaps in practices for process and environment items were identified. Context specific exemplars and reducing the items used at each handover audit can enhance usability. Further research is needed to develop context specific exemplars and undertake additional reliability testing using a wide range of handover settings. What is already known about this topic•The quality of nurse bedside handover is influenced by complex interactions related to the content, processes used and the work environment.•Standardised handover tools to guide training, coaching and audit provide a consistent framework of behaviours, content and processes in line with expectations of quality practice.•Reliable and valid handover quality improvement tools are necessary to provide quality data to improve patient safety and quality of care, however these are seldom examined in the context of daily practice. Contribution of the paper•High content and face validity of the handover audit tool provides guidance to standardise practices to improve the quality of nurse bedside handover related to content, process and environment across different clinical contexts.•This research demonstrated nurse handover practices related to content were most frequently addressed, but gaps related to handover process and environmental safety emerged.•Tool reliability was acceptable for frequently observed content items, however practice improvement is required before further reliability analysis can evaluate process and environment items.

Development of an imaging mitigation strategy for patient enrolment in the tanezumab nerve growth factor inhibitor (NGF-ab) program with a focus on eligibility assessment.

Nerve growth factor antibodies (NGF-ab) have shown promising analgesic efficacy. Aim was to describe reader training efforts and present reliability data focusing on radiographic eligibility in the tanezumab program. A multi-step process was used for reader calibration and reliability testing. First, a reference standard set of cases was created and diagnostic performance was evaluated. A second exercise focused on agreement of ordinal assessment (Kellgren-Lawrence grading) of radiographic osteoarthritis. Subsequently, 11 readers were trained and read a test set of 100 cases focused on eligibility assessments. Additional reliability testing and calibration of five core readers assessing eligibility of 30 cases was performed 3 and 6 months after study start. Sensitivity for the reference standard readings ranged from 0.50 to 0.90 and specificity from 0.40 to 0.83. Overall agreement for Kellgren-Lawrence grading ranged from 71.4% to 82.9%. For the 11 reader exercise, in 76% of cases at least 8 of 11 readers agreed on eligibility status. For the reliability testing 3 months after study start, in 80.0% of cases at least 4 of 5 readers agreed on eligibility with a κ = 0.43 (95% CI: 0.32-0.54). For the reliability testing after 6 months, in 83.3% of cases at least 4 of 5 readers agreed on eligibility with a κ = 0.52 (95% CI: 0.41-0.63). After intense efforts spent in the development of an imaging program for an NGF-ab clinical program, the achieved reliability for eligibility assessment is substantial but not perfect. Ongoing efforts of calibration prior to including additional readers to the program and during study conduct between current readers will be needed to ensure agreement on potential adverse events and radiographic disease severity.

Evaluating nursing hours per patient day as a nurse staffing measure.

To identify the techniques used to measure nurse staffing and to evaluate the reliability, validity and limitations of nursing hours per patient day (NHPPD). Numerous studies have attempted to identify appropriate nurse staffing levels; however, variations in nurse staffing measures may have caused inconsistent findings regarding the relationships between nurse staffing and quality of care. Seventeen studies using nurse staffing measures were reviewed. Six common nurse staffing measures were identified: nurse-to-patient ratios, full-time equivalents, NHPPD, skill mix, nurse-perceived staffing adequacy and nurse-reported number of assigned patients. Among nurse staffing measures, NHPPD is the most frequently used and is considered to be highly beneficial. This measure shows some evidence of high inter-rater reliability. The predictive validity of NHPPD for patient falls is high, whereas that for pressure ulcers is low. For NHPPD to be applied more effectively as a nurse staffing measure, there is a need for additional reliability testing in various types of units with large sample sizes; further validity research for additional patient outcomes; appropriate adjustments in its application to capture variations in the characteristics of nurses, patients and hospital units; and a consistent data collection procedure.

An international pilot study of asynchronous teleconsultation for oropharyngeal dysphagia

We investigated whether an expert's consultation provided via telemedicine could improve the quality of care for patients with dysphagia. A trained clinician completed videofluoroscopic swallowing studies (VFSS) of 17 consecutive patients in a Greek hospital. The videofluoroscopic images were then stored on a website for independent review by an expert Speech and Language Pathologist in the US. An extra Rater evaluated 20% of all data for additional reliability testing. Eight diagnostic indicators of swallowing impairment and an overall subjective severity index were recorded for each study. Clinicians were also asked to choose from ten common treatment options for patients with dysphagia. There was good inter-rater agreement for most of the diagnostic indicators examined (ranging from 78% to 90%; kappa = 0.52-0.71) between all three Raters. Agreement on overall severity ratings was exact for more than half of the patients and within one-point on the 4-point scale for all other patients except one. However, the quality of care would have been substandard for more than half of the patients if teleconsultation had not been employed. In settings where a swallowing expert is not available and real-time telemedicine is not feasible, the use of asynchronous teleconsultation can produce better quality of care for patients with dysphagia.

Flip Chip Reliability of GaAs on Si Thinfilm Substrates Using AuSn Solder Bumps

AbstractAu/Sn solder bumps are mainly used for flip chip assembly of compound semiconductors in optoelectronic and RF applications. They allow a fluxless assembly which is required to avoid contamination of optical interfaces and the metallurgy is well suited to the final gold metallization on GaAs or InP. Flip chip assembly experiments were carried out using two layer Au/Sn bumps as plated without prior bump reflow. An RF and reliability test vehicles comprise a GaAs chip which was flip chip soldered on a silicon substrate. Temperature cycling tests with and without underfiller were performed. The different failure modes for underfilled and nonunderfilled samples were discussed and compared. Additional reliability tests were performed with flip chip bonding by gold thermocompression for comparison. The test results and the failure modes are discussed in detail.