Individual Rates Research Articles

Large language models can support generation of standardized discharge summaries – A retrospective study utilizing ChatGPT-4 and electronic health records

ObjectiveTo evaluate whether psychiatric discharge summaries (DS) generated with ChatGPT-4 from electronic health records (EHR) can match the quality of DS written by psychiatric residents. MethodsAt a psychiatric primary care hospital, we compared 20 inpatient DS, written by residents, to those written with ChatGPT-4 from pseudonymized residents’ notes of the patients’ EHRs and a standardized prompt. 8 blinded psychiatry specialists rated both versions on a custom Likert scale from 1 to 5 across 15 quality subcategories. The primary outcome was the overall rating difference between the two groups. The secondary outcomes were the rating differences at the level of individual question, case, and rater. ResultsHuman-written DS were rated significantly higher than AI (mean ratings: human 3.78, AI 3.12, p < 0.05). They surpassed AI significantly in 12/15 questions and 16/20 cases and were favored significantly by 7/8 raters. For “low expected correction effort”, human DS were rated as 67 % favorable, 19 % neutral, and 14 % unfavorable, whereas AI-DS were rated as 22 % favorable, 33 % neutral, and 45 % unfavorable. Hallucinations were present in 40 % of AI-DS, with 37.5 % deemed highly clinically relevant. Minor content mistakes were found in 30 % of AI and 10 % of human DS. Raters correctly identified AI-DS with 81 % sensitivity and 75 % specificity. DiscussionOverall, AI-DS did not match the quality of resident-written DS but performed similarly in 20% of cases and were rated as favorable for “low expected correction effort” in 22% of cases. AI-DS lacked most in content specificity, ability to distill key case information, and coherence but performed adequately in conciseness, adherence to formalities, relevance of included content, and form. ConclusionLLM-written DS show potential as templates for physicians to finalize, potentially saving time in the future.

Rating pome fruit quality traits using deep learning and image processing.

Quality assessment of pome fruits (i.e. apples and pears) is used not only for determining the optimal harvest time but also for the progression of fruit-quality attributes during storage. Therefore, it is typical to repeatedly evaluate fruits during the course of a postharvest experiment. This evaluation often includes careful visual assessments of fruit for apparent defects and physiological symptoms. A general best practice for quality assessment is to rate fruit using the same individual rater or group of individual raters to reduce bias. However, such consistency across labs, facilities, and experiments is often not feasible or attainable. Moreover, while these visual assessments are critical empirical data, they are often coarse-grained and lack consistent objective criteria. Granny, is a tool designed for rating fruit using machine-learning and image-processing to address rater bias and improve resolution. Additionally, Granny supports backward compatibility by providing ratings compatible with long-established standards and references, promoting research program continuity. Current Granny ratings include starch content assessment, rating levels of peel defects, and peel color analyses. Integrative analyses enhanced by Granny's improved resolution and reduced bias, such as linking fruit outcomes to global scale -omics data, environmental changes, and other quantitative fruit quality metrics like soluble solids content and flesh firmness, will further enrich our understanding of fruit quality dynamics. Lastly, Granny is open-source and freely available.

Benchmarking Human–AI collaboration for common evidence appraisal tools

Background and ObjectiveIt is unknown whether large language models (LLMs) may facilitate time- and resource-intensive text-related processes in evidence appraisal. The objective was to quantify the agreement of LLMs with human consensus in appraisal of scientific reporting (Preferred Reporting Items for Systematic reviews and Meta-Analyses [PRISMA]) and methodological rigor (A MeaSurement Tool to Assess systematic Reviews [AMSTAR]) of systematic reviews and design of clinical trials (PRagmatic Explanatory Continuum Indicator Summary 2 [PRECIS-2]) and to identify areas where collaboration between humans and artificial intelligence (AI) would outperform the traditional consensus process of human raters in efficiency. Study Design and SettingFive LLMs (Claude-3-Opus, Claude-2, GPT-4, GPT-3.5, Mixtral-8x22B) assessed 112 systematic reviews applying the PRISMA and AMSTAR criteria and 56 randomized controlled trials applying PRECIS-2. We quantified the agreement between human consensus and (1) individual human raters; (2) individual LLMs; (3) combined LLMs approach; (4) human–AI collaboration. Ratings were marked as deferred (undecided) in case of inconsistency between combined LLMs or between the human rater and the LLM. ResultsIndividual human rater accuracy was 89% for PRISMA and AMSTAR, and 75% for PRECIS-2. Individual LLM accuracy was ranging from 63% (GPT-3.5) to 70% (Claude-3-Opus) for PRISMA, 53% (GPT-3.5) to 74% (Claude-3-Opus) for AMSTAR, and 38% (GPT-4) to 55% (GPT-3.5) for PRECIS-2. Combined LLM ratings led to accuracies of 75%–88% for PRISMA (4%–74% deferred), 74%–89% for AMSTAR (6%–84% deferred), and 64%–79% for PRECIS-2 (29%–88% deferred). Human–AI collaboration resulted in the best accuracies from 89% to 96% for PRISMA (25/35% deferred), 91%–95% for AMSTAR (27/30% deferred), and 80%–86% for PRECIS-2 (76/71% deferred). ConclusionCurrent LLMs alone appraised evidence worse than humans. Human–AI collaboration may reduce workload for the second human rater for the assessment of reporting (PRISMA) and methodological rigor (AMSTAR) but not for complex tasks such as PRECIS-2.

Four hundred Greek idiomatic expressions: Ratings for subjective frequency, ambiguity, and decomposability.

Idioms differ from other forms of figurative language because of their dimensions of subjective frequency, ambiguity (possibility of having a literal interpretation), and decomposability (possibility of the idiom's words to assist in its figurative interpretation). This study focuses on the Greek language and aims at providing the first corpus of 400 Greek idioms rated for their dimensions by 113 native Greek students, aged 19 to 39 years. The study aimed at (1) rating all idioms for their degree of subjective frequency, ambiguity, and decomposability, and (2) investigating the relationships between these dimensions. Three different assessments were conducted, during which the participants were asked to evaluate the degree of idioms' subjective frequency, ambiguity, and decomposability. The idioms were selected from a dictionary of Greek idioms titled "Dictionary of Idioms in Modern Greek" (Vlaxopoulos, 2007). This study resulted in the first database of Greek idioms assessed for their dimensions. The intraclass correlation coefficient (ICC) (two-way mixed, absolute agreement) demonstrated high internal consistency in the ratings given for each dimension, for the same idiom, by the different individual raters. Correlational analyses showed that subjective frequency was positively moderately correlated with decomposability, and positively weakly correlated with ambiguity, while decomposability was positively moderately correlated with ambiguity.

Inter-Rater and Intrarater Reliability of Radiographs in the Diagnosis of Pediatric Scaphoid Fractures.

Pediatric scaphoid fractures can be challenging to diagnose on plain radiograph. Rates of missed scaphoid fractures can be as high as 30% to 37% on initial imaging and overall sensitivity ranging from 21% to 97%. Few studies, however, have examined the reliability of radiographs in the diagnosis of scaphoid fractures, and none are specific to the pediatric population. Reliability, both between different specialists and for individual raters, may elucidate some of the diagnostic challenges. We conducted a 2-iteration survey of pediatric orthopedic surgeons, plastic surgeons, radiologists, and emergency physicians at a tertiary children's hospital. Participants were asked to assess 10 series of pediatric wrist radiographs for evidence of scaphoid fracture. Inter-rater and intrarater reliability was calculated using the intraclass correlation coefficient of 2.1. Forty-two respondents were included in the first iteration analysis. Inter-rater reliability between surgeons (0.66; 95% confidence interval, 0.43-0.87), radiologists (0.76; 0.55-0.92), and emergency physicians (0.65; 0.46-0.86) was "good" to "excellent." Twenty-six respondents participated in the second iteration for intrarater reliability (0.73; 0.67-0.78). Sensitivity (0.75; 0.69-0.81) and specificity (0.78; 0.71-0.83) of wrist radiographs for diagnosing scaphoid fractures were consistent with results in other studies. Both inter-rater and intrarater reliability for diagnosing pediatric scaphoid fractures on radiographs was good to excellent. No significant difference was found between specialists. Plain radiographs, while useful for obvious scaphoid fractures, are unable to reliably rule out subtle fractures routinely. Our study demonstrates that poor sensitivity stems from the test itself, and not rater variability.

Diagnoses and Treatment Recommendations-Interrater Reliability of Uroflowmetry in People with Multiple Sclerosis.

Uroflowmetry (UF) is an established procedure in urology and is recommended before further investigations of neurogenic lower urinary tract dysfunction (NLUTD). Some authors even consider using UF instead of urodynamics (UD). Studies on the interrater reliability of UF regarding treatment recommendations are rare, and there are no relevant data on people with multiple sclerosis (PwMS). The aim of this study was to investigate the interrater reliability (IRR) of UF concerning diagnosis and therapy in PwMS prospectively. UF of 92 PwMS were assessed by 4 raters. The diagnostic criteria were normal findings (NFs), detrusor overactivity (DO), detrusor underactivity (DU), detrusor-sphincter dyssynergia (DSD) and bladder outlet obstruction (BOO). The possible treatment criteria were as follows: no treatment (NO), catheter placement (CAT), alpha-blockers, detrusor-attenuating medication, botulinum toxin (BTX), neuromodulation (NM), and physiotherapy/biofeedback (P/BF). IRR was assessed by kappa (κ). κ of diagnoses were NFs = 0.22; DO = 0.17; DU = 0.07; DSD = 0.14; and BOO = 0.18. For therapies, the highest κ was BTX = 0.71, NO = 0.38 and CAT = 0.44. There is a high influence of the individual rater. UD should be subject to the same analysis and a comparison should be made between UD and UF. This may have implications for the value of UF in the neuro-urological management of PwMS, although at present UD remains the gold standard for the diagnostics of NLUTD in PwMS.

Between-rater reliability for using radar technology to quantify maximal horizontal deceleration performance in NCAA division 1 American football and female lacrosse athletes.

With recent increases in the popularity of studying the physical construct of horizontal deceleration performance in team-sport athletes, the aim of the present study was to assess the inter-rater and intra-rater reliability of processing and quantifying horizontal deceleration ability using radar technology. Data from 92 NCAA Division 1 athletes from two different athletic teams (American football and Lacrosse) were used for the present investigation. All athletes performed two trials of the modified acceleration to deceleration assessment (ADA), which consisted of a maximal 10 m sprint acceleration, followed by a rapid deceleration. Four individual raters manually processed raw, radar-derived instantaneous velocity data for the ADA, and an automated script was used to calculate metrics of interest. Primary study findings suggest moderate to excellent levels of agreement (ICC = 0.56-0.91) for maximal horizontal deceleration metrics between the four individual raters. The intra-rater analyses revealed poor to excellent consistency (ICC = 0.31-0.94) between ADA trials, with CV%'s ranging from 3.1% to 13.2%, depending on the respective metric and rater. Our data suggests that if a foundational understanding and agreement of manual data processing procedures for radar-derived data is given between raters, metrics may be interpreted with moderate to excellent levels of confidence. However, when possible, and when using the Stalker ATS radar technology, authors recommend that practitioners use one trained individual to manually process raw data. Ideally, this process should become fully automated, based on selected filters or algorithms, rather than the subjectivity of the rater.

Reliability of a Modified Watson-Jones Classification for Tibial Tubercle Fractures in Children.

The first case report describing a pediatric fracture of the tibial tubercle was published in 1852 in the Medical Times and Gazette, a Journal of Medical Science, literature, criticism, and news. A century later, in 1955, Sir Watson-Jones introduced the first classification system for this fracture. Despite the existence of several classification systems for pediatric tibial tubercle fractures in the medical literature, a crucial aspect of their validation-reliability and reproducibility-has not yet been systematically evaluated. In this study, a modified Watson-Jones (mWJ) classification system of tibial tubercle fractures in children was assessed for intraobserver and interobserver variability. Using the mWJ classification, 3 board-certified pediatric orthopaedic surgeons and 3 orthopaedic surgery residents, classified thirty tibial tubercle fractures based on anteroposterior and lateral radiographs on 2 separate occasions in a 2-week duration. Further comparison was made to evaluate the impact of advanced imaging, specifically computed tomography or magnetic resonance imaging, on diagnostic reliability and reproducibility. The study found substantial intraobserver reliability of the mWJ classification based on radiographs alone, with a Cohen weighted kappa (κ w ) coefficient of 0.733. When advanced imaging was utilized, the reliability of the classification improved to κ w = 0.783. Similarly, interobserver reliability demonstrated substantial consistency among observers when using radiographs alone (κ w = 0.69) and improved agreement with advanced imaging (κ w = 0.75). Notably, there was no significant difference in reliability scores between senior-level attendings and residents when analyzed as separate groups. Fracture classification systems are clinically relevant tools that help organize and transfer knowledge efficiently, provide treatment guidance, propose prognostic expectations, and improve communication in academic literature. The present study demonstrated substantial reproducibility of an mWJ fracture classification system both between and within individual surgeon raters. Level III-diagnostic.

Fecal Calprotectin as a Biomarker of Depression Severity: A Systematic Review

Introduction: Depression is a debilitating mental health condition. Although the underlying pathophysiology of depression is poorly understood, emerging evidence suggests that the gut-brain-immune axis may contribute to depressive symptoms. Recent literature has established that depression is associated with systemic low-grade inflammation. According to the leaky-gut hypothesis, this systemic inflammation is thought to be caused by increased permeability—a by-product of mucosal inflammation. Accordingly, some research has demonstrated that fecal calprotectin (FC)—a marker of mucosal inflammation—is positively associated with depressive symptoms. However, with some research showing no correlation between the two, the literature has a lack of consensus on the topic. The aim of the current systematic review was to clarify the association between FC and depression. Methods: A search was conducted in Ovid MEDLINE, Embase, and APA PsycInfo using MeSH terms related to fecal calprotectin and depression. All English studies with human participants were screened for eligibility by two individual raters, excluding case studies and review papers. Selected articles were subsequently used in the data extraction to look for relevant study characteristics and their findings regarding the correlation between fecal calprotectin and depression. Additionally, the JBI tool was used to assess risk of bias. Results: A total of 7 articles were included in the study, comprising 1184 patients. The review found that there is no consensus on whether calprotectin and depression are correlated; therefore, no evidence suggesting utility of calprotectin as a biomarker of depression was found. Discussion: Despite the conflicting findings of the study, there is still some evidence to suggest that calprotectin could be useful as a biomarker. Some of the included studies found that the time of sample collection in relation to the ranking of depression symptoms has an effect on the degree of correlation. Conclusion: This study found that fecal calprotectin is a weak biomarker of depression and is impractical in the general population due to the collection method. Alternatives include serum calprotectin and markers of neuroinflammation. Other studies looking at the role of salivary biomarkers in depression may prove to be promising.

A Web-Based and Mobile Intervention Program Using a Spaced Education Approach for Workplace Mental Health Literacy: Cluster Randomized Controlled Trial.

Workplace mental health is an important global health concern. This unblinded, phase-III, wait-listed cluster randomized controlled trial aimed to examine the effectiveness of a mobile health (mHealth) psychoeducation program using a spaced education approach on mental health literacy (MHL) in the workplace. The main interest of this paper was the immediate and 3-month medium-term effect of the program on the MHL of workers. The purposely built mHealth platform was also evaluated as a health-related app. The mHealth platform was designed using the principle of spaced education as a psychoeducation intervention program, with various modules of web-based and mobile materials presented to the participant in a progressive manner. Short quizzes at the end of each module ensured adequate learning, and successful completion qualified the learner to progress to the next level. The trial recruited 456 employees of specific industries with high levels of work-related stress. Participants who were nested in different offices or units were allocated into the intervention and wait-listed control groups using a block randomization process, with the office or unit as the cluster. A separate sample of 70 individual raters were used for the evaluation of the mHealth platform. The Australian National MHL and Stigma Survey and the Mobile Apps Rating Scale were completed through a web-based self-reported survey to assess MHL and evaluate the app. The trial and follow-up data were analyzed by a generalized linear latent and mixed model with adjustments for the clustering effect of work sites and repeated measures. Of the 456 participants in the trial, 236 (51.8%) responded to the follow-up survey. Most MHL outcomes obtained significant results immediately after the intervention and across time. After adjusting for the clustering effect, the postintervention weighted mean scores were significantly higher in the intervention group than the control group for correct recognition of a mental health problem, help seeking, and stigmatization by 0.2 (SE 0.1; P=.003), 0.9 (SE 0.2; P<.001), and 1.8 (SE 0.4; P<.001), respectively. After adjusting for the clustering effect, significant differences across time were found in help-seeking intention (P=.01), stigmatization (P<.001), and social distancing (P<.001). The evaluation of the mHealth program resulted in average scores of the 4 major domains ranging from 3.8 to 4.2, with engagement having the lowest score. The mHealth psychoeducation intervention program using this platform had immediate and 3-month medium-term effects of retaining and improving MHL. The platform was evaluated to have satisfactory performance in terms of functionality, aesthetics, information content, and utility in enhancing MHL. It is anticipated that ongoing development in digital health will provide great benefits in improving the mental health of the global population.

AIxSuture: vision-based assessment of open suturing skills

PurposeEfficient and precise surgical skills are essential in ensuring positive patient outcomes. By continuously providing real-time, data driven, and objective evaluation of surgical performance, automated skill assessment has the potential to greatly improve surgical skill training. Whereas machine learning-based surgical skill assessment is gaining traction for minimally invasive techniques, this cannot be said for open surgery skills. Open surgery generally has more degrees of freedom when compared to minimally invasive surgery, making it more difficult to interpret. In this paper, we present novel approaches for skill assessment for open surgery skills.MethodsWe analyzed a novel video dataset for open suturing training. We provide a detailed analysis of the dataset and define evaluation guidelines, using state of the art deep learning models. Furthermore, we present novel benchmarking results for surgical skill assessment in open suturing. The models are trained to classify a video into three skill levels based on the global rating score. To obtain initial results for video-based surgical skill classification, we benchmarked a temporal segment network with both an I3D and a Video Swin backbone on this dataset.ResultsThe dataset is composed of 314 videos of approximately five minutes each. Model benchmarking results are an accuracy and F1 score of up to 75 and 72%, respectively. This is similar to the performance achieved by the individual raters, regarding inter-rater agreement and rater variability. We present the first end-to-end trained approach for skill assessment for open surgery training.ConclusionWe provide a thorough analysis of a new dataset as well as novel benchmarking results for surgical skill assessment. This opens the doors to new advances in skill assessment by enabling video-based skill assessment for classic surgical techniques with the potential to improve the surgical outcome of patients.

Detecting Randomness Effect among Raters in Physics Essay Items using Many-Facet Rasch Measurement

This study sought to detect randomness effects among raters in physics essay items using Many-Facet Rasch Measurement. The research design adopted for this study is descriptive research design based on survey method. The population of the study comprised eighty-eight (88) public schools in all the local government areas with a physics student population of 3,642 students and ninety-four (94) physics teachers in all the Senior Secondary Schools in Uyo Senatorial District for the 2022/2023 academic session. Three hundred and sixty-four (364) SSS3 physics students and 37 physics teachers from the 31 selected secondary schools in Uyo Senatorial District were sampled using multistage sampling technique for effective selection. The multi-stage sampling technique was adopted for the study. The instrument used for data collection was Physics Achievement Test (PAT) obtained from WAEC and NECO 2020 Physics Essay items with reliability coefficients of 0.91 and 0.90 respectively. The finding revealed that most of the individual raters commit randomness effect when rating physics essay Items. It also revealed there is a significant difference at the rater's group level exhibiting randomness effect when rating physics essay items, which implies that there is no group-level randomness effect present among raters when rating physics essay items. We concluded that rater effects are sources of variance in performance ratings that are associated with the raters' behaviour and not the actual performance of the ratee. It was recommended that raters should follow the rating guidelines to reduce the impact of randomness in their ratings and provide more accurate and objective evaluations.

Bubonic plague: can the size of buboes be accurately and consistently measured with a digital calliper?

IntroductionConducting clinical research on treatments for emerging infectious diseases is often complicated by methodological challenges, such as the identification of appropriate outcome measures to assess treatment response and the lack of validated instruments available to measure patient outcomes. In bubonic plague, some studies have assessed bubo size as an indicator of treatment success, a measure widely assumed to be indicative of recovery. Evaluating this outcome however is challenging as there is no validated method for measuring bubo size. The aim of this study is to assess the accuracy and inter- and intra-rater agreement of artificial bubo measurements using a digital calliper to understand whether a calliper is an appropriate measurement instrument to assess this outcome.MethodsStudy technicians measured 14 artificial buboes made from silicone overlaid with artificial silicone skin sheets over the course of two training sessions. Each artificial bubo was measured by each study technician once per training session, following a Standard Operating Procedure. The objectives of this study are to (i) evaluate the accuracy of individual measurements against the true size of the artificial bubo when using a digital calliper, (ii) understand whether the characteristics of the artificial bubo influence measurement accuracy and (iii) evaluate inter- and intra-rater measurement agreement.ResultsIn total, 14 artificial buboes ranging from 52.7 to 121.6 mm in size were measured by 57 raters, generating 698 measurements recorded across two training sessions. Raters generally over-estimated the size of the artificial bubo. The median percentage difference between the measured and actual bubo size was 13%. Measurement accuracy and intra-rater agreement decreased as the size of the bubo decreased. Three quarters of all measurements had a maximum of 25% difference from another measurement of the same artificial bubo. Inter-rater agreement did not vary with density, size or presence of oedema of the artificial bubo.ConclusionsThe results of this study demonstrate the challenges for both individual and multiple raters to repeatedly generate consistent and accurate measurements of the same artificial buboes with a digital calliper.

Evaluating mobile applications for estimating soil properties: Quality of current apps, limitations and future directions

The estimation of soil properties is an important factor in agriculture. With the advancement of smartphones and artificial intelligence, estimating soil properties such as the Munsell soil color, pH, soil organic matter and moisture is tangible, alleviating the need for laboratories, which are time-consuming and costly. While there are several smartphone applications available in different app stores that claim to provide this functionality, an evaluation of the limitations and quality is required for these apps. The objective of this study is to identify existing soil property estimation apps, devise a soil apps rating scale to evaluate the functionality along with the software quality characteristics, and provide design guidelines for improving apps of this type. A thorough search was conducted in two prominent app stores: the Google Play and Apple App store. The selected apps were then rated by four individual raters. A total of 337 apps were found in the search and from this list, 32 selected apps were reviewed using the devised app rating scale. The rating scale evaluates the apps against ten sub-scales: app metadata, aesthetics, general app features, performance and efficiency, usability, functionality, subjective quality, transparency, user comments, and the perceived impact of the app on users. The inter-rater and intra-rater reliability are calculated along with the internal consistency of the rating scale to ensure the validity of the results. The evaluation results that returned the highest-ranking mean sub-scales were performance, usability, and aesthetics. Adversely, the lowest-ranging sub-scales were functionality and general features. There was also an evident absence of the functionality to calculate any type of soil properties in all but four of the apps reviewed. Several key design considerations were identified, such as the need for automated estimation of soil properties, an improvement of general features, and the inclusion of a shared soil database.

Investigating The Central Tendency Effect Among Raters in Complex Problem-Solving Skills in Physics Essay Items: An Application of Many-Facet Rasch Measurement

The study sought to investigate the central tendency effect among raters in complex problem-solving skills in physics essay items using Many-Facet Rasch Measurement. This became necessary because the central tendency effect can affect students' scores. The research design adopted for this study is a descriptive research design based on the survey method. The population of the study comprised eighty-eight (88) public schools in all the local government areas with a physics student population of 3642 students and ninety-four (94) physics teachers in all the Senior Secondary Schools in Uyo Senatorial District for the 2022/2023 academic session. 364 SS3 physics students and 37 physics teachers from the 31 selected secondary schools in Uyo Senatorial District were sampled using a multistage sampling technique for effective selection. The multi-stage sampling technique was adopted for the study. The instrument used for data collection was the Physics Achievement Test (PAT) obtained from WAEC and NECO 2020 Physics Essay items. However, the reliability coefficient of 0.91 and 0.90 respectively. The finding revealed that there was no central tendency effect error committed by individual raters when rating complex problem-solving skills in WAEC and NECO 2020 physics Essay Items. Also, reveals there is a significant difference at the rater's group level exhibiting central tendency effect when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items which implies that there is no group-level central tendency effect present among raters when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items. The researchers concluded that rater effects are sources of variance in performance ratings that are associated with the rater's behaviour and not the actual performance of the ratee. It was recommended that to prevent the central tendency effect, it is important for superior raters to regularly observe their subordinates throughout the evaluation period and keep notes for future reference.

Inter-Rater Agreement on Cincinnati Prehospital Stroke Scale (CPSS) and Prehospital Acute Stroke Severity Scale (PASS) Between EMS Providers, Neurology Residents and Neurology Consultants.

To examine the agreement between emergency medical service (EMS) providers, neurology residents and neurology consultants, using the Cincinnati Prehospital Stroke Scale (CPSS) and the Prehospital Acute Stroke Severity Scale (PASS). Patients with stroke, transient ischemic attack (TIA) and stroke mimic were included upon primary stroke admission or during rehabilitation. Patients were included from June 2018 to September 2019. Video recordings were made of patients being assessed with CPSS and PASS. The recordings were later presented to the healthcare professionals. To determine relative and absolute interrater reliability in terms of inter-rater agreement (IRA), we used generalisability theory. Group-level agreement was determined against a gold standard and presented as an area under the curve (AUC). The gold standard was a consensus agreement between two neurology consultants. A total of 120 patient recordings were assessed by 30 EMS providers, two neurology residents and two neurology consultants. Using the CPSS and the PASS, a total of 1,800 assessments were completed by EMS providers, 240 by neurology residents and 240 by neurology consultants. The overall relative and absolute IRA for all items combined from the CPSS and PASS score was 0.84 (95% CI 0.80; 0.87) and 0.81 (95% CI 0.77; 0.85), respectively. Using the CPSS, the agreement on a group-level resulted in AUCs of 0.83 (95% CI 0.78; 0.88) for the EMS providers and 0.86 (95% CI 0.82; 0.90) for the neurology residents when compared with the gold standard. Using the PASS, the AUC was 0.82 (95% CI 0.77; 0.87) for the EMS providers and 0.88 (95% CI 0.84; 0.93) for the neurology residents. The high relative and absolute inter-rater agreement underpins a high robustness/generalisability of the two scales. A high agreement exists across individual raters and different groups of healthcare professionals supporting widespread applicability of the stroke scales.

Hazard Assessment and Remediation Tool for Simulation-Based Healthcare Facility Design Testing.

To develop an objective, structured observational tool to enable identification and measurement of hazards in the built environment when applied to audiovisual recordings of simulations by trained raters. Simulation-based facility design testing is increasingly used to optimize safety of healthcare environments, often relying on participant debriefing or direct observation by human factors experts. Hazard categories were defined through participant debriefing and detailed review of pediatric intensive care unit in situ simulation videos. Categories were refined and operational definitions developed through iterative coding and review. Hazard detection was optimized through the use of structured coding protocols and optimized camera angles. Six hazard categories were defined: (1) slip/trip/fall/injury risk, impaired access to (2) patient or (3) equipment, (4) obstructed path, (5) poor visibility, and (6) infection risk. Analysis of paired and individual coding demonstrated strong overall reliability (0.89 and 0.85, Gwet's AC1). Reliability coefficients for each hazard category were >0.8 for all except obstructed path (0.76) for paired raters. Among individual raters, reliability coefficients were >0.8, except for slip/trip/fall/injury risk (0.68) and impaired access to equipment (0.77). Hazard Assessment and Remediation Tool (HART) provides a framework to identify and quantify hazards in the built environment. The tool is highly reliable when applied to direct video review of simulations by either paired raters or trained single clinical raters. Subsequent work will (1) assess the tool's ability to discriminate between rooms with different physical attributes, (2) develop strategies to apply HART to improve facility design, and (3) assess transferability to non-ICU acute care environments.

Automated detection of lacunar infarcts from MRI images

AbstractBackgroundWhen MRI images of patients with memory problems are assessed, the presence of lacunar infarcts is evaluated. Lacunar infarcts, fluid‐filled cavities of 3‐15 mm in diameter, are one possible result of cerebral small vessel disease and therefore relevant in dementia diagnostics. In this study, we propose an automated tool for detecting lacunar infarcts on MRI images.MethodWe adapted our previous approach [1] in three ways: 1) post‐processing steps were added to the convolutional neural network segmentation pipeline, e.g., to exclude enlarged perivascular spaces and segmentations located inside sulci or ventricles, 2) memory clinic data from the Amsterdam Dementia Cohort and PredictND cohort were included in addition to the LADIS cohort data, and 3) the ground truth was defined by three experienced neuroradiologists (A, B, C) instead of one rater. T1 and T2‐FLAIR images were available for all cases and T2 for a subset in visual rating. The dataset was composed of 393 subjects and each expert rated 1156 potential lacunar infarcts as “1 = no”, “2 = probably no”, “3 = probably yes” or “4 = yes”. A consensus (CO) was created by calculating the average and using a cutoff at 2.5. The agreement was measured between individual raters (A‐B, B‐C, A‐C), between automatic rating and individual raters (AU‐A, AU‐B, AU‐C), and between the automatic rating and the consensus rating (AU‐CO) using Kappa. Correlation coefficient for the number of lacunar infarcts per subject was defined. Cross‐validation was used in training the algorithm.ResultOut of 1156 candidate regions evaluated, 540 were considered as lacunar infarcts based on consensus. The individual raters agreed with each other in 74% of cases whether the shown lesion was lacunar infarct (kappa = 0.48). Correspondingly, automatic rating agreed with the individual raters in 74% of cases (kappa = 0.47). When automatic rating was compared with consensus, agreement/accuracy was 78% (kappa = 0.55). The correlation coefficient for the number of lacunar infarcts detected automatically from the whole image and visually (consensus) was 0.85 (Figure 1).ConclusionThis study suggests that the agreement in detecting lacunar infarcts automatically is comparable to inter‐rater variability between experienced neuroradiologists. [1] Jokinen et al. Stroke 2020 Nov 9; 50(1)

MRI May Be More Valuable than Pelvic Radiographs in the Assessment of Paediatric Borderline Acetabular Dysplasia.

The osseous acetabular index (OAI) and cartilaginous acetabular index (CAI) is often used in diagnosing acetabular dysplasia (AD) in children. We examined the reliability of OAI and CAI in AD diagnostics and compared OAI measurements obtained from radiographs versus MRI. Four raters performed retrospective repeated measurements of the OAI and CAI on pelvic radiographs and MRI scans of 16 consecutive patients (mean age 5 years (2-8)) examined for borderline AD during a period of 2½ years. In MRI, the image selected for analysis by the raters was also registered. Spearman's correlation, scatter plots, and Bland-Altman (BA) plots were analysed for correlation between OAI on pelvic radiographs (OAIR) and MRI scans (OAIMRI), while intra- and interrater reliability was assessed for OAIR, OAIMRI, CAI, and MRI image selection using intraclass correlation coefficients (ICC). ICC values for inter- and intrarater reliability of OAIR, OAIMRI, and CAI were all above 0.65, with no significant differences observed. ICC values (CI) for individual raters' MRI image selection was 0.99 (0.998-0.999). The mean difference (95% CI) between OAIR and OAIMRI was -0.99 degrees (-1.84; -0.16), while the mean absolute difference (95% CI) between OAIR and OAIMRI was 3.68 degrees (3.17; 4.20). Absolute differences between OAIR and OAIMRI was independent of pelvic positioning or time interval between radiographs and MRI scans. OAI and CAI had high Intrarater reliability but mediocre interrater reliability. There was an absolute difference of 3.7 degrees in OAI between pelvic radiographs and MRI scans.

Development of the McMaster Embalming Scale (MES) to assess embalming solutions for surgical skills training.

Human cadavers used for surgical training are embalmed using various methods to facilitate tissue storage and longevity while preserving the natural characteristics required to achieve high fidelity functional task alignment. However, there are no standardized means to evaluate the suitability of embalming solutions for this purpose. The McMaster Embalming Scale (MES) was developed to assess the extent to which embalming solutions allow tissues to achieve physical and functional correspondence to clinical contexts. The MES follows a five-point Likert scale format and evaluates the effect of embalming solutions on tissue utility in seven domains. This study aims to determine the reliability and validity of the MES by presenting it to users after performing surgical skills on tissues embalmed using various solutions. A pilot study of the MES was conducted using porcine material. Surgical residents of all levels and faculty were recruited via the Surgical Foundations program at McMaster University. Porcine tissue was unembalmed (fresh- frozen) or embalmed using one of seven solutions identified in the literature. Participants were blinded to the embalming method as they completed four surgical skills on the tissue. After each performance, participants evaluated their experience using the MES. Internal consistency was evaluated using Cronbach's alpha. Domain to total correlations and a g-study were also conducted. Formalin-fixed tissue achieved the lowest average scores, while fresh frozen tissue achieved the highest. Tissues preserved using Surgical Reality Fluid (Trinity Fluids, LLC, Harsens Island, MI) achieved the highest scores among embalmed tissues. The Cronbach's alpha scores varied between 0.85-0.92, indicating a random sample of new raters would offer similar ratings using the MES. All domains except odor were positively correlated. The g-study indicated that the MES is able to differentiate between embalming solutions, but an individual rater's preference for certain tissue qualities also contributes to the variance in scores captured. This study evaluated the psychometric characteristics of the MES. Future steps to this investigation include validating the MES on human cadavers. This article is protected by copyright. All rights reserved.