Average Measures Intraclass Correlation Coefficient Research Articles

586. Let's Have a Chat: How Well Does an Artificial Intelligence Chatbot Answer Clinical Infectious Diseases Questions?

Abstract Background ChatGPT is an artificial intelligence tool used by practitioners to answer clinical questions. It’s unknown whether ChatGPT provides quality responses to infectious diseases (ID)-specific questions. This study surveyed ID pharmacist subject matter experts (SMEs) to assess the quality of ChatGPT responses. Methods The primary outcome was percentage of ChatGPT responses considered useful. Secondary outcomes were SME’s ratings of correctness, completeness, and safety (C/C/S). One hundred clinically encountered questions by ID pharmacists were assembled and internally validated. Questions were entered into ChatGPT version 3.5 and responses were recorded. Definitions for useful and C/C/S were based on prior definitions and literature. A 0-10 rating scale for C/C/S was developed and validated for interrater reliability using a random sample. Questions with ChatGPT responses were sent to five SMEs for evaluation. Ordinal and categorical variables were assessed for interrater reliability using an average measures intraclass correlation coefficient (ICC) and Fleiss Multirater Kappa (FMK), respectively. SMEs’ responses were compared using the Kruskal-Wallis and Chi-square tests for ordinal and categorical variables, respectively. A post-hoc analysis was performed to identify the location of differences between SME ratings based on question difficulty and category for C/C/S using a Mann-Whitney U test with Bonferroni correction. Results SMEs considered 41.8% of responses useful. Median (IQR) ratings for C/C/S were 7 (IQR 4-9), 5 (IQR 3-8), and 8 (IQR 4-10), respectively. The FMK for useful was 0.379 (95% CI 0.317-0.441) indicating fair agreement, and ICC were 0.820 (95% CI 0.758-0.870), 0.745 (95% CI 0.656-0.816), and 0.833 (95% CI 0.775-0.880) for C/C/S, respectively, indicating substantial agreement. No significant difference was observed between SME responses for the percentage of responses considered useful. Neither question category nor difficulty resulted in a difference in SMEs’ ratings for C/C/S or percentage of responses considered useful. Conclusion Fewer than half of ChatGPT responses were considered useful by SMEs. However, responses were mostly considered correct and safe, but often deemed incomplete. Disclosures Conan MacDougall, PharmD, MAS, Merck: Grant/Research Support Elias Chahine, Pharm.D., Seqirus: Advisor/Consultant|Seqirus: Honoraria Wesley D. Kufel, Pharm.D., BCPS, BCIDP, Merck & Co.: Grant/Research Support|Shionogi, Inc: Grant/Research Support

Let's Have a Chat: How Well Does an Artificial Intelligence Chatbot Answer Clinical Infectious Diseases Pharmacotherapy Questions?

It is unknown whether ChatGPT provides quality responses to infectious diseases (ID) pharmacotherapy questions. This study surveyed ID pharmacist subject matter experts (SMEs) to assess the quality of ChatGPT version 3.5 (GPT-3.5) responses. The primary outcome was the percentage of GPT-3.5 responses considered useful by SME rating. Secondary outcomes were SMEs' ratings of correctness, completeness, and safety. Rating definitions were based on literature review. One hundred ID pharmacotherapy questions were entered into GPT-3.5 without custom instructions or additional prompts, and responses were recorded. A 0-10 rating scale for correctness, completeness, and safety was developed and validated for interrater reliability. Continuous and categorical variables were assessed for interrater reliability via average measures intraclass correlation coefficient and Fleiss multirater kappa, respectively. SMEs' responses were compared by the Kruskal-Wallis test and chi-square test for continuous and categorical variables. SMEs considered 41.8% of responses useful. Median (IQR) ratings for correctness, completeness, and safety were 7 (4-9), 5 (3-8), and 8 (4-10), respectively. The Fleiss multirater kappa for usefulness was 0.379 (95% CI, .317-.441) indicating fair agreement, and intraclass correlation coefficients were 0.820 (95% CI, .758-.870), 0.745 (95% CI, .656-.816), and 0.833 (95% CI, .775-.880) for correctness, completeness, and safety, indicating at least substantial agreement. No significant difference was observed among SME responses for percentage of responses considered useful. Fewer than 50% of GPT-3.5 responses were considered useful by SMEs. Responses were mostly considered correct and safe but were often incomplete, suggesting that GPT-3.5 responses may not replace an ID pharmacist's responses.

A large language model in solving primary healthcare issues: A potential implication for remote healthcare and medical education.

Access to quality health care is essential, particularly in remote areas where the availability of healthcare professionals may be limited. The advancement of artificial intelligence (AI) and natural language processing (NLP) has led to the development of large language models (LLMs) that exhibit capabilities in understanding and generating human-like text. This study aimed to evaluate the performance of a LLM, ChatGPT, in addressing primary healthcare issues. This study was conducted in May 2023 with ChatGPT May 12 version. A total of 30 multiple-choice questions (MCQs) related to primary health care were selected to test the proficiency of ChatGPT. These MCQs covered various topics commonly encountered in primary healthcare practice. ChatGPT answered the questions in two segments-one is choosing the single best answer of MCQ and another is supporting text for the answer. The answers to MCQs were compared with the predefined answer keys. The justifications of the answers were checked by two primary healthcare professionals on a 5-point Likert-type scale. The data were presented as number and percentage. Among the 30 questions, ChatGPT provided correct responses for 28 yielding an accuracy of 93.33%. The mean score for explanation in supporting the answer was 4.58 ± 0.85. There was an inter-item correlation of 0.896, and the average measure intraclass correlation coefficient (ICC) was 0.94 (95% confidence interval 0.88-0.97) indicating a high level of interobserver agreement. LLMs, such as ChatGPT, show promising potential in addressing primary healthcare issues. The high accuracy rate achieved by ChatGPT in answering primary healthcare-related MCQs underscores the value of these models as resources for patients and healthcare providers in remote healthcare settings. This can also help in self-directed learning by medical students.

Assessing the Capability of Large Language Models in Naturopathy Consultation.

Background The rapid advancements in natural language processing have brought about the widespread use of large language models (LLMs) across various medical domains. However, their effectiveness in specialized fields, such as naturopathy, remains relatively unexplored. Objective The study aimed to assess the capability of freely available LLM chatbots in providing naturopathy consultations for various types of diseases and disorders. Methods Five free LLMs (viz., Gemini, Copilot, ChatGPT, Claude, and Perplexity) were used to converse with 20 clinical cases (simulation of real-world scenarios). Each case had the case details and questions pertinent to naturopathy. The responses were presented to three naturopathy doctors with > 5 years of practice. The answers were rated by them on a five-point Likert-like scale for language fluency, coherence, accuracy, and relevancy. The average of these four attributes is termed perfection in his study. Results The overall score of the LLMs were Gemini 3.81±0.23, Copilot 4.34±0.28, ChatGPT 4.43±0.2, Claude 3.8±0.26, and Perplexity 3.91±0.28 (ANOVA F [3.034, 57.64] = 33.47, P <0.0001. Together, they showed overall ~80% perfection in consultation. The average measure intraclass correlation coefficient among the LLMs for the overall score was 0.463 (95% CI = -0.028 to 0.76), P = 0.03. Conclusion Although the LLM chatbots could help in providing naturopathy and yoga treatment consultation with approximately an overall fair level of perfection, their solution to the user varies across different chatbots and there was very low reliability among them.

Reliability testing of the Health of the Nation Outcome Scales 2018.

The Health of the Nation Outcome Scales (HoNOS) is a widely used clinical measure designed to rate and monitor the outcomes of service users accessing specialist mental healthcare. Since its development (in 1996), numerous research studies have confirmed the HoNOS captures the aspects of care that it purports to (validity), and that clinicians' ratings are consistent both over time, and between different raters (reliability). In 2018, the HoNOS was reviewed with updates made to some terminology and other revisions intended to remove ambiguity in the guidance for raters. However, although the new version (HoNOS 2018) was accompanied by a recommendation that its validity and reliability be re-tested this was not undertaken. To our knowledge, this is the first study to re-assess the updated tool's reliability by measuring the level of agreement between different raters. Our findings confirm that there is an acceptable level of consistency between student mental health nurses that have been trained to use the (new) HoNOS 2018. The HoNOS is nationally mandated for use by all specialist mental healthcare providers in the UK. Our findings provide some assurance that, with appropriate update training and monitoring of organisational-level data sets, the original HoNOS glossary can safely be replaced with the HoNOS 2018 to ensure more contemporary routine outcome measurement can occur. INTRODUCTION: The Health of the Nation Outcome Scales (HoNOS) is a well-established clinician rated outcome measure for use in mental health services. Following an international review, an updated version (HoNOS 2018) was published with a recommendation that its psychometric properties be re-tested prior to widespread implementation. To date, only one such study has been published. To test the inter-rater agreement levels for HoNOS 2018. Third-year student mental health nurses received training to complete the HoNOS 2018. Following this timetabled session, they were each invited to independently rate two, randomly selected, videos of (simulated) patient interviews. The resulting data were then analysed to calculate the tool's internal consistency and inter-rater agreement levels. The 55 participants provided 106 ratings from four vignettes. Cronbach's alphas and McDonalds omegas confirmed the revised tool's internal consistency was acceptable. Average measure intraclass correlation coefficients for the four patient vignettes indicated excellent reliability. This study provides initial assurance that the HoNOS 2018 is a reliable clinician rated outcome measure suitable for use in routine clinical practice by relatively inexperienced mental health practitioners with limited training.

Validity and reliability of Arabic version of pediatric migraine disability assessment scale (Child Self-Report versus Parent Proxy-Report): a multi-center study

BackgroundPediatric Migraine Disability Assessment (PedMIDAS) is one of the most frequently used questionnaires to assess disability from migraine in pediatric patients. This work aimed to evaluate the validity and test–retest reliability of the Arabic version of the child self-report versus the parent proxy report PedMIDAS. We also aimed to test the agreement between children's and parents' reports of the scale.MethodsPedMIDAS was subjected to translation and back-translation, then applied to 112 pediatric patients fulfilling the migraine diagnostic criteria. This cross-sectional study was conducted on two visits, one week apart. At visit 1, the following data were obtained from the included pediatric patients: disease duration, migraine type, current treatment regimen, monthly migraine days (MMD) during the last month preceding the enrollment, and migraine intensity using the visual analogue scale. Then, each child and his parent were independently asked to fill out PedMIDAS and Child Self-Report of the Pediatric Quality of Life Inventory™ 4.0 (PedsQL™) to test the convergent validity of PedMIDAS. At visit 2, each child was requested to complete PedMIDAS again, and so was the parent to evaluate test–retest reliability.ResultsCronbach’s alpha was estimated to be 0.94 for each instrument. For the child-self report PedMIDAS, the average measure intraclass correlation coefficient (ICC) value was 0.992 (95%CI = 0.989–0.995), while it was estimated to be 0.990 for the parent-proxy report with 95%CI = 0.985–0.993, indicating excellent test–retest reliability for both instruments. The child-self report and the parent-proxy report PedMIDAS scores were significantly correlated with MMD, VAS, and all domains of the corresponding PedsQL, supporting convergent validity for both instruments. Agreement between parent and child on disability grading categories of PedMIDAS was substantial (κ = 0.644).ConclusionThe Arabic version of PedMIDAS was a valid and reliable instrument to assess disability from migraine in Arabic-speaking pediatric patients with migraine. Parent reports can be valuable as a complement to child reports for a comprehensive assessment of migraine.

P467 Towards AI-Augmented Clinical Decision Making: An Examination of ChatGPT's Utility in Acute Ulcerative Colitis Presentations

Abstract Background ChatGPT is a large language model based chatbot created by OpenAI. Since its release, ChatGPT has gained widespread attention among the healthcare community regarding its potential utility as a medical practice tool. In addition, ChatGPT can be adapted to serve as a clinical-decision support tool. In this study we explored the potential of ChatGPT as a decision support tool for acute Ulcerative Colitis (UC) presentations in the setting of the emergency department (ED). Methods Our investigation centered around 20 distinct acute UC presentations to the ED, accumulated over two years. Case summaries - embodying crucial data points such as symptoms, vital signs, and laboratory results - were processed by ChatGPT. For each case, we asked ChatGPT to assess disease severity based on the TrueLove and Witts classification, substituting erythrocyte sedimentation rate ≥30 with C-Reactive protein ≥12. Furthermore, it was to recommend hospitalization or outpatient care for each case based on the disease severity. The answers were compared with assessments made by our department's gastroenterologists and the actual decision made by the physician in the ED. Results Overall, ChatGPT categorized 12, 7 and 1 patient with severe, moderate and mild disease, respectively. For each case, ChatGPT supplied a detailed answer depicting severity of every variable of the criteria and an overall severity classification (table 1). Compared to our gastroenterologists’ assessments, ChatGPT graded 16/20 (80%) of the patients with the same severity. A high degree of reliability was found between the two assessments as the average measure intra-class correlation coefficient of absolute agreement was 0.839 (95% confidence interval 0.588-0.937, F= 5.95, p&amp;lt;0.001). Inconsistencies in four cases stemmed primarily from inaccurate cut-off values for systemic variables. Following severity assessment, ChatGPT leaned towards hospitalization for 16 out of 18 (88.9%) patients. For two moderate UC cases, however, it could not provide a decisive recommendation. Comparatively, only 12 out of the 20 patients were hospitalized in actual clinical practice. Conclusion In this unique study, findings suggest that Chat-GPT, has potential as a clinical decision-support tool in assessing UC severity and recommending suitable settings for further treatment. While this concept warrants further investigation and validation, its ability to evaluate a clinical scenario based on established criteria could greatly benefit the field of Inflammatory bowel disease and gastroenterology.

PROJECT, PERSUADE, AND PROTECT: ADULT CHILDREN’S PREDICTION OF PARENTS’ HEALTH STATE VALUATIONS

Abstract Serious illness decision making for older adults frequently involves their adult children. Thus, it is important that adult children are familiar with parents’ illness preferences. In this mixed methods analysis (total N = 114), we examined concordance between older adults’ valuations of illness states, their children’s predictions of their valuations, reasoning behind children’s predictions, and children’s reactions to discordance between their predictions and parents’ valuations. Thirty-eight older parents (M age= 75.8) provided valuations for five different illness states. Two adult children (M age= 46.0) independently predicted their parent’s answers. Family triads participated in a semi-structured conversation about their responses. One-way random effects, absolute agreement, average measures intraclass correlation coefficients (ICCs) showed that average concordance within families was poor for incontinence (ICC = .172, 95% CI [-.497, .572]), dialysis (ICC = .300, 95% CI [-.264, .639]), tube feeding (ICC = .320, 95% CI [-.228, .649]), and chronic pain (ICC = .438, 95% CI [-.015, .710]), and moderate for Alzheimer disease (ICC = .556, 95% CI [.198,.771]). Results from qualitative thematic analysis indicated that children often projected their own values when attempting to predict parents’ valuations. When faced with discordance, many children attempted to persuade parents to change their valuations, while also making assertions to protect the parent’s autonomy. In summary, adult children navigating surrogate or shared decision making may face challenges inferring parent values versus their own and balancing their input with parent autonomy. Future research and interventions in late-life families could target these challenges.

The development and preliminary evaluation of the Genetic Counseling Skills Checklist.

Genetic counseling (GC) relies on communication to help people understand and adapt to genetic contributions of disease, and there is need for a practical and reliable method of comprehensively documenting GC communication skills without intensive coding. To this end, we created a novel process measure called the Genetic Counseling Skills Checklist (GCSC), utilizing previously validated measures, communication/counseling frameworks, and prior research findings. A multistage iterative process was used to develop, evaluate, and modify the GCSC to improve its clarity, usability, and content validity. To assess interrater reliability, randomly assigned, untrained individuals (i.e., coders) used the GCSC version 3 to code multiple simulated GC sessions. Average measures intraclass correlation coefficients (ICCs) were calculated for each of the 8 GCSC process categories using one-way, random effects models. After relatively minor modifications to the GCSC, two pairs of experienced coders used GCSC version 4 to independently code additional GC sessions and Cohen's Kappa coefficients (κ) were calculated to assess interrater reliability for each process category. The GCSC contains five to eight items within each category and demonstrates good content validity given its ability to capture nearly all GC skills that genetic counselors reported using in a prior qualitative study. Interrater reliability of GCSC version 3 among coders with limited experience was moderate or good for 6 out of the 8 process categories as evidenced by ICCs ranging from 0.55 to 0.86. Average interrater reliability of GCSC version 4 among one pair of experienced coders was strong for all eight process categories (κ ranging from 0.82 to 0.94); among the second pair of experienced coders scores were strong for six categories (κ ranging from 0.80 to 0.87) and moderate for two categories (κ of 0.77 and 0.78). The results suggest the need for training and experience to assure adequate interrater reliability across GCSC coders. Future work is needed to create a formalized training program for coders, complete a larger study to further validate the measure, and use the GCSC to document variability in skills used across providers and sessions.

Software compatibility analysis for quantitative measures of [18F]flutemetamol amyloid PET burden in mild cognitive impairment

RationaleAmyloid-β (Aβ) pathology is one of the earliest detectable brain changes in Alzheimer’s disease pathogenesis. In clinical practice, trained readers will visually categorise positron emission tomography (PET) scans as either Aβ positive or negative. However, adjunct quantitative analysis is becoming more widely available, where regulatory approved software can currently generate metrics such as standardised uptake value ratios (SUVr) and individual Z-scores. Therefore, it is of direct value to the imaging community to assess the compatibility of commercially available software packages. In this collaborative project, the compatibility of amyloid PET quantification was investigated across four regulatory approved software packages. In doing so, the intention is to increase visibility and understanding of clinically relevant quantitative methods.MethodsComposite SUVr using the pons as the reference region was generated from [18F]flutemetamol (GE Healthcare) PET in a retrospective cohort of 80 amnestic mild cognitive impairment (aMCI) patients (40 each male/female; mean age = 73 years, SD = 8.52). Based on previous autopsy validation work, an Aβ positivity threshold of ≥ 0.6 SUVrpons was applied. Quantitative results from MIM Software’s MIMneuro, Syntermed’s NeuroQ, Hermes Medical Solutions’ BRASS and GE Healthcare’s CortexID were analysed using intraclass correlation coefficient (ICC), percentage agreement around the Aβ positivity threshold and kappa scores.ResultsUsing an Aβ positivity threshold of ≥ 0.6 SUVrpons, 95% agreement was achieved across the four software packages. Two patients were narrowly classed as Aβ negative by one software package but positive by the others, and two patients vice versa. All kappa scores around the same Aβ positivity threshold, both combined (Fleiss’) and individual software pairings (Cohen’s), were ≥ 0.9 signifying “almost perfect” inter-rater reliability. Excellent reliability was found between composite SUVr measurements for all four software packages, with an average measure ICC of 0.97 and 95% confidence interval of 0.957–0.979. Correlation coefficient analysis between the two software packages reporting composite z-scores was strong (r2 = 0.98).ConclusionUsing an optimised cortical mask, regulatory approved software packages provided highly correlated and reliable quantification of [18F]flutemetamol amyloid PET with a ≥ 0.6 SUVrpons positivity threshold. In particular, this work could be of interest to physicians performing routine clinical imaging rather than researchers performing more bespoke image analysis. Similar analysis is encouraged using other reference regions as well as the Centiloid scale, when it has been implemented by more software packages.

Performance of an MRI scoring system for inflammation of joints and entheses in peripheral SpA: post-hoc analysis of the CRESPA trial.

The aim of this study was to investigate the reliability, validity, and sensitivity to change of a novel MRI scoring system in early peripheral SpA (pSpA). MRI of the pelvis and lower extremities was performed before initiation of the TNF inhibitor golimumab in 56 patients and repeated in 46 patients who achieved sustained clinical remission after 24, 36 or 48 weeks. Three readers applied a semi-quantitative MRI scoring system for lower-extremity joint and entheseal inflammation. Four lesion types were assessed: entheseal osteitis, entheseal soft-tissue inflammation, joint osteitis, and joint synovitis/effusion. MRI response was defined as a decrease in MRI lower-extremity inflammation index (sum of scores from 75 sites, each scored 0-3) above the smallest detectable change (SDC). At follow-up, the MRI index decreased in 34 of 46 patients (74%), and 15 (33%) patients achieved MRI response, i.e. a decrease above SDC of 2.8. When restricting the analysis to patients with clinical involvement of lower-extremity sites that were assessed by MRI, 13 of 28 (46%) achieved MRI response. Interreader reliability was very good, with an average-measure intraclass correlation coefficient of 0.92 (95% CI: 0.85-0.95) for status scores and 0.89 (0.80-0.94) for change in scores. The MRI index correlated with other measures of disease activity, including CRP [Spearman's rho 0.41 (0.23-0.56)], swollen joint count of 6 joints [0.47 (0.27-0.63)], tender enthesis count of 14 entheses [0.32 (0.12-0.50)] and pain score [0.28 (0.08-0.46)], all P < 0.05. The proposed MRI lower-extremity inflammation index demonstrated reliability, validity, and sensitivity to change in patients with early pSpA. Clinicaltrials.gov, http://clinicaltrials.gov, NCT01426815.

Correlations in radiographic and MAKO Total Knee Robotic-Assisted Surgery intraoperative limb coronal alignment.

Robotic-assisted arthroplasty has become increasingly established in recent years. The aim of the study is to determine if intraoperative coronal alignment during robotic-assisted total knee arthroplasty correlates with radiographic alignment. We prospectively compared the pre- and postoperative limb alignment values measured on long leg standing radiographs with intraoperative robotic-assisted measurements for 100 patients who underwent primary total knee arthroplasty. Two-tailed bivariate Pearson correla- tions were performed to evaluate the strength of the association between radiographic and robotic- assisted alignment. The intraclass correlation coeffi- cient (ICC) was used to estimate interrater reliability. There was a male/female ratio of 1.16 and the mean age was 67 years (range 42-88). Robotic-assisted measurements slightly overestimated the degree of varus relative to radiographs. Radiographic and robotic-assisted measurements were strongly correlated (r = 0.915, p < 0.001) preoperatively, with a difference of 1.6 ± 3.2°. The average measure ICC was 0.996 with a 95% confidence interval from 0.995 to 0.997 (p < 0.001). Postoperatively a bigger difference was measured (3.1° ± 1.9°), comparing radiographic and MAKO alignment. A moderate correlation was observed between the postoperative radiographic and MAKO outcome alignment (r = 0.604, p < 0.001). The average measure ICC was 0.977 with a 95% confidence interval from 0.967 to 0.984 (p < 0.001). There is a strong correlation in the preoperative set- ting between radiographic and robotic-assisted lower limb alignment and a moderate correlation in the post-operative setting. The values measured by the MAKO Total Knee application were considerably more in varus.

Reliability of a three-dimensional spinal proprioception assessment for patients with adolescent idiopathic scoliosis

PurposeAlthough it is evident that some patients with adolescent idiopathic scoliosis (AIS) have proprioceptive deficit in peripheral joints, knowledge on the proprioceptive function of the deformed spine is limited. Nonetheless, spinal proprioception in AIS may be affected three-dimensionally, prior studies only focussed on evaluating peripheral proprioception in single plane. Therefore, this study aimed to develop a novel spinal proprioception assessment using three-dimensional motion analysis in patients with AIS.MethodsParticipants were included if they had a primary diagnosis of AIS who did not receive or failed conservative treatments. Three trunk repositioning tests involving flexion-extension, lateral-flexion, and axial-rotation were conducted. A three-dimensional kinematics of the trunk was used as the outcome measures. The proprioceptive acuity was quantified by the repositioning error. The intra-examiner and test-retest reliability were analysed by the intraclass correlation coefficient (ICC).ResultsFifty-nine patients with AIS were recruited. Regarding the trunk flexion–extension test, the single measure ICC showed moderate reliability (0.46) and the average measures ICC demonstrated good reliability (0.72). As for the trunk lateral-flexion test, the reliability of single measure and average measures ICC was moderate (0.44) and good (0.70) reliability, respectively. For the trunk axial-rotation test, the single measure ICC indicated fair reliability (0.32), while the average measures ICC showed moderate reliability (0.59).ConclusionThis is the first study to evaluate the reliability of novel three-dimensional spinal proprioception assessments in patients with AIS. The trunk flexion-extension repositioning test may be preferable clinical test given its highest reliability.

Reliability and validity of the Slovenian Translation of the Functional Oral Intake Scale (FOIS-SI)

Background: It is crucial that each and every stroke patient gets a systematic assessment of swallowing disorders, performed by a dysphagia specialist. Therefore, there is a need to use a standardized scale among professionals. The aim of this study was to translate, adapt, and obtain the statistical characteristics of the Functional Oral Intake Scale (FOIS), which is an observer-rated dysphagia severity scale primarily developed for stroke patients, into the Slovenian language.&#x0D; Methods: We performed a 5-step translation and adaptation process. Six speech and language pathologists had evaluated 30 clinical records of patients with an acute ischemic stroke in the carotid artery territory with the Slovenian translation of the Functional Oral Intake Scale (FOIS-SI). Every clinical record included a speech and language clinical bedside assessment of dysphagia, the sex, age, National Institutes of Health Stroke Scale (NIHSS), comorbidities of patients and lesion location. We examined interrater and intrarater reliability and additionally convergent validity of FOIS-SI with the theoretically related IDDSI-Functional Diet Scale (IDDSI-FDS).&#x0D; Results: The average measure Intraclass Correlation Coefficient (ICC) for the interrater reliability (ICC=0.959) and the intrarater reliability (ICC=0.979 and ICC=0.991) for both raters was excellent. The FOIS-SI scores correlated very high with the IDDSI-FDS (rs=0.927).&#x0D; Conclusion: The results of this study show that the Slovenian translation of the FOIS has an excellent interrater and intrarater reliability and very high validity, which implies the FOIS-SI can be used as a tool to quantitatively assess the severity of dysphagia in clinical practice and research in patients with stroke in carotid artery territory across the country in all medical and care settings.

Very high interrater reliability for manual segmentation of the medial perirhinal cortex

AbstractBackgroundIntracellular neurofibrillary tangles (NFT) belong to the main characteristics of Alzheimer’s disease (AD) pathology and occur first in the medial perirhinal cortex (mPRC; also referred to as “transentorhinal region”). Adjacent regions such as lateral perirhinal cortex (lPRC), entorhinal cortex (ERC), and hippocampus are affected at a later stage of NFT progression. Therefore, mPRC integrity potentially serves as a promising imaging biomarker in early AD. Due to the high anatomic variability of the collateral sulcus, which defines mPRC boundaries, it is important to evaluate the interrater reliability of manual segmentation. Today, automated segmentation protocols only exist for the whole PRC (i.e. mPRC and lPRC together). However, discrete analysis of mPRC integrity is crucial to detect AD related pathology as early as possible.MethodForty‐four adults (20 men; mean age = 69.2 ± 10.4 years) from different diagnostic groups (e.g. normal controls, mild cognitive impairment, Alzheimer's dementia or major depression) were included in the analyses. All subjects received structural magnetic resonance imaging in a 3‐Tesla scanner (MAGNETOM Verio, Siemens). Regions of interest (mPRC, lPRC, and ERC) were drawn by‐hand by two raters (SK, NAH) blinded for diagnoses. Bilateral cortical thickness estimates were extracted using FreeSurfer. Intraclass correlation coefficients (ICC) were calculated in SPSS based on a single‐rating, absolute‐agreement, 2‐way mixed‐effects model.ResultA high degree of reliability was found between both raters with an average measure ICC of .876 for the left mPRC (F[43,43]=14.954, p&lt;.001, 95% confidence interval [CI] =.784 ‐ .930) and .873 for the right mPRC (F[43,43]=14.472, p&lt;.001, 95% CI=.779 ‐ .929). Similar results were found for lPRC (left: ICC=.984; F[43,43]=118.834, p&lt;.001, 95% CI=.970 ‐ .991; right: ICC=.953; F[43,43]=40.344, p&lt;.001, 95% CI=.915 ‐ .974) and ERC (left: ICC=.969; F[43,43]=62.657, p&lt;.001, 95% CI=.944 ‐ .983; right: ICC=.961; F[43,43]=49.390, p&lt;.001, 95% CI=.930 ‐ .979).ConclusionThe ICC analysis showed a very high interrater reliability for manual mPRC segmentation as well as for lPRC and ERC. These results indicate that the applied segmentation protocol, which considers collateral sulcus variability, allows different raters to achieve highly similar results and may serve as a reference to establish future automated segmentation protocols.

Reliability of forearm medial-anterior surface dimensional changes at different isometric hand grip forces

Indirect techniques of predicting hand grip force are fundamental to develop hand control systems for assistive devices. The purpose of this study was to determine the reliability of 3D position of forearm surface at different isometric hand grip forces. Three-dimensional motion analysis was used to measure displacement of 24 discrete and standardized surface markers placed on forearm in 20 healthy participants. The relative displacements were measured for isometric grip forces at 0%, 5%, 20% 50% and 80% of maximum voluntary contraction (MVC). Intraclass correlation coefficient (ICC) for relative radial displacement (RRD) of each marker was calculated. Averaged single measure ICC of 24 markers at five grip forces was 0.61; while the highest averaged single measure ICC (0.80) for all markers was achieved at 80% of MVC and the lowest (0.47) at 0% of MVC. The average measure ICC for each grip force across the 24 markers also increased with grip force from 0.80 at 0% of MVC to the maximum of 0.95 at 80% of MVC. In conclusion, RRD showed moderate and high ICCs for single and average measures respectively. Overall, this study suggests that the reliable dimensional changes of 3D positions of forearm surface might be considered as an indirect and non-invasive method to predict hand grip force in future.

The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources.

Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations. We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool. As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability. Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC=0.68, 95% confidence interval [CI]=0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI= 0.77 to 0.95). We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.

Interobserver reliability of canine urine specific gravity assessed by analog or digital refractometers.

Refractometry is utilized routinely to evaluate canine urine specific gravity (USG) in veterinary clinical settings. We aimed to determine if the magnitude of interobserver reliability when assessing canine USG via refractometry could impact clinical judgment. USG was determined in 38 dogs by 3 registered veterinary technicians (RVTs) using both an optical analog refractometer and a digital refractometer. Summary statistics were reported, interobserver reliability was assessed via intraclass correlation coefficient (ICC) analysis through a 2-way mixed-effects model, and agreement between RVT pairs was compared through Bland-Altman plots. The median analog refractometer USG measurement was 1.018 (range: 1.004-1.040) and for the digital refractometer was 1.0176 (1.0035-1.0357). The analog refractometer average measure ICC was 0.995 (95% CI: 0.992, 0.997; p < 0.001). The digital refractometer average measure ICC was 0.999 (95% CI: 0.999, 1.000; p < 0.001). Strong agreement between all pairs of RVTs was seen via Bland-Altman plots for both analog and digital refractometers, with 95% CIs spanning no more than 0.002 in either the positive or negative direction for all pairings. The interobserver variability in canine USG measurements by RVTs was trivial and did not impact clinical judgment and decision-making.

Inter-Observer and Intra-Observer Reliability of 2D Radiograph-Based Valgus Cut Angle Measurement in Preoperative Planning for Primary Total Knee Arthroplasty.

BackgroundPreoperative planning is imperative for a successful total knee replacement. Determining the valgus angle for the distal femoral cut from the preoperative 2D radiograph is a very inexpensive method that can be used to achieve the native knee alignment during a total knee arthroplasty (TKA).ObjectiveThe aim of this study was to document the intra-observer and inter-observer reliability of the valgus angle determination for distal femur cut from a preoperative digital radiograph in a TKA.MethodsA total of 20 patients with bilateral grade 3-4 primary osteoarthritis were assessed independently by a medical student, five surgeons, of which one was a consultant with more than 20 years of experience, and four residents in varied levels of training. Full-length (pelvis to toes) weight-bearing radiographs of both lower limbs were obtained prior to the surgery. The measurements were made thrice at more than 24 hours interval without any possible knowledge of their own previous measurement or that of other surgeons. We assessed the angle between the mechanical and anatomical axis of both femurs.ResultsThe single measures intraclass correlation was found to be 0.733, showing that there is a moderate reliability among the six raters when single measures were considered, whereas the average measures intraclass correlation was found to be 0.943, showing that there is an excellent reliability among the six raters when average measures were considered.For intra-observer reliability, the single measures intraclass correlation showed that there is a good-to-excellent reliability between the three trials when single measures was considered. The averages measures intraclass correlation showed that there is an excellent reliability between the three trials when average measures was considered.ConclusionA very inexpensive method of determining valgus angle for distal femur cut was found to have a moderate-to-excellent inter-observer reliability and a good-to-excellent intra-observer reliability.

Assessing Resident Diagnostic Skills Using a Modified Bronchiolitis Score.

Resident milestones are objective instruments that assess the resident's growth, progression in knowledge, and clinical diagnostic reasoning; but they rely on the subjective appraisal of the supervising attending. Little is known about the use of standardized instruments that may complement the evaluation of resident diagnostic skills in the academic setting. Evaluate a modified bronchiolitis severity assessment tool by appraising the inter-rater variability and reliability between pediatric attendings and pediatric residents. Cross-sectional study of children under 24 months of age who presented to a Community Hospital's Emergency Department with bronchiolitis between January-June 2014. A paired pediatric attending and resident evaluated each patient. Evaluation included age-based respiratory rate (RR), retractions, peripheral saturation, and auscultation. Cohen's kappa (K) measured inter-rater agreement. Inter-rater reliability (IRR) was assessed using a one-way random, average measures intra-class correlation (ICC) to evaluate the degree of consistency and magnitude of disagreement between inter-raters. Value of >0.6 was considered substantial for kappa and good internal consistency for ICC. Twenty patients were evaluated. Analysis showed fair agreement for the presence of retractions (K=0.31), auscultation (K=0.33), and total score (K=0.3). The RR (ICC=0.97), SpO2 (ICC=1.0), auscultation (ICC=0.77), and total score (ICC=0.84) were scored similarly across both raters, indicating excellent IRR. Identification of retractions had the least agreement across all statistical analysis. The use of a standardized instrument, in conjunction with a trained resident-teaching staff, can help identify deficiencies in clinical competencies among residents and facilitate the learning process for the identification of pertinent clinical findings.