Substantial Inter-rater Reliability Research Articles

Dose reduction potential in cone-beam CT imaging of upper extremity joints with a twin robotic x-ray system

Cone-beam computed tomography is a powerful tool for 3D imaging of the appendicular skeleton, facilitating detailed visualization of bone microarchitecture. This study evaluated various combinations of acquisition and reconstruction parameters for the cone-beam CT mode of a twin robotic x-ray system in cadaveric wrist and elbow scans, aiming to define the best possible trade-off between image quality and radiation dose. Images were acquired with different combinations of tube voltage and tube current–time product, resulting in five scan protocols with varying volume CT dose indices: full-dose (FD; 17.4 mGy), low-dose (LD; 4.5 mGy), ultra-low-dose (ULD; 1.15 mGy), modulated low-dose (mLD; 0.6 mGy) and modulated ultra-low-dose (mULD; 0.29 mGy). Each set of projection data was reconstructed with three convolution kernels (very sharp [Ur77], sharp [Br69], intermediate [Br62]). Five radiologists subjectively assessed the image quality of cortical bone, cancellous bone and soft tissue using seven-point scales. Irrespective of the reconstruction kernel, overall image quality of every FD, LD and ULD scan was deemed suitable for diagnostic use in contrast to mLD (very sharp/sharp/intermediate: 60/55/70%) and mULD (0/3/5%). Superior depiction of cortical and cancellous bone was achieved in FDUr77 and LDUr77 examinations (p < 0.001) with LDUr77 scans also providing favorable bone visualization compared to FDBr69 and FDBr62 (p < 0.001). Fleiss’ kappa was 0.618 (0.594–0.641; p < 0.001), indicating substantial interrater reliability. In this study, we demonstrate that considerable dose reduction can be realized while maintaining diagnostic image quality in upper extremity joint scans with the cone-beam CT mode of a twin robotic x-ray system. Application of sharper convolution kernels for image reconstruction facilitates superior display of bone microarchitecture.

The "CIV Classification," a New Proposal for the Architectural Grading of Vulvar Lichen Sclerosus.

The purpose of this cross-sectional study was to prepare a reliable and easy-to-use architectural classification for vulvar lichen sclerosus (VLS) aimed at defining the morphological patterns of this condition. An expert panel composed by 7 physicians with expertise in clinical care of vulvar conditions outlined the architectural criteria for the definition of VLS severity (phimosis of the clitoris, involvement of the interlabial sulci, narrowing of the vulvar introitus), identifying 5 grades to build up a classification. Thirteen physicians with 2-30 years expertise in vulvar diseases (nonexpert group) were asked to evaluate 3-5 pictures from 137 patients. Each physician individually assigned a grade to each case, according to the previously mentioned criteria. Interrater reliability was analyzed by means of intraclass correlation coefficient (ICC). The reliability concerning the 2 classifications of each rater was analyzed by means of κ statistic. Intraobserver and interobserver reliability in vivo was analyzed by means of κ index. This study provides a new classification of VLS, based on defined anatomical criteria and graded into mutually exclusive progressive classes.The ICC analysis showed a substantial interrater reliability of the classification, ICC = 0.89 (0.87-0.91), both in the expert panel and in the nonexpert group (ICC = 0.92 and 0.87, respectively). An "almost perfect" intraobserver and interobserver reliability was achieved among physicians in vivo (κ = 0.93). Our classification showed a high reliability. It is easy to use, and it can be applied in clinical practice and eventually, in the evaluation of regenerative and cosmetic surgery.

Inter-rater Reliability and Construct Validity of the Lerner Defense Scale in Clinical and Non-clinical Groups.

The current study investigated the inter-rater reliability and the construct validity of the Rorschach Lerner Defense Scale (LDS). In particular, it aimed to explore the inter-rater reliability, analyzing the most frequent coding mistakes in an attempt to improve the coding guidelines, and to investigate the ability of the scale to distinguish between individuals with neurotic-level and borderline-level personality organization, according to the Psychodynamic Diagnostic Manual-2 (PDM-2), and non-clinical subjects. Eighty clinical subjects and 80 non-clinical ones participated in the study. Among the clinical subjects, 40 have borderline-level personality organization and 40 have neurotic-level personality organization. Non-clinical subjects were drawn from an archival dataset of non-clinical individuals who previously participated in a Rorschach normative study. The LDS showed substantial inter-rater reliability; however, guidelines could be improved, specifically with regard to the threshold for coding Devaluation and Idealization at level 1. Furthermore, more examples should be included in the manual about the coding of Projective Identification and Denial. The LDS distinguished borderline-level subjects from both the non-clinical and neurotic groups with regard to Devaluation and Projective Identification, with borderline-level personality organization subjects reporting higher scores than either of the two other groups. Only the Denial scale discriminated between the non-clinical and neurotic group, with the latter reporting higher scores of high-level Denial.

7T MRI Differentiates Remyelinated from Demyelinated Multiple Sclerosis Lesions.

ObjectiveTo noninvasively assess myelin status in chronic white matter lesions of multiple sclerosis (MS), we developed and evaluated a simple classification scheme based on T1 relaxation time maps derived from 7‐tesla postmortem and in vivo MRI.MethodsUsing the MP2RAGE MRI sequence, we classified 36 lesions from 4 postmortem MS brains as “long‐T1,” “short‐T1,” and “mixed‐T1” by visual comparison to neocortex. Within these groups, we compared T1 times to histologically derived measures of myelin and axons. We performed similar analysis of 235 chronic lesions with known date of onset in 25 MS cases in vivo and in a validation cohort of 222 lesions from 66 MS cases, investigating associations with clinical and radiological outcomes.ResultsPostmortem, lesions classified qualitatively as long‐T1, short‐T1, and mixed‐T1 corresponded to fully demyelinated, fully remyelinated, and mixed demyelinated/remyelinated lesions, respectively (p ≤ 0.001). Demyelination (rather than axon loss) dominantly contributed to initial T1 prolongation. We observed lesions with similar characteristics in vivo, allowing manual classification with substantial interrater and excellent intrarater reliability. Short‐T1 lesions were most common in the deep white matter, whereas long‐T1 and mixed‐T1 lesions were prevalent in the juxtacortical and periventricular white matter (p = 0.02) and were much more likely to have paramagnetic rims suggesting chronic inflammation (p < 0.001). Older age at the time of lesion formation portended less remyelination (p = 0.007).Interpretation7‐tesla T1 mapping with MP2RAGE, a clinically available MRI method, allows qualitative and quantitative classification of chronic MS lesions according to myelin content, rendering straightforward the tracking of lesional myelination changes over time. ANN NEUROL 2021;90:612–626

P38. A novel fusion classification system for the advent of bioactive interbody implants

BACKGROUND CONTEXT Historically, intervertebral fusion and biologically-mediated stability required bone bridging exclusively. The advent of bioactive interbody devices, which allow for appositional integration of bone at the endplate-implant interface, demands an evolution in our radiographic assessment of such fusion processes. Specifically, acknowledgement that endplate bone adhesion to the surface of these implant confers stability, and potentially an effective ‘fusion’, warrants strong consideration within a new classification scheme. PURPOSE We present a novel interbody fusion classification system that addresses inadequacies if traditional grading schemes in order to accommodate the integration capability of bioactive intervertebral cages. STUDY DESIGN/SETTING Classification System Reliability Survey. PATIENT SAMPLE A total of 37 surgeon respondents. OUTCOME MEASURES Intraclass correlation coefficients for inter- and intra-rater reliability assessments. METHODS A novel grading system for intervertebral fusion was developed by the authors (Figure) and included fusion consolidation through and/or around the intervertebral device, as well as the apposition of bone to the surface of the implants (ingrowth/ongrowth). Ten patient cases with 1year CT scans were selected for the survey, including coronal and sagittal reconstruction views which were provided to the respondents. Surveys were administered twice >2weeks apart and with cases in differential order. Intraclass Correlation Coefficient (ICC) was used to calculate inter and intrarater reliability. RESULTS A total of 51 spine surgeons participated, with 37 of them completing surveys on two separate occasions. For the entire cohort, ICC inter-rater reliability (95% CI) average was 0.618 (0.435-0.834; p<0.001), while the intra-rater reliability average was 0.778 (SD=0.211). Four respondents had outlier intra-rater reliability scores averaged 0.259. When removing those 4 respondents (10.8% of the respondents), and analyzing the more consistent 90% of the respondents, ICC inter-rater reliability average was 0.628 (0.445-0.841; p=<0.001), and intra-rater reliability average was 0.831 (SD=0.125). CONCLUSIONS Our novel CT-based interbody fusion classification system demonstrated substantial inter-rater reliability, and excellent intra-rater reliability when applied to a spectrum of real cases. Future assessment of interbody fusions should consider utilization of this new system in order to address the important contribution of appositional ongrowth/ingrowth, along with traditional bone healing through and around devices. FDA DEVICE/DRUG STATUS This abstract does not discuss or include any applicable devices or drugs. Historically, intervertebral fusion and biologically-mediated stability required bone bridging exclusively. The advent of bioactive interbody devices, which allow for appositional integration of bone at the endplate-implant interface, demands an evolution in our radiographic assessment of such fusion processes. Specifically, acknowledgement that endplate bone adhesion to the surface of these implant confers stability, and potentially an effective ‘fusion’, warrants strong consideration within a new classification scheme. We present a novel interbody fusion classification system that addresses inadequacies if traditional grading schemes in order to accommodate the integration capability of bioactive intervertebral cages. Classification System Reliability Survey. A total of 37 surgeon respondents. Intraclass correlation coefficients for inter- and intra-rater reliability assessments. A novel grading system for intervertebral fusion was developed by the authors (Figure) and included fusion consolidation through and/or around the intervertebral device, as well as the apposition of bone to the surface of the implants (ingrowth/ongrowth). Ten patient cases with 1year CT scans were selected for the survey, including coronal and sagittal reconstruction views which were provided to the respondents. Surveys were administered twice >2weeks apart and with cases in differential order. Intraclass Correlation Coefficient (ICC) was used to calculate inter and intrarater reliability. A total of 51 spine surgeons participated, with 37 of them completing surveys on two separate occasions. For the entire cohort, ICC inter-rater reliability (95% CI) average was 0.618 (0.435-0.834; p<0.001), while the intra-rater reliability average was 0.778 (SD=0.211). Four respondents had outlier intra-rater reliability scores averaged 0.259. When removing those 4 respondents (10.8% of the respondents), and analyzing the more consistent 90% of the respondents, ICC inter-rater reliability average was 0.628 (0.445-0.841; p=<0.001), and intra-rater reliability average was 0.831 (SD=0.125). Our novel CT-based interbody fusion classification system demonstrated substantial inter-rater reliability, and excellent intra-rater reliability when applied to a spectrum of real cases. Future assessment of interbody fusions should consider utilization of this new system in order to address the important contribution of appositional ongrowth/ingrowth, along with traditional bone healing through and around devices.

The interrater and intrarater reliability of the flexibility and strength tests included in the Sport Science Lab® screening protocol amongst professional rugby players.

BackgroundConsidering the injury incidence rate (IR) associated with elite-level rugby, measures to reduce players’ injury risk are important. Establishing scientifically sound, pre-season musculoskeletal screening protocols forms part of injury prevention strategies.ObjectiveTo determine the interrater and intrarater reliability of the flexibility and strength tests included in the Sport Science Lab® (SSL) screening protocol.MethodsWe determine the interrater and intrarater reliability of 11 flexibility and nine strength tests. Twenty-four injury-free, elite, adult (> 18 years), male rugby players were screened by two raters on two occasions. To establish intrarater and interrater reliability, Gwet’s AC1, AC2 and intraclass correlation coefficients (ICC) were used for the analysis of binary, ordinal and continuous variables, respectively. Statistical significance was set at 95%.ResultsFlexibility tests which require lineal measurement had at least substantial interrater (ICC = 0.70–0.96) and intrarater reliability (ICC = 0.89–0.97). Most of the flexibility tests with binary outcomes attained almost perfect interrater and intrarater reliability (Gwet’s AC1 = 0.8–0.97). All strength tests attained at least substantial interrater (Gwet’s AC2 = 0.73–0.96) and intrarater (Gwet’s AC2 = 0.67–0.97) reliability.ConclusionThe level of interrater and intrarater reliability of most of the flexibility and strength tests investigated supports their use to quantify various aspects of neuromusculoskeletal qualities and possible intrinsic risk factors amongst elite rugby players.Clinical implicationsEstablishing the reliability of tests, is one step to support the inclusion thereof in official screening protocols. Results of our study, verify the reliability of the simple, clinically friendly strength and flexibility tests included and therefore support their use as preparticipation screening tools for rugby players. Further investigation as to the association thereof to athletes’ injury risk and performance is warranted.

Surgeon vs Pathologist for Prediction of Pancreatic Fistula: Results from the Randomized Multicenter RECOPANC Study

Surgically assessed pancreatic texture has been identified as the strongest predictor of postoperative pancreatic fistula. However, texture is a subjective parameter with no proven reliability or validity. Therefore, a more objective parameter is needed. In this study, we evaluated the fibrosis level at the pancreatic neck resection margin and correlated fibrosis and all clinico-pathologic parameters collected over the course of the Pancreatogastrostomy vs Pancreatojejunostomy for RECOnstruction (RECOPANC) study. The RECOPANC trial was a multicenter randomized prospective trial of patients undergoing pancreatoduodenectomy. There were 261 hematoxylin and eosin-stained slides allocated for histopathologic analyses. Pancreatic fibrosis was scored from 0 to III (no fibrosis up to severe fibrosis) by 2 blinded independent pathologists. All variables possibly associated with POPF were entered into a generalized linear model for multivariable analysis. The fibrosis grade and pancreatic texture were scored in all 261 patients. In POPF B/C (postoperative pancreatic fistula grade B or C) patients, 71% had a soft pancreas, and fibrosis grades were distributed as follows: 48% with score 0, 28% with score I, 20% with score II, and 7% with score III, respectively. Fibrosis grading showed substantial inter-rater reliability (kappa= 0.74) and correlated positively with hard pancreatic texture (p < 0.05). In univariable analysis, area under the curve (AUC) for POPF B/C prediction was higher for fibrosis grade than for pancreatic texture (0.71 vs 0.59). In multivariate analysis, the following predictors were selected: sex, surgeon volume, pancreatic texture, and fibrosis grade. However, the addition of pancreatic texture only led to an incremental improvement (AUC 0.794 vs 0.819). Histologically evaluated pancreatic fibrosis is an easily applicable and highly reproducible POPF predictor and superior to surgically evaluated pancreatic texture. Future studies might use fibrosis grade for risk stratification in pancreatoduodenectomy.

The PCOC Symptom Assessment Scale (SAS): A valid measure for daily use at point of care and in palliative care programs.

BackgroundVery few measures are used successfully as part of routine care within national palliative care outcome programs. Only a handful of studies examine these measures. The aim of this study is to evaluate the validity of a measure used in a national outcomes program: the Palliative Care Outcomes Collaboration Symptom Assessment Scale (PCOC SAS).MethodsA retrospective multi-site cohort study with secondary analysis of routinely collected patient-level data to assess PCOC SAS’s internal consistency, construct validity, reliability, interpretability, acceptability and sensitivity. The analyses used two sets, with data collected by inpatient and community palliative care services registered with the Australian national PCOC.ResultsDataset one included 1,117 patients receiving palliative care from 21 services. Dataset two included 5,294 patients receiving palliative care from 119 PCOC services. PCOC SAS demonstrated the ability to detect and discriminate distress by palliative care phase, functional status and diagnosis. Excellent and good convergent and discriminant validity were demonstrated. Fair through to substantial inter-rater and intra-rater reliability levels were evidenced. Sufficient interpretability resulted along with necessary levels of acceptability and sensitivity.ConclusionPCOC SAS is a valid and reliable patient-reported outcome measure suitable for use in routine clinical care with patients requiring palliative and or end-of-life care, including in national outcomes programs.

Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies

BackgroundIn the context of the volume of mixed- and multi-methods studies in health services research, the present study sought to develop an appraisal tool to determine the methodological and reporting quality of such studies when included in systematic reviews. Evaluative evidence regarding the design and use of our existing Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was synthesised to enhance and refine it for application across health services research.MethodsSecondary data were collected through a literature review of all articles identified using Google Scholar that had cited the QATSDD tool from its inception in 2012 to December 2019. First authors of all papers that had cited the QATSDD (n=197) were also invited to provide further evaluative data via a qualitative online survey. Evaluative findings from the survey and literature review were synthesised narratively and these data used to identify areas requiring refinement. The refined tool was subject to inter-rater reliability, face and content validity analyses.ResultsKey limitations of the QATSDD tool identified related to a lack of clarity regarding scope of use of the tool and in the ease of application of criteria beyond experimental psychological research. The Quality Appraisal for Diverse Studies (QuADS) tool emerged as a revised tool to address the limitations of the QATSDD. The QuADS tool demonstrated substantial inter-rater reliability (k=0.66), face and content validity for application in systematic reviews with mixed, or multi-methods health services research.ConclusionOur findings highlight the perceived value of appraisal tools to determine the methodological and reporting quality of studies in reviews that include heterogeneous studies. The QuADS tool demonstrates strong reliability and ease of use for application to multi or mixed-methods health services research.

Assessing fidelity of a community based psychosocial intervention for people with mild dementia within a large randomised controlled trial

BackgroundUnderstanding intervention delivery as intended, particularly in complex interventions, should be underpinned by good quality fidelity assessment. We present the findings from a fidelity assessment embedded as part of a trial of a complex community-based psychosocial intervention, Journeying through Dementia (JtD). The intervention was designed to equip individuals with the knowledge and skills to successfully self-manage, maintain independence, and live well with dementia and involves both group and individual sessions. The methodological challenges of developing a conceptual framework for fidelity assessment and creating and applying purposely designed measures derived from this framework are discussed to inform future studies.MethodsA conceptual fidelity framework was created out of core components of the intervention (including the intervention manual and training for delivery), associated trial protocols and pre-defined fidelity standards and criteria against which intervention delivery and receipt could be measured. Fidelity data collection tools were designed and piloted for reliability and usability. Data collection in four selected sites (fidelity sites) was via non-participatory observations of the group aspect of the intervention, attendance registers and interventionist (facilitator and supervisor) self-report.ResultsInterventionists from all four fidelity sites attended intervention training. The majority of group participants at the four sites (71%) received the therapeutic dose of 10 out of 16 sessions. Weekly group meeting attendance (including at ‘out of venue’ sessions) was excellent at 80%. Additionally, all but one individual session was attended by the participants who completed the intervention. It proved feasible to create tools derived from the fidelity framework to assess in-venue group aspects of this complex intervention. Results of fidelity assessment of the observed groups were good with substantial inter-rater reliability between researchers KAPPA 0.68 95% CI (0.58–0.78). Self-report by interventionists concurred with researcher assessments.ConclusionsThere was good fidelity to training and delivery of the group aspect of the intervention at four sites. However, the methodological challenges of assessing all aspects of this complex intervention could not be overcome due to practicalities, assessment methods and ethical considerations. Questions remain regarding how we can assess fidelity in community-based complex interventions without impacting upon intervention or trial delivery.Trial registrationISRCTN17993825.

Respiratory Distress Observation Scale Italian Version: Cultural-Linguistic Validation and Psychometric Properties.

Dyspnea can be assessed using self-rating scales but, as death approaches, self-reporting becomes difficult. The validated Respiratory Distress Observation Scale measures dyspnea distress. The aim of this study was to develop the Italian version of the Respiratory Distress Observation Scale and to examine its psychometric properties. This was a cross-sectional study, analyzing cultural and linguistic validation, content validity, and psychometric properties. Eighty-nine palliative care subjects were enrolled to validate the Italian version of the Respiratory Distress Observation Scale. Patients had an average age of 74.5 (SD, 11.6) years, and 52% (n = 46) were female. Fourteen experts in palliative care evaluated the Italian Respiratory Distress Observation Scale in terms of the content validity ratio and the content validity index (CVI). The Italian Respiratory Distress Observation Scale was reported with an internal consistency (Cronbach α value) of .72 and an overall substantial interrater reliability (Cohen κ method). The Italian Respiratory Distress Observation Scale achieved a scale-level CVI of 93%, an items-level CVI of greater than 86%, and a minimum content validity ratio value of 0.71. A weak positive correlation was found between the Respiratory Distress Observation Scale and the Dyspnea Visual Analog Scale scores (0.374; P < .001). The Italian Respiratory Distress Observation Scale showed good reliability and validity for patients in palliative care. The Respiratory Distress Observation Scale measured respiratory distress in patients nearest to death.

Breast Implant-Associated Anaplastic Large Cell Lymphoma Online Education Tools: A Multimetric Analysis.

In light of highly publicized media coverage on breast implant recalls and Food and Drug Administration hearings on breast implant safety, online searches of these topics have surged. It is thus critical to determine whether such searches are providing meaningful information for those who use them. Patient/laywomen-directed online education materials on breast implant-associated anaplastic large cell lymphoma (BIA-ALCL) must not only be comprehensible but also accurate, actionable, and culturally concordant, especially as more diverse populations turn to the internet for breast implant-related information. This study assessed the overall suitability of BIA-ALCL patient-directed educational text and video-based materials online. This was a cross-sectional, multimetric investigation of online text- and video-based patient-directed educational materials for BIA-ALCL using multiple validated measures. Two reviewers assessed each resource. Kruskal-Wallis and Fisher exact analyses were used as appropriate to compare text- and video-based online resources. In total, 30 websites and 15 videos were evaluated, more than half (56%) of which were from academic/physician or governmental/nonprofit sources. Overall, website and video content, as well as quality, varied by source. Academic/physician or governmental/nonprofit sources tended to be more comprehensive and have higher-quality information than commercial/media outlet sources. Median actionability of websites was 38%, well below the threshold of 70% used in the literature to identify actionable resources. The median suitability score for BIA-ALCL websites was 57%, which is considered "adequate" based on published thresholds. The mean overall Cultural Sensitivity Assessment Tool score for websites was 2.4; Cultural Sensitivity Assessment Tool scores higher than 2.5 are generally regarded as culturally sensitive. In general, videos were more understandable than websites. Substantial interrater reliability across the validated tools used in this study was noted using Pearson correlation coefficients. Online resources varied in content and quality by source. As BIA-ALCL becomes an increasingly salient topic among both providers and patients, it is important to empower women with accurate information about this implant-associated cancer. Of available resources, providers should refer patients or those seeking more information to websites from governmental/academic organizations (".gov" or ".org" domains) and videos from academic/physician or governmental sources, given that among high-quality resources, these were most clear/comprehensible. Overall, there is a need for improved online content on this topic.

Inter-Rater Reliability of the FAST-ED in the Out-of-Hospital Setting

Introduction: Patients experiencing a large vessel occlusion stroke (LVOS) may require endovascular-capable centers and benefit from direct transport to such facilities, creating a need for an accurate prehospital assessment. The Field Assessment Stroke Triage for Emergency Destination (FAST-ED) is a secondary scale to identify LVOS. Currently, there is limited prospective evidence validating the use of the FAST-ED in the prehospital environment. This study aimed to evaluate the inter-rater reliability of the FAST-ED between patient care providers in the prehospital setting. Methods: This prospective study was conducted between 4/1/2018 and 7/1/2018 in a single municipal EMS agency that staffs two providers per ambulance with at least one being a paramedic. Patients were included based on paramedic impression that the patient was both having a stroke and greater than 18 years old. Each provider independently performed and documented a FAST-ED assessment on eligible patients. Data analysis consisted of performing inter-rater reliability using Cohen’s Kappa on the FAST-ED score between primary and secondary providers. The FAST-ED was analyzed on an item level, an aggregate level (cumulative of all items), and using the defined cut point of ≥4. A sub-analysis determined if inter-rater reliability changed across provider certification. Results: There were 231 patients included in this analysis with an average age of 68.5 years and 135 (58.4%) female. Inter-rater reliability varied across individual items in the scale from 90.1% agreement to 82.5%. When analyzing inter-rater reliability of the aggregate FAST-ED score, the scale demonstrated 70.1% agreement (Kappa 0.66), considered substantial agreement. FAST-ED scores were analyzed using a cut point of ≥4. When using this cut point, there was 92.2% (Kappa 0.81) agreement between primary and secondary caregiver, demonstrating almost perfect agreement. Agreement was substantial across provider certifications including paramedics and EMTS. Conclusion: This study demonstrated high inter-rater reliability of the FAST-ED scale when performed in the prehospital setting on patients suspected of having a stroke. There were minimal differences in reliability based on provider certification, and item level analysis indicated substantial inter-rater reliability.

Interobserver Agreement of Lung Ultrasound Findings of COVID-19.

BackgroundLung ultrasound (LUS) has received considerable interest in the clinical evaluation of patients with COVID‐19. Previously described LUS manifestations for COVID‐19 include B‐lines, consolidations, and pleural thickening. The interrater reliability (IRR) of these findings for COVID‐19 is unknown.MethodsThis study was conducted between March and June 2020. Nine physicians (hospitalists: n = 4; emergency medicine: n = 5) from 3 medical centers independently evaluated n = 20 LUS scans (n = 180 independent observations) collected from patients with COVID‐19, diagnosed via RT‐PCR. These studies were randomly selected from an image database consisting of COVID‐19 patients evaluated in the emergency department with portable ultrasound devices. Physicians were blinded to any patient information or previous LUS interpretation. Kappa values (κ) were used to calculate IRR.ResultsThere was substantial IRR on the following items: normal LUS scan (κ = 0.79 [95% CI: 0.72–0.87]), presence of B‐lines (κ = 0.79 [95% CI: 0.72–0.87]), ≥3 B‐lines observed (κ = 0.72 [95% CI: 0.64–0.79]). Moderate IRR was observed for the presence of any consolidation (κ = 0.57 [95% CI: 0.50–0.64]), subpleural consolidation (κ = 0.49 [95% CI: 0.42–0.56]), and presence of effusion (κ = 0.49 [95% CI: 0.41–0.56]). Fair IRR was observed for pleural thickening (κ = 0.23 [95% CI: 0.15–0.30]).DiscussionMany LUS manifestations for COVID‐19 appear to have moderate to substantial IRR across providers from multiple specialties utilizing differing portable devices.The most reliable LUS findings with COVID‐19 may include the presence/count of B‐lines or determining if a scan is normal. Clinical protocols for LUS with COVID‐19 may require additional observers for the confirmation of less reliable findings such as consolidations.

Validation of the acoustic breathiness index to the Brazilian Portuguese language

Objective To assess the concurrent validity and the diagnostic accuracy of the Acoustic Breathiness Index (ABI) in Brazilian Portuguese. Methods The counting numbers 1–20 and the vowel /a/ of 150 subjects were recorded (37 vocally healthy and 113 with dysphonia). The analyzed samples were the counting number 1–11 and 3 s of the sustained vowel. Nine voice specialists performed the perceptual judgment of the degree of breathiness. The Spearman Correlation and the receiver operating characteristic (ROC) curve were used to assess ABI’s concurrent validity and diagnosis accuracy. Results Results from five listeners were chosen for the study analyses due to moderate and substantial intra-rater reliability (Cohen’s Kappa values = 0.520–0.772) and moderate inter-rater reliability (Fleiss Kappa = 0.353). The ABI presented a high concurrent validity (r = 0.746); 55.6% of the breathiness vocal deviation can be explained by the acoustic analysis (r 2 = 0.556). The ROC curve presented good diagnostic accuracy (85.2%). At a threshold of 2.94, the sensitivity was 75.3% and the specificity was 93.4%. Conclusion The ABI is a valid tool for screening and patient’s follow-up regarding breathy vocal qualities in the Brazilian Portuguese language.

Moderate to Substantial Inter-Rater Reliability in the Assessment of Cranial Bone Mobility Restrictions.

Background: The World Health Organization benchmarks for osteopathic training consider cranial osteopathy as an important manual skill. Studies of cranial manual therapy have exhibited poor reliability. The aim of this study was to investigate the inter-rater reliability of the manual mobility tests of the spheno-occipital synchondrosis (SOS), and the temporal, parietal, and frontal bones, as assessed in osteopathic manual therapy. Methods: Twenty-one adults were assessed on a single day by three experienced osteopaths using a standard assessment protocol. Before data collection, the osteopaths participated in a consensus training, which included establishing the criteria for identifying a cranial bone mobility restriction; the application of the seven-step palpation method; a pretesting practice; a fine-tuning palpation training; and a calibration period before the assessment of the subjects. Three subjects were assessed simultaneously with the evaluators rotating to assess each subject. The evaluators were blinded to the subject by a curtain, and each other's assessments. Each bone was rated as restricted or not restricted. The authors applied the Landis and Koch classification to describe the magnitude of inter-rater reliability. Results: Moderate reliability was established for a lateral strain of the SOS (Fleiss' generalized kappa 0.48), substantial reliability was established for the other SOS strain patterns (Fleiss' generalized kappa 0.62-0.75), and almost perfect reliability for temporal, parietal, and frontal bone (Fleiss' generalized kappa 0.81-0.96). Conclusion: The results demonstrate consistency when three experienced osteopaths evaluate cranial bone mobility restrictions. The results highlight the importance of consensus training and rigorous methodology in manual therapy reliability studies.

Measuring Caregiver Person-Centered Communication for Persons With Dementia via Video Observations

Communication is fundamental for daily care activities in nursing homes (NHs). Second-by-second behavioral coding of video observations is an ideal approach to examine the interactive nature of communication but requires a reliable coding scheme. Recent studies have adapted the Peron-Centered Behavioral Inventory (PCBI) and Task-Centered Behavioral Inventory (TCBI) to analyze caregiver communication during mealtime interactions, but their use for coding general daily caregiving activities has not been widely evaluated. This pilot study adapted the PCBI and TCBI of video observations and determined their inter-rater reliability for measuring caregiver verbal communication with persons with dementia (PwD). We analyzed videos from a randomized controlled trial of an intervention to improve caregiver communication in NHs. We selected one 1-minute segment from 12 videos that included interactions of caregiver-resident dyads. One research assistant transcribed caregivers’ verbal communication and segmented the communication into utterances. Two other research assistants independently coded each utterance using the adapted PCBI and TCBI. The coding scheme was expanded by modifying the existing operational definitions, adding three new codes, and developing a coding decision guide. Residents were Caucasian (100%), mean age 86 years with dementia and resistive behaviors. The adapted PCBI and TCBI had an inter-rater reliability of Kappa=0.656 (p<.001) across the 12 videos. Overall, our adapted PCBI and TCBI showed substantial inter-rater reliability. The results support the use of our adapted PCBI and TCBI to distinguish between person-centered and task-centered communication in video observations, which, in turn, allows for sequential analysis to examine the impact of caregiver communication on PwD.

Inter-Rater Reliability of a Pressure Injury Risk Assessment Scale for Home Care: A Multicenter Cross-Sectional Study.

PurposeThe aim of the current study was to assess the inter-rater reliability and agreement of the Pressure Injury Primary Risk Assessment Scale for Home Care (PPRA-Home), a risk assessment scale recently developed for Japan-specific social welfare professionals called care managers, to predict pressure injury risk in geriatric individuals who require long-term home care needs.MethodsA multicenter cross-sectional study was conducted at 30 home-based geriatric support services facilities located at four local districts in Japan. Eligible participants were individuals who needed partial or full assistance for daily living under Japan’s long-term care insurance system (care levels 1–5). The degree of agreement and kappa coefficient were calculated for each item and the total score, after which inter-rater reliability was determined. The effect of the participant’s care level on reliability was also evaluated as secondary analysis.ResultsA total of 96 participants were assessed by 83 care managers (two assessors scored each participant). The degree of agreement and calculated kappa coefficient of the PPRA-Home total score were 59% and 0.72, respectively, with the inter-rater reliability for the total score determined to be “Substantial”. Our subgroup analysis showed that the inter-rater reliability differed according to the participant’s care level. Accordingly, the kappa coefficient for the total score was lower in subgroup “care level 1–3” than in subgroup “care level 4–5” (0.51 and 0.76, respectively).ConclusionOur result showed that the PPRA-Home has substantial inter-rater reliability for evaluation of risks of pressure injury development at home care. However, some research focusing on intra-later reliability and validity of the PPRA-Home with adequate sample sizes are required to provide categorical conclusions on whether it can be used for the risk assessment scale in actual clinical settings.

A Paging Training Program for a Fourth-Year Internship Readiness Course.

IntroductionNew medical interns face a steep learning curve as they must manage complex medical scenarios, many of which they have only seen before in a classroom setting. To ameliorate these challenges, medical schools are increasingly including courses designed to address the transition from student to doctor. One of the biggest challenges for new interns is learning to triage and manage nursing pages, so we designed a mock paging program incorporated within our fourth-year transitions course.MethodsWe developed a database of clinical scenarios to occur via telephone between a nurse and a medicine intern. Throughout the 2-week course, these cases were administered to 40 fourth-year medical students by Master's level nursing students and nurse evaluators. The nurses used checklists to evaluate medical student management and communication, and at the end of the phone encounter students received immediate feedback. We used an observational prospective design, using a within subjects method with repeated measures.ResultsData from a total of 216 phone calls were analyzed for 36 students. No statistically significant improvement of checklist scores was observed. Substantial interrater reliability was observed for the four observed cases with a Fleiss-Kappa of .76. Student comments indicated the activity was helpful for preparing them to answer pages.DiscussionOur paging program offered students the chance to simulate being on call, as well as the opportunity to receive immediate feedback. It did not show improvement in checklists across time. Limitations included a small sample size and few common variables across the cases.

Interobserver Agreement of Lung Ultrasound Findings of COVID-19

Background: Lung ultrasound (LUS) has received considerable interest in the clinical evaluation of patients with COVID-19. Previously described LUS manifestations for COVID-19 include B-lines, consolidations, and pleural thickening. The interrater reliability (IRR) of these findings for COVID-19 is unknown.Methods: This study was conducted at conducted at two academic medical centers between 03/2020-06/2020. Nine physicians (hospitalists: n=4; emergency medicine: n=5) independently evaluated n=20 LUS scans (n=180 independent observations) collected from RT-PCR confirmed COVID-19 patients. These studies were randomly selected from an image database consisting of COVID-19 patients evaluated in the emergency department with portable ultrasound devices. Physicians were blinded to any patient information or previous LUS interpretation. Kappa values (κ) were used to calculate IRR.Findings: There was substantial IRR on the following items: normal LUS scan (κ=0.79 [95% CI: 0.72-0.87]), presence of B-lines (κ=0.79 [95% CI: 0.72-0.87]), >=3 B-lines observed (κ=0.72 [95% CI: 0.64-0.79]). Moderate IRR was observed for the presence of any consolidation (κ=0.57 [95% CI: 0.50-0.64]), subpleural consolidation (κ=0.49 [95% CI: 0.42-0.56]), and presence of effusion (κ=0.49 [95% CI: 0.41-0.56]). Fair IRR was observed for pleural thickening (κ=0.23 [95% CI: 0.15-0.30]).Interpretation: Many LUS manifestations for COVID-19 appear to have moderate to substantial IRR across providers from multiple specialties utilizing differing portable devices.Trial Registration: This analysis was conducted as part of an ongoing study investigating the role of POCUS for COVID-19 that is being conducted at four medical centers in the United States (Clinicaltrials.gov Registration: NCT04384055). Funding Statement: None.Declaration of Interests: Andre Kumar, MD, MEd is a paid consultant for Vave Health, which manufactures one of the ultrasound devices used in this study. His consultant duties include providing feedback on product development. The other authors do not have any items to disclose.Ethics Approval Statement: This study has received institutional review board (IRB) approval at all participating sites.