Low Inter-rater Reliability Research Articles

Assessing Fears of Negative Consequences in Children With Symptoms of Avoidant Restrictive Food Intake Disorder.

Fear of Aversive Consequences (FOAC), such as choking or vomiting, is an important associated feature of Avoidant/Restrictive Food Intake Disorder (ARFID). However, the manifestation of FOAC in young children is poorly understood. This study aimed to describe the fears of children with ARFID symptoms and examine the concordance between parent and child ratings of fear. Child-reported FOAC was assessed using an interview designed for children between 6 and 10 years old, the Gustatory Avoidance and Gastrointestinal Stress Symptoms (GAGSS). Parents were administered a semi-structured diagnostic interview regarding their child's symptoms, the Pica, ARFID, and Rumination Interview. Among 68 children with ARFID diagnoses or symptoms (41.2% female, 85.3% White, mean age = 8.2 years, SD = 1.1 years; range 5.2-9.9 years), 91.2% of children endorsed at least one fear relative to 26.5% of parents. Among parent-child dyads, 36.8% disagreed about the child's fear of stomach pain (κ = 0.12) and 48.5% disagreed about the child's fear of vomiting, (κ = 0.08), both indicating low inter-rater reliability. On average, children endorsed 4.3 (SD = 2.3) fears out of 9 options. The most frequently endorsed fears were that food will "taste bad," (n = 43, 63.2%), "make you gag" (n = 37, 54.4%), and "look disgusting" (n = 36, 52.9%). Findings highlight ways in which fear may manifest in children with ARFID that are not easily discernable by adults. Greater precision in depicting childhood fears may facilitate the earlier detection of problematic eating behaviors.

Reliability of Suicide Risk Estimates: A Vignette Study

Objective Suicide risk assessments are obligatory when patients express a death wish in clinical practice. Yet, suicide risk estimates based on unguided risk assessments have been shown to be of low reliability. Since generalizability of previous studies is limited, the current study aimed to assess inter-rater and intra-rater reliability of risk estimates conducted by psychotherapists and psychology students using written case vignettes. Method In total, N = 256 participants (psychology students, psychotherapists) were presented with 24 case vignettes describing patients at either low, moderate, severe or extreme risk of suicide. Participants were asked to assign a level of risk to each single vignette at a baseline assessment and again at a follow-up assessment two weeks later. Results Risk estimates showed a low inter-rater reliability, both for students (AC1 = .35) and for psychotherapists (AC1 = .44). Intra-rater reliability was moderate for psychotherapists (AC1 = .59) and rather low for psychology students (AC1 = .47). In general, intra- and intra-rater reliability were highest for vignettes displaying “low” and “extreme” risk. Conclusions The results highlight that the reliability of unguided suicide risk assessments is questionable. Standardized risk assessment protocols are therefore recommended. Nonetheless, even reliable risk estimation does not imply predictive validity of risk estimates for future suicidal behavior.

Which North American spine society disc herniation morphology descriptors are most associated with improvements in clinical outcomes after microdiscectomy?

The North American Spine Society (NASS) assembled the first ever comprehensive naming system for describing lumbar disc disease, including lumbar disc herniation. The objectives of this study were (1) to determine which NASS descriptors are most predictive of independent patient-reported outcomes after microdiscectomy and (2) to identify the inter-rater reliability of each NASS descriptor. Adult patients (≥18 years) who underwent a lumbar microdiscectomy from 2014-2021 were retrospectively identified. Patient-reported outcome measures (PROMs) were collected at preoperative, 3-month, and 1-year postoperative time points. Lumbar disc herniations were evaluated and classified on preoperative MRI using the NASS lumbar disc nomenclature specific to disc herniation. About 213 microdiscectomy patients were included in the final analysis. Herniation descriptors exhibiting the greatest reliability included sequestration status (κ=0.83), axial disc herniation area (κ=0.83), and laterality (κ=0.83). The descriptor with the lowest inter-rater reliability was direction of migration (κ=0.53). At 3 months, a sequestered herniation was associated with lower odds of achieving the minimal clinically important difference (MCID) for ODI (p=.004) and MCS (p=.032). At 12 months, a similar trend was observed for Oswestry Disability Index (ODI) MCID achievement (p=.001). At 3 months, a herniation with larger axial area was a predictor of MCID achievement in ODI (p=.004) and the mental component summary (MCS) (p=.009). Neither association persisted at 12 months; however, larger axial disc herniation area was able to predict MCID achievement in the Visual Analogue Scale (VAS) leg (p=.031) at 12 months. The utility of the NASS nomenclature system in predicting postoperative outcomes after microdiscectomy has yet to be studied. We showed that sequestration status and disc area are both reliable and able to predict the odds of achieving MCID in certain clinical outcomes at 3 months and 12 months after surgery. Hence, preoperative imaging analysis of lumbar disc herniations may be useful in accurately setting patient expectations.

Inter-rater reliability of ACS-NSQIP colorectal procedure coding in Canada

BackgroundThe American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) uses Current Procedural Terminology (CPT) codes for risk-adjusted calculations. This study evaluates the inter-rater reliability of coding colorectal resections across Canada by ACS-NSQIP surgical clinical nurse reviewers (SCNR) and its impact on risk predictions. MethodsSCNRs in Canada were asked to code simulated operative reports. Percent agreement and free-marginal kappa correlation were calculated. The ACS-NSQIP risk calculator was utilized to illustrate its impact on risk prediction. ResultsResponses from 44 of 150 (29.3 ​%) SCNRs revealed 3 to 6 different codes chosen per case, with agreement ranging from 6.7 ​% to 62.3 ​%. Free-marginal kappa correlation ranged from moderate agreement (0.53) to high disagreement (−0.17). ACS-NSQIP risk calculator predicted large absolute differences in risk for serious complications (0.2 ​%–13.7 ​%) and mortality (0.2 ​%–6.3 ​%). ConclusionThis study demonstrated low inter-rater reliability in coding ACS-NSQIP colorectal procedures in Canada among SCNRs, impacting risk predictions.

Low Inter-Rater Reliability and Reproducibility of Neck Reflex/“Adler-Langer” Points in Neural Therapy Diagnostics but Increased Pressure Pain Threshold after Therapy: Results of a Randomized Controlled Observer-Blind Trial

Background: Neck reflex points or Adler-Langer points are commonly used in neural therapy to detect so-called interference fields. Chronic irritations or inflammations in the sinuses, teeth, tonsils, or ears are supposed to induce tension and tenderness of the soft tissues and short muscles in the upper cervical spine. The individual treatment strategy is based on the results of diagnostic Adler-Langer point palpation. This study investigated the inter- and intra-rater reliability and explored treatment effects. Methods: We performed a randomized controlled trial with 104 inpatients (80.8% female, 51.8 ± 12.74 years) of a German department for internal and integrative medicine. Patients were randomized to individual neural therapy according to the pathological findings (n = 48) or no treatment (n = 56). In each patient, three experienced raters (20–45 years of experience in neural therapy) and two novice raters (medical students) rated Adler-Langer points rigidity on a standardized rating scale (“strong,” “weak,” “none”). The patients independently evaluated the tenderness on palpation of the eight points using the same scale. Pressure pain thresholds were assessed at the eight Adler-Langer points. All patients were retested after 30 min. The five raters were blinded to treatment allocation and assessments of the other raters. Video recordings were obtained to assess the consistency of the areas tested by the different raters. Results: Agreement between patients and raters (Cohen’s kappa = 0.161–0.400) and inter-rater reliability were low (Fleiss kappa = 0.132–0.150). Moreover, the individual agreement (pre-post comparisons in untreated patients) was similarly low even in experienced raters (Cohen’s kappa = 0.099–0.173). Video documentation suggests that raters do not place their fingers in the correct segments (percentage of correct position: 42.0–60.6%). Pressure pain thresholds at five of the eight Adler-Langer points showed significant changes after treatment compared to none in the control group. Conclusion: Under this artificial experimental setting, this method of Adler-Langer point palpation has not proven to be a reliable diagnostic tool. But it could be shown that, as claimed by the method, the tenderness in five of eight Adler-Langer points decreased after neural therapy.

More Evidence of Low Inter-Rater Reliability of a High-Stakes Performance Assessment of Teacher Candidates

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen’s kappa and Gwet’s AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments.

Assessing the performance of Bing Chat artificial intelligence: Dental exams, clinical guidelines, and patients’ frequent questions

ObjectivesBing Chat is a large language model artificial intelligence (AI) with online search and text generating capabilities. This study assessed its performance within the scope of dentistry in: (a) tackling exam questions for dental students, (ii) providing guidelines for dental practitioners, and (iii) answering patients’ frequently asked questions. We discuss the potential of clinical tutoring, common patient communication and impact on academia. MethodsWith the aim of assessing AI's performance in dental exams, Bing Chat was presented with 532 multiple-choice questions and awarded scores based on its answers. In evaluating guidelines for clinicians, a further set of 15 questions, each with 2 follow-up questions on clinical protocols, was presented to the AI. The answers were assessed by 4 reviewers using electronic visual analog scale. In evaluating answers to patients’ frequently asked questions, another list of 15 common questions was included in the session, with respective outputs assessed. ResultsBing Chat correctly answered 383 out of 532 multiple-choice questions in dental exam part, achieving a score of 71.99 %. As for outlining clinical protocols for practitioners, the overall assessment score was 81.05 %. In answering patients’ frequently asked questions, Bing Chat achieved an overall mean score of 83.8 %. The assessments demonstrated low inter-rater reliability. CONCLUSIONSThe overall performance of Bing Chat was above the regularly adopted passing scores, particularly in answering patient's frequently asked questions. The generated content may have biased sources. These results suggest the importance of raising clinicians’ awareness of AI's benefits and risks, as well as timely adaptations of dental education curricula, and safeguarding its use in dentistry and healthcare in general. Clinical significanceBing Chat AI performed above the passing threshold in three categories, and thus demonstrated potential for educational assistance, clinical tutoring, and answering patients’ questions. We recommend popularizing its benefits and risks among students and clinicians, while maintaining awareness of possible false information.

Reliability of Markerless Motion Capture Systems for Assessing Movement Screenings.

Movement screenings are commonly used to detect unfavorable movement patterns. Markerless motion capture systems have been developed to track 3-dimensional motion. To determine the reliability of movement screenings assessed using a markerless motion capture system when comparing the results of multiple systems and multiple collection periods. Descriptive laboratory study. The inter- and intrarater reliability of a commercially available markerless motion capture system were investigated in 21 recreationally active participants aged between 18 and 22 years. A total of 39 kinematic variables arising from 10 fundamental upper and lower body movements typical of a screening procedure in sports performance were considered. The data were statistically analyzed in terms of relative error via the intraclass correlation coefficient (ICC) and absolute error via the residual standard error (RSE). Both inter- and intrarater reliability ICCs were at least moderate across all variables (ICC, >0.50), with most movements and corresponding variables having excellent reliability (ICC, >0.90). Although maximum knee valgus angles were the kinematic variables with the lowest interrater reliability (ICCs, 0.59-0.82) and moderate relative agreement, there was agreement in absolute terms with an RSE of <1.3°. Findings indicated that markerless motion capture provides reliable measurements of joint position during a movement screen, which allows for a more objective evaluation of the direction and subsequent success of interventions. However, practitioners should consider relative and absolute agreements when applying information provided by these systems. Markerless motion capture systems may assist clinicians by reliably assessing movement screenings using different systems over different collection periods.

Inter-rater reliability in the Testing methods of the International Dysphagia Diet Standardization Initiative (IDDSI) in Slovenian language

Background: Texture modification of drinks and food is a widespread clinical intervention for managing dysphagia. The descriptive and experiential approach to measurement is mainly used by healthcare professionals, caregivers, and patients. This study examines the inter-rater reliability of the International Dysphagia Diet Standardization Initiative (IDDSI) Testing methods on different food samples from the hospital’s central kitchen.&#x0D; Methods: Nine previously trained healthcare professionals of different professional backgrounds and working experiences independently evaluated 34 food samples in IDDSI levels 4, 5, 6, and 7.1, using a Slovenian translation of the IDDSI Audit Tools. The 12 descriptive food characteristics were observed during the two consecutive days of data collection, representing 7 IDDSI Testing Methods. The Fleiss’ Kappa was used as a measure of agreement.&#x0D; Results: Perfect agreement among raters was found for the Fork drip test regarding both observed parameters: dripping and holding shape. All ratings of participants were completely the same. Therefore, Fleiss’ Kappa is considered (κ =1) in both cases. Moderate agreement (κ = 0.583) was found in particle size assessment. Fair agreement was observed in the Fork Pressure Test (κ = 0.361) and Finger Test, indicating bolus separation (κ = 0.328), pressure (κ = 0.305), and moisture (κ = 0.297). Fair agreement was also found for the Fork/Spoon Separation Test (κ = 0.265) and the Spoon Tilt Test in parameters of holding shape on the spoon (κ =0.233) and stickiness (κ = 0.327). No statistically significant results were found in the assessment of adhesiveness (κ = 0.059) in the Spoon Tilt Test and assessment of a typical pattern (κ = 0.039) in the Fork Pressure Test.&#x0D; Conclusions: The study proved the Fork Drip Test to be a comprehensive and highly reliable assessment tool, while the other IDDSI Testing Methods showed low inter-rater reliability. The study highlights the risks of low agreement among raters. It draws attention to the IDDSI descriptors, which should be improved with additional instructions or standardised training protocol for learning the IDDSI Testing Methods.

CTA Supplemented by CTP Increases Interrater Reliability and Endovascular Treatment Use in Patients with Aneurysmal SAH.

Cerebral vasospasm is a common complication of aneurysmal SAH and remains a risk factor for delayed cerebral ischemia and poor outcome. The interrater reliability of CTA in combination with CTP has not been sufficiently studied. We aimed to investigate the reliability of CTA alone and in combination with CTP in the detection of cerebral vasospasm and the decision to initiate endovascular treatment. This is a retrospective single-center study including patients treated for aneurysmal SAH. Inclusion criteria were a baseline CTA and follow-up imaging including CTP due to suspected vasospasm. Three neuroradiologists were asked to grade 15 intracranial arterial segments in 71 cases using a tripartite scale (no, mild <50%, or severe >50% vasospasm). Raters further evaluated whether endovascular treatment should be indicated. The ratings were performed in 2 stages with a minimum interval of 6 weeks. The first rating included only CTA images, whereas the second rating additionally encompassed CTP images. All raters were blinded to any clinical information of the patients. Interrater reliability for per-segment analysis of vessels was highly variable (κ = 0.16-0.61). We observed a tendency toward higher interrater reliability in proximal vessel segments, except for the ICA. CTP did not improve the reliability for the per-segment analysis. When focusing on senior raters, the addition of CTP images resulted in higher interrater reliability for severe vasospasm (κ = 0.28; 95% CI, 0.10-0.46 versus κ = 0.46; 95% CI, 0.26-0.66) and subsequently higher concordance (κ = 0.23; 95% CI, -0.01-0.46 versus κ = 0.73; 95% CI, 0.55-0.91) for the decision of whether endovascular treatment was indicated. CTA alone offers only low interrater reliability in the graduation of cerebral vasospasm. However, using CTA in combination with CTP might help, especially senior neuroradiologists, to increase the interrater reliability to identify severe vasospasm following aneurysmal SAH and to increase the reliability regarding endovascular treatment decisions.

Evaluating insomnia queries from an artificial intelligence chatbot for patient education.

We evaluated the accuracy of ChatGPT in addressing insomnia-related queries for patient education and assessed ChatGPT's ability to provide varied responses based on differing prompting scenarios. Four identical sets of 20 insomnia-related queries were posed to ChatGPT. Each set differed by the context in which ChatGPT was prompted: no prompt, patient-centered, physician-centered, and with references and statistics. Responses were reviewed by 2 academic sleep surgeons, 1 academic sleep medicine physician, and 2 sleep medicine fellows across 4 domains: clinical accuracy, prompt adherence, referencing, and statistical precision, using a binary grading system. Flesch-Kincaid grade-level scores were calculated to estimate the grade level of the responses, with statistical differences between prompts analyzed via analysis of variance and Tukey's test. Interrater reliability was calculated using Fleiss's kappa. The study revealed significant variations in the Flesch-Kincaid grade-level scores across 4 prompts: unprompted (13.2 ± 2.2), patient-centered (8.1 ± 1.9), physician-centered (15.4 ± 2.8), and with references and statistics (17.3 ± 2.3, P < .001). Despite poor Fleiss kappa scores, indicating low interrater reliability for clinical accuracy and relevance, all evaluators agreed that the majority of ChatGPT's responses were clinically accurate, with the highest variability on Form 4. The responses were also uniformly relevant to the given prompts (100% agreement). Eighty percent of the references ChatGPT cited were verified as both real and relevant, and only 25% of cited statistics were corroborated within referenced articles. ChatGPT can be used to generate clinically accurate responses to insomnia-related inquiries. Alapati R, Campbell D, Molin N, etal. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J Clin Sleep Med. 2024;20(4):583-594.

Tutor assessment of medical students in problem-based learning sessions.

Problem-based learning (PBL) is a method of learning that has been adopted in different curricula of different disciplines for more than 30 years; the assessment of the students in PBL sessions in medical schools is fundamental to ensure students' attainment of the expected outcomes of conducting PBL sessions and in providing the students with the feedback that help them to develop and encourage their learning. This study investigated the inter-rater reliability of the tutor assessment in assessing medical students' performance in their PBL tutorial sessions. This study was conducted in the College of Medicine (COM), in the academic year 2021-2022. The study involved ten raters (tutors) of two genders who assessed 33 students in three separate PBL tutorial sessions. The PBL sessions were prerecorded and shown to the 10 raters for their assessment of PBL sessions. This study showed that male raters gave higher scores to students compared with female raters. In addition, this investigation showed low inter-rater reliability and poor agreement among the raters in assessing students' performance in PBL tutorial sessions. This study suggests that PBL tutor assessment should be reviewed and evaluated; this should be performed with consideration of using assessment domains and criteria of performance. Thus, we recommend that 360-degree assessment including tutor, self, and peer assessment should be used to provide effective feedback to students in PBL tutorial sessions.

Assessment of laparoscopic fundoplication with endoscopy: room for improvement.

Gastroesophageal reflux disease affects a significant portion of the Australian and world population. Minimally invasive laparoscopic fundoplication is a highly effective treatment in appropriately selected patients, with a 90% satisfaction rate. However, up to 5% will undergo revisional surgery. Endoscopy is an important investigation in the evaluation of persistent or new symptoms after fundoplication. Our study sought to evaluate the inter-rater reliability and variability in assessing fundoplication with endoscopy. Upper gastrointestinal (UGI) surgeons and gastroenterologists were invited to join the cohort study through their professional membership with two societies based in Australia. Participants completed a two part 25-item multiple choice questionnaire, involving the analysis of ten static endoscopic images post-fundoplication. A total of 101 participants were included in the study (64 UGI surgeons and 37 gastroenterologists). Over 95% of participants were consultant level, working in non-rural tertiary hospitals. Total accuracy for all 10 cases combined was 76% for UGI surgeons and 69.9% for gastroenterologists. In three of the 10 cases, UGI surgeons performed significantly better than gastroenterologists (p < 0.05). When assessing performance across each of the 4 questions for each case, UGI surgeons were more accurate than gastroenterologists in describing the integrity of the wrap (p = 0.014). Inter-rater reliability was low across both groups for most domains (kappa < 1). Our study confirms low inter-rater reliability between endoscopists and large variations in reporting. UGI surgeons performed better than gastroenterologists in certain cases, usually when describing the integrity of the fundoplication. Our study provides further support for the use of a standardized reporting system in post-fundoplication patients.

Low Inter-Rater Reliability of Calculating the 4T's Score for Heparin Induced Thrombocytopenia

Introduction Heparin induced thrombocytopenia (HIT) is an immune-mediated drug reaction that can cause thromboembolism in the setting of thrombocytopenia following heparin exposure. The “4T's score” has been validated to determine the pre-test probability of HIT and to assist in decision making around ordering testing for HIT. The 4T's scoring system requires an individual clinician to determine each component of the score. The objective of our study is to investigate the inter-rater reliability of calculating the 4T's score among clinicians. Methods Through retrospective query of Northwestern Enterprise Data Warehouse, we identified patients who had a HIT antibody (Ab) test ordered between 10/2019 and 10/2022 after implementation of a clinical decision support (CDS) tool that asked clinicians to calculate a 4T's score as part of HIT PF-4 Ab orders. From this cohort, an independent clinician randomly selected 15 patients. Four raters, including an attending hematologist, a hospital medicine attending, a hematology/oncology fellow, and an internal medicine resident, performed manual chart review of the randomly selected subjects to calculate a 4T's score. Data collected for each case included individual components of the 4T's score and overall 4T's score from each rater. In addition, we compared raters' scores to the 4T's score entered by the ordering clinician for the patient. We then categorized the numerical scores into validated pre-test probability categories: 0-3 as low, 4-5 as intermediate, and ≥6 as high. Inter-rater reliability for the categories was calculated using the Fleiss kappa statistic. Our study was approved by the Northwestern University Institutional Review Board. Results Of 15 cases selected, 5 each were scored as low, intermediate, and high probability 4T's scores by the ordering clinician (associated with the HIT Ab order). The overall agreement between the score categories for the 5 clinicians (4 raters and the ordering clinician) was 50.7%, with a Fleiss Kappa statistic of 0.26 (95% CI [0.07-0.45]), indicating poor inter-rater reliability (Figure 1). Excluding the ordering clinician, the overall agreement in score category between the 4 raters was 58.9%, with a Fleiss Kappa statistic of 0.38 (95% CI [0.14-0.62]). The same 4T's score pre-test probability category was calculated by the 4 raters in 5 cases, with only one case in which all four raters calculated the same numerical 4T's score. There was one case in which all 5 clinicians calculated the same 4T's score probability category, but none with the same numerical 4T's score. The hematology/oncology fellow had highest inter-rater agreement with the original clinical (Kappa 0.30 [-0.09-0.69]), whereas the internal medicine attending and resident had the lowest (both with Kappa 0.00 [-0.37-0.37]). Of the 15 4T's scores, 12/15 ordered by the hematology attending, 5/15 by the internal medicine attending, 12/15 by the hematology/oncology fellow, and 11/15 by the internal medicine resident were lower than those calculated by the original ordering clinicians. Conclusion Our study demonstrates poor inter-rater reliability of HIT 4T's score calculation, across levels of training and specialty. Importantly, poor inter-rater reliability was seen across 4T's categories, which has implications for clinical management of patients undergoing evaluation for HIT. This suggests that different strategies are necessary to help clinicians better use the 4Ts score.

Development and initial evaluation of a treatment integrity measure for low-intensity group psychoeducational interventions.

Despite the importance of assessing the quality with which low-intensity (LI) group psychoeducational interventions are delivered, no measure of treatment integrity (TI) has been developed. To develop a psychometrically robust TI measure for LI psychoeducational group interventions. This study had two phases. Firstly, the group psychoeducation treatment integrity measure-expert rater (GPTIM-ER) and a detailed scoring manual were developed. This was piloted by n=5 expert raters rating the same LI group session; n=6 expert raters then assessed content validity. Secondly, 10 group psychoeducational sessions drawn from routine practice were then rated by n=8 expert raters using the GPTIM-ER; n=9 patients also rated the quality of the group sessions using a sister version (i.e. GPTIM-P) and clinical and service outcome data were drawn from the LI groups assessed. The GPTIM-ER had excellent internal reliability, good test-retest reliability, but poor inter-rater reliability. The GPTIM-ER had excellent content validity, construct validity, formed a single factor scale and had reasonable predictive validity. The GPTIM-ER has promising, but not complete, psychometric properties. The low inter-rater reliability scores between expert raters are the main ongoing concern and so further development and testing is required in future well-constructed studies.

The Traffic Light System: A user-friendly alternative for gait data representation

BackgroundInstrumented gait analysis is an established procedure in biomechanical assessment, requiring specially-trained analysts to interpret the complex graphical output generated. Research questionDoes a new method of visual representation of lower limb kinematic gait analysis data provide a reliable and valid method of interpretation of biomechanical data for healthcare professionals? MethodsAn innovative system based on the Traffic Lights System (TLS) was developed. Simulated abnormal gait was captured using a 16-camera optoelectronic motion capture system, and the results were presented in both the Traditional Graphical System (TGS) format and the new TLS. An online form was filled by health professionals who attempted to interpret normal and abnormal motion in the joints presented in the 2 output formats. ResultsOut of 26 raters, 18 preferred the new system because of its user-friendliness and its ease of interpretation. 2 raters preferred the TGS, with one of these raters clarifying that the preference is due to colour blindness. For intra-rater reliability, 2 trained raters provided a second response for the TGS (Cronbach’s Alpha ranging between 0.733 and 0.918), whilst the TLS resulted in Cronbach’s Alpha between 0.817 and 1.00 amongst 3 untrained raters. The Fleiss Multi-rater Kappa Test demonstrated low inter-rater reliability amongst raters in the TGS, whereas the overall Fleiss Multi-rater Kappa values of the TLS surpassed the TGS in all 3 studies. SignificanceThis study showed that whilst trained health professionals have high intra-rater reliability in interpreting traditional gait analysis results, those professionals inexperienced in the system, do not always comprehend the complex graphs generated by the system when presenting gait analysis data. When these graphs are transformed into coloured outputs representing the extent of the movement, the TLS has demonstrated high validity and high intra- and inter-rater reliability, significantly exceeding those of the TGS, especially in untrained health professionals.

A webcam-based machine learning approach for three-dimensional range of motion evaluation.

Joint range of motion (ROM) is an important quantitative measure for physical therapy. Commonly relying on a goniometer, accurate and reliable ROM measurement requires extensive training and practice. This, in turn, imposes a significant barrier for those who have limited in-person access to healthcare. The current study presents and evaluates an alternative machine learning-based ROM evaluation method that could be remotely accessed via a webcam. To evaluate its reliability, the ROM measurements for a diverse set of joints (neck, spine, and upper and lower extremities) derived using this method were compared to those obtained from a marker-based optical motion capture system. Data collected from 25 healthy adults demonstrated that the webcam solution exhibited high test-retest reliability, with substantial to almost perfect intraclass correlation coefficients for most joints. Compared with the marker-based system, the webcam-based system demonstrated substantial to almost perfect inter-rater reliability for some joints, and lower inter-rater reliability for other joints (e.g., shoulder flexion and elbow flexion), which could be attributed to the reduced sensitivity to joint locations at the apex of the movement. The proposed webcam-based method exhibited high test-retest and inter-rater reliability, making it a versatile alternative for existing ROM evaluation methods in clinical practice and the tele-implementation of physical therapy and rehabilitation.

Feature Fusion of Time-frequency and Deep Learning Features for Epileptic Seizure Detection using EEG Signals

A persistent brain's neurological state is epilepsy, characterised by recurring seizure. Brain electrical activity is measured using EEG signals, which can be used to detect and diagnose significant brain problems such as Epilepsy, Autism, Alzheimer’s etc. However, manual EEG data processing is time-consuming, requires highly skilled clinicians, and is associated with low inter-rater reliability (IRA). A computer-aided diagnosis approach for epileptic seizure detection from multichannel EEG recordings by fusing the time-frequency features and the deep learning features extracted from Convolutional Neural Network-Gated Recurrent Unit (CNN-GRU) model using canonical correlation analysis (CCA) method is provided in this study. Deep Learning features are extracted using CNN-GRU layers, motivated by recent advancements in image classification and optimised for use with EEG data. We have also extracted time-frequency features such as spectral entropies and Sub Band energies from Empirical mode decomposition (EMD) and Hilbert Marginal Spectrum (HMS). We used CHBMIT dataset to carry out the results and showed that the method proposed for fusing the time-frequency features and deep learning has given better performance.

Reliability of confocal corneal microscopy for measurement of dendritic cell density in suspected small fiber neuropathy.

Dendritic cells (DCs) and their contacts with corneal nerves are described in animal models of nerve damage. Dendritic cell density (DCD) is a potential marker of immune activity in suspected small-fiber neuropathy (SFN). Here, we aim to evaluate the intra- and inter-rater reliability of DCD measurements in suspected SFN. This retrospective study collected DCD from confocal microscopy images from the corneal sub-basal epithelium of the eye from 48 patients (mean age 49.6 ± 12.1 y, 61% female). Two examiners, each blinded to the other's examinations and measurements, assessed DCD to evaluate inter-rater reliability. For intra-rater reliability, the first examiner performed a second measurement after 14 days. DCs were classified into two cell morphological subtypes: mature and immature. Test-retest reliability for total DCD showed excellent agreement, with an intraclass correlation coefficient of 0.96 and inter-rater reliability intraclass correlation coefficient of 0.77. The immature cell subtype showed excellent intra-rater reliability but lower inter-rater reliability. We found that DCD measurements in the corneal sub-basal epithelium are sufficiently reliable for consideration in clinical studies of patients with suspected SFN.

Paper 24: Reliability of Posterior Tibial Slope Measurement on Magnetic Resonance Imaging Versus Computed Tomography

Objectives: Increased posterior tibial slope (PTS) has been linked to higher rates of anterior cruciate ligament (ACL) injury and ACL graft failure post-reconstruction. In recent years, more specific investigations have delineated distinct roles of the medial tibial plateau (MTP) and lateral tibial plateau (LTP) geometries on anteroposterior and rotatory stability of the knee. While the diagnostic workup of patients with ACL insufficiency typically consists of radiographs and magnetic resonance imaging (MRI), computed tomography (CT), which is the gold standard for evaluating the osseous geometry of the proximal tibia, is not routinely obtained. With increased understanding of the roles of the MTP and LTP geometries on tibiofemoral kinematics, accurate sagittal slope measurements of both the MTP and LTP will be crucial for further defining the treatment algorithm for ACL insufficiency. It is currently unknown whether traditional MRI can be used to accurately measure PTS and how well these measurements correlate to those on CT. Therefore, the purpose of this study was to compare PTS measurements of the MTP and LTP on MRI versus CT. Methods: After Institutional Review Board approval was obtained, an institutional picture archiving and communication system imaging database was retrospectively queried for patients that received concurrent MRI and CT imaging of the same knee within a one-year interval. Knees with significant arthrosis (Kellgren-Lawrence grade &gt;2), deformity, proximal tibia fracture, or artifact that obscured visualization of proximal tibia landmarks were excluded. On paired MRI and CT studies, PTS of the MTP and LTP were measured using a previously described, validated method (Hudek et al, Clin Orthop Relat Res, 2009) by two independent raters (Figure 1). Interrater reliability of PTS measurements was assessed using the intraclass correlation coefficient (ICC). Intermethod agreement between MRI and CT measurements was assessed using the ICC and Bland-Altman analysis. Results: A total of 41 knees in 40 patients (mean age, 37.9±13.4 years, 78.0% male) met final inclusion criteria. Interrater reliability was moderate for MRI measurements (ICC = 0.62-0.73) and high for CT measurements (ICC = 0.83-0.86). Mean measurements of PTS at the MTP were 3.3±2.8° (range, -4.7-8.4°) and 3.7±3.1° (range, -3.8-10.4°) on MRI and CT, respectively. Mean measurements of PTS at the LTP were 4.5±3.8° (range, -8.0-10.6°) and 5.2±3.6° (range, -3.6-15.7°) on MRI and CT, respectively. The mean absolute differences in PTS measurements between MRI and CT were 2.6±2.1° and 2.6±2.6° at the MTP and LTP, respectively. Poor intermethod agreement between MRI and CT was observed at the MTP (ICC = 0.31-0.41), while moderate intermethod agreement was observed at the LTP (ICC = 0.61-0.62). Bland-Altman plots demonstrated high variability and minimal bias of PTS measurements on MRI compared to CT (-0.9° [95% Limit of Agreement (LOA) -7.5-5.6°] for MTP; -1.0° [95% LOA -7.2-5.2°] for LTP) (Figure 2). Conclusions: With increased understanding of the roles of the medial and lateral tibial plateau geometries on tibiofemoral stability in the setting of ACL insufficiency, accurate measurements of PTS are necessary to guide treatment decisions. In this study, PTS measurements on MRI demonstrated lower interrater reliability compared to PTS measurements on CT. Poor-to-moderate agreement was observed for measurements of PTS of the MTP and LTP between paired MRI and CT studies, with high variability and minimal bias. Therefore, measurements of medial and lateral PTS may not be reliable on traditional MRI.