Fair Reliability Research Articles

S104. THE RELIABILITY AND VALIDITY OF THE CORE SCHIZOPHRENIA SYMPTOMS SCALE OF THE STANDARD FOR CLINICIANS’ INTERVIEW IN PSYCHIATRY

BackgroundCore schizophrenia symptoms (CSS) include delusions, hallucinations and disorganization and have been included in the diagnostic criteria of schizophrenia since Kraeplin (Kendler 2016). More than 40 subtypes of delusions, hallucinations and disorganization have been described as symptoms and signs of schizophrenia. There is a need to derive a short list of the core symptoms and signs of schizophrenia that are reliable, valid and useful for clinicians in clinical settings and clinical research.MethodsThe Standard for Clinicians’ Interview in Psychiatry (SCIP) is a new diagnostic interview designed to be used by clinicians (psychiatrists and experienced mental health professionals) in clinical settings and clinical research. The SCIP is a valid and reliable tool and was tested in an international multisite study in three countries (USA, Canada and Egypt) between 2000 and 2012 (Aboraya, El-Missiry et al. 2014, Aboraya 2015, Aboraya 2016, Aboraya, Nasrallah et al. 2016). A total of 700 patients were interviewed at William R. Sharpe Jr. Hospital in Weston, West Virginia (670 patients) and Chestnut Ridge Center in Morgantown, West Virginia (30 patients). Mean patient age was 34, 59% male, 95% White and 34% had less than 12 years of education. The SCIP includes 38 items covering subtypes of delusions, hallucinations and disorganization. The 38 items were shortened by removing items with low prevalence, low sensitivity or low item-rest correlation (< 0.4). The reliability and validity of the remaining items was recalculated with repetitive iterations. The final model was developed with input from experts. The result is the Core Schizophrenia Symptoms (CSS) Scale which has 18 items: 6 items measuring hallucinations, 8 items measuring delusions and 4 items measuring disorganization. The items were scored with binary and Likert-type scales ranging from 0 to 3. The reliability of the CSS scale was measured using the kappa coefficient for inter-rater reliability of the CSS individual items and Cronbach’s alpha for internal consistency of the CSS dimension. The validity of the CSS scale was assessed using Receiver Operating Characteristic (ROC) curves to determine the best clinical cut-off point for the CSS scale that maximizes sensitivity and specificity of the scale against the SCIP diagnosis of schizophrenia (the reference standard).ResultsTable (1) shows stable kappa values and standard error of 15 CSS items. Nine items have good reliability (kappa > 0.7), three items have fair reliability (kappa values range from 0.5 to 0.7) and three items have poor reliability (kappa < 0.5). Table (2) shows the internal consistency of the CSS dimension using Cronbach’s alpha and one-sided 95% confidence interval. The Cronbach’s alpha is 0.8317, indicating excellent internal consistency. Table (3) shows the sensitivity and specificity of the Core Schizophrenia Symptoms (CSS) scale. At a cut-off of one or more positive items, sensitivity is 95.06% and specificity is 88.94%; at a cut-off of two or more positive items, sensitivity is 90.12% and specificity is 89.39%.DiscussionThe Core Schizophrenia Symptoms (CSS) Scale is reliable at the level of individual items and at the dimensional level. In addition, the CSS scale is a valid scale that differentiates between schizophrenia and non-schizophrenia cases in a clinical population.

The gluteal muscles in end-stage osteoarthritis of the hip: intra- and interobserver reliability and agreement of MRI assessments of muscle atrophy and fatty degeneration

To assess the intra- and interobserver reliability and agreement of muscular atrophy and fatty degeneration of the gluteal muscles on magnetic resonance imaging (MRI) in patients with end-stage osteoarthritis of the hip. Thirty-nine patients with end-stage osteoarthritis of the hip, who scheduled for hip replacement surgery, were included in the study. The cross-sectional areas of the gluteus medius and minimus muscles were manually circumscribed and automatically calculated, while intramuscular fatty degeneration was assessed using the semi-quantitative classification system of Goutallier. Two independent observers performed all the evaluations. Good agreement and excellent reliability were found for the cross-sectional areas of the gluteus medius and gluteus minimus (intraclass correlation coefficient [ICC] for intra-observer 0.97 and 0.93, ICC for interobserver 0.98 and 0.95). Fair reliability was found for fatty degeneration of the gluteus muscles (mean kappa 0.23 and mean prevalence-adjusted kappa 0.53). The average proportion of agreement for the raters was 0.71. The present study demonstrated substantial agreement and excellent intra- and interobserver reliability for the measurements of the cross-sectional areas of the gluteus medius and gluteus minimus. Assessment of fatty degeneration had fair intra- and interobserver reliability, but acceptable agreement.

Rapid and Accurate Behavioral Health Diagnostic Screening: Initial Validation Study of a Web-Based, Self-Report Tool (the SAGE-SR).

BackgroundThe Structured Clinical Interview for DSM (SCID) is considered the gold standard assessment for accurate, reliable psychiatric diagnoses; however, because of its length, complexity, and training required, the SCID is rarely used outside of research.ObjectiveThis paper aims to describe the development and initial validation of a Web-based, self-report screening instrument (the Screening Assessment for Guiding Evaluation-Self-Report, SAGE-SR) based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) and the SCID-5-Clinician Version (CV) intended to make accurate, broad-based behavioral health diagnostic screening more accessible within clinical care.MethodsFirst, study staff drafted approximately 1200 self-report items representing individual granular symptoms in the diagnostic criteria for the 8 primary SCID-CV modules. An expert panel iteratively reviewed, critiqued, and revised items. The resulting items were iteratively administered and revised through 3 rounds of cognitive interviewing with community mental health center participants. In the first 2 rounds, the SCID was also administered to participants to directly compare their Likert self-report and SCID responses. A second expert panel evaluated the final pool of items from cognitive interviewing and criteria in the DSM-5 to construct the SAGE-SR, a computerized adaptive instrument that uses branching logic from a screener section to administer appropriate follow-up questions to refine the differential diagnoses. The SAGE-SR was administered to healthy controls and outpatient mental health clinic clients to assess test duration and test-retest reliability. Cutoff scores for screening into follow-up diagnostic sections and criteria for inclusion of diagnoses in the differential diagnosis were evaluated.ResultsThe expert panel reduced the initial 1200 test items to 664 items that panel members agreed collectively represented the SCID items from the 8 targeted modules and DSM criteria for the covered diagnoses. These 664 items were iteratively submitted to 3 rounds of cognitive interviewing with 50 community mental health center participants; the expert panel reviewed session summaries and agreed on a final set of 661 clear and concise self-report items representing the desired criteria in the DSM-5. The SAGE-SR constructed from this item pool took an average of 14 min to complete in a nonclinical sample versus 24 min in a clinical sample. Responses to individual items can be combined to generate DSM criteria endorsements and differential diagnoses, as well as provide indices of individual symptom severity. Preliminary measures of test-retest reliability in a small, nonclinical sample were promising, with good to excellent reliability for screener items in 11 of 13 diagnostic screening modules (intraclass correlation coefficient [ICC] or kappa coefficients ranging from .60 to .90), with mania achieving fair test-retest reliability (ICC=.50) and other substance use endorsed too infrequently for analysis.ConclusionsThe SAGE-SR is a computerized adaptive self-report instrument designed to provide rigorous differential diagnostic information to clinicians.

Reliability of the Dutch Cleft Speech Evaluation Test and Conversion to the Proposed Universal Scale.

The Dutch cleft speech evaluation test (DCSET) has been implemented by the speech-language pathologists nationwide in the Netherlands since 2003, but the inter- and intrarater reliability was unknown. Two speech-language pathologists experienced in evaluating cleft speech assessed audio recordings of 20 children with cleft speech using the DCSET, and reassessed the recordings 2 weeks later. Intra- and interrater reliability was calculated, but found to be unacceptable after the first phase of this study using audio recordings. Following consensus training and some modifications in the scoring scales, the study was repeated using video recordings of 20 different children with cleft speech. Results from the second phase of this study using standardized video recordings showed fair, moderate, and good reliability on different subsets of the DCSET. Intrarater reliability (Kappa 0.59-1.00) was greater than interrater reliability (Kappa 0.33-0.79). Interrater reliability agreement was good (Kappa 0.63-0.79) for consonant production errors and speech understandability and acceptability, moderate (Kappa 0.59) for the resonance of the nasal passage, and fair (Kappa 0.33-0.37) for the resonance of the mixed and denasal passages. Subsequently, an algorithm was made to convert the DCSET scales to universal scales for international comparison of cleft speech as suggested by Henningsson et al in 2008.

Intertester reliability of clinical shoulder instability and laxity tests in subjects with and without self-reported shoulder problems

ObjectiveFirst, to investigate the intertester reliability of clinical shoulder instability and laxity tests, and second, to describe the mutual dependency of each test evaluated by each tester for identifying self-reported...

Psychometric properties of MDHAQ/RAPID3 in patients with systemic lupus erythematosus.

The Multidimensional Health Assessment Questionnaire (MDHAQ) is a patient-reported outcome (PRO) tool that includes the Routine Assessment of Patient Index Data 3 (RAPID3), an index that can be calculated at the point of care. The objective of this study was to perform psychometric analyses of MDHAQ/RAPID3 to study its measurement properties in systemic lupus erythematosus (SLE). The MDHAQ was completed by 161 SLE patients in routine care, along with LupusPRO (a disease-specific PRO). The SLE disease-specific activity index (SELENA-SLEDAI) and damage (SDI) were assessed. Data from 70 patients with rheumatoid arthritis who had completed MDHAQ during their routine medical care were used as controls to compare the results of Physical Function (FN) domain exploratory factor analysis. Internal consistency reliability (ICR) for FN items was calculated using Cronbach's α. Validity of MDHAQ/RAPID3 was evaluated for content validity and construct validity. Responsiveness of the RAPID3 to changes in disease activity anchors was assessed. The ICR of the 10 physical function items on Cronbach's α was 0.88. Exploratory factor analysis revealed cross-loadings of three FN items. RAPID3 showed a strong correlation with LupusPRO health-related quality of life score (rho -0.68 (p < 0.001)), indicating convergent validity. RAPID3 scores did not correlate with disease activity indices or SDI. After adjustment for fibromyalgia status, a weak correlation with the Physician's Global Assessment (PGA) (rho = 0.31, p = 0.008) was noted. RAPID3 could differentiate between SLE patients based on flare status. RAPID3 was not responsive to changes in PGA, SELENA-SLEDAI or SELENA-Flare Index. MDHAQ/RAPID3 has fair reliability and validity in SLE.

Test-Retest Reliabilty of Exercise-Induced Hypoalgesia After Aerobic Exercise.

Exercise increases pressure pain thresholds (PPTs) in exercising and nonexercising muscles, known as exercise-induced hypoalgesia (EIH). No studies have investigated the test-retest reliability of change in PPTs after aerobic exercise. Primary objectives were to compare the effect on PPTs after an incremental bicycling exercise compared with quiet rest and to investigate the relative and absolute test-retest reliability of the test stimulus (PPT) and the absolute and relative EIH response in exercising and nonexercising muscles. Laboratory. In two sessions, PPTs in the quadriceps and trapezius muscles were assessed before and after 15 minutes of quiet rest and 15 minutes of bicycling in 34 healthy subjects. Habitual physical activity was assessed by the International Physical Activity Questionnaire (IPAQ). Bicycling increased PPTs in exercising and nonexercising muscles in both sessions (P < 0.05). The magnitude of the EIH response in the exercising muscle was, however, larger in the second compared with the first session (P < 0.015). PPTs showed excellent (intraclass correlation [ICC] ≥ 0.84) within-session and between-session test-retest reliability. The EIH response in exercising and nonexercising muscles demonstrated fair (ICC = 0.45) between-session relative test-retest reliability, but agreement in EIH responders between sessions was not significant (quadriceps: κ = 0.24, P = 0.15; trapezius: κ = 0.01, P = 0.97). Positive correlations between the IPAQ score and PPTs were found (quadriceps: r = 0.44, P = 0.009; trapezius: r = 0.31, P = 0.07) before exercise. No significant association was found between IPAQ and EIH. Incremental bicycling exercise increased PPTs with fair relative and absolute reliability of the EIH response. These data might have an impact on future studies investigating EIH and for clinicians designing exercise programs for pain relief.

A critical review of the Motor-Free Visual Perception Test—fourth edition (MVPT-4)

ABSTRACTThe Motor-Free Visual Perception Test—fourth edition (MVPT-4) is a revised version of the Motor-Free Visual Perception Test—third edition. The MVPT-4 is used to assess the visual-perceptual ability of individuals aged 4.0 through 80+ years via a series of visual-perceptual tasks that do not require a motor response. Test norms were derived from a normative sample closely matching the United States population. Reliability and validity of the MVPT-4 was described and a critique of the MVPT-4’s psychometric properties is presented. The MVPT-4 was critiqued based on the “Quality Criteria for Health Status Questionnaires” (QCHSQ) and the “Consensus-based Standards for the Selection of Health Measurement Instruments” (COSMIN). According to the QCHSQ, the MVPT-4 exhibits good levels of content validity, construct validity, and interpretability, although information about its reliability, criterion validity, responsiveness, and floor and ceiling effects are poor. According to the COSMIN, the MVPT-4 demonstrated poor measurement error, structural validity, cross-cultural validity, criterion validity, responsiveness, and generalizability; fair internal consistency, reliability, hypothesis testing, and interpretability; and excellent content validity.Practical implications of the use of the MVPT-4 in clinical practice and research are outlined. Further research is needed to determine the cross-cultural generalizability, predictive validity, responsiveness, clinical utility, and dimensionality of the MVPT-4.

Evaluating the reliability and construct validity of the Eckardt symptom score as a measure of achalasia severity

Achalasia is a disease of mechanical esophageal dysfunction characterized by dysphagia, chest pain, regurgitation, and malnutrition. The Eckardt symptom score (ESS) is the gold standard self-report assessment tool. Current guidelines outline a three-step approach to patient reported outcomes measure design. Developed prior to these policies, the ESS has not undergone rigorous testing of its reliability and validity. Adult achalasia patients retrospectively identified via a patient registry were grouped based on treatment history. Patients were grouped PREPOST (completed ESS, GERDQ, brief esophageal dysphagia questionnaire, NIH PROMIS Global Health, high resolution manometry, timed barium esophagram prior to treatment and after) and POST (completed measures only after treatment). Clinical characteristics, treatment type and date were obtained via medical record. Standardized psychometric analyses for reliability and construct validity were performed. 107 patients identified; 83 POST and 24 PREPOST. The ESS has fair internal consistency and split-half reliability with a single factor structure. Dysphagia accounts for half the variance in ESS, while chest pain and weight loss account for 10% each. Pre-post-surgical assessment demonstrates improvements in ESS, except for weight loss. Effect sizes range from 0.24 to 2.53, with greatest change in regurgitation. Validity of the ESS is supported by modest correlations with GERDQ, HRQOL, and physiological data. The ESS demonstrates fair reliability and validity, with a single factor structure mostly explained by dysphagia. Based on psychometric findings, weight loss and chest pain items may be decreasing ESS reliability and validity. Further assessment of the ESS under FDA guidelines is warranted.

Test-retest reliability of the Wii Balance Board for assessing standing balance in young people with intellectual disability

Objectives: The main objective was to identify the test-retest reliability of the Wii Balance Board (WBB) for assessing standing balance when administered to a population of people with intellectual disability (ID). A secondary objective was to provide information regarding the reliability of the WBB, taking into account the severity of cognitive impairment.Methods: The WBB was administered two times to a group of 50 people (mean age: 20.44 ± 5.73 years) with mild (n=32), moderate (n=6), or severe ID (n=12) within a 15-day period. The relative reliability of the WBB for the ‘standing balance with open eyes’ test was analyzed by means of intraclass correlation coefficients (ICCs). The standard error measurement and the minimum detectable change were used to identify absolute reliability.Results: For the whole sample, the WBB showed a fair (ICC=0.71), poor (ICC=0.66), and excellent (ICC=0.99) relative reliability for assessing weight-bearing asymmetry, center of pressure, path length, and sway area, respectively. In people with moderate/severe ID, statistically significant differences between the WBB test and retest scores were found. The absolute reliability of the WBB was considered poor.Conclusion: The WBB showed accurate adequate test-retest reliability for assessing standing balance in young people with ID. When using this device with people with moderate/severe ID, it is advisable to perform several testing trials, in order to avoid a possible learning effect. Standard error of measurement and minimal detectable change were high, indicating that this device might not be sensitive enough to properly identify static balance changes in this population.

The Assignment of American Society of Anesthesiologists Physical Status Classification for Adult Polytrauma Patients: Results From a Survey and Future Considerations.

The American Society of Anesthesiologists (ASA) physical status (PS) classification system assesses the preoperative health of patients. Previous studies demonstrated poor interrater reliability and variable ASA PS scores, especially in trauma scenarios. There are few studies that evaluated the assignment of ASA PS scores in trauma patients and no studies that evaluated ASA PS assignment in severely injured adult polytrauma patients. Our objective was to assess interrater reliability and identify sources of discrepancy among anesthesiologists and trauma surgeons in designating ASA PS scores to adult polytrauma patients. A link to an online survey containing questions assessing attitudes regarding ASA PS classification, demographic information, and 8 fictional trauma cases was e-mailed to anesthesiologists and trauma surgeons. The participants were asked to assign an ASA PS score to each scenario and explain their choice. Rater-versus-reference and interrater reliability, beyond that expected by chance, among respondents was analyzed using the Fleiss kappa analysis. A total of 349 participants completed the survey. All 8 cases had inconsistent ASA PS scores; several cases had scores ranging from I to VI and variable emergency (E) designations. Using weighted kappa (Kw) analysis for a subset of 201 respondents (101 trauma surgeons [S] and 100 anesthesiologists [A]), we found moderate (Kw = 0.63; SE = 0.024; 95% confidence interval, 0.594-0.666; P < .001) interrater-versus-reference reliability. The interrater reliability was fair (Kw = 0.43; SE = 0.037; 95% confidence interval, 0.360-0.491; P < .001). This study demonstrates fair interrater reliability beyond that expected by chance of the ASA PS scores among anesthesiologists and trauma surgeons when assessing adult polytrauma patients. Although the ASA PS is used in some trauma risk stratification models, discrepancies of ASA PS scores assigned to trauma cases exist. Future modifications of the ASA PS guidelines should aim to improve the interrater reliability of ASA PS scores in trauma patients. Further studies are warranted to determine the value of the ASA PS score as a trauma prognostic metric.

A tonic heat test stimulus yields a larger and more reliable conditioned pain modulation effect compared to a phasic heat test stimulus.

The interest in conditioned pain modulation (CPM) as a clinical tool for measuring endogenously induced analgesia is increasing. There is, however, large variation in the CPM methodology, hindering comparison of results across studies. Research comparing different CPM protocols is needed in order to obtain a standardized test paradigm. The aim of the study was to assess whether a protocol with phasic heat stimuli as test-stimulus is preferable to a protocol with tonic heat stimulus as test-stimulus. In this experimental crossover study, we compared 2 CPM protocols with different test-stimulus; one with tonic test-stimulus (constant heat stimulus of 120-second duration) and one with phasic test-stimuli (3 heat stimulations of 5 seconds duration separated by 10 seconds). Conditioning stimulus was a 7°C water bath in parallel with the test-stimulus. Twenty-four healthy volunteers were assessed on 2 occasions with minimum 1 week apart. Differences in the magnitude and test-retest reliability of the CPM effect in the 2 protocols were investigated with repeated-measures analysis of variance and by relative and absolute reliability indices. The protocol with tonic test-stimulus induced a significantly larger CPM effect compared to the protocol with phasic test-stimuli (P < 0.001). Fair and good relative reliability was found with the phasic and tonic test-stimuli, respectively. Absolute reliability indices showed large intraindividual variability from session to session in both protocols. The present study shows that a CPM protocol with a tonic test-stimulus is preferable to a protocol with phasic test-stimuli. However, we emphasize that one should be cautious to use the CPM effect as biomarker or in clinical decision making on an individual level due to large intraindividual variability.

A novel and simple classification for ligamentum teres pathology based on joint hypermobility.

Ligamentum teres (LT) pathology (including synovitis, partial and complete tears) is common at the time of hip arthroscopy with a reported prevalence of 51–90%. Currently, there are four published classifications of LT injuries and tears. The majority focuses on differentiating partial from full thickness tears, whereas a more recently published classification also incorporates the presumed underlying mechanism of pathology. A recent review of the current classification systems found that all are deficient for lack of inclusion of what constitutes a normal ligament, lack of inclusion of synovitis as a source of pathology and lack of inclusion of hypermobility as part of the treatment algorithm. Also, the two most commonly used classification systems have only fair inter-observer reliability. Recent work has found that underlying joint hypermobility plays an important role in LT pathology and that the addition of capsular plication/suture at the time of surgery for LT pathology improves outcomes and reduces re-tear rates. In order to address these problems which have been identified with the currently available classification systems, we propose a novel and simple classification for LT pathology based on underlying joint hypermobility [as assessed by the Beighton test score (BTS)]. LT pathology is used to divide all patients into four types: 0 normal (which includes minor fraying), 1 synovitis (which would also include minor fraying), 2 partial tear and 3 complete tear. Further, all types are subdivided into two groups: Group A patients have no clinical evidence of joint hypermobility (BTS < 3), whereas Group B patients do have clinical evidence of joint hypermobility (BTS ≥ 4). On the basis of this classification system and the available literature, we have also developed a treatment algorithm for LT pathology.

Validation of subjective rating scales for assessment of surgical workspace during laparoscopy.

Recently, studies have focused on how to optimize laparoscopic surgical workspace by changes in intra-abdominal pressure, level of muscle relaxation or body position, typically evaluated by surgeons using subjective rating scales. We aimed to validate two rating scales by having surgeons assess surgical workspace in video sequences recorded during laparoscopic surgery. Video sequences were obtained from laparoscopic procedures. Eight experienced surgeons assessed the video sequences on a categorical 5-point scale and a numerical 10-point rating scale. Intraclass correlations coefficients (ICC) and 95% confidence intervals (CI) were calculated for intra- and inter-rater reliability. The 5-point rating scale had an intra-rater ICC of 0.76 (0.69; 0.83) and an inter-rater ICC of 0.57 (0.45; 0.68), corresponding to excellent and fair reliability, respectively. The 10-point scale had an intra-rater ICC of 0.86 (0.82; 0.89) and an inter-rater ICC of 0.54 (0.39; 0.68), corresponding to excellent and fair as well. All surgeons used the full range of the 5-point scale, but only one surgeon used the full range of the 10-point scale. In conclusion, both scales showed excellent intra-rater and fair inter-rater reliability for assessing surgical workspace in laparoscopy. The 5-point surgical rating scale had all categories employed by all surgeons.

A Novel MRI-Based Classification System for Tibial Eminence Fractures in Pediatric Patients: An Improvement from Meyers and McKeever?

The most common system for classifying pediatric tibial spine fractures is based on X-Rays. We propose an MRI-based system. Our results show a moderate intra-rater and fair inter-rater reliability of the new system. In 1/3 of our cases, assessment using MRI substantially modified the treatment. This new system may be a helpful tool for planning the treatment of tibial spine fractures in children. The Meyers and McKeever Classification (MMC) is typically used to classify pediatric tibial spine fractures on plain radiographs. While MRI images can provide important information about the fractures, there is currently no system for classifying these fractures on MRI. The aims of this study were: (1) propose a new MRI-based system for classifying tibial spine fractures and (2) assess how often using an MRI-based system changes the treatment plan compared to the MMC, which is based on X-Rays. Our hypothesis is that an MRI-based system allows for more precise identification and treatment of tibial spine fractures. The new MRI-based system has three grades.-Grade I fractures are defined as non- or minimally-displaced fractures with < 2 mm of displacement.-Grade II fractures are posterior-hinged fractures with > 2 mm displacement of the anterior aspect of the fracture and < 2 mm displacement of the posterior aspect of the fragment.-Grade III fractures are displaced fractures (> 2 mm of displacement of the posterior aspect of the fragment), any tibial spine fracture that results in meniscal entrapment (where the meniscus or intra-meniscal ligament is underneath a fracture fragment), or any tibial spine fracture extending to the weight-bearing surface of the medial or lateral tibial plateau with > 2 mm of displacement. We considered three scenarios as “changes in treatment”: when use of the MRI-system (1) allowed raters to identify a fracture that had been graded as “no fracture” using X-Rays, (2) changed the treatment from non-operative to operative, and (3) changed the treatment from operative to non-operative. The number of cases that experienced a change was reported as raw counts and percentages. Four independent observers (one attending orthopaedic surgeon, one attending radiologist, a senior orthopaedic surgery resident, and a junior orthopaedic surgery resident) classified the images of 20 patients according to the MMC using plain X-Rays. Each observer graded images at two time points at least one week apart. Next, raters repeated their assessments on MRIs using the MRI-based system. The inter-rater and intra-rater reliability of each system was assessed (Stata 14.0 software). Based on the Altman threshold, there was moderate intra-rater correlation between the observers when using MRI-based classification (avg = 0.60; range = 0.45-0.75) and a fair inter-rater correlation (k = 0.33). Following the evaluation of MRIs with the new system, we observed “changes in treatment” in 33% (26 of 80) of grades after the first grading and 34% (27 of 80) after the second compared to the initial evaluation using X-Rays. In 11 grades (20%), MRI recognized previously unnoticed fractures; in 21 grades (40%), MRI assessment upgraded the lesion to operative treatment; in 21 grades (40%) formerly operative candidates were re-assigned to conservative treatment. There is moderate intra-rater and fair inter-rater reliability of the new system. In one third of the cases, assessment using MRI substantially modified the treatment recommendations. This MRI-based system may become a helpful tool for diagnosing and planning the treatment of tibial spine fractures in children.

Generating Rasch-based activity of daily living measures from the Spinal Cord Injury Longitudinal Aging Study.

Retrospective Longitudinal Study. (1) To determine whether the Spinal Cord Injury Activities of Daily Living (SCI_ADL) measure shows adequate item-level and precision psychometrics; (2) to investigate whether the SCI_ADL measure effectively detects ADL changes across time; (3) to describe self-care task(s) participants can and cannot do across time. Two Midwestern hospitals and 1 Southeastern specialty hospital in 1993. All participants were adults with traumatic SCI of at least 1-year duration at enrollment. We used 20-year (1993-2013) retrospective longitudinal data and categorized participants into three injury levels: C1-C4 (cervical; n=50), C5-C8 (n=126) and T1-S5 (thoracic, lumbar and sacral; n=168). We first examined psychometrics of the SCI_ADL with factor and Rasch analyses; then we investigated longitudinal change of SCI_ADL scores at three time points over 20 years (1993, 2003 and 2013) using generalized linear mixed modeling and post hoc analyses. The SCI_ADL measure demonstrated unidimensionality, person strata of 2.9, high Cronbach's α (0.93) and fair person reliability (0.76). T1-S5 had the highest measures, following C5-C8 and C1-C4 at three time points (P<0.05). The C1-C4 and T1-S5 groups showed significant decreases from 2003 to 2013; however, none of the three groups showed significant differences from 1993 to 2003 (P<0.05). The SCI_ADL measure could detect longitudinal ADL changes of the population with SCI across time. The C1-C4 group decreased the most in ADLs, indicating higher need of long-term services and rehabilitation.

Diagnostic performance of central vein sign for multiple sclerosis with a simplified three-lesion algorithm

Background: Detection of a “central vein sign” (CVS) on FLAIR* magnetic resonance imaging (MRI) is highly specific and sensitive for multiple sclerosis (MS). We evaluated the specificity and sensitivity of simplified CVS algorithms for MS diagnosis. Methods: MRIs from 10 participants with MS without additional comorbidities for MRI white matter abnormalities; 10 with MS and additional comorbidities for white matter abnormalities; 10 with migraine, white matter abnormalities, and no additional comorbidities; and 10 who had previously been erroneously diagnosed with MS were evaluated. 3T MRI T2-FLAIR and T2*-weighted sequences were acquired to create FLAIR* images. Three MS physician reviewers, blinded to diagnosis, evaluated two different algorithms: (1) three lesions pre-selected on FLAIR were subsequently evaluated for CVS on FLAIR*(select3). (2) FLAIR* was evaluated for up to three lesions with CVS (select3*). Results: For select3, average specificity across reviewers for MS was 0.98 and sensitivity 0.52 and a correct prediction of diagnosis demonstrated kappa = 0.29. For select3*, specificity was 0.81, sensitivity was 0.83, and kappa was 0.31. Conclusion: A simplified determination of CVS in three white matter lesions on 3T FLAIR* MRI demonstrated good specificity and sensitivity and fair inter-rater reliability for a diagnosis of MS and with further study, may be a candidate for clinical application.

Eyeballing: the use of visual appearance to diagnose 'sick'.

Prior studies suggest that clinicians can categorise patients in an emergency room as 'sick' or 'not sick' using rapid visual assessment. The rapid nature of these decisions suggests clinicians are relying on pattern recognition or System 1 processing; however, this has not been studied experimentally. In this study, we explore the accuracy of these decisions using patient disposition (discharge, admission to ward or admission to critical care) as an objective outcome, and collect evidence to argue for the use of System 1 processing in the 'sick' or 'not sick' decision process. Fourteen practising emergency physicians reviewed 25 videos of patients presenting to the emergency room. They were asked to predict patient disposition (discharge, admission to ward or admission to critical care) and estimate whether they were 'sick' or 'not sick' using a continuous slider on a 'sick' scale from 'not sick' (0) to 'sick' (100). We collected decision time and asked physicians to identify how they came to the decision using a continuous slider on a 'system processing' scale from 'knew immediately' (0) to 'deliberated intently' (1). Inter-rater reliability judging 'sick' was computed as an intraclass correlation coefficient (ICC) of 0.54. Agreement among physicians in predicting disposition was 68% with ICC of 0.44, and accuracy at predicting disposition was 55%. Physicians made their decision in an average of 10 - 11 seconds and rated 70% of their decisions as < 0.5 on the scale from 'knew immediately' (0) to 'deliberated intently' (1). Experienced emergency physicians are able to visually assess patients rapidly and predict disposition in a very short time, albeit with fair reliability and lower accuracy than reported previously. Subjectively, they reported that the majority of decisions were on the side of 'knew immediately', consistent with the application of System 1 processing.

Tactile acuity testing at the neck: A comparison of methods

BackgroundInterest in measurement of tactile acuity in musculoskeletal practice has emerged following its link to functional reorganization of the somatosensory cortex in ongoing pain states. Several tactile acuity measurement methods have been described but have not been thoroughly investigated in the cervical region. ObjectiveThis study examined reliability, concurrent validity and responsiveness of four tests of tactile acuity—Two-point discrimination, Point-to-point, Graphesthesia, and Localisation tests—at the cervical region. MethodForty-two healthy participants were included. In Part 1 (n = 22), participants' tactile acuity was assessed at two time points, 30 min apart, to determine the test-retest reliability and concurrent validity of each of the tests. In Part 2 (n = 20), participants received five daily tactile training sessions, delivered via a vibro-tactile device. Tactile acuity was assessed pre- and post-training to examine responsiveness of each test. ResultsTwo-point discrimination demonstrated excellent test-retest reliability (ICC = 0.85, SEM = 3.7 mm), Point-to-point and Localisation tests demonstrated good reliability (ICC = 0.60, SEM = 2.8 mm; ICC = 0.60, SEM = 8.8%), and Graphesthesia demonstrated fair reliability (ICC = 0.48, SEM = 1.9/20). There was no significant correlation among measures. Only Graphesthesia failed to show responsiveness to change following training. ConclusionThe reliability of Two-point discrimination appears superior to other examined tests of tactile acuity, however measurement variability should be considered. Two-point discrimination, Point-to-point, and Localisation tests appear responsive to change, although testing in clinical samples is needed. The lack of concurrent validity among tests suggests that they cannot be used interchangeably.

Inter-rater reliability and double reading analysis of an automated three-dimensional breast ultrasound system: comparison of two independent examiners.

Breast ultrasound could be a valuable tool complementary to mammography in breast cancer screening. Automated 3D breast ultrasound (ABUS) addresses challenges of hand-held ultrasound and could allow double reading analysis of ultrasound images. This trial assesses the inter-rater reliability and double reading analysis of an ABUS system. To assess the reproducibility and diagnostic validity of the ABUS system, SomoV™, a blinded double reading analysis, was performed in 1019 patients (2038 breasts) by two examiners (examiner A/B) and compared to single reading results, as well as to the reference standard regarding its diagnostic validity. Cohen's kappa coefficients were calculated to measure the inter-rater reliability and agreement of the different diagnostic modalities. Patient comfort and time consumption for image acquisition and reading were analyzed descriptively as secondary objectives. Analysis of inter-rater reliability yielded agreement in 81.6% (κ=0.37; p<0.0001) showing fair agreement. Single reading analysis of SomoV™ exams (examiner A/examiner B) compared to reference standard showed good specificity (examiner A: 88.3%/examiner B: 84.5%), fair inter-rater agreement (examiner A: κ=0.31/examiner B: κ=0.31), and adequate sensitivity (examiner A: 53.1%/examiner B: 64.2%). Double reading analysis yielded good sensitivity and specificity (73.7 and 77.7%). Mammography (n=1911) alone detected 160 of 176 carcinomas (sensitivity 90.1%). Adding SomoV™ to mammography would have detected 12 additional carcinomas, resulting in a higher sensitivity of 97.7%. SomoV™ is a promising technique with good sensitivity, high patient comfort, and fair inter-examiner reliability. It allows double reading analysis that, in combination with mammography, could increase detection rates in breast cancer screening.