Training as a prerequisite for reliable use of NIH Stroke Scale.

Abstract

HomeStrokeVol. 29, No. 6Training as a Prerequisite for Reliable Use of NIH Stroke Scale Free AccessOtherPDF/EPUBAboutView PDFView EPUBSections ToolsAdd to favoritesDownload citationsTrack citationsPermissions ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toFree AccessOtherPDF/EPUBTraining as a Prerequisite for Reliable Use of NIH Stroke Scale Susanne Schmülling, Martin Grond and Jobst Rudolf Peter Kiencke Susanne SchmüllingSusanne Schmülling Klinik und Poliklinik für Neurologie, Universität zu Köln Search for more papers by this author , Martin GrondMartin Grond Klinik und Poliklinik für Neurologie, Universität zu Köln Search for more papers by this author and Jobst RudolfJobst Rudolf Klinik und Poliklinik für Neurologie, Universität zu Köln Search for more papers by this author Peter KienckePeter Kiencke Institut für Medizinische Statistik, Informatik und Epidemiologie, Universität zu Köln, Köln, Germany Search for more papers by this author Originally published1 Jun 1998https://doi.org/10.1161/01.STR.29.6.1258Stroke. 1998;29:1258–1259To the Editor: Before new therapies for ischemic stroke are established, their safety and effectiveness must be proved. In particular, the numerous multicenter acute stroke trials currently being performed require a valid, efficient, and reliable measure of patient status and outcome after treatment. Interrater variation in the assessment of neurological deficits could imply that important effects of the treatment remain concealed, which in turn may have a misleading influence on therapeutic decisions. A commonly used yardstick for measuring the outcome of neurological deficits in stroke patients is the National Institutes of Health Stroke Scale (NIHSS).123 Not only experienced neurologists can reliably apply the NIHSS; it can be used as well by nonneurologists or even nonphysicians (eg, study nurses),4567 provided the raters are well trained and given detailed instructions. As far as the NINDS study is concerned, the investigators were video trained and required to take an examination.1The question, however, of whether the NIHSS provides precise and reliable data when applied without an intensive training program has not yet been raised. We therefore investigated the reliability of the NIHSS as used by trained and untrained raters in 22 stroke patients in the Neurological Department at the University Hospital of Cologne. Diagnosis was confirmed by CT. Eighteen patients were suffering from ischemic and 4 from hemorrhagic stroke; 3 of the strokes were infratentorial and 19 supratentorial; 13 lesions were left and 6 right hemispheric. Five patients were obtunded or comatose, and 5 suffered from severe aphasia.Four neurologists in our department independently assessed the patients’ neurological status. Two raters were experienced in using the NIHSS, video trained and instructed by the material of the NINDS-group (available from B.C. Tilley, PhD, Biostatistics and Research Epidemiology, Henry Ford Health Science Center, 1 Ford Place, Suite 3E, Detroit, MI 48202). The other two were inexperienced in the application of the NIHSS and were given no information other than the original NIHSS examination form. In this form the instructions for rating are very short and do not go into detail on how to handle problematic cases, such as aphasic, comatose, or unresponsive patients. To minimize a possible bias from a training or fatigue effect in the patients, untrained and trained raters were assigned in random order. To reduce the impact of fluctuation on the patients’ neurological state, evaluation had to be performed within a close time window (<90 minutes) only in the state of subacute stroke (>12 hours after symptom onset).5To assess interobserver reliability, the κ statistic was used for every individual item to be examined with use of SPSS for Windows 7.0 (SPSS Inc) and BMDP 7.0 Dynamic (BMDP Statistical Software, Inc). The degree of interrater agreement based on κ is considered excellent if κ>0.80, substantial if κ is between 0.61 and 0.80, moderate if between 0.41 and 0.60, fair if between 0.21 and 0.40, and slight or poor if κ≤0.20.8 Additionally, we compared the total scores because they are generally used for assessment of outcome in clinical studies. However, this is not strictly valid, because the NIHSS does not represent a collection of numerical ratings but rather one of ordinal ratings.7The results regarding the reliability achieved among trained raters with mean κ=0.61 (SD=0.17) show substantial interrater reliability. As far as the untrained group is concerned, κ was 0.33 (SD=0.22), indicating fair interrater reliability. Between trained and untrained raters, the unweighted κ was 0.45 (SD=0.2), indicating moderate agreement. Thus, the reliability achieved among untrained raters and that achieved among trained and untrained raters is substantially poorer than the result achieved among trained raters.Furthermore, reliability of individual items differed substantially between trained and untrained raters. Among trained raters, fair agreement was found only in 2 items, limb ataxia (κ=0.34) and neglect (κ=0.32); there were no items with poor agreement. Among the untrained raters, the items ataxia (−0.03), gaze (κ=0.06), visual fields (κ=−0.02), and dysarthria (κ=0.18) were poorly reliable; furthermore, 6 items were only fairly reliable. The mean total score for the trained raters was 11.81 (SD=10.02; range, 2 to 40); the maximum difference between scores was 3 points in 5 patients, 2 points in 4 patients, and 1 point in 7 patients. Identical score was reached in 5 patients (1 missing). Among untrained raters, the maximum difference was 10 points, and a difference of ≥4 points was found in 4 patients. Between trained and untrained raters, the difference of total scores reached ≥4 points in 12 patients.The results of the present study suggest that good interrater reliability of the NIHSS34567 depends on adequate training of the raters. Interobserver reliability among trained raters was substantial, with a mean of κ=0.61, which is comparable to prior study results (κ between 0.51 and 0.69).34567 By contrast, only fair reliability was achieved among untrained raters (κ=0.33).The discrepancies in interobserver reliability achieved among untrained observers and among untrained and trained observers (κ=0.45) are alarming. They might not only influence study results but also therapeutical decisions. Especially in a large multicenter trial effected on an international basis with numerous raters of different countries, high reliability of the assessment score as a primary outcome measure is crucial. Our study demonstrates that even within one and the same department only fair reliability can be reached without an adequate training program.Apart from the difficulties in the assessment of some individual items, as reported in former studies,34567 one of the major problems the group of the untrained raters faced was the lack of instruction necessary for assessment of the 10 comatose, obtunded, unresponsive, and aphasic patients. In 8 of these patients, there were substantial differences (≥4 or more points) in total scores between trained and untrained raters, because assessment of several individual items such as gaze, visual fields, and dysarthria in those patients is especially problematic if no detailed instruction is provided.In conclusion, without any systematic training program and knowledge of detailed instructions, the NIHSS can not reliably be applied. Therefore, a standardized use of the NIHSS is mandatory. References 1 The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke. N Engl J Med.1995; 333:1581–1587.CrossrefMedlineGoogle Scholar2 Hacke W, Kaste M, Fieschi C, Toni D, Lesaffre E, von Kummer R, Boysen G, Bluhmki E, Höxter G, Mahagne MH, Hennerici M, for the ECASS Study Group. Intravenous thrombolysis with recombinant tissue plasminogen activator for acute hemispheric stroke. JAMA.1995; 274:1017–1025.CrossrefMedlineGoogle Scholar3 Albanese MA, Clarke WR, Adams HP, Woolson RF, and TOAST Investigators. Ensuring reliability of outcome measures in multicenter clinical trials of treatments for acute ischemic stroke. Stroke.1994; 25:1746–1751.CrossrefMedlineGoogle Scholar4 Brott T, Adams HP, Olinger CP, Marler JR, Barsan WG, Biller J, Spilker J, Holleran R, Eberle R, Hertzberg V, Rorick M, Moomaw CJ, Walker M. Measurements of acute cerebral infarction: a clinical examination scale. Stroke.1989; 20:864–870.CrossrefMedlineGoogle Scholar5 Goldstein LB, Bartels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol.1989; 46:660–662.CrossrefMedlineGoogle Scholar6 Goldstein LB, Samsa GP. Reliability of the National Institutes of Health Stroke Scale: extension to non-neurologists in the context of a clinical trial. Stroke.1997; 28:307–310.CrossrefMedlineGoogle Scholar7 Lyden P, Brott T, Tilley B, Welch KMA, Mascha EJ, Levine S, Haley EC, Grotta J, Marler J, and the NINDS TPA Stroke Study Group. Improved reliability of the NIH Stroke Scale using video training. Stroke.1994; 25:2220–2226.CrossrefMedlineGoogle Scholar8 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics.1977; 33:159–174.CrossrefMedlineGoogle Scholar Previous Back to top Next FiguresReferencesRelatedDetailsCited By McLoughlin A, Olive P and Lightbody C (2022) Reliability of the National Institutes of Health Stroke Scale, British Journal of Neuroscience Nursing, 10.12968/bjnn.2022.18.Sup5.S3, 18:Sup5, (S3-S10), Online publication date: 1-Oct-2022. Laurent D, Small C, Goutnik M and Hoh B (2022) Ischemic Stroke Acute Care Neurosurgery by Case Management, 10.1007/978-3-030-99512-6_13, (159-172), . Ma Y, Ou Z and Ou J (2021) Ischemic Attack Encyclopedia of Gerontology and Population Aging, 10.1007/978-3-030-22009-9_1058, (2775-2780), . Ma Y, Ou Z and Ou J (2020) Ischemic Attack Encyclopedia of Gerontology and Population Aging, 10.1007/978-3-319-69892-2_1058-1, (1-6), . Boling B and Keinath K (2018) Acute Ischemic Stroke, AACN Advanced Critical Care, 10.4037/aacnacc2018483, 29:2, (152-162), Online publication date: 15-Jun-2018. Lyden P (2017) Using the National Institutes of Health Stroke Scale, Stroke, 48:2, (513-519), Online publication date: 1-Feb-2017.Lee C, Venketasubramanian N, Wong K, Chen C, Navarro J, Gan H, Lao A, Baroque A, Lokin J, Hiyadan J, Sarfati M, Fangonillo R, Ambasing N, Chua C, San Jose M, Advincula J, Berame E, Canete M, Young S, Mamauag M, Tay S, Pande S, Thirugnanam U, Singh R, Chang H, De Silva D, Chan B, Sharma V, Luen T, Poungvarin N, Muengtaweepongsa S, Towanabut S, Suwanwela N, Chotickanuchit S, Chankrachang S, Nitinun S, de Silva H, Ranawake U, Wijekoon N and Eow G (2015) Comparison Between the Original and Shortened Versions of the National Institutes of Health Stroke Scale in Ischemic Stroke Patients of Intermediate Severity, Stroke, 47:1, (236-239), Online publication date: 1-Jan-2016. Ortiz G and L. Sacco R (2014) National Institutes of Health Stroke Scale ( NIHSS ) Wiley StatsRef: Statistics Reference Online, 10.1002/9781118445112.stat06823 Nanri Y, Yakushiji Y, Hara M, Eriguchi M, Okada R, Yukitake M and Hara H (2013) Stroke Scale Items Associated with Neurologic Deterioration within 24 Hours after Recombinant Tissue Plasminogen Activator Therapy, Journal of Stroke and Cerebrovascular Diseases, 10.1016/j.jstrokecerebrovasdis.2012.10.006, 22:7, (1117-1124), Online publication date: 1-Oct-2013. Siegler J, Boehme A, Kumar A, Gillette M, Albright K and Martin-Schild S (2013) What Change in the National Institutes of Health Stroke Scale Should Define Neurologic Deterioration in Acute Ischemic Stroke?, Journal of Stroke and Cerebrovascular Diseases, 10.1016/j.jstrokecerebrovasdis.2012.04.012, 22:5, (675-682), Online publication date: 1-Jul-2013. Oh M, Yu K, Lee J, Jung S, Ko I, Shin J, Cho S, Choi H, Kim H and Lee B (2012) Validity and Reliability of a Korean Version of the National Institutes of Health Stroke Scale, Journal of Clinical Neurology, 10.3988/jcn.2012.8.3.177, 8:3, (177), . Giantin V, Semplicini A, Franchin A, Simonato M, Baccaglini K, Attanasio F, Toffanello E and Manzato E (2011) Outcome after acute ischemic stroke (AIS) in older patients: Effects of age, neurological deficit severity and blood pressure (BP) variations, Archives of Gerontology and Geriatrics, 10.1016/j.archger.2010.11.002, 52:3, (e185-e191), Online publication date: 1-May-2011. Lokk J, Roghani R and Delbari A (2011) Effect of methylphenidate and/or levodopa coupled with physiotherapy on functional and motor recovery after stroke - a randomized, double-blind, placebo-controlled trial, Acta Neurologica Scandinavica, 10.1111/j.1600-0404.2010.01395.x, 123:4, (266-273), Online publication date: 1-Apr-2011. Adams H (2011) Clinical Scales to Assess Patients with Stroke Stroke, 10.1016/B978-1-4160-5478-8.10021-1, (307-333), . I Zraick R, C O'Neill A, C Robinson G, K McWeeny E and J Hutton T (2010) Training speech-language pathology graduate clinicians to administer the National Institutes of Health Stroke Scale, International Journal of Therapy and Rehabilitation, 10.12968/ijtr.2010.17.8.49289, 17:8, (416-423), Online publication date: 1-Aug-2010. Meyer B and Lyden P (2009) The Modified National Institutes of Health Stroke Scale: its Time has Come, International Journal of Stroke, 10.1111/j.1747-4949.2009.00294.x, 4:4, (267-273), Online publication date: 1-Aug-2009. Ortiz G and Sacco R (2008) National Institutes of Health Stroke Scale (NIHSS) Wiley Encyclopedia of Clinical Trials, 10.1002/9780471462422.eoct400 Meyer M (2008) Stroke Quick Score: a visual aid in scoring neurologic deficits for acute stroke with National Institutes of Health Stroke Scale, The American Journal of Emergency Medicine, 10.1016/j.ajem.2007.04.027, 26:2, (189-190), Online publication date: 1-Feb-2008. Caneda M, Fernandes J, Almeida A and Mugnol F (2006) Confiabilidade de escalas de comprometimento neurológico em pacientes com acidente vascular cerebral, Arquivos de Neuro-Psiquiatria, 10.1590/S0004-282X2006000400034, 64:3a, (690-697), Online publication date: 1-Sep-2006. Anemaet W (2002) Using Standardized Measures to Meet the Challenge of Stroke Assessment, Topics in Geriatric Rehabilitation, 10.1097/00013614-200212000-00006, 18:2, (47-62), Online publication date: 1-Dec-2002. Yaeger E, Morris D, Rosamond W and Evenson K (2000) Use of the National Institutes of Health Stroke Scale in the Emergency Department Setting, Annals of Emergency Medicine, 10.1016/S0196-0644(00)70043-6, 35:6, (628-629), Online publication date: 1-Jun-2000. Odderson I (1999) The National Institutes of Health Stroke Scale and Its Importance in Acute Stroke Management, Physical Medicine and Rehabilitation Clinics of North America, 10.1016/S1047-9651(18)30162-1, 10:4, (787-800), Online publication date: 1-Nov-1999. June 1998Vol 29, Issue 6 Advertisement Article InformationMetrics Copyright © 1998 by American Heart Associationhttps://doi.org/10.1161/01.STR.29.6.1258 Originally publishedJune 1, 1998 PDF download Advertisement