Experienced Raters Research Articles

A study of the reliability of the Nociception Coma Scale

Objective: In this study, we investigated the reliability of the Nociception Coma Scale which has recently been developed to assess nociception in non-communicative, severely brain-injured patients. Design: Prospective cross-sequential study. Setting: Semi-intensive care unit and long-term brain injury care. Subjects: Forty-four patients diagnosed as being in a vegetative state (n=26) or in a minimally conscious state (n=18). Interventions: Patients were assessed by two experts (rater A and rater B) on two consecutive weeks to measure inter-rater agreement and test-retest reliability. Main measures: Total scores and subscores of the Nociception Coma Scale. Results: We performed a total of 176 assessments. The inter-rater agreement was moderate for the total scores (k = 0.57) and fair to substantial for the subscores (0.33 ≤ k ≤ 0.62) on week 2. The test-retest reliability was substantial for the total scores (k = 0.66) and moderate to almost perfect for the subscores (0.53 ≤ k ≤ 0.96) for rater A. The inter-rater agreement was weaker on week 1, whereas the test-retest reliability was lower for the least experienced rater (rater B). Conclusions: This study provides further evidence of the psychometric qualities of the Nociception Coma Scale. Future studies should assess the impact of practical experience and background on administration and scoring of the scale.

Automated segmentation and quantification of white matter hyperintensities in acute ischemic stroke patients with cerebral infarction.

White matter hyperintensities (WMHs) of presumed vascular origin are common in ageing population, especially in patients with acute cerebral infarction and the volume has been reported to be associated with mental impairment and the risk of hemorrhage from antithrombotic agents. WMHs delineation can be computerized to minimize human bias. However, the presence of cerebral infarcts greatly degrades the accuracy of WMHs detection and thus limits the application of computerized delineation to patients with acute cerebral infarction. We propose a computer-assisted segmentation method to depict WMHs in the presence of cerebral infarcts in combined T1-weighted, fluid attenuation inversion recovery, and diffusion-weighted magnetic resonance imaging (MRI). The proposed method detects WMHs by empirical threshold and atlas information, with subtraction of white matter voxels affected by acute infarction. The method was derived using MRI from 25 hemispheres with WMHs only and 13 hemispheres with both WMHs and cerebral infarcts. Similarity index (SI) and correlation were utilized to assess the agreement between the new automated method and a gold standard visually guided semi-automated method done by an expert rater. The proposed WMHs segmentation approach produced average SI, sensitivity and specificity of 83.142±11.742, 84.154±16.086 and 99.988±0.029% with WMHs only and of 68.826±14.036, 74.381±18.473 and 99.956±0.054% with both WMHs and cerebral infarcts in the derivation cohort. The performance of the proposed method with an external validation cohort was also highly consistent with that of the experienced rater.

Development and Validation of an Objective Balance Error Scoring System

Limited access to sophisticated technology and the unreliability of simple tools prevent accurate and reliable human standing balance assessments outside research laboratory settings. The goal of this study was to develop and validate a simple objective balance assessment tool that provides an accurate, reliable, and affordable alternative to currently available laboratory and clinical methods. Thirty healthy subjects were filmed performing the Balance Error Scoring System (BESS) while wearing inertial measurement units (IMU) measuring linear accelerations and angular velocities from seven locations of the body: forehead, sternum, waist, right and left wrist, and right and left shin. Each video was scored by four experienced BESS raters, whose mean scores were used to develop an algorithm computing objective BESS (oBESS) scores solely from IMU data. Interrater reliability and accuracy of oBESS scores were assessed using intraclass correlations (ICC). Raters displayed low variability in scoring (ICC3,1 = 0.91). The oBESS was able to produce scores with accurate fit to raters (ICC3,1 = 0.92) and predicted individual BESS scores (ICC3,1 = 0.90) using data from one IMU placed at the forehead. oBESS was unable to produce accurate scores (ICC3,1 = 0.68) when using IMU data from the subset of conditions (firm surface only) used in popular concussion identification protocols. The oBESS can reliably predict total BESS scores in healthy subjects. Pending further validation, oBESS could represent a valid tool to assess balance by offering an objective and reliable alternative to the current scoring methods of the BESS.

Static Versus Dynamic Musculoskeletal Ultrasound for Detection of Plantar Plate Pathology.

Diagnostic Level II, Prospective.

Diffusion-weighted Imaging at b1000 for Identifying Intracerebral Hemorrhage: Preliminary Sensitivity, Specificity, and Inter-rater Variability

Noncontrast computed tomography (NCCT) is the gold standard to detect intracerebral hemorrhage (ICH) and ischemic stroke (IS) in patients presenting with acute focal syndromes. Diffusion-weighted magnetic resonance imaging (DW-MRI) obtained at b1000 is highly sensitive to identify acute IS but its sensitivity and specificity to detect ICH has not been systematically studied. Patients with a diagnosis of ICH on NCCT were prospectively enrolled and underwent DW-MRI at b1000. Patients with suspected ischemia and a negative NCCT served as controls. All diffusion-weighted imaging (DWI) scans were evaluated blindly by 4 experienced raters. Sensitivity, specificity, and inter-rater variability of the DWI b1000 scans for detection of ICH were determined. In this preliminary pilot study, 15 patients with ICH and 17 patients with IS were included. All ICH lesions seen on NCCT showed a typical pattern on DW-MRI at b1000 with a hypointense core surrounded by a hyperintense rim. ICH volumes and size were similar on NCCT and MRI. All cases of IS were identified on the DWI scans but none were apparent on NCCT. The mean sensitivity and specificity of DW-MRI at b1000 for ICH were 94% and 93.5%, respectively, and the inter-rater variability for ICH detection on DWI was excellent (κ=.84). DW-MRI at b1000 has a diagnostic yield similar to NCCT for detecting ICH and superior to NCCT for detecting IS.Therefore, DW-MRI may be considered as the initial screening tool for imagingpatients presenting with focal neurologic symptoms suggestive of stroke.

Spine Curve Measures Taken in Individuals with Osteoporosis and Osteopenia Using the IONmed Mobile Phone Application Are Highly Consistent with Digital Inclinometer Measures

Introduction. This study determined the agreement between measures of standing posture in individuals at risk of osteoporotic fracture obtained using the IONmed mobile phone application and a digital inclinometer. Methods. One experienced rater assessed 20 adults, aged 55–88 years, attending an outpatient osteoporosis clinic. A standardized protocol was duplicated to acquire a single measure of spine curvature (lumbosacral angle, lordosis, kyphosis) using the IONmed mobile phone application and a digital inclinometer. Interdevice agreement was determined using the intraclass correlation coefficient (ICC) with 95% confidence intervals (CI) and standard error of the measurement (SEM) with 95% CI and Bland-Altman plots. Results. ICC (95% CI) for lumbosacral angle, lordosis, and kyphosis is 0.97 (0.92, 0.99), 0.97 (0.92, 0.99), and 0.99 (0.98, 1.0), respectively. SEM (95% CI) for lumbosacral angle, lordosis, and kyphosis is 0.99° (0.75, 1.44), 1.33° (1.01, 1.94), and 1.23° (0.94, 1.80), respectively. Measurement bias is small. Conclusions. In our sample, the IONmed mobile phone application provides measures highly consistent with the digital inclinometer measures of standing posture. This method offers clinicians a quick, convenient way of identifying alterations in spine alignment which require followup and remediation.

Postharvest noninvasive assessment of fresh chestnut (Castanea spp.) internal decay using computer tomography images

X-ray computed tomography (CT) is an effective noninvasive tool to visualize fresh agricultural commodities’ internal components and quality attributes, including those of chestnuts (Castanea spp). There is no procedure to automatically, effectively and efficiently classify fresh commodities from a continuous inline flow through a CT system. If the information obtained by CT scanning of fresh agricultural commodities is to be used in an industrial application (e.g. inline sorting), automated interpretation of CT images is essential. For this purpose, an image analysis method (algorithm) for the automatic classification of CT images obtained from 2848 fresh chestnuts (cv. ‘Colossal’ and ‘Chinese seedlings’), during the harvesting years from 2009 to 2012, was developed and tested. Classification accuracy was evaluated by comparing the classes obtained from six CT images per chestnut to their internal quality assessment. An experienced human rater performed internal quality assessment by visually and invasively rating fresh chestnut internal decay severity (quality) into 5-, 3- and 2-classes.After CT image preprocessing, cropping and segmentation, 1194 grayscale intensity and textural features were extracted from six resultant CT images per sample. Relevant features were selected using a sequential forward selection algorithm with the Fisher discriminant objective function. 86, 155 and 126 features were effective in designing a quadratic discriminant classifier with a 4-fold cross-validation with a performance accuracy of 85.9%, 91.2% and 96.1% for 5, 3 and 2 classes, respectively. This method is accurate and objective in determining fresh chestnut internal quality, and the methodology is applicable to automatic noninvasive inline CT sorting system development.

Development and validation of a standard area diagram set to aid assessment of severity of loquat scab on fruit

A standard area diagram set (SAD) to aid visual assessment of loquat scab (caused by Fusicladium eriobotryae) severity on fruit was developed and evaluated for improving accuracy, precision and reliability of visual estimates. The SAD set contains eight black and white diagrams of diseased fruit with severity values from 2 % to 98 %. To evaluate the SADs, a group of 20 raters (comprising 10 ‘experienced’ and 10 ‘inexperienced’ raters) assessed the same set of 50 images three times, the first without SADs and the second and third using the SADs as an aid. Only for the group of inexperienced raters did SADs significantly improve accuracy (bias correction factor, C b = 0.93 without SADs and 0.98 with SADs), precision (correlation coefficient, r = 0.88 without SADs and r = 0.96 with SADs) and overall agreement (Lin’s concordance correlation coefficient, ρ c = 0.82 without SADs and ρc with SADs = 0.95) of the estimates. Accuracy and precision of the estimates by inexperienced raters were significantly higher than those obtained by the experienced raters, especially for the second assessment with SADs. Inter-rater reliability was improved when SADs were used by inexperienced raters, whereas a high degree of intra-rater reliability was obtained by both experienced and inexperienced raters when using SADs. The SADs developed in this study were useful for obtaining more accurate, precise and reliable assessments of loquat scab for inexperienced raters, and should be used as an aid for assessing scab in epidemiological studies or monitoring for decision-making purposes.

Cohesion and Coherence in the Writings of Iranain EFL Students

This study was conducted in to investigate the relationship of cohesive chains and chain interaction to the coherence of texts. To do this, the following procedure was followed. First, 95 EFL students were asked to write a composition on a particular topic. These texts were scored by three experienced raters based on their perceived degree of coherence. The texts were then ordered from high to low. Thirty high-rated texts were labeled as group A, and thirty low-rated texts were labeled as group B. These texts were analyzed for the presence and frequency of cohesive chain and chain interaction based on the model proposed by Hassan (1989). After collecting the data and statistically analyzing them, the results showed that the texts getting higher coherence scores had a higher proportion of relevant tokens to peripheral ones and a higher proportion of central tokens to non-central ones in comparison to the low-rated texts. Therefore, cohesive chain and chain interaction can be used as an indicator of coherence and has pedagogical and theoretical implications.

Abstract T MP23: M2 Vessel Diameter and Clot Length Influence Degree of Successful Recanalization Within 6 Hours Of Iv-tpa Treatment

Introduction: Occlusions of the M2 branch show recanalization rates of 44% at 2 hours post treatment with IV-tPA when measured by TCD however no studies have looked at recanalization rates in early (<6h) time windows with angiography. Objectives: We aim to define recanalization rates for M2 occlusions with IV-tPA in an early time window (<6H) using CTA and explore the role of M2 size and clot length in predicting successful recanalization. Methods: patients with acute intracranial occlusions were prospectively enrolled into the InterrSect study and had a baseline CTA at time of inclusion and within 6 hours of receiving IV-tPA. Baseline CTA was analyzed on the OSIRIX platform; two experienced raters measured M2 diameter at the proximal clot interface and M1 diameter at mid-segment on axial MIP images. Two experienced raters independently analyzed clot length and recanalization on follow-up CTA (75/81) and first run angiograms (6/81). Clot length was stratified as (long>15mm) or (short0.8) or (Low <= 0.8). Results: 81 patients with M2 occlusions treated with IV-tPA were included in the study. Successful recanalization was achieved in 63.5% of patients with IV-tPA within 4-6 hours of bolus. Figure1 demonstrates a significant relationship between clot lengths, ratio M2/M1 to early recanalization (Fishers exact p= 0.035). Using general linear modelling, the rate of early recanalization is 1.9 times (p= 0.018, 95%CI 1.12-3.37) higher in the Short clot-Low M2/M1 ratio group when compared to the group with long clot and high ratio after adjusting for age. Conclusion: Small M2 clots treated with IV-tPA achieve early recanalization exceeding 80%; large M2 clots in comparison achieve recanalization rates below 50%. The later clots could be included in clinical trials comparing IV with endovascular therapy.

Observational skills assessment score: reliability in measuring amount and quality of use of the affected hand in unilateral cerebral palsy

BackgroundThe Observational Skills Assessment Score (OSAS) measures amount and quality of use of the affected hand in children with unilateral Cerebral Palsy (CP) in bimanual activities and could therefore be a valuable addition to existing assessment tools. The OSAS consists of tasks that are age appropriate and require use of the affected hand.MethodsTo measure the agreement and reliability of the OSAS a convenience sample of two groups of 16 children with unilateral spastic CP (2.5-6 and 12–16 years old), performed age specific bimanual tasks in 2 measurement sessions. Three experienced raters took part in testing and 8 in scoring. Intra class correlation (ICC) values for intra- and inter-rater reliability, and the mean and standard deviation of the differences between measurements were calculated. For test-retest reliability beside ICC scores, Smallest Detectable Differences (SDDs) were calculated in 16 older and 10 younger children.ResultsGenerally, there seems to be good agreement between repeated measurements of the OSAS, as indicated by the small SDDs on most scales for quality of movement, compared to the range of their scales. This indicates potentially good sensitivity to change if used for patient evaluation purposes. The exceptions were the ‘quality of reach’ score for all tasks, and all quality scores for the stacking blocks task for the young children. As used in the present study, the OSAS has good discriminative capacity within patient populations as indicated by the high ICCs for most quality scores. Measuring the amount of use does not seem to be useful for either discrimination or evaluation.ConclusionIn general, the OSAS seems to be a reliable tool for assessing the quality of use of the affected hand in bimanual activities in younger and older children with unilateral CP. Some modifications may improve its usefulness and efficiency.

Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale

Recent literature suggests a revival of interest in single-case methodology (e.g., the randomised n-of-1 trial is now considered Level 1 evidence for treatment decision purposes by the Oxford Centre for Evidence-Based Medicine). Consequently, the availability of tools to critically appraise single-case reports is of great importance. We report on a major revision of our method quality instrument, the Single-Case Experimental Design Scale. Three changes resulted in a radically revised instrument, now entitled the Risk of Bias in N-of-1 Trials (RoBiNT) Scale: (i) item content was revised and increased to 15 items, (ii) two subscales were developed for internal validity (IV; 7 items) and external validity and interpretation (EVI; 8 items), and (iii) the scoring system was changed from a 2-point to 3-point scale to accommodate currently accepted standards. Psychometric evaluation indicated that the RoBiNT Scale showed evidence of construct (discriminative) validity. Inter-rater reliability was excellent, for pairs of both experienced and trained novice raters. Intraclass correlation coefficients of summary scores for individual (experienced) raters: ICCTotalScore = .90, ICCIVSubscale = .88, ICCEVISubscale = .87; individual (novice) raters: ICCTotalScore = .88, ICCIVSubscale = .87, ICCEVISubscale = .93; consensus ratings between experienced and novice raters (ICCTotalScore = .95, ICCIVSubscale = .93, ICCEVISubscale = .93. The RoBiNT Scale thus shows sound psychometric properties and provides a comprehensive yet efficient examination of important features of single-case methodology.

SAT0422 Increased Supraspinatus Thickness with Reduced Acromohumeral Distance in People with Mild and Moderate Rotator Cuff Tendinopathy: A Reliability and Comparative Study

BackgroundSubacromial impingement syndrome has historically been ascribed to extrinsic compression of the supraspinatus tendon (SsT) due to altered acromial morphology. More recently intrinsic tendon pathology has been proposed as an...

Pecan Scab Severity-Effects of Assessment Methods.

Pecan scab is caused by the fungus Fusicladium effusum, and is the most destructive disease of pecan in the United States. Accurate and reliable disease assessments are needed to ensure that data provide a measure of actual disease intensity. The Horsfall-Barratt (H-B) category scale and its derivatives are commonly used to assess disease. Estimates using the H-B scale were compared with nearest percent estimate (NPEs) for rating disease severity of pecan scab on valves of fruit. Both inexperienced and experienced raters were included in the experiment. Lin's concordance correlation showed that agreement using NPEs was variable (ρc = 0.57 to 0.96), whereas estimates of disease severity using the H-B scale had similar agreement among most raters (ρc = 0.59 to 0.98). Converted values of NPEs to the H-B midpoints (NPEH-B) also provided a similar range (ρc = 0.61 to 0.96). Neither experienced nor inexperienced raters were consistently better using any of the three methods. Bootstrap analysis indicated that, among experienced raters, precision (r) and agreement (ρc) were often reduced when using the H-B scale compared with NPEs. There was no consistent effect of converting NPEs to NPEH-B midpoint values compared with actual H-B values. Inter-rater reliability using the H-B scale was never better than NPEs. Bootstrap analysis indicated no difference in the length of time needed to assess disease but regression analysis suggested that raters who were inherently fast in assessing disease with NPEs were often slower when using the H-B scale; conversely, raters who were slow assessing with NPEs were often faster when using the H-B scale. Thus, there appears to be no advantage in accuracy or reliability or reduction in time when inexperienced or experienced raters used a category rating scale to assess pecan scab.

Interobserver Agreement for Confocal Imaging of Ampullary Lesions

Malignant ampullary lesions can be difficult to classify by endoscopy alone. Probe-based confocal laser endomicroscopy (pCLE) permits in vivo assessment of mucosal structures in the gastrointestinal tracts in the real time. The objective of this pilot multicenter study was to assess the interobserver agreement and variance in interpretation of pCLE of ampullary lesions. Twelve pCLE video clips of ampullary lesions were distributed to 6 gastrointestinal specialists at 5 medical centers, blinded to final pathologic results. Six variables were assessed for interobserver agreement using κ statistics. Variables included an epithelial outer border with irregular thickness, dark epithelium without discernable individual cells, heterogenously distributed elongated crypts, reduced number of goblet cells, neovascularization, and final diagnosis. The overall interobserver agreement for all observers was poor to slight for all variables (κ=0.02, 0.05, -0.01, 0.04, 0.018) except for the first variable with fair degree of agreement (κ=0.27). On the basis of experience, 3 observers were classified as less experienced, whereas 3 were classified as most experienced. Upon stratification, the less experienced observers had poor interobserver agreement for all variables, except 1. The most experienced observers had poor agreement for 2 variables, slight agreement for 3 variables, and fair agreement for the final diagnosis variable. The overall interpersonal agreement on pCLE for ampullary lesions was poor. The interobserver agreement was not substantially improved for experienced raters. Further standardization of pCLE image criteria is needed for ampullary lesions. Standardized training may improve interrater reliability to an acceptable level.

FASTER: an unsupervised fully automated sleep staging method for mice

Identifying the stages of sleep, or sleep staging, is an unavoidable step in sleep research and typically requires visual inspection of electroencephalography (EEG) and electromyography (EMG) data. Currently, scoring is slow, biased and prone to error by humans and thus is the most important bottleneck for large-scale sleep research in animals. We have developed an unsupervised, fully automated sleep staging method for mice that allows less subjective and high-throughput evaluation of sleep. Fully Automated Sleep sTaging method via EEG/EMG Recordings (FASTER) is based on nonparametric density estimation clustering of comprehensive EEG/EMG power spectra. FASTER can accurately identify sleep patterns in mice that have been perturbed by drugs or by genetic modification of a clock gene. The overall accuracy is over 90% in every group. 24-h data are staged by a laptop computer in 10 min, which is faster than an experienced human rater. Dramatically improving the sleep staging process in both quality and throughput FASTER will open the door to quantitative and comprehensive animal sleep research.

Surface electrical stimulation in dysphagic parkinson patients: A randomized clinical trial

A new treatment for oropharyngeal dysphagia in Parkinson's disease was evaluated in the present study. Prospective randomized controlled trial. The study describes the effects of surface electrical stimulation (SES) of the neck (submental region) in dysphagic Parkinson patients using different intensities of electrical current. Quasi-random allocation was performed when assigning patients to treatment groups. Three groups consisting of dysphagic patients with idiopathic Parkinson's disease (N = 90) received daily treatment for 15 days with periods of no treatment during the weekend. All three received traditional logopedic dysphagia treatment. In addition, two groups received SES, either motor-level or sensory-level stimulation. A standardized measurement protocol, including fiber optic endoscopic evaluation of swallowing (FEES) and videofluoroscopy of swallowing (VFS), was performed before and after therapy. A team of experienced raters was blinded to the treatment group and to the moment of measurement. Intrarater and interrater reliability were calculated. Using proportional odds models (POMs), some of the visuoperceptual ordinal outcome variables showed significant improvement in all groups following treatment. Following 15 days of SES of the submental region, few significant effects were found, suggesting a therapy effect of traditional logopedic dysphagia treatment without any additional influence of SES. On the grounds of this study, it is concluded that further research is needed on the exact mechanism of SES and its effects on the neural pathways involved in swallowing.

Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation: Revisiting Research Conventions

This mixed-methods study examines the effects of rating scale length and rater experience on listeners' judgments of second-language (L2) speech. Twenty experienced and 20 novice raters, who were randomly assigned to 5-point or 9-point rating scale conditions, judged speech samples of 38 newcomers to Canada on numerical rating scales for comprehensibility, accentedness, and fluency. Results yielded high Cronbach's alpha coefficients and no group differences for rating scale length or rater experience. However, Rasch category probability plots revealed that raters had difficulty differentiating between scale steps, particularly in mid-scale range—a challenge that was exacerbated in the 9-point scale condition. Evidence from verbal protocols and posttask interviews suggested that experienced and novice raters adopted strategies to either draw on or offset their perceived experience with L2 speech in conducting their ratings. Implications for L2 pronunciation research are discussed, including the need to draw on raters' perceptions to develop a greater understanding of major listener perceptual constructs in L2 pronunciation and to move beyond adherence to the native speaking standard in data collection procedures.

Interrater Reliability of the Star Excursion Balance Test

Dynamic postural control has gained popularity as a more useful assessment of function than static postural control. One measurement of dynamic postural control that has increased in frequency of use is the Star Excursion Balance Test (SEBT). Although the intrarater reliability of the SEBT is excellent, few authors have determined interrater reliability. Preliminary evidence has shown poor reliability between assessors. To determine interrater reliability using a group of investigators at 2 testing sites. A corollary purpose was to examine the interrater reliability when using normalized and nonnormalized performance scores on the SEBT. Descriptive laboratory study. University research laboratory. A total of 29 healthy participants between 18 and 50 years of age. Participants were evaluated by 5 raters at 2 testing sites. After participants performed 4 practice trials, each rater assessed 3 test trials in the anterior, posteromedial, and posterolateral reaching directions of the SEBT. Normalized and nonnormalized (leg-length) reaching distances were analyzed. Additionally, the mean and maximum values from the 3 test trials were analyzed, producing a total of 16 variables. For all 16 measures, the interrater reliability was excellent. For the normalized maximum excursion distances, the intraclass correlation coefficients (1,1) ranged from 0.86 to 0.92. Reliability for the nonnormalized measurements was stronger, ranging from 0.89 to 0.94. When the raters have been trained by an experienced rater, the SEBT is a test with excellent reliability when used across multiple raters in different settings. This information adds to the body of knowledge that exists regarding the usefulness of the SEBT as an assessment tool in clinical and research practice. Establishing excellent interrater reliability with normalized and nonnormalized scores strengthens the evidence for using the SEBT, especially at multiple sites.

Interrater Reliability of 2 Definitions of Seizure Adequacy in Electroconvulsive Therapy

Clear evidence of a generalized seizure is regarded as an essential component of a therapeutic electroconvulsive therapy (ECT) session. Recent criteria, in particular, the Royal College of Psychiatry ECT Handbook (2005), suggest that the pattern of electroencephalographic (EEG) seizure is more important than the duration of the seizure when assessing seizure adequacy. We examined the reliability of using EEG criteria compared to simple measures of seizure duration by reviewing 100 sample EEG printouts obtained during ECT sessions in a Scottish hospital ECT unit. The EEGs were independently assessed by 3 clinicians (a "control" ECT consultant, an experienced ECT therapist, and a less experienced trainee) blind to each other's ratings. While there was good interrater reliability for measures of seizure duration, kappa scores for EEG criteria were much better for the more than the less experienced rater (κ = 0.68 vs 0.50). Even greater differences were seen for individual components: polyspike activity (κ = 0.80 vs 0.55), spike and wave complexes (κ = 0.80 vs 0.38), and postictal suppression (κ = 0.63 vs 0.35). The implications of these finding are briefly discussed in the context of developing rational criteria for assessing ECT efficacy that rely upon the consistent and replicable delivery of standardized ECT techniques and consequent training needs of ECT practitioners.