Video Raters Research Articles

Functional movement screen test: A reliable screening test for young elite ice hockey players

ObjectivesTo determine inter-rater and intra-rater reliability of the Functional Movement Screen (FMS) test among young elite hockey players. DesignReliability study. SettingInter-rater reliability was evaluated by two raters in the field. All performances were videotaped. Two other raters evaluated the videos once and then again 6 weeks later to determine intra-rater reliability. A weighted kappa statistic was used to analyze intra-rater and inter-rater reliability of each FMS sub-test, while an intra-class correlation coefficient (ICC) was calculated for the total score. ParticipantsTwenty-eight male hockey players aged 13–16. Main OutcomeFMS total and sub-tests scores. ResultsThe video raters demonstrated excellent intra-rater reliability for the total score, with an ICC of 0.96 (95% CI; 0.92–0.98) and 0.96 (95% CI; 0.91–0.98). The field raters achieved excellent inter-rater reliability for the total score, with an ICC of 0.96 (95% CI; 0.92–0.98). Sub-test analysis showed good agreement among all four raters for five of the seven main sub-tests. ConclusionFMS is a reliable test for young elite hockey players. Further research should be done to assess the predictive validity of the FMS test within this population so that physiotherapists may eventually use it as an injury prevention tool.

Reliability and Validity of Swallows as a Measure of Breast Milk Intake in the First Days of Life

Breastfeeding assessment in the hospital to determine adequacy of feeds remains controversial. Swallow evaluation is integral to current assessment tools, but the literature is not clear about whether the number of swallows is an accurate indicator of breast milk intake in early postpartum. To determine the reliability and validity of swallows as a measure of breast milk intake in the first days of a newborn's life. Thirty mother-baby dyads were observed at one breastfeeding; pre- and postfeed weights were done, bedside audible swallows were counted, and feeds were videotaped for independent rating. Milk intake was determined from weight change, adjusted for insensible water loss. Number of swallows was significantly and positively correlated with breast milk intake (r = .71). Number of swallows alone, however, accounted for only 50.8% (R (2) = .508) of the variation in milk intake. Infant age was the best predictor of milk intake (β(age) = .56 vs β(No. swallows) = .36), accounting for 68% of the variation in milk intake when combined with swallows. The majority of infants took in 10 g or less of breast milk (77%); length of feeds was variable. In-room (at bedside) lactation consultant (LC) and video rater LC swallow counts were significantly and positively correlated (r = .93; P < .01). However, the 2 lactation consultant swallow numbers include clinically important discrepancies in a Bland-Altman analysis. Number of swallows was not supported as a reliable or valid indicator of milk intake and adequacy of a feed in the first few days of life.

Agreement on Diagnosis Among Patients, Companions, and Professionals After a Dementia Evaluation

A diagnosis of dementia is challenging to deliver and to hear; yet, agreement on diagnosis is essential for effective treatment for dementia. We examined consensus on the results of an evaluation of dementia in 90 patients assessed at an Alzheimer's Disease Research Center. Diagnostic impressions were obtained from 5 sources: (1) the physician's chart; (2) the patient who was evaluated; (3) a companion present at the evaluation; (4) a diagnostic summary written by a nurse present during the evaluation; and (5) raters who watched a video of the diagnostic disclosure conversation. Overall, diagnostic consensus was only moderate. Patients and companions exhibited just fair agreement with one another. Agreement was better between physicians and companions compared with that between physicians and patients, although it was imperfect between physicians and video raters and the written summary. Agreement among sources varied by dementia severity, with the lowest agreement occurring in instances of very mild dementia. This study documents discrepancies that can arise in diagnostic communication, which could influence adjustment to a diagnosis of dementia and decisions regarding future planning and care.

Effectiveness of IV Cannulation Skills Laboratory Training and Its Transfer into Clinical Practice: A Randomized, Controlled Trial

BackgroundThe effectiveness of skills laboratory training is widely recognized. Yet, the transfer of procedural skills acquired in skills laboratories into clinical practice has rarely been investigated. We conducted a prospective, randomised, double-blind, controlled trial to evaluate, if students having trained intravenous (IV) cannulation in a skills laboratory are rated as more professional regarding technical and communication skills compared to students who underwent bedside teaching when assessed objectively by independent video assessors and subjectively by patients.Methodology and Principal Findings84 volunteer first-year medical students were randomly assigned to one of two groups. Three drop-outs occurred. The intervention group (IG; n = 41) trained IV cannulation in a skills laboratory receiving instruction after Peyton's ‘Four-Step Approach’. The control group (CG; n = 40) received a bedside teaching session with volunteer students acting as patients. Afterwards, performance of IV cannulation of both groups in a clinical setting with students acting as patients was video-recorded. Two independent, blinded video assessors scored students' performance using binary checklists (BC) and the Integrated Procedural Protocol Instrument (IPPI). Patients assessed students' performance with the Communication Assessment Tool (CAT) and a modified IPPI. IG required significantly shorter time needed for the performance on a patient (IG: 595.4 SD(188.1)s; CG: 692.7 SD(247.8)s; 95%CI 23.5 s to 45.1 s; p = 0.049) and completed significantly more single steps of the procedure correctly (IG: 64% SD(14) for BC items; CG: 53% SD(18); 95%CI 10.25% to 11.75%; p = 0.004). IG also scored significantly better on IPPI ratings (median: IG: 3.1; CG: 3.6; p = 0.015;). Rated by patients, students' performance and patient-physician communication did not significantly differ between groups.ConclusionsTransfer of IV cannulation-related skills acquired in a skills laboratory is superior to bedside teaching when rated by independent video raters by means of IPPI and BC. It enables students to perform IV cannulation more professionally on volunteer students acting as patients.

The evaluation of music therapy process in the intersubjective perspective: the music therapy rating scale. A pilot study.

This study presents a tool (Music Therapy Rating Scale [MTRS]) to evaluate the progression of the relationship between the patient and the therapist during music therapy sessions. The rating scale was developed from an intersubjective framework and from an improvisational music therapy technique. It mainly consists of two behavioral classes: non-verbal communication (NVC) and sonorous-musical communication (SMC). Five levels for each class are given (from 0 to 4). Every 15 seconds, two sets of two music therapists coded the behaviors in seven sessions with different patients. The patients were seven children (four males; three females) aged 3–10 years (mean age = 6.28), diagnosed with Pervasive Developmental Disorder. The reliability indices were computed and there was a substantial agreement in the assessments of the video raters (non-verbal relationship: ρ = 0.70; sonorous-musical relationship: ρ = 0.77). A digital output of the program to use the MTRS is presented and clinical implications are discussed.

Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped Performances

The Global Operative Assessment of Laparoscopic Skill (GOALS) has been shown to meet high standards for direct observation. The purpose of this study was to investigate the reliability and validity of GOALS when applied to blinded, videotaped performances. Five novice surgeons and 5 experienced surgeons were each evaluated by 2 observers during a laparoscopic cholecystectomy. Subsequently, 4 laparoscopists (V1 to V4) evaluated the videotaped procedures using GOALS. Two of the raters (V1 and V3) had prior experience using GOALS. The interrater reliabilities between video raters (VRs) and between VRs and direct raters (DRs) were calculated using the intraclass correlation coefficient. Construct validity was assessed using 2-way analysis of variance. Interrater reliability between the 4 VRs and the 2 DRs was 0.72. The intraclass correlation coefficient for the 4 VRs was 0.68 and for each VR compared with the mean DR was 0.86, 0.39, 0.94, and 0.76, respectively. All raters, except V2, differentiated between novice and experienced groups (P values ranged from .01 to .05). These data suggest that GOALS can be used to assess laparoscopic skill based on videotaped performances but that rater training may play an important role in ensuring the reliability and validity of the instrument. Experience with the tool in the operating room may improve the reliability of video rating and could be of value in training evaluators.

A Coding Scheme for the Evaluation of the Relationship in Music Therapy Sessions

This study presents a coding system for observation and monitoring of changes in the interactive behaviour between patient and therapist during music therapy sessions. The coding scheme was developed from a psychodynamic framework and mainly consists of four behavioural classes: Verbal Communication, Nonverbal Communication, Countenance, and Sonorous Musical Communication. The 15 minutes in the middle of each videotape concerning the first active music therapy session--based on the sonorous musical improvisation--were coded. Subjects were children (4 boys; 3 girls) ages 3 to 10 years (M age = 6.3), diagnosed with Pervasive Developmental Disorder, and seven therapists. The method for data collection was continuous recording, applied through The Observer Video-Pro 5.0. For the reliability indexes there was a substantial agreement between assessments by video raters.

A rapid method for mass screening for parkinsonism

Epidemiology studies of parkinsonism employ a variety of techniques for unbiased sampling of populations. No current method permits mass screening of all subjects in a population for parkinsonism by movement disorders specialists. We developed and piloted a new approach to facilitate accurate and efficient screening of large populations for diagnosis of parkinsonism and provide data on sensitivity and specificity. We evaluated 2081 welders referred for medical-legal screening. Subjects were video taped using a standardized protocol, and videos were rated on the Unified Parkinson's Disease Rating Scale motor subsection 3 (UPDRS3). A “video rater” viewed video tapes and entered ratings through a web-based database. An “in-person” examiner performed a UPDRS3 examination in a randomly selected subgroup of 48 workers drawn from the 2081. We developed quantitative diagnostic criteria for parkinsonism that established minimum diagnostic thresholds based upon UPDRS3 scores and compared these criteria with diagnosis by an in-person examiner. Specificity of these criteria compared to in-person examination was 91–100% but sensitivity was 56%. A threshold UPDRS3 score greater than nine provided 100% sensitivity and 81% specificity. Liberal criteria identified 266 (13.1%) subjects with probable parkinsonism and 220 (10.8%) subjects with definite parkinsonism. Conservative criteria identified 260 (12.8%) with probable parkinsonism and 122 (6%) with definite parkinsonism. Our screening method permits rapid assessment of parkinsonian signs. An absolute UPDRS3 score greater than nine provided the best combination of sensitivity and specificity for the diagnosis of parkinsonism, while quantitative exam-based criteria for cardinal parkinsonian signs maximized specificity. Parkinsonism as diagnosed by our criteria was common in this group of welders.

New adaptation of measurement to improve the reliability of hand-held dynamometers used for large muscle strength tests

To determine inter-rater and intra-rater reliability of the Functional Movement Screen (FMS) test among young elite hockey players.Reliability study.Inter-rater reliability was evaluated by two raters in the field. All performances were videotaped. Two other raters evaluated the videos once and then again 6 weeks later to determine intra-rater reliability. A weighted kappa statistic was used to analyze intra-rater and inter-rater reliability of each FMS sub-test, while an intra-class correlation coefficient (ICC) was calculated for the total score.Twenty-eight male hockey players aged 13–16.FMS total and sub-tests scores.The video raters demonstrated excellent intra-rater reliability for the total score, with an ICC of 0.96 (95% CI; 0.92–0.98) and 0.96 (95% CI; 0.91–0.98). The field raters achieved excellent inter-rater reliability for the total score, with an ICC of 0.96 (95% CI; 0.92–0.98). Sub-test analysis showed good agreement among all four raters for five of the seven main sub-tests.FMS is a reliable test for young elite hockey players. Further research should be done to assess the predictive validity of the FMS test within this population so that physiotherapists may eventually use it as an injury prevention tool.

Objective assessment of technical skills in cardiac surgery

Reduced training time combined with no rigorous assessment for technical skills makes it difficult for trainees to monitor their competence. We have developed an objective bench-top assessment of technical skills at a level commensurate with a junior registrar in cardiac surgery. Forty cardiothoracic surgeons were recruited for the study, consisting of 12 junior trainees (year 1-3), 15 senior trainees (year 4-6) and 13 consultants. The assessment consisted of four key tasks on standardised bench-top models: aortic root cannulation, vein-graft to aorta anastomosis, vein-graft to Left Anterior Descending (LAD) anastomosis and femoral triangle dissection. An expert surgeon was present at each station to provide passive assistance and rate performance on a validated global rating scale giving rise to a total possible score of 40. Three expert surgeons repeated the ratings retrospectively, using blinded video recordings. Data analysis employed non-parametric tests. Both live and video scores differentiated significantly between performances of all groups of surgeons for all four stations (P < 0.01) (median live and video score for LAD; Junior 19,17; Senior 29,22; Consultant 36,28). Correlations between live and blinded rating were high (r = 0.67-0.84; P < 0.001) as was inter-rater reliability between the three expert video raters (alpha = 0.81). The use of bench-top tasks to differentiate between cardiac surgeons of differing technical abilities has been validated for the first time. Furthermore, it is unnecessary to perform post-hoc video rating to obtain objective data. These measures can provide formative feedback for surgeons-in-training and lead to the development of a competency-based technical skills curriculum.

Mother-child interaction on video compared with infant observation: Is five minutes enough time for assessment?

In this small-scale preliminary study, we compared the correlation between assessments based on short videorecordings and infant observation. Five mother-infant dyads were assessed when the child was 1 year of age according to the Parent-Child Early Relational Assessment method (PCERA, 65 items) developed by Roseanne Clark (1985), using both observations covering 1 year and a 5-min videorecording. The agreement between two video raters was sufficient (κ = 0.41-1.00 or proportion of agreement: 4/5-5/5) in 54 items. In 42 (78%) of these items, there was at least moderate agreement (κ = 0.41-1.00 or proportion of agreement 4/5-5/5), and in another 7 (13%) items fair (κ = 0.21-0.40 or proportion of agreement 3/5) agreement between the video and observation ratings, and both methods revealed the areas of strength and areas of concern of the dyads. Only a few items describing negative interactive style were assessed as identifying concern, but such items identified the dyad which was assessed to have the most constant problems. Most items describing positive interactive style and identifying the areas of concern in the interactive style according to both observation and video assessment were connected with the parental and dyadic engagement in the interaction. Five items with slight or poor (κ ≤ 0.20 or proportion of agreement 1/5-2/5) agreement between video rater and observer gave rise to questions about how the videorecording situation affected the behavior of the dyads, and about the limits of assessment methods and assessment of dyads with children of different ages and dyads from different cultures. ©2005 Michigan Association for Infant Mental Health.

Evaluating motor function in children with Down syndrome: validity of the GMFM.

The Gross Motor Function Measure (GMFM) was developed and validated originally by Russell and colleagues as an evaluative assessment of gross motor function in children with cerebral palsy (CP). The present study reports the results of reliability and validity testing of the GMFM for use with children with Down syndrome (DS). One hundred and twenty-three children with DS were assessed twice over a 6-month period, using the GMFM and the motor scale of the Bayley Scales of Infant Development - second edition (BSID-II). In addition to the usual method of scoring the GMFM using only observed motor behaviours (standard score), parent reports of children's activities not seen by the assessor on the day of testing were also obtained and a second score (reported score) was calculated for each GMFM assessment. Test-retest and interrater reliabilities were excellent (all >0.90). Observed correlations between change on GMFM and judgements of change made independently by parents, intervenors, and masked video raters were lower than hypothesized. However, the pattern of change scores in predefined age and severity subgroups supported the contention that the GMFM was able to detect differential amounts of change as predicted. The GMFM was shown to be relatively more responsive to change in gross motor function than the motor scale of the BSID-II. The 'reported' scores on the GMFM demonstrated better evidence of reliability, validity, and responsiveness than the standard scoring method and this approach is recommended for use when assessing children with DS.