Novice Raters Research Articles

Reliability of Using a Handheld Tablet and Application to Measure Lower-Extremity Alignment Angles.

Landing kinematics have been identified as a risk factor for knee injury. Detecting atypical kinematics in clinical settings is important for identifying individuals at risk for these injuries. To determine the reliability of a handheld tablet and application (app) for measuring lower-extremity kinematics during drop vertical-jump landings. Measurement reliability. Laboratory. 23 healthy young adults with no lower-extremity injuries and no contraindications for jumping and landing. Subjects performed 6 drop vertical jumps that were captured with an iPad2 and analyzed with a KinesioCapture app by 2 novice and 2 experienced raters. Three trials each were captured in the frontal and sagittal planes. Frontal-plane projection angles, knee flexion, and hip flexion at initial contact and maximum knee flexion were measured. ICC and SEM were calculated to determine intertrial and interrater reliability. One-way ANOVAs were used to examine differences between the measured angles of the raters. Average intertrial reliability ranged from .71 to .98 for novice raters and .77 to .99 for experienced raters. SEMs were 2.3-4.3° for novice raters and 1.6-3.9° for experienced raters. Interrater ICC2,1 was.39-.98 for the novice raters and .69-.93 for the experienced raters. SEMs were smallest with the experiences raters, all less than 1.5°. A handheld tablet and app is promising for evaluating landing kinematics and identifying individuals at risk for knee injury in a clinical setting. Intertrial reliability is good to excellent when using average trial measures. Interrater reliability is fair to excellent depending on experience level. Multiple trials should be assessed by a single rater when assessing lower-extremity mechanics with a handheld tablet and app, and results may vary with experience level or training.

Bony abnormalities of the hip joint: a new comprehensive, reliable and radiation-free measurement method using magnetic resonance imaging.

The objective of this study was to develop comprehensive and reliable radiation-free methods to quantify femoral and acetabular morphology using magnetic resonance imaging (MRI).Thirty-two hips [16 subjects, 6 with intra-articular hip disorder (IAHD); 10 controls] were included. A 1.5-T magnetic resonance system was used to obtain three-dimensional fat-suppressed gradient-echo images at the pelvis and distal femora. After acquisition, pelvic images were post-processed to correct for coronal, axial and sagittal rotation. Measurements performed included acetabular version (AV), femoral version (FV), lateral center-edge angle (LCEA), femoral neck angle (FNA) and alpha angle (AA) at 3, 2, 1 and 12 a.m. Two experienced raters, a musculoskeletal radiologist and an orthopedic physical therapist, and a novice rater, a research assistant, completed reliability testing. Raters measured all hips twice with minimum 2 weeks between sessions. Intra-class Correlation Coefficients (ICCs) were used to determine rater reliability; standard error of measurements was reported to estimate the reasonable limits of the expected error in the different raters’ scores.Inter-rater reliability was good to excellent for all raters for AV, FV, FNA and LCEA (ICCs: 0.82–0.98); good to excellent between experienced raters (ICCs: 0.78–0.86) and poor to good between novice and experienced raters (ICCs: 0.23–0.78) for AA. Intra-rater reliability was good to excellent for all raters for AV, FV and FNA (ICCs: 0.93–0.99); for one experienced and novice rater for LCEA (ICCs: 0.84–0.89); moderate to excellent for the experienced raters for AA (ICCs: 0.72-0.89). Intra-rater reliability was poor for the second experienced rater for LCEA (ICC: 0.56), due to a single measurement error and for the novice rater for AA (ICCs: 0.17–0.38).We described MRI methods to comprehensively assess femoral and acetabular morphology. Measurements such as AV, FV and FNA and the LCEA can be made reliably by both experienced and novice raters; however, the AA measurement was reliable only among experienced raters.

Direct Behavior Rating: An evaluation of time-series interpretations as consequential validity.

Direct Behavior Rating (DBR) is a repeatable and efficient method of behavior assessment that is used to document teacher perceptions of student behavior in the classroom. Time-series data can be graphically plotted and visually analyzed to evaluate patterns of behavior or intervention effects. This study evaluated the decision accuracy of novice raters who were presented with single-phase graphical plots of DBR data. Three behaviors (i.e., academically engaged, disruptive, and respectful) and three graphical trends (i.e., positive, no trend, and negative) were analyzed by 27 graduate and five undergraduate participants who had minimal visual analysis experience. All graphs were unique, with data points arranged to form one of three "true" trends. Raters correctly classified graphs with positive, no, and negative trends an average of 76, 98, and 67% of instances. The generalized linear mixed model was used to handle significance tests for the categorical data. Results indicate that accuracy was influenced by the trend direction, with the most accurate ratings in the no trend condition. Despite the significant effect for trend direction, the current study provides empirical evidence for accuracy of DBR trends and interpretations. Novice raters and visual analysts yielded accurate decisions regarding the trend of plotted data for student behavior.

Reliability and validity of goniometric iPhone applications for the assessment of active shoulder external rotation

ABSTRACTPurpose/hypothesis: The purpose of this study was to determine the reliability and validity of two smartphone applications: (1) GetMyROM – inclinometery-based and (2) DrGoniometry – photo-based in the measurement of active shoulder external rotation (ER) as compared to standard goniometry (SG). Participants: Ninety-four Texas Woman's University Doctor of Physical Therapy students from the School of Physical Therapy – Houston campus, were recruited to participate in this study. Materials/methods: Two iPhone applications were compared to SG using both novice and experienced raters. Active shoulder ER range of motion was measured over two time periods in random order by blinded novice and experienced raters. Results: Intra-rater reliability using novice raters for the two applications ranged from an intraclass correlation coefficient (ICC) of 0.79 to 0.81 with SG at 0.82. Inter-rater reliability (novice/expert) for the two applications ranged from an ICC of 0.92 to 0.94 with SG at 0.91. Concurrent validity (when compared to SG) ranged from 0.93 to 0.94. There were no significant differences between the novice and experienced raters. Conclusion: Both applications were found to be reliable and comparable to SG. A photo-based application potentially offers a superior method of measurement as visualizing the landmarks may be simplified in this format and it provides a record of measurement. Clinical relevance: Further study using patient populations may find the two studied applications are useful as an adjunct for clinical practice.

Pair assessment of pupil writing: A dialogic approach for studying the development of rater competence

Abstract This paper reports on rating during the development of a Norwegian sample-based national assessment of L1 writing as a key competency. This assessment is to be officially introduced in August 2014. Novice members of a national rater panel to assess Year 8 pupils’ texts were studied during three of their successive training sessions: in June 2011, November 2011 and April 2012. My purpose was to conduct an exploratory investigation into how the rating practice of novice raters might develop during such a preparatory stage. The raters in this study mainly assessed in pairs, and data sources were assessment dialogues. The analysis of transcripts showed that rater behaviour changed only to a minor extent towards an increased use of shared assessment resources. The quality of the assessment dialogues did not change much either, leaving the impression that raters often reached consensus without much discussion. Since reliable scoring is a collective task, however, it is argued that a more balanced use of resources, both those attained from teacher practice and those attained from being a member of a national panel, may together with more exploration-oriented dialogue be necessary to achieve sufficient reliability.

Board 520 - Technology Innovations Abstract Development and Validation of Simulators for Pediatric Inguinal Hernia Management in France (Submission #427)

Introduction/Background Inguinal hernia is one of the most common conditions seen in pediatric practice, with 3-5% of newborns experiencing inguinal hernias. An incarcerated or strangulated hernia can result in intestinal necrosis and testicular or ovarian atrophy, while improper repair techniques can lead to recurrence, testicular torsion, or injury of spermatic cord structures. Reduction of hernia contents back into the abdomen is a non-operative procedure that involves a series of simple but specific maneuvers that can prevent incarceration until surgery can be performed.1 No simulation models currently exist to teach an open approach to pediatric inguinal hernia repair nor maneuvers of inguinal hernia reduction. Because inguinal hernias are common, every physician should be adept at performing these procedures. The objective of this project is to develop and evaluate validity evidence for two simulators: a surgical simulator for pediatric inguinal hernia repair and a clinical simulator for pediatric inguinal hernia reduction. Methods Surgical Simulator: The surgical simulator consists of a plastic base with a pigmented silicone rubber covering. The spermatic cord model is secured inside the base and consists of a balloon representing the hernia sac, two elastic strings representing the spermatic vessels and vas deferens, and plastic wrap and silicone representing surrounding tissues. Fourteen novice raters and seventeen expert raters (N=31) from five different institutions in France (Nice, Paris, Strasbourg, Limoges, and Angers) performed the inguinal hernia repair. Participants completed a self-report rating scale ranging from 1 (Don’t know) to 5 (Very realistic, no changes needed). Validity evidence relevant to test content and internal structure was evaluated using the many-facet Rasch analysis. Analysis indicated that Attendings had higher ratings (Observed Average (OA)= 4.4/5.0) than Interns (OA=4.0) and Residents (OA=3.8), p=.001, and that there were overall differences when comparing ratings across institutions, with higher ratings associated with participants from Limoges (OA=4.56), followed by ratings from Nice (OA=4.25), Angers (OA=4.01), Strasbourg (OA=3.89), and Paris (OA=3.64), p=.01. Ratings across seven domains (Realism of materials, Realism of experience, Value, Internal anatomy, Ability to perform tasks, Relevance, Physical attributes) ranged from 3.91 to 4.41. The observed global rating was 2.37/4.0. Clinical Simulator: The clinical simulator consists of a modified baby doll with a pigmented silicone rubber covering. The hernia model inside the baby doll is represented by a sponge inside of a balloon, which can be manipulated from one end of the balloon to the other. Forty-eight novice raters, twenty expert raters, and ten non-identified raters (N=86) from the five institutions in France performed the inguinal hernia reduction. Participants completed a similar self-report rating scale ranging from 1 to 5. Validity evidence relevant to test content and internal structure was evaluated using the many-facet Rasch analysis. Analysis indicated no statistical differences when comparing Attending-specialists (Observed Average (OA)= 3.9/5.0) to Attending-generalists (OA=3.9) and Residents (OA=3.8), p=.97, but that there were overall differences when comparing ratings across institutions, with higher ratings associated with participants from Limoges and Angers (OA=3.9), followed by ratings from Strasbourg and Nice (OA=3.8), and Paris (OA=3.7), p=.001. Ratings across six domains (Ability to perform tasks, Realism of experience, Realism of materials, Physical attributes, Relevance, Value) ranged from 3.5 to 4.2. The observed global rating was 2.4/4.0. Results: Conclusion For the surgical simulator, participants agreed that the model has a great deal of value as a training and testing tool and is highly relevant to their clinical practice. For the clinical simulator, participants agreed that the model has some value as a training and testing tool and is relevant to their clinical practice. For both models, the observed global ratings indicate some need for improvement prior to implementation. Ratings of individual items within each domain will help us identify areas for improvement.

Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale

Recent literature suggests a revival of interest in single-case methodology (e.g., the randomised n-of-1 trial is now considered Level 1 evidence for treatment decision purposes by the Oxford Centre for Evidence-Based Medicine). Consequently, the availability of tools to critically appraise single-case reports is of great importance. We report on a major revision of our method quality instrument, the Single-Case Experimental Design Scale. Three changes resulted in a radically revised instrument, now entitled the Risk of Bias in N-of-1 Trials (RoBiNT) Scale: (i) item content was revised and increased to 15 items, (ii) two subscales were developed for internal validity (IV; 7 items) and external validity and interpretation (EVI; 8 items), and (iii) the scoring system was changed from a 2-point to 3-point scale to accommodate currently accepted standards. Psychometric evaluation indicated that the RoBiNT Scale showed evidence of construct (discriminative) validity. Inter-rater reliability was excellent, for pairs of both experienced and trained novice raters. Intraclass correlation coefficients of summary scores for individual (experienced) raters: ICCTotalScore = .90, ICCIVSubscale = .88, ICCEVISubscale = .87; individual (novice) raters: ICCTotalScore = .88, ICCIVSubscale = .87, ICCEVISubscale = .93; consensus ratings between experienced and novice raters (ICCTotalScore = .95, ICCIVSubscale = .93, ICCEVISubscale = .93. The RoBiNT Scale thus shows sound psychometric properties and provides a comprehensive yet efficient examination of important features of single-case methodology.

Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation: Revisiting Research Conventions

This mixed-methods study examines the effects of rating scale length and rater experience on listeners' judgments of second-language (L2) speech. Twenty experienced and 20 novice raters, who were randomly assigned to 5-point or 9-point rating scale conditions, judged speech samples of 38 newcomers to Canada on numerical rating scales for comprehensibility, accentedness, and fluency. Results yielded high Cronbach's alpha coefficients and no group differences for rating scale length or rater experience. However, Rasch category probability plots revealed that raters had difficulty differentiating between scale steps, particularly in mid-scale range—a challenge that was exacerbated in the 9-point scale condition. Evidence from verbal protocols and posttask interviews suggested that experienced and novice raters adopted strategies to either draw on or offset their perceived experience with L2 speech in conducting their ratings. Implications for L2 pronunciation research are discussed, including the need to draw on raters' perceptions to develop a greater understanding of major listener perceptual constructs in L2 pronunciation and to move beyond adherence to the native speaking standard in data collection procedures.

Rating Performance of EFL Teachers in Writing Assessment: Comparison of Experienced and Novice Raters

Rating Performance of EFL Teachers in Writing Assessment: Comparison of Experienced and Novice Raters

Investigating an online rater training program: product and process

The Diagnostic English Language Needs Assessment (DELNA) programme at the University of Auckland has traditionally offered face-to-face training to both new raters and to experienced raters of writing who need to undergo refresher training in order to increase inter-rater and intra-rater reliability. However, for reasons of practicality and convenience, an online program to offer refresher training for experienced raters was developed and revised after trialling in 2003-04 (Elder, Barkhuizen, Knoch & von Randow, 2007). The study reported on in this paper arose out of the need to establish the validity of using this online DELNA program to train new raters (i.e., novice raters). Rater-training outcomes were investigated through the analysis of quantitative data. However, there was also a focus on the ‘process of rating’ using qualitative data, that is, think-aloud protocol, to identify rating behaviours associated with more vs. less reliable ratings. Results suggest that this program has potential to train novice raters to rate reliably but that future research needs to investigate more rigorously the impact of a program like this one on subsequent rating. Nvivo analysis of the think-aloud protocols highlighted key aspects of the rating behaviours of more and less reliable raters.

Comparison of Functional Ankle Motion Measures in Modern Dancers

Ankle injuries are the most common lower extremity injury in dance and sports, often resulting in limitation of dorsiflexion or plantar flexion. Accurate assessment of any limitation in range of motion is an important factor in implementing effective preventative and rehabilitative regimens. Ankle range of motion has traditionally been quantified with goniometers. However, standard goniometry may not be an adequate method of assessing plantar flexion range of motion in dancers. An alternative technique using inclinometers to quantify ankle plantar flexion has been reported, but reliability and inter-instrument correlations for this method are limited. The dorsiflexion lunge distance method has been used to assess ankle dorsiflexion. Although shown to be reliable, this method has not been objectively correlated. The purpose of this study was to determine the most clinically appropriate tool for assessing functional ankle dorsiflexion and plantar flexion in dancers. The aims were to: 1. determine reliability of the weightbearing ankle dorsiflexion lunge method using inclinometer, goniometer, and distance; 2. assess the reliability of inclinometer and goniometer measurements of non-weightbearing active plantar flexion; 3. conduct inter-method and experienced versus novice rater correlations; and 4. determine if a relationship exists between dorsiflexion distance (cm) and dorsiflexion inclinometer angle (degrees) measurements. Twenty-six modern dancers (age: 20.2 ±1.8 years) participated in the study. Four raters measured weightbear-ing dorsiflexion in a lunge position using a goniometer, two inclinometer placements, and a distance measurement. They also measured active ankle plantar flexion using an inclinometer placed on the anterior talonavicular joint and a goniometer placed at the lateral ankle. Intra- and inter-rater reliability and inter-method correlations were calculated with Intra-class Correlation Coefficients (ICC) and standard error of measurement (SEM). The relationship of dorsiflexion distance to angle was determined using grouped linear regression (p < 0.05). Dorsiflexion and plantar flexion intra- and inter-rater ICCs for inclinometer, goniometer, and distance ranged from 0.84 to 0.99. The SEM for angular measures ranged from 1oto 3°, and linear measures from 0.3 cm to 0.9 cm. Inter-method correlations ranged from 0.55 to 0.89. There were no differences between inclinometer and goniometer measurement dorsiflexion means. However, inclinometer plantar flexion values were greater than goniometric values (p < 0.001). Experienced raters recorded greater goniometric values compared to novice raters (p < 0.01). There was no consistent linear relationship between dorsiflexion lunge distance measure and inclinometer degrees. It is concluded that functional ankle DF in modern dancers is best quantified using an inclinometer (posterior placement) in the weightbearing lunge position. Non-weightbearing active ankle PF in modern dancers is best quantified with inclinometer placement on the dorsum of the foot. The distance method cannot be compared directly to angular measurement, is subject-specific, and cannot be used as a normative measure to compare DF range between subjects, populations, or age groups.

Reliability of novice raters in using the movement system impairment approach to classify people with low back pain

Observational cross sectional study.To examine the inter-rater reliability of novice raters in using the Movement System Impairment (MSI) approach system and to explore the patterns of disagreement in classification errors. The inter-rater reliability of individual tests items used in the MSI approach is moderate to good; however, the reliability of the classification algorithm has been tested only preliminarily.Using previously recorded patient data (n = 21), 13 novice raters classified patients according to the MSI schema. The overall percent agreement using the kappa statistic as well as the agreement/disagreement among pair-wise comparisons in classification assignments were examined.There was an overall 87.4% agreement in the pairs of classification judgments with a kappa coefficient of 0.81 (95% CI: 0.79, 0.83). Raters were most likely to agree on the classification of Flexion (100%) and least likely to agree on the classification of Rotation (84%).The MSI classification algorithm can be learned by novice users and with training, their inter-rater reliability in applying the algorithm for classification judgments is good and similar to that reported in other studies. However, some degree of error persists in the classification decision-making associated with the MSI system, in particular for the Rotation category.

Reliability of a Treatment-Based Classification System for Subgrouping People With Low Back Pain

Observational, cross-sectional reliability study. To examine the interrater reliability of novice raters in their use of the treatment-based classification (TBC) system for low back pain and to explore the patterns of disagreement in classification errors. Although the interrater reliability of individual test items in the TBC system is moderate to good, some error persists in classification decision making. Understanding which classification errors are common could direct further refinement of the TBC system. Using previously recorded patient data (n = 24), 12 novice raters classified patients according to the TBC schema. These classification results were combined with those of 7 other raters, allowing examination of the overall agreement using the kappa statistic, as well as agreement/disagreement among pairwise comparisons in classification assignments. A chi-square test examined differences in percent agreement between the novice and more experienced raters and differences in classification distributions between these 2 groups of raters. Among 12 novice raters, there was 80.9% agreement in the pairs of classification (κ = 0.62; 95% confidence interval: 0.59, 0.65) and an overall 75.5% agreement (κ = 0.57; 95% confidence interval: 0.55, 0.69) for the combined data set. Raters were least likely to agree on a classification of stabilization (77.5% agreement). The overall percentage of pairwise classification judgments that disagreed was 24.5%, with the most common disagreement being between manipulation and stabilization (11.0%), followed by a mismatch between stabilization and specific exercise (8.2%). Additional refinement is needed to reduce rater disagreement that persists in the TBC decision-making algorithm, particularly in the stabilization category. J Orthop Sports Phys Ther 2012;42(9):797-805, Epub 7 June 2012. doi:10.2519/jospt.2012.4078.

Disentangling accent from comprehensibility

The goal of this study was to determine which linguistic aspects of second language speech are related to accent and which to comprehensibility. To address this goal, 19 different speech measures in the oral productions of 40 native French speakers of English were examined in relation to accent and comprehensibility, as rated by 60 novice raters and three experienced teachers. Results showed that both constructs were associated with many speech measures, but that accent was uniquely related to aspects of phonology, including rhythm and segmental and syllable structure accuracy, while comprehensibility was chiefly linked to grammatical accuracy and lexical richness.

The Functional Movement Screen: A Reliability Study

Reliability study. To determine intrarater test-retest and interrater reliability of the Functional Movement Screen (FMS) among novice raters. The FMS is used by various examiners to assess movement and predict time-loss injuries in diverse populations (eg, youth to professional athletes, firefighters, military service members) of active participants. Unfortunately, critical analysis of the reliability of the FMS is currently limited to 1 sample of active college-age participants. Sixty-four active-duty service members (mean ± SD age, 25.2 ± 3.8 years; body mass index, 25.1 ± 3.1 kg/m2) without a history of injury were enrolled. Participants completed the 7 component tests of the FMS in a counterbalanced order. Each component test was scored on an ordinal scale (0 to 3 points), resulting in a composite score ranging from 0 to 21 points. Intrarater test-retest reliability was assessed between baseline scores and those obtained with repeated testing performed 48 to 72 hours later. Interrater reliability was based on the assessment from 2 raters, selected from a pool of 8 novice raters, who assessed the same movements on day 2 simultaneously. Descriptive statistics, weighted kappa (κw), and percent agreement were calculated on component scores. Intraclass correlation coefficients (ICCs), standard error of the measurement, minimal detectable change (MDC95), and associated 95% confidence intervals (CIs) were calculated on composite scores. The average ± SD score on the FMS was 15.7 ± 0.2 points, with 15.6% (n = 10) of the participants scoring less than or equal to 14 points, the recommended cutoff for predicting time-loss injuries. The intrarater test-retest and interrater reliability of the FMS composite score resulted in an ICC3,1 of 0.76 (95% CI: 0.63, 0.85) and an ICC2,1 of 0.74 (95% CI: 0.60, 0.83), respectively. The standard error of the measurement of the composite test was within 1 point, and the MDC95 values were 2.1 and 2.5 points on the 21-point scale for interrater and intrarater reliability, respectively. The interrater agreement of the component scores ranged from moderate to excellent (κw = 0.45-0.82). Among novice raters, the FMS composite score demonstrated moderate to good interrater and intrarater reliability, with acceptable levels of measurement error. The measures of reliability and measurement error were similar for both intrarater reliability that repeated the assessment of the movement patterns over a 48-to-72-hour period and interrater reliability that had 2 raters assess the same movement pattern simultaneously. The interrater agreement of the FMS component scores was good to excellent for the push-up, quadruped, shoulder mobility, straight leg raise, squat, hurdle, and lunge. Only 15.6% (n = 10) of the participants were identified to be at risk for injury based on previously published cutoff values.

Reliability, Standard Error, and Minimum Detectable Change of Clinical Pressure Pain Threshold Testing in People With and Without Acute Neck Pain

Clinical measurement. To evaluate the intrarater, interrater, and test-retest reliability of an accessible digital algometer, and to determine the minimum detectable change in normal healthy individuals and a clinical population with neck pain. Pressure pain threshold testing may be a valuable assessment and prognostic indicator for people with neck pain. To date, most of this research has been completed using algometers that are too resource intensive for routine clinical use. Novice raters (physiotherapy students or clinical physiotherapists) were trained to perform algometry testing over 2 clinically relevant sites: the angle of the upper trapezius and the belly of the tibialis anterior. A convenience sample of normal healthy individuals and a clinical sample of people with neck pain were tested by 2 different raters (all participants) and on 2 different days (healthy participants only). Intraclass correlation coefficient (ICC), standard error of measurement, and minimum detectable change were calculated. A total of 60 healthy volunteers and 40 people with neck pain were recruited. Intrarater reliability was almost perfect (ICC = 0.94-0.97), interrater reliability was substantial to near perfect (ICC = 0.79-0.90), and test-retest reliability was substantial (ICC = 0.76-0.79). Smaller change was detectable in the trapezius compared to the tibialis anterior. This study provides evidence that novice raters can perform digital algometry with adequate reliability for research and clinical use in people with and without neck pain.

The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters

Raters are central to writing performance assessment, and rater development – training, experience, and expertise – involves a temporal dimension. However, few studies have examined new and experienced raters’ rating performance longitudinally over multiple time points. This study uses operational data from the writing section of the MELAB (n = 20,662 ratings), an international exam of English proficiency, to investigate the rating quality of new and experienced raters over three time periods of 12 to 21 months. Rating quality was operationalized in terms of rater severity and consistency, and estimates of those modeled using multi-facet Rasch methodology. Results indicate that, within one particular rating context, (1) novice raters, where initially differing in performance, learn to rate appropriately relatively quickly, (2) raters are able to maintain rating quality over time, and (3) rating volume and rating quality may be related. Implications for rater preparation, rater certification, and the notion of expert rater are discussed.

Clock Drawing in the Montreal Cognitive Assessment: Recommendations for Dementia Assessment

Background: Clock drawing is part of the Montreal Cognitive Assessment (MoCA) test but may have administration and scoring limitations. We assessed (1) the reliability of the MoCA clock criteria relative to a published error scoring approach, (2) whether command-only administration could distinguish dementia from cognitively intact individuals and (3) the value of adding a clock copy condition to the MoCA. Methods: Three novice raters and clocks from dementia and control participants were used to assess the 3 aims. Results: MoCA interrater and intrarater reliability were low (i.e. intraclass correlation coefficient = 0.12–0.31) and required repeat training. Clocks drawn to command classified dementia at chance. Inclusion of a copy condition demonstrated expected dementia subgroup patterns. Conclusion: Reliable clock scoring with MoCA criteria requires practice. Supplementing a clock copy to the standard MoCA test (takes <1 min) will improve dementia assessment.

Inter‐Rater Reliability of Ultrasound Imaging of the Trunk Musculature Among Novice Raters

The purpose of this study was to determine the inter-rater reliability of ultrasound imaging for assessing trunk muscle morphologic characteristics at rest and while contracted among different pairs of novice raters. The secondary purpose was to compare 3 different measurement techniques for assessing lateral abdominal muscle thickness. A single-group repeated measures reliability study was conducted on 21 healthy participants (mean ± SD, 21.5 ± 4.4 years; 5 female and 16 male) without low back pain. Ultrasound images of the transversus abdominis, internal oblique, rectus abdominis, and lumbar multifidus muscles were obtained by different pairs of novice raters in a counterbalanced order. All raters received a standardized training program before obtaining measurements. The intraclass correlation coefficient (1, 3) point estimates ranged from 0.86 to 0.94; the standard error of the measurement ranged from 0.04 to 0.16 cm for the thickness values and 0.67 cm(2) for the cross-sectional area of the rectus abdominis muscle. There was no meaningful difference between the different measurement techniques used to analyze the lateral abdominal muscles. Good to excellent reliability was obtained for all measures by novice raters. Minimal differences in reliability were noted between the different measurement techniques to assess lateral abdominal muscle thickness.

The effect of viewing angle on wrist posture estimation from photographic images using novice raters

Observational assessment of wrist posture using photographic methods is theoretically affected by camera view angle. A study was conducted to investigate whether wrist flexion/extension and radial/ulnar deviation postures were estimated differently by raters depending on the viewing angle and compared to predictions using a quantitative 2D model of parallax. Novice raters ( n = 26) estimated joint angles from images of wrist postures photographed from ten different viewing angles. Results indicated that ideal views, orthogonal to the plane of motion, produced more accurate estimates of posture compared to non-ideal views. The neutral (0°) posture was estimated the most accurately even at different viewing angles. Raters were more accurate than model predictions. Findings demonstrate a need for more systematic methods for collecting and analyzing photographic data for observational studies of posture. Renewed caution in interpreting existing studies of wrist posture where viewing angle was not controlled is advised.