Assess Interrater Reliability Research Articles

Prospective validation study of an epilepsy seizure risk system for outpatient evaluation.

We conducted clinical testing of an automated Bayesian machine learningalgorithm (Epilepsy Seizure Assessment Tool [EpiSAT]) for outpatient seizure risk assessment using seizure counting data, and validated performance against specialized epilepsy clinicianexperts. We conducted a prospective longitudinal study of EpiSAT performance against 24 specialized clinician expertsat three tertiary referral epilepsy centers in the United States. Accuracy, interrater reliability, and intra-rater reliability of EpiSATfor correctly identifyingchanges inseizure risk (improvements, worsening, or no change) were evaluated using 120 seizures from four synthetic seizure diaries (seizure risk known) and 120 seizures from four real seizure diaries (seizure risk unknown). The proportion of observed agreement between EpiSAT and clinicians was evaluated to assess compatibility of EpiSAT with clinical decision patterns by epilepsy experts. EpiSAT exhibited substantial observed agreement (75.4%) with clinicians for assessing seizure risk. The mean accuracy of epilepsy providers for correctly assessing seizure risk was 74.7%. EpiSAT accurately identified seizure risk in 87.5% of seizure diary entries, corresponding to a significant improvement of 17.4% (P=.002). Clinicians exhibited low-to-moderate interrater reliability for seizure risk assessment (Krippendorff's α=0.46) with good intrarater reliability across a 4- to 12-week evaluation period (Scott's π=0.89). These results validate the ability of EpiSAT to yield objective clinical recommendations on seizure risk which follow decision patterns similar to those from specialized epilepsy providers, but with improved accuracy and reproducibility. This algorithm may serve as a useful clinical decision support system for quantitative analysis of clinical seizure frequency in clinical epilepsy practice.

Differences of the Glasgow Coma Scale and the Full Outline of Unresponsiveness Scores on Consciousness Level Examination

Consciousness is an integration between the sensory system, reasoning, imagination and emotions, and memory in receiving and processing information from the internal and external environment. Due to the large spectrum and causes of impaired consciousness, several methods are needed to check the level of consciousness, they are Glasgow Coma Scale and the Full Outline of Unresponsiveness Score. This article summarizes the differences in the use of the Glasgow Coma Scale and Full Outline of Unresponsiveness score methods in measuring the level of consciousness. The method used is literature searching from various national and international journals and using a digital library such as www.ncbi.nlm.nih.gov and Google Scholar with keyword Full Outline of Unresponsiveness (FOUR) score AND/OR Glasgow Coma Scale. The literature used totaled 14 articles which published in 2006 to 2019. The analytical method used is a systematic literature review method by systematically identifying, reviewing, and investigating research in terms of its title, approach, objectives, results, and discussion with a particular topic focus. The results of this literature study indicate that there are differences in the assessment components, inter-rater reliability, and differences in the predictive ability of mortality in the Glasgow Coma Scale method and the Full Outline of Unresponsiveness score so that the Full Outline of Unresponsiveness score can be an alternative or complement to Glasgow Coma Scale in measuring the level of consciousness.

Evaluating the Measurement Properties of the ScanCourse, a Dual-Task Assessment of Visual Scanning.

The ScanCourse is used by occupational therapists to evaluate visual scanning ability during locomotion. Its measurement properties have not been examined. To assess the interrater reliability, test-retest reliability, and construct validity of the ScanCourse. This study involved data collection at two time points. To assess test-retest reliability, the ScanCourse was administered twice within a 2-week period. To assess interrater reliability, a second rater was present for one session. To assess level of agreement, a Bland-Altman plot was created. To assess absolute reliability, the standard error of measurement was calculated. To evaluate construct validity, the results of the ScanCourse were compared with results of the Bells Test and Trail Making Test A and B. Rehabilitation hospital. Forty-one patients with neurological impairments. The ScanCourse (participants identify numbered cards placed on both sides of a hallway at various heights during locomotion). The ScanCourse was found to have excellent interrater reliability (intraclass correlation coefficient [ICC] [1,1] = .998; 95% confidence interval [CI] [.996-.999]), test-retest reliability (ICC [1,1] = .912; 95% CI [.811-.959]), a high level of agreement, and a low standard error of measurement (.503), and it was found to be significantly correlated with Trails A (rs = -.436, p = .009) and B (rs = -.364, p = .029). The assessment was found to have strong measurement properties, and it is therefore an appropriate tool for assessing dual-task visual scanning among those with neurological impairments. This research demonstrates that the ScanCourse is reliable between raters and over time and that scores on the measure vary as anticipated with scores on a related measure, which provides evidence of its validity. These findings support its use in practice.

Concurrent validity and inter-rater reliability of a teleconference assessment of calf raise exercise fidelity

Introduction: Telerehabilitation uses telecommunications technology (telephone or teleconference) to provide rehabilitation services at a distance that replace or supplement face-to-face care. Clinical outcomes from exercise interventions are likely to be related to Implementation fidelity. Exercise fidelity (e.g. posture, tempo, resistance, pain response) is a fundamental aspect of exercise prescription by physiotherapists (PTs). However, the utility of evaluating exercise fidelity via telerehabiltiation has not been assessed. To bridge this knowledge gap, this study aims to evaluate the concurrent validity and inter-rater reliability of assessing exercise fidelity in Achilles tendinopathy (AT) rehabilitation via videoconference.

Interobserver reliability of Diméglio and Pirani score and their subcomponents in the evaluation of idiopathic clubfoot in a clinical setting: a need for improved scoring systems.

PurposeDiméglio (DimS) and Pirani (PirS) scores are the most commonly used scoring systems for evaluation of clubfoot, with many centres performing both. Interobserver reliability of their global score has been rated high in a few studies, but agreement of their subcomponents has been poorly investigated. The aim of the study was to assess interrater reliability of global scores and of items in a clinical setting and to analyse overlapping features of the two scores.MethodsFifty-six consecutive idiopathic clubfeet undergoing correction using the Ponseti method were independently evaluated at each casting session by two trained paediatric orthopaedic surgeons using both scores. Interobserver reliability of collected data was analysed; a kappa coefficient > 0.60 was considered adequate.ResultsFor DimS and PirS, the Pearson correlation coefficients were 0.87 and 0.91 (p < .0001) respectively, and kappa coefficients were 0.23 and 0.31. Among subcomponents, kappa values were rated > 0.60 only for equinus and curvature of lateral border in PirS; muscular abnormality in DimS was rated 0.74 but a high prevalence index (0.94) indicated influence of scarce prevalence of this feature. All other items showed k < 0.60 and were considered to be improved.For overlapping features: posterior and medial crease showed similar agreement in the two systems, items describing equinus and midfoot adduction were much more reliable in PirS than in DimS.ConclusionsIn a clinical setting, despite a high correlation of evaluations for total scores, the interobserver agreement of DimS and PirS was not adequate and only a few items were substantially reliable. Simultaneous use of two scores seemed redundant and some overlapping features showed different reliability according to criterion or scale used. Future scoring systems should improve these limitations.Level of EvidenceLevel I – Diagnostic studies

The Head Control Scale: Development, inter-rater reliability, and utility.

This study introduces a new scale for the assessment of head control called the Head Control Scale (HCS). The purpose of this study was to establish interrater reliability of the HCS and to determine its usefulness in a clinical setting. The HCS assesses head control in four positions (prone, supine, pull to sit, and supported sitting) on a 0-4 rating scale. The authors used both a focus group and pilot testing to refine the scale to its final version, which was then used to assess interrater reliability. Twenty-six therapists used the HCS to evaluate head control of five subjects of varying ages and abilities who were videotaped spending 30-40 seconds in each position. Participants also completed a post-rating survey. Fleiss's weighted kappa coefficient is excellent for the prone (0.82), pull to sit (0.83), and sitting (0.88) positions as well as for the scale overall (kappa = 0.91). It can be described as fair to good for supine (kappa = 0.68). The HCS has high interrater reliability and users report it to be a needed tool, applicable to clinical practice, and easy to use. The results of this study indicate that the HCS has great potential for clinical use.

Validity and Reliability of a Nutrition Screening Tool in Identifying Malnutrition Among Hospitalized Adult Patients.

Malnutrition among hospitalized patients is closely associated with various medical complications. This study aimed to determine the validity and reliability of a 3-Minute Nutrition Screening (3-MinNS) tool in identifying the risk of malnutrition among hospitalized patients that can be administered by healthcare professionals. A cross-sectional study was conducted between January and December 2012. A total of 350 adult patients in a teaching hospital were screened for risk of malnutrition using 3-MinNS and Subjective Global Assessment (SGA). To assess interrater reliability, each patient was screened for risk of malnutrition using 3-MinNS by 2 different nurses on 2 different occasions within 24 hours after admission. To assess the validity of 3-MinNS, the level of risk of malnutrition identified by the nurses using 3-MinNS was compared with the risk of malnutrition as assessed by a dietitian using SGA within 48 hours after the patients' enrolment into the study. The sensitivity, specificity, and predictive values were calculated in detecting patients at risk of malnutrition. Interrater reliability was determined using κ statistics. Using SGA, the estimated prevalence of moderate to severe malnutrition was 36.3% (127/350). There was 94% proportional agreement between 2 nurses using 3-MinNS, and interrater reliability was substantial (κ = 0.79, P < .001). The analysis showed that 3-MinNS had moderate sensitivity (61.4%-68.5%) but high specificity (95.1%). The 3-MinNS is a reliable and valid screening tool for use by healthcare professionals for identifying newly admitted medical and surgical patients who are at risk of malnutrition.

Could Residents Adequately Assess the Severity of Hidradenitis Suppurativa? Interrater and Intrarater Reliability Assessment of Major Scoring Systems

A wide variety of assessment tools have been proposed for hidradenitis suppurativa (HS) until now, but none of them meets the criteria for an ideal score. Because there is no gold standard scoring system, the choice of the measure instrument depends on the purpose of use and even on the physician’s experience in the subject of HS. The aim of this study was to assess the intrarater and interrater reliability of 6 scoring systems commonly used for grading severity of HS: the Hurley Staging System, the Refined Hurley Staging, the Hidradenitis Suppurativa Severity Score System (IHS4), the Hidradenitis Suppurativa Severity Index (HSSI), the Sartorius Hidradenitis Suppurativa Score and the Hidradenitis Suppurativa Physician’s Global Assessment Scale (HS-PGA). On the scoring day, 9 HS patients underwent a physical examination and disease severity assessment by a group of 16 dermatology residents using all evaluated instruments. Then, intrarater reliability was calculated using intraclass correlation coefficient (ICC), and interrater variability was evaluated using the coefficient of variation (CV). In all 6 scorings the ICCs were >0.75, indicating high intrarater reliability of all presented scales. The study has also demonstrated moderate agreement between raters in most of the evaluated measure instruments. The most reproducible methods, according to CVs, seem to be the Hurley staging, IHS4, and HSSI. None of the 6 evaluated scoring systems showed a significant advantage over the other when comparing ICCs, and all the instruments seem to be very reliable methods. The interrater reliability was usually good, but the most repeatable results between researchers were obtained for the easiest scales, including Hurley scoring, IHS4 and HSSI.

Nurses' understanding and experience of applying painful stimuli when assessing components of the Glasgow Coma Scale.

To evaluate nurses' application, understanding and experience of applying painful stimuli when assessing components of the Glasgow Coma Scale. The Glasgow Coma Scale has been subjected to much scrutiny and debate since its publication in 1974. However, criticism, confusion and misunderstandings in relation to the use of painful stimuli and its application remain. An absence of evidence-informed guidance on the use and duration of application of painful stimuli remains, with the potential to negatively impact on decision-making, delay responsiveness to neurological deterioration and result in adverse incidents. This international study used an online self-reported survey design to ascertain neuroscience nurses' perceptions and experiences around the application of painful stimuli as part of a GCS assessment (n=273). The STROBE checklist was used. Data revealed varied practices and a sense of confusion from participants. Anatomical sites for the assessment of pain varied, but most respondents identified the trapezius grip/pinch in assessing eye-opening and motor responses. Most respondents identified they assess eye-opening and motor responses together and apply pain for <6s to elicit a response. Witnessed complications secondary to applying a painful stimulus were varied and of concern. Neuroscience nurses in this study clearly required evidence-informed guidelines to underpin practice both in applying painful stimuli and in managing the experience of the person in their care and the family response. A standardised approach to education is necessary to ensure greater interrater reliability of assessment not only within nursing but across professions. Results of this study illustrate inconsistency and confusion when using the Glasgow Coma Scale in practice; this has the potential to compromise care. Clarity around the issues highlighted is necessary. Moreover, these results can inform future guidelines and education required for supporting nurses in practice.

The Developing Assessment Instrument In Critical Thinking Ability For Fifth Grade Of Elementary School In Thematic Learning

The background of this study is about the availibility of the test assessment instrument to measure the critical thinking ability of fifth grade elementary school students which is limitted. Therefore, it needs to be developed. This development research used the Borg &amp; Gall development method. The purpose of this study was to develop an assessment instrument in essay. The assessment instrument was validated by 5 Experts Judgment by using the Aiken'V formula. The results of trials using the construct validity of the Confirmatory Factor Analysis (CFA), interater reliability used Two Ways Anova proved by the Ebel formula and internal consistent reliability with the Alpha Cronbach formula. The result of expert judgment validation showed that ≥0.8 which mean that 10 items were in valid category, construct validity results from large-scale test analysis by using LISREL 8.8 namely Confirmatory Factor fulfilled the testing of goodness of fit GFI value 0,93, CFI=0,97 and NFI=0,91. The three assessment criteria have value &gt; 0,90. It can be concluded that the construct validity is met. The construct validity is also evidenced by the loading factor of 10 items. All of which have a price of &gt; 0,3. The interrater reliability assessment instruments based on Expert Judgment coefficient value was 0,62. The expert agreement indicated that the rating given by each rater is reliable or consistent between one another and internal consistency of small scale test results of coefficient value of alpha 0,861, large scale 0.813. Practicality of assessment instruments based on expert judgment had mean 37.2 and included in practical category. The profile of critical thinking skills of fifth grade elementary school students category “Medium”. The Conclusions was the assessment instruments in critical thinking skills of fifth grade students on thematic learning were tested in validity, reliability and suitable to be used.

Scoring Using the Malnutrition Screening Tool: Dietetic Intern and Registered Nurse

To assess the number of patients that receive malnutrition screening and the accuracy of said screening on patients admitted to a large research-based teaching hospital. At the University of Virginia Health System (UVAHS), the Malnutrition Screening Tool (MST) is intended to be completed by registered nurses (RNs) within 24 hours of adult inpatient admission. The recent small, random audits at UVAHS have shown that 75% of patients are screened for malnutrition. More robust data are needed to better quantify the percentage of patients that receive an MST by nursing. Additionally, accuracy of the MST by nursing will be useful in determining whether education and/or systems changes are needed to improve screening processes. Malnutrition screening is an important first step in identifying hospitalized inpatients at risk for increased lengths of stay, mortality, and morbidity. The dietetic intern completed the MST using a paper form one day per week for 7 weeks. A list of newly admitted adult patients was gathered from the electronic medical report inpatient length of stay at UVAHS within 1 hour of the dietetic intern conducting MST screenings. A total of 85 patients were screened by the dietetic intern. After the MST was conducted, information was gathered from patients medical records: age, sex, admission date and time, unit, primary diagnosis, nursing completion of the MST, and nursing MST score (if applicable). A Weighted Cohens Kappa test was used to assess interrater reliability between the dietetic intern and nursing, including the probability that the scores were the same by chance. Nursing screened 67 out of the 85 patients (78.8%) seen by the dietetic intern. The Weighted Cohens Kappa test showed moderate interrater reliability between dietetic intern and nursing scores (Kappa score = 0.498). Out of a total of 85 patients screened for malnutrition by the dietetic intern, 20 out of the 85 (24%) were not triggered to see a registered dietitian nutritionist (RDN) due to nursing either not filling out the MST or filling it out incorrectly. Twelve out of the 67 (18%) patients screened by RNs as not being at risk were at risk for malnutrition based on dietetic intern screening. Eight out of 18 (44%) patients that nursing did not screen were at risk for malnutrition based on dietetic intern screening. Nursing screened 78.8% of patients for malnutrition using the MST. This is slightly higher than 75% that was seen in recent small random audits completed at UVAHS. Twenty out of 85 (24%) patients were missed due to nursing either not filling out the MST or filling it out incorrectly. Results indicate a potential knowledge deficit amongst nursing regarding how to conduct malnutrition screening. Results also indicate a need for improved compliance with use of the MST to ensure at-risk patients are referred to RDNs.

External Multicenter Study of Reliability and Reproducibility for Lower Cervical Spine Injuries Classification Systems-Part 1: A Comparison of Morphological Schemes.

Study Design:Multicenter observational survey study.Objectives:To quantify and compare the inter- and intraobserver reliability of Allen-Fergusson (A-F), Harris, Argenson, and AOSpine (AOS) classifications for cervical spine injuries, in a multicentric survey of neurosurgeons with different levels of experience.Methods:We used data of 64 consecutive patients. Totally, 37 surgeons (from 7 centers), were included in the study. The initial assessment was returned by 36 raters. The second assessment performed after 1.5 months included 24 raters.Results:We received 15 111 answers for 3840 evaluations. Raters reached a fair general agreement of the A-F scale, while the experienced group achieved κ = 0.39. While all groups showed moderate interrater reliability for primary assessment of Harris scale (κ = 0.44), the κ value for experts decreased from 0.58 to 0.49. The Argenson scale demonstrated moderate and substantial agreement among all raters (κ = 0.47 and κ = 0.55, respectively). The AOS scheme primary assessment general kappa value for all types of injuries and across all raters was 0.49, reaching substantial agreement among experts (κ = 0.62) with moderate agreement across beginner and intermediate groups (κ = 0.48 and κ = 0.44, respectively). The second assessment general agreement kappa value reached 0.56.Conclusions:We found the highest values of interobserver agreement and reproducibility among surgeons with different levels of experience with Argenson and AOSpine classifications. The AOSpine scale additionally incorporated more detailed description of compression injuries and facet-joint fractures. Agreement levels reached for Allen-Fergusson and Harris scales were fair and moderate, respectively, indicating difficulty of their application in clinical practice, especially by junior specialists.

An Assessment of Face Value, Usability, and Interrater Reliability of the Bronchiolitis Severity Score (BSS) for Children with Acute Bronchiolitis

An Assessment of Face Value, Usability, and Interrater Reliability of the Bronchiolitis Severity Score (BSS) for Children with Acute Bronchiolitis

An Assessment of Face Value, Usability, and Interrater Reliability of the Bronchiolitis Severity Score (BSS) for Children with Acute Bronchiolitis

Purpose: Respiratory scoring metrics for infants admitted with acute viral bronchiolitis have yet to undergo definitive validation for clinical use such as determining appropriate level of care, disease severity, or appropriate weaning of respiratory support. We aim to assess the Bronchiolitis Severity Score (BSS), a metric derived from the Seattle Pediatric Respiratory Score (Figure 1), to determine its face validity, usability, and interrater reliability. Methods: To assess interrater reliability and usability, we prospectively assessed …

Coronal Flexion Versus Coronal Neutral Sonographic Views in Infantile DDH: An Important Source of Variability.

Ultrasound (US) is the preferred imaging modality for the diagnosis and treatment of infantile developmental dysplasia of the hip (DDH). Currently accepted indices that distinguish normal from dysplastic hips in the coronal plane include percent femoral head coverage (FHC), α angle, and β angle. Recent data suggests that significant user and interscan variability may exist for these metrics. Less studied, however, is potential variability because of patient positioning, specifically coronal flexion versus coronal neutral views. The purpose of this study was to compare standard DDH indices between coronal US views with hips in flexion versus neutral positions, for hips of varying DDH severity. This retrospective study included normal infants and those treated for different severities of DDH. Coronal flexion and coronal neutral US images from the same study were evaluated at diagnosis, early treatment, start of weaning, and treatment resolution. FHC, α, and β angles were measured on both views at each time point and compared. Inter-rater and intra-rater reliability assessments were performed for all metrics. Among the 168 hips in this study (45 normals, 45 Ortolani positive, 17 Barlow positive, and 61 stable dysplasia), median FHC was significantly lower in coronal flexion compared with coronal neutral for normals and all 3 severities of DDH at each time point (mean decrease 8.4%; range 5.5% to 10.9%; P<0.01). Alpha angle also decreased on coronal flexion views, observed for all hip types, but only at certain time points (mean decrease 3.3 degrees; range 0 to 7.5 degrees; P<0.01 to 0.35). β angles demonstrated less variability between views, but also had poor reliability. Coronal flexion and coronal neutral views demonstrated significant differences in FHC and α angle across a spectrum of DDH severities and treatment time points. Flexion views may represent a "baby Barlow" test, revealing subtle instability as evidenced by the significant and consistent decrease in FHC across all hips. Indices measured in flexion, therefore, may represent more stringent criteria for defining normal hips. Level III-diagnostic study.

PC-FACS

PC-FACS

Visual Function Classification System for children with cerebral palsy: development and validation.

To develop and validate the Visual Function Classification System (VFCS), which was created to classify how children with cerebral palsy (CP) use visual abilities in daily life. The process of development and validation of the VFCS involved four phases: (1) drafting of the five levels from the analysis of literature and clinical experience; (2) validation of constructs and revision of the levels for concept meaningfulness, using nominal group process; (3) refinement by international Delphi survey; and(4) assessment of interrater reliability among professionals and with caregivers, and of test-retest reliability. Five nominal groups involved 29 participants; 65 people completed the first round and 51 the second round of the Delphi survey. Construct validity was demonstrated within an expert group and external validation through several stakeholders, with the involvement of patients and families to ensure meaningfulness of the concept. Discussions continued until consensus was reached about the construct and content of the five levels. Participants in the reliability study included 29 professionals, 39 parents, and a total sample of 160 children with CP (mean age [SD] 6y 6mo [3y 4mo]; median 5y 7mo, range 1-19y). Absolute interrater agreement among professionals was 86% (weighted κ=0.88; 95% confidence interval [CI] 0.83-0.93). Test-retest reliability was high (weighted κ=0.97; 95% CI 0.95-0.99). Parent-professional interrater reliability on 39 children was moderate (weighted κ=0.51; 95% CI 0.39-0.63). The VFCS has been appropriately constructed and provides a reliable system to classify visual abilities of children with CP both in clinical and in research settings. The Visual Function Classification System is a valid and reliable system. It classifies visual abilities of children with cerebral palsy in clinical and research settings. At a clinical level, it can be used to harmonize communication among professionals and identify patients' strengths and weaknesses. In research settings, it can be used to stratify patients, define natural history evolution, and interpret intervention studies.

Interrater And Intrarater Reliability Of The Standard Assessment Of Tackling Technique (SATT) On Secondary School Football Athletes

BACKGROUND: Eighteen years after Guskiewicz et al. (2000) began studying sport related concussion in college and high school football players, head and spine injuries remain a very serious concern for the sport. An upward trend in serious catastrophic brain and spinal cord injuries still exists (Meehan and Landry, 2015), despite greater awareness of signs and symptoms, improved injury evaluation, more accurate symptom assessment, cautious return to play decisions, better helmet technology, new tackling styles, updated coach education, and rule changes limiting contact in practice (Yang et al., 2017). A consistent, cost effective method to evaluate and screen unsafe tackling behaviors has yet to be established. The Standard Assessment of Tackling Technique (SATT) is an observational tool designed to score movement proficiency on five critical elements of an American football tackle. Each element is rated on a four step, ordinal scale from zero - did not occur to three - performed with proficiency. PURPOSE: of this study was to evaluate interrater and intrarater reliability of the SATT. METHODS:Fifteen healthy subjects were videotaped while performing a tackle proficiency assessment (TPA) drill on a square blocking dummy. Twelve players (n=12) completed all three TPA sessions spaced 7 days apart, three players did not complete all trials and their data was removed. A total of 36 videos were independently scored by two, trained raters. Training required completion of a forty-five-minute session with explanation of the scoring rubric and scoring sample videos until 80% scoring agreement was obtained. RESULTS: Interrater reliability was good for sessions 1 (ICC = 0.801; 95% confidence interval [CI]: 0.446-0.938) and 2 (ICC = 0.856; 95% CI: 0.575-0.956) and moderate for session 3 (ICC = 0.602; 95% CI: 0.076-0.867). The individual SATT components showed Arm Rip was the least reliable component (ICC = 0.40; 95% CI: 0.31-0.51) and leg drive was the most reliable component (ICC = 0.95; 95% CI: 0.92-0.97). Intrarater reliability was moderate to good for both raters (ICC = 0.57 - 0.79). CONCLUSIONS: The results showed that the SATT can be independently scored by two raters, following a 45 minute training session and more experienced raters demonstrate increased intrarater reliability.

The validity and inter-rater reliability of project assessment in mathematics learning

[English]: A common criticism of project assessment is the subjectivity and inconsistency of raters in scoring. In the present article, we provide the result of validity and inter-rater reliability test of the project assessment instrument. The instrument with a rubric was used to assess students’ project task in grade eight for function and relation topic. The task was adopted from mathematics textbooks used in the schools. The instrument has been tested to 10 raters/teachers and 94 grade eight students from three schools (in Surabaya and Gresik). Data were collected through the project assessment sheet along with its rubric as the scoring guidance for the teachers. Construct validity was analyzed through confirmatory factor analysis, while a reliability test was conducted by using inter-rater reliability method with the Intraclass Correlation Coefficient. The result of the validity test showed that the instrument did not fulfill the criteria of construct validity. It is indicated by the different number of factors between the initial construction and the empirical test result. In term of inter-rater reliability, the instrument is highly reliable. The findings indicate the need for testing the non-test assessment instrument provided on mathematics textbooks, so the aspects of its assessment sheet fulfill the valid and reliable criteria.&#x0D; Keywords: Validity, Reliability, Inter-rater, Project assessment, Project task&#x0D; [Bahasa]: Subjektivitas dan kurang konsistennya guru/rater dalam proses penyekoran merupakan kritik yang umum ditujukan pada penilaian proyek dalam pembelajaran matematika. Oleh karena itu, artikel ini menyajikan hasil uji validitas konstruk dan reliabilitas inter-rater instrumen penilaian projek. Instrumen yang dilengkapi rubrik tersebut digunakan untuk menilai tugas proyek siswa kelas VIII SMP pada materi relasi dan fungsi. Tugas diambil dari buku matematika yang digunakan di sekolah. Instrumen diujicobakan pada 10 raters/guru dan 94 siswa di tiga sekolah berbeda di Kota Surabaya dan Kabupaten Gresik. Data dikumpulkan melalui lembar penilaian projek yang dilengkapi rubrik penilaian sebagai pedoman guru dalam melakukan penskoran. Validitas konstruk dianalisis dengan menggunakan Confirmatory Factor Analysis, sedangkan reliabilitas inter-rater dianalisis dengan Intraclass Correlation Coefficient. Hasil uji validitas menunjukkan bahwa instrumen penilaian yang digunakan tidak valid secara konstruk. Ketidakvalidan ditandai dengan perbedaan banyaknya faktor hasil konstruksi awal dengan hasil uji empiris. Dari sisi reliabilitas inter-rater, instrumen penilaian proyek yang digunakan reliabel. Temuan ini mengindikasikan perlunya dilakukan pengujian instrumen penilaian non tes pada buku matematika, sehingga aspek-aspek dalam lembar penilaian menjadi valid dan reliabel.&#x0D; Kata kunci: Validitas, Reliabilitas, Inter-rater, Penilaian proyek, Tugas proyek

A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool

ObjectivesThe objectives of this study were to determine the interrater reliability (IRR) of assessment of multiple systematic reviews (AMSTAR) 2 for reviews of pharmacological or psychological interventions for the treatment of major depression, to compare it to that of AMSTAR and risk of bias in systematic reviews (ROBIS), and to assess the convergent validity between the appraisal tools. Study Design and SettingTwo groups of four raters were each assigned one of two samples of 30 systematic reviews. All eight raters applied AMSTAR 2 to their sample. Each group also applied either AMSTAR or ROBIS. Fleiss' kappa and Gwet's AC1 were calculated, and agreement between the tools was assessed. ResultsThe median kappa values as a measure of IRR indicated a moderate agreement for AMSTAR 2 (median = 0.51), a substantial agreement for AMSTAR (median = 0.62), and a fair agreement for ROBIS (median = 0.27). Validity results showed a positive association for AMSTAR and AMSTAR 2 (r = 0.91) as well as ROBIS and AMSTAR 2 (r = 0.84). For the overall rating, AMSTAR 2 showed a high concordance with ROBIS and a lower concordance with AMSTAR. ConclusionThe IRR of AMSTAR 2 was found to be slightly lower than the IRR of AMSTAR and higher than the IRR of ROBIS. Validity measurements indicate that AMSTAR 2 is closely related to both ROBIS and AMSTAR.