Inter-rater Reliability Analysis Research Articles

Quantification of torsional stiffness in running footwear: proposed methodology

There is evidence to suggest that certain extrinsic factors, such as footwear, may contribute to risk of running injury. Torsional shoe stiffness quantifies the resistance of a shoe to twisting about its long axis between the heel and toe box, and has been previously linked to changes in foot and ankle kinetics and kinematics during running and cutting motions. Despite the potential for torsional stiffness to influence gait and injury risk, torsional stiffness of running shoes is rarely reported in biomechanical research, and methods of quantifying shoe torsional stiffness are not currently standardised. The purpose of this study was to propose and validate a device for the quantification of torsional stiffness of a variety of types and sizes of running footwear. The device includes the measurement of shoe length. This provides required information for a more comprehensive calculation of torsional stiffness that represents the product of the area moment of inertia and the modulus of rigidity, both of which can be useful to characterise a particular type of shoe based on its shape and material properties. The results of inter-rater and intra-rater reliability analyses suggest that the device is useful for repeatable measurements of torsional stiffness in both the medial and lateral directions, as well as heel thickness.

Development of the Multi-Dimensional Analysis of Patient Outcome Predictions (MD.POP) during medical encounters

ObjectiveOur first objective was to develop the Multi-Dimensional analysis of Patient Outcome Predictions (MD.POP), an interaction analysis system that assesses how HCPs discuss precisely and exclusively patient outcomes during medical encounters. The second objective was to study its interrater reliability. MethodThe MD.POP was developed by consensus meetings. Forty simulated medical encounters between physicians and an actress portraying a patient were analysed. Interrater reliability analysis was conducted on 20 of those simulated encounters. ResultsThe MD.POP includes six dimensions: object, framing, value, domain, probability and form of POP. The coding method includes four steps: 1) transcription of the encounter, 2) POP identification, 3) POP dimension coding and 4) POP scoring. Descriptive analyses show that the MD.POP is able to describe verbal expressions addressing the patient’s outcomes. Statistical analyses show excellent interrater reliability (Cohen’s Kappa ranging from 0.92 to 0.94). ConclusionThe MD.POP is a reliable interaction analysis system that assesses how HCPs discuss patient medical, psychological or social outcomes during medical encounters. Practical implicationThe MD.POP provides a measure for researchers to study how HCPs communicate with patients about potential outcomes. Results of such studies will allow to provide recommendations to improve HCP’s communication about patients’ outcomes.

Prevalence and reliability of treatment-based classification for subgrouping patients with low back pain

Objectives: To observe the distribution of patients who presented with low back pain (LBP) and to determine the between therapists’ interrater reliability of assessments in a private outpatient setting using treatment-based classification (TBC) subgroups.Methods: An observational and methodological study was conducted. Four hundred and twenty-nine patients (231 male; 198 female) presenting LBP symptoms and referred to conservative treatment were assessed by 13 physical therapists who conducted a 60-min examination process utilizing TBC subgroups. Interrater reliability analyses from six raters were assessed using Fleiss’ kappa and previously recorded data (n = 30).Results: In this study, 65.74% of patients were classified in only one subgroup, the most prevalent being stabilization (21.91%), followed by extension (15.38%), traction (11.89%), flexion (10.96%), manipulation (5.13%), and lateral shift (0.47%). Approximately 20.98% of patients were classified in two subgroups, where the most frequent overlaps were flexion + stabilization (7.46%), extension + stabilization (6.06%), flexion + traction (4.20%), extension + manipulation (1.86%), and 13.29% of patients were not classified in any TBC subgroup. Analysis of interrater reliability showed a kappa value of 0.62 and an overall agreement of 66% between raters.Discussion: LBP is a heterogeneous clinical condition and several classification methods are proposed in the attempt to observe better outcomes for patients. Eighty-five percent of patients assessed were able to be classified when using the TBC assessment and reliability analysis showed a substantial agreement between raters.Level of Evidence: 2c.

Objective breast symmetry analysis with the breast analyzing tool (BAT): improved tool for clinical trials

PurposeObjective cosmetic analysis is important to evaluate the cosmetic outcome after breast surgery or breast radiotherapy. For this purpose, we aimed to improve our recently developed objective scoring software, the Breast Analyzing Tool (BAT®).MethodsA questionnaire about important factors for breast symmetry was handed out to ten experts (surgeons) and eight non-experts (students). Using these factors, the first-generation BAT® software formula has been modified and the breast symmetry index (BSI) from 129 women after breast surgery has been calculated by the first author with this new BAT® formula. The resulting BSI values of these 129 breast cancer patients were then correlated with subjective symmetry scores from the 18 observers using the Harris scale. The BSI of ten images was also calculated from five observers different from the first author to calculate inter-rater reliability. In a second phase, the new BAT® formula was validated and correlated with subjective scores of additional 50 women after breast surgery.ResultsThe inter-rater reliability analysis of the objective evaluation by the BAT® from five individuals showed an ICC of 0.992 with almost no difference between different observers. All subjective scores of 50 patients correlated with the modified BSI score with a high Pearson correlation coefficient of 0.909 (p < .001) which was better compared to the old software (r = 0.769; p < .001).ConclusionsThe modified BAT® software improves the correlation between subjective and objective BSI values, and may be a new standard for trials evaluating breast symmetry.

Validation of the Pediatric Sedation State Scale.

Development and validation of the Pediatric Sedation State Scale (PSSS) is intended to specifically meet the needs of pediatric procedural sedation providers to measure effectiveness and quality of care. The PSSS content was developed through Delphi methods utilizing leading pediatric sedation experts and published guidelines on procedural sedation in children. Video clips were created and presented to study participants, who graded the state of patients during procedures by using the PSSS to evaluate inter- and intrarater reliability by determining the intraclass correlation coefficient. We also compared the PSSS to the Observational Scale of Behavioral Distress-revised during 4 clinically relevant phases of a laceration repair procedure. Six sedation states were defined for the PSSS. Each state was assigned a numerical value with higher numbers for increasing activity states. We included behaviors associated with adequate and inadequate sedation and adverse events associated with excessive sedation. Analysis of interrater and intrarater reliability revealed an intraclass correlation coefficient of 0.994 (95% confidence interval: 0.986-0.998) and 0.986 (95% confidence interval: 0.970-0.995), respectively. Criterion validity was confirmed with respect to the Observational Scale of Behavioral Distress-revised (Spearman r = 0.96). Construct validity was indicated by significant differences in PSSS scores (P < .001) between 4 phases of a procedure, each having a different degree of painful or distressing stimuli. The PSSS is a 6-point scale that is a valid measure of the effectiveness and quality of procedural sedation in children within the limits of the testing method used in this study.

Mathematics Classroom Observation Protocol for Practices (MCOP2): A validation study

ABSTRACTThis article reports the robust validation process of the Mathematics Classroom Observation Protocol for Practices (MCOP2). The development of the instrument took into consideration both direct and dialogic instruction encompassing classroom interactions for the development of conceptual understanding, specifically examining teacher facilitation and student engagement. Instrument validation involved feedback from 164 external experts for content validity, interrater and internal reliability analyses, and structure analyses via scree plot analysis and exploratory factor analysis. Taken collectively, these results indicate that the MCOP2 measures the degree to which actions of teachers and students in mathematics classrooms align with practices recommended by national organizations and initiatives using a two-factor structure of teacher facilitation and student engagement.

Developing a digital photography-based method for dietary analysis in self-serve dining settings

Current population-based methods for assessing dietary intake, including food frequency questionnaires, food diaries, and 24-h dietary recall, are limited in their ability to objectively measure food intake. Digital photography has been identified as a promising addition to these techniques but has rarely been assessed in self-serve settings. We utilized digital photography to examine university students' food choices and consumption in a self-serve dining hall setting. Research assistants took pre- and post-photos of students’ plates during lunch and dinner to assess selection (presence), servings, and consumption of MyPlate food groups. Four coders rated the same set of approximately 180 meals for inter-rater reliability analyses; approximately 50 additional meals were coded twice by each coder to assess intra-rater agreement. Inter-rater agreement on the selection, servings, and consumption of food groups was high at 93.5%; intra-rater agreement was similarly high with an average of 95.6% agreement. Coders achieved the highest rates of agreement in assessing if a food group was present on the plate (95–99% inter-rater agreement, depending on food group) and estimating the servings of food selected (81–98% inter-rater agreement). Estimating consumption, particularly for items such as beans and cheese that were often in mixed dishes, was more challenging (77–94% inter-rater agreement). Results suggest that the digital photography method presented is feasible for large studies in real-world environments and can provide an objective measure of food selection, servings, and consumption with a high degree of agreement between coders; however, to make accurate claims about the state of dietary intake in all-you-can-eat, self-serve settings, researchers will need to account for the possibility of diners taking multiple trips through the serving line.

Adolescent and Young Adult Cancer Survivorship Educational Programming: A Qualitative Evaluation.

BackgroundThis program evaluation considers the need for increased professional and patient education for adolescent and young adult (AYA) cancer survivorship. Due to the high incidence of late effects of cancer treatment among AYA cancer survivors, knowledge sharing and communications are needed throughout the transition from cancer care into community care. AYA survivors are likely to need developmentally appropriate psychosocial care as well as extensive follow-on surveillance by physicians who are educated and aware of the likely chronic conditions and late effects that may occur in these patients.ObjectiveThe objective of this study was to evaluate the outcomes of the After Cancer Care Ends, Survivorship Starts for Adolescent and Young Adults (ACCESS AYA) programming. The intent of the ACCESS AYA program was to build health literacy around AYA survivorship issues and to stimulate improved communications between survivors and health care providers. This paper addresses the central research question of “How did the ACCESS AYA program increase health literacy, communications, and understanding among AYA survivors and providers?”MethodsThe primarily qualitative evaluation included a brief introductory survey of participant awareness and effectiveness of the ACCESS AYA project serving as a recruitment tool. Survey respondents were invited to participate in in-depth interviews based on interview guides tailored to the different stakeholder groups. The evaluation used the Atlas Ti qualitative database and software for coding and key word analyses. Interrater reliability analyses were assessed using Cohen kappa analysis with Stata 12.1 (StataCorp LLC) software.ResultsThe key themes, which included survivor wellbeing, health care professional education, cancer advocates role and education, hospital and community-based resources, and the role of societal support, are presented in a concept map. The interrater reliability scores (ranging from 1 to minus 1) were .893 for first cycle coding and .784 for the second cycle. In the brief quantitative survey based on a scale of 1 to 5 with 5 as high, the 22 respondents rated their level of awareness of the project with a mean 3.2 (CI 3.02-3.45) and project effectiveness with a mean of 4 (CI 3.72-4.27).ConclusionsThis study contributes to understanding of the ACCESS AYA survivor community in central Texas and the health care professionals and advocates who aid them in their efforts to a new normal life and wellbeing in their survivorship. The results of the evaluation highlight the need to continue to build both survivor and professional resources to address the unique impact of cancer on AYA cancer survivors.

Examiner characteristics and interrater reliability in a communication OSCE

ObjectiveTo identify inter-individual examiner factors associated with interrater reliability in a summative communication OSCE in the 4th study year. MethodsThe OSCE consists of 4 stations assessed with a 4-item 5-point global rating instrument. A bivariate secondary analysis of interrater reliability in relation to 4 examiner factors (gender, profession, OSCE experience, examiner training) was conducted. Intraclass correlation coefficients (ICC) were calculated and compared between examiner dyads of different similarity. Results169 pairwise ratings from 19 different examiners in 16 dyads were analysed. Interrater reliability is significantly higher in examiner dyads of same vs. different gender (ICC=0.76 (95%CI=0.65-0.83) vs. ICC=0.41 (95%CI=0.21-0.57)), in dyads of two clinicians vs. non-clinical/mixed professions (ICC=0.72 (95%CI=0.56-0.83) vs. ICC=0.57 (95%CI=0.41-0.69)), and in dyads with high vs. low/mixed OSCE experience (ICC=0.73 (95%CI 0.50-0.87) vs. ICC=0.56 (95%CI=0.41-0.69)). Participation in recent examiner training had no influence on ICCs. ConclusionBetter concordance of ratings between clinically active examiners might be a hint for context specificity of good communication. Higher interrater reliability between examiners with same gender may indicate gender-specific communication concepts. Practice implicationsMedical faculties introducing summative assessment of communication competence should focus the influence of examiner characteristics on interrater reliability.

Comparison of Radiographs and Computed Tomography for the Screening of Anterior Inferior Iliac Spine Impingement

To compare radiographic and 3-dimensional (3D) computed tomography (CT) imaging modalities for the screening of anterior inferior iliac spine (AIIS) impingement by establishing imaging measurement related to the AIIS. Anteroposterior and false-profile radiographs and 3D CT scans were obtained on 10 human cadaveric pelvises. On the anteroposterior view for each methodology, 2 measurements were calculated: distance to the most lateral AIIS from the 12 o'clock position on the acetabular rim, and the angle between the lateral AIIS and the sagittal plane. On the false-profile view for each methodology, 2 measurements were calculated: distance to the anterior AIIS from the 12 o'clock position on the acetabular rim, and the angle between the anterior AIIS and the sagittal plane. Inter-rater and intrarater reliability analyses were performed for both methods in addition to an intermethod analysis. The radiographic false-profile view was the most repeatable orientation, with intraclass correlation coefficients showing excellent reproducibility in both inter-rater (angle: 0.980, distance: 0.883) and intrarater (angle: 0.995, distance: 0.995) analyses. The mean distance from the 12 o'clock position of the acetabular rim to the most anterior/lateral aspect of the AIIS was 41.4mm and 16.0mm on the radiographic false-profile and anteroposterior views, respectively. Intermethod analysis showed a systematic, quantitative bias between modalities (anteroposterior view:-4.1mm, 6.7°; false-profile view:-0.1mm, 8.3°), which will remain relatively consistent as evidenced by the strong individual reproducibility of each measurement. AIIS morphology in relation to the acetabular rim 12 o'clock position and its angle relative to the sagittal plane can be quantitatively determined using either radiographic or 3D CT imaging modalities. Radiographic evaluation may be a valuable tool in the screening of AIIS impingement.

Fostering and assessing infographic design for learning: the development of infographic design criteria

In this ever more digital and visual world, it has become more vital that students are encouraged to create content during the learning process through effective visualization of their knowledge. Infographics are an effective method for such visualization. The current study therefore proposes an infographic design rubric (IDR) as a criteria-based framework to be used in the learning process for the creation and evaluation of infographics. The IDR was developed based on the infographic design model (IDM), a process of how to design infographics for teachers and learners using the three main components of content generation, visual design generation and digital design. The development process was conducted in the five design cycles of criteria determination, validity and reliability studies involving colleagues, university students, pre-service teachers and teachers. IDR criteria were generated based on content generation and visual design generation dimensions of IDM. The content generation dimension focuses on the quality and organization of the information presented, while the visual design generation dimension investigates the visual features of the infographic. Results of the inter-rater reliability analysis represent significant intraclass correlations. The proposed IDR is considered to be an important step for visualization in the learning process.

Validation of bioelectrical impedance analysis for assessing dry weight of dialysis patients in Pakistan.

Accurate dry weight estimation (DW) to achieve euvolemia is one of the key objectives of hemodialysis (HD). While conventionally DW is estimated by clinical examination, bioelectrical impedance analysis (BIA) has been proposed as an objective method to determine DW and has been tested extensively in the Western population. We aim to validate BIA for determining DW in a Pakistani population against the conventional clinician's method. This is a single-center validation study conducted at two outpatient HD units of Aga Khan Hospital, Karachi. One hundred and forty-eight DW readings of patients who were on maintenance HD were taken both by BIA technology and by clinical assessment. The clinician was blinded to readings obtained by BIA. Data analysis was carried out using the Statistical Package for Social Sciences (SPSS) version 19. Median age of patients was 63 years (range 12-89). Nearly 54.1% of the samples were female (n = 80). Spearman's correlation between the clinician's estimate and BIA-derived DW showed a correlation coefficient of 0.982, which was statistically significant (P <0.001). This association remained significant when stratified analysis was carried out by dividing the sample into subgroups according to age, gender, body mass index, and total body water content. Inter-rater reliability analysis using the kappa statistics showed almost perfect agreement between the two methods, κ = 0.929 (95% confidence interval, 0.878-0.980, P <0.001). BIA has been validated as a tool for DW assessment of HD patients in Pakistan in comparison to clinical method.

Presentation of the Coding and Assessment System for Narratives of Trauma (CASNOT): Application in Spanish Battered Women and Preliminary Analyses.

This study introduces a new coding system, the Coding and Assessment System for Narratives of Trauma (CASNOT), to analyse several language domains in narratives of autobiographical memories, especially in trauma narratives. The development of the coding system is described. It was applied to assess positive and traumatic/negative narratives in 50 battered women (trauma-exposed group) and 50 nontrauma-exposed women (control group). Three blind raters coded each narrative. Inter-rater reliability analyses were conducted for the CASNOT language categories (multirater Kfree coefficients) and dimensions (intraclass correlation coefficients). High levels of inter-rater agreement were found for most of the language domains. Categories that did not reach the expected reliability were mainly those related to cognitive processes, which reflects difficulties in operationalizing constructs such as lack of control or helplessness, control or planning, and rationalization or memory elaboration. Applications and limitations of the CASNOT are discussed to enhance narrative measures for autobiographical memories.

Inter-Rater Reliability and User-Friendliness of the Delirium Observation Screening Scale.

Delirium is a common and often unrecognized complication of hospitalized elderly patients. Currently, there is no evidence for inter-rater reliability studies between registered nurses in the literature. Furthermore, the user-friendliness of the Delirium Observation Screening Scale (DOSS) has not been tested in Austria. A quantitative cross-sectional design with a convenience sample of 141 patients and 36 nurses in an Austrian hospital. Analysis of rater-agreement and inter-rater reliability on item level, total score as well as category of delirium risk demonstrated very high agreement. In contrast, no or only fair kappa coefficient were determined. The user-friendliness of the scale was partially satisfactory. The very high absolute agreement speaks for the reliability of DOSS although the kappa paradox became obvious. The results of the presented study relate only to the tested setting.

Barriers and Health Beliefs Related to Weight Management Among Veterans With Human Immunodeficiency Virus

The success of antiretroviral therapy has led to dramatic changes in causes of morbidity and mortality among U.S. Veterans with human immunodeficiency virus (HIV). Among the 25,000 Veterans treated for HIV, 70% are over age 50 and the rate of obesity has doubled in this population. Veterans with HIV have a 50% increased risk of myocardial infarction yet have limited presence in prevention-related programs designed to lower cardiovascular disease risk. This mixed methods study (focus groups, Schwarzer and Renner physical activity, and nutrition self-efficacy questionnaires) was used to explore factors related to health behavior and identify barriers that overweight Veterans with HIV face in enrolling in the MOVE weight management program. Institutional review board approval was granted before the start of the study. All participants were recruited from the Infectious Disease clinic if they met national inclusion criteria for the MOVE weight management program and had not previously participated in the program. Transcribed audio recordings were independently analyzed and coded by four of the researchers using an exploratory process to obtain consensus regarding themes. An interrater reliability analysis for the Kappa statistic was performed to determine consistency among raters. The relationship between physical activity self-efficacy scores and nutrition self-efficacy scores was tested using Spearman's correlation coefficient. The median age of the sample was 56 with high rates of diabetes (36%), hypertension (73%), hyperlipidemia (36%), and tobacco use history (82%). External barriers to participation were discussed in addition to 8 other themes, which influence treatment engagement for Veterans with obesity and HIV including adaptation, stigma, self-management, and support. Veterans held strong beliefs about responsibility and commitment to their health and wanted to assume an active and informed role in their health care. Veterans with high levels of perceived self-efficacy indicated intention to overcome barriers to improve their nutrition and increase their physical activity. Refer to the full manuscript online to see the results in tables. Despite the chronic life-threatening nature of their condition, Veterans with HIV display a remarkable ability to adapt and commit to their treatment regimen. However, the dual stigma of obesity and HIV was a significant barrier to participation in weight management. This group placed high value on exercise over eating healthier and the importance of social support particularly from their Veteran peers. Focus groups allowed for fluid interaction between group members and researchers, rich conversation, and allowed additional clarification and exploration of topics. One unanticipated effect of the focus groups was that participants may feel less isolated after being a part of the discussion and may develop supportive relationships with their peers. It is possible that participants demonstrated more positive behavioral adaptation or other possible sources of bias. The study findings provide insight into health beliefs and barriers to weight management for all populations struggling with chronic disease and stigma. Data collected will inform future recruitment and retention strategies to engage Veterans with HIV in prevention-related programs designed to enhance long-term health and wellness.

Resident characteristics to evaluate during recruitment and interview: a Delphi study

Background and objectives: The recruitment and interview process for medical residency programmes is a time- and resource-intensive effort. There is very little research to guide programmes when evaluating residency candidates. This study represents one step in identifying candidate characteristics to assess during the recruitment and interview process. Methods: Sixteen expert interviewers from 14 family medicine residency programmes in North Carolina participated in a three-round Delphi study to build consensus around a ranked list of successful resident candidate characteristics. An interrater reliability analysis produced average pair-wise agreement and Krippendorff’s Alpha coefficients. Results: Clinical skills, medical knowledge, interpersonal and communication skills, critical thinking, and professional and ethical behaviour were the highest ranked characteristics. Average pair-wise agreement for rounds two and three were 6.30 and 11.04%, respectively. Conclusions: Residency programmes may benefit from using an empirically studied list of characteristics to evaluate candidate applications and interviews. Future research should include national surveys of expert interviewers from a variety of residency programmes and a longitudinal study to correlate interview evaluations using the ranked list with measures of residency success.

Methodological Criteria for Scoring Clustering and Switching in Verbal Fluency Tasks

Abstract The objective of this study was to describe an adaptation to Brazilian Portuguese of the methodological criteria for analysis of clustering and switching in semantic verbal fluency (SVF) and phonemic verbal fluency (PVF) tasks. The adaptation process consisted of six steps, including the selection of the clustering and switching variables based on data from a sample of 419 children and the analysis of inter-rater reliability (six raters). The following variables were scored: the total number of words generated the raw number of clusters, the mean cluster size, and the raw number of switches. There was a significant association between raters (intra-class correlation coefficients between 0.95 and 0.99), showing that the analytical method was reliable. Our study provides an evaluation of SVF and PVF tasks that goes beyond the overall score, making it possible to investigate the cognitive processes underlying this neuropsychological function.

Interrater reliability of Violence Risk Appraisal Guide scores provided in Canadian criminal proceedings.

Published research suggests that most violence risk assessment tools have relatively high levels of interrater reliability, but recent evidence of inconsistent scores among forensic examiners in adversarial settings raises concerns about the "field reliability" of such measures. This study specifically examined the reliability of Violence Risk Appraisal Guide (VRAG) scores in Canadian criminal cases identified in the legal database, LexisNexis. Over 250 reported cases were located that made mention of the VRAG, with 42 of these cases containing 2 or more scores that could be submitted to interrater reliability analyses. Overall, scores were skewed toward higher risk categories. The intraclass correlation (ICCA1) was .66, with pairs of forensic examiners placing defendants into the same VRAG risk "bin" in 68% of the cases. For categorical risk statements (i.e., low, moderate, high), examiners provided converging assessment results in most instances (86%). In terms of potential predictors of rater disagreement, there was no evidence for adversarial allegiance in our sample. Rater disagreement in the scoring of 1 VRAG item (Psychopathy Checklist-Revised; Hare, 2003), however, strongly predicted rater disagreement in the scoring of the VRAG (r = .58). (PsycINFO Database Record

Validitas dan Reliabilitas Ujian SOCA (Students Oral Case Analysis): Studi di Salah Satu Fakultas Kedokteran di Indonesia

Background: Medical faculty has to make sure that the students meet the minimal competence needed using apropriate exam. While the exam itself should facilitate students to learn. Oral examination has known for its ability to facilitate students learn but low in validity and reliability. Medical faculty of Lampung University (FK Unila) apply the student oral case analysis (SOCA) exam as one of block assessment component, as with MCQ, tutorial, and laboratory exam. This study aimed to evaluate validity and reliability of SOCA examination at FK UnilaMethod: Video of 65 students doing SOCA examination and 28 question rubrics had taken when odd semester exam year 2014-2015 has been carying out at FK Unila. Video and question rubrics were assessed by 5 panelis and analysed using Lawshe's content validity ratio (CVR) to determinate its content validity. Students performance on the video were re-assessed by another assessor to see inter-rater reliability, than analysed using kappa Cohen. Two expert in medical education assessed the cognitive comlpexity of the question rubrics. Data of SOCA's student's mark from year II, III, and IV were analysed for construct valdity and internal consistency.Results: 93,7% of the overall question in 65 video were valid (CVR&gt;99%) and 71,8% question number in 28 question rubrics also valid according to 5 panelis. SOCA cognitive complexity were at level of analyse, know how and 4a. Inter-rater reliability analysis showed 0,549 (moderate agreement) kappa value. Mann Whitney analysis for construct validity showed no significant difference of all year. Cronbach alpha analysis showed internal consistency at the point 0,575.Conclusion: FK Unila's SOCA of odd semester examination year 2014-2015 has sufficient content validity, sufficient cognitive complexity and sufficent inter-rater reliability but lack in construct validity and internal consistency. Keywords: SOCA, validity, reliability

Bronchiolitis Score of Sant Joan de Déu: BROSJOD Score, validation and usefulness

To validate the bronchiolitis score of Sant Joan de Déu (BROSJOD) and to examine the previously defined scoring cutoff. Prospective, observational study. BROSJOD scoring was done by two independent physicians (at admission, 24 and 48 hr). Internal consistency of the score was assessed using Cronbach's α. To determine inter-rater reliability, the concordance correlation coefficient estimated as an intraclass correlation coefficient (CCC) and limits of agreement estimated as the 90% total deviation index (TDI) were estimated. An expert opinion was used to classify patients according to clinical severity. A validity analysis was conducted comparing the 3-level classification score to that expert opinion. Volume under the surface (VUS), predictive values, and probability of correct classification (PCC) were measured to assess discriminant validity. About 112 patients were recruited, 62 of them (55.4%) males. Median age: 52.5 days (IQR: 32.75-115.25). The admission Cronbach's α was 0.77 (CI95%: 0.71; 0.82) and at 24 hr it was 0.65 (CI95%: 0.48; 0.7). The inter-rater reliability analysis was: CCC at admission 0.96 (95%CI 0.94-0.97), at 24 h 0.77 (95%CI 0.65-0.86), and at 48 hr 0.94 (95%CI 0.94-0.97); TDI 90%: 1.6, 2.9, and 1.57, respectively. The discriminant validity at admission: VUS of 0.8 (95%CI 0.70-0.90), at 24 h 0.92 (95%CI 0.85-0.99), and at 48 hr 0.93 (95%CI 0.87-0.99). The predictive values and PCC values were within 38-100% depending on the level of clinical severity. There is a high inter-rater reliability, showing the BROSJOD score to be reliable and valid, even when different observers apply it. Pediatr Pulmonol. 2017;52:533-539. © 2016 Wiley Periodicals, Inc.