Inter-rater Reliability Agreement Research Articles

Quality of Novel Coronavirus Related Health Information over the Internet: An Evaluation Study.

Background The novel coronavirus disease (COVID-19) has spread globally from its epicenter in Hubei, China, and was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. The most popular search engine worldwide is Google, and since March 2020, COVID-19 has been a global trending search term. Misinformation related to COVID-19 from these searches is a problem, and hence, it is of high importance to assess the quality of health information over the internet related to COVID-19. The objective of our study is to examine the quality of COVID-19 related health information over the internet using the DISCERN tool. Methods The keywords included in assessment of COVID-19 related information using Google's search engine were “Coronavirus,” “Coronavirus causes,” “Coronavirus diagnosis,” “Coronavirus prevention,” and “Coronavirus management”. The first 20 websites from each search term were gathered to generate a list of 100 URLs. Duplicate sites were excluded from this search, allowing analysis of unique sites only. Additional exclusion criteria included scientific journals, nonoperational links, nonfunctional websites (where the page was not loading, was not found, or was inactive), and websites in languages other than English. This resulted in a unique list of 48 websites. Four independent raters evaluated the websites using a 16-item DISCERN tool to assess the quality of novel coronavirus related information available on the internet. The interrater reliability agreement was calculated using the intracluster correlation coefficient. Results Results showed variation in how the raters assigned scores to different website categories. The .com websites received the lowest scores. Results showed that .edu and .org website category sites were excellent in communicating coronavirus related health information; however, they received lower scores for treatment effect and treatment choices. Conclusion This study highlights the gaps in the quality of information that is available on the websites related to COVID-19 and study emphasizes the need for verified websites that provide evidence-based health information related to the novel coronavirus pandemic.

The Reliability of Radiographic Evaluation of the Posterior Tibial Slope in Skeletally Immature Patients

Objectives:An association exists between increased posterior tibial slope and anterior cruciate ligament (ACL) injuries in pediatric patients with open physes. Additionally, an increased posterior tibial slope is also associated with increased odds of a further ACL injury after ACL reconstruction. Reliable radiographic measurement techniques are important for investigating limb alignment prior to and following pediatric ACL reconstruction. There have been multiple methods described to measure tibial slope, however, it is unknown if these are reliable in the pediatric population given the altered and developing proximal tibia anatomy during skeletal maturation. The purpose of this study is to evaluate the intra- and interobserver reliability of previously described posterior tibial slope measurements from lateral radiographs of skeletally immature patients.Methods:A retrospective chart review was performed including patients age 6-18 years old with available lateral knee radiographs and no prior surgery or musculoskeletal pathology. 130 patients (ten in each age group) were analyzed by three reviewers. Measurements were made using the Centricity Enterprise Web PACS System (Version 3.0; GE Medical Systems, Barrington, Illinois). The posterior tibial slope was measured using three previously described methods: the anterior tibial cortex (ATC), posterior tibial cortex (PTC), and the proximal tibia anatomic axis (TPAA) (Figure 1). The radiographs were graded by each reviewer twice, performed two weeks apart. The intra- and interobserver agreements were determined using the intraclass correlation coefficient (ICC) with the second set of measurements used for interobserver agreement. ICC estimates and their 95% confident intervals were calculated using SAS statistical package (Version 9; SAS Institute, Cary, North Carolina) based on an individual ratings, absolute-agreement, two-way mixed-effects model. As described by Landis and Koch, the interpretation of the ICC was as follows—slight: 0.00 to 0.20; fair: 0.21 to 0.40; moderate: 0.41 to 0.60; substantial: 0.61 to 0.80; almost perfect agreement: 0.81 to 1.00.Results:There were 130 patients included with an average age of 12 years old (range 6-18 years) with 47.7% (n=62) male patients. The mean measurements were ATC: 12.3 degrees, PTC 7.2 degrees, and TPAA: 9.3 degrees. Measures of intra-observer agreement met almost perfect agreement criteria among all three reviewers for all three methods of measuring the posterior tibial slope with a mean of 0.88 (range, 0.86-0.92) for ATC, 0.85 (range, 0.82-0.87) for PTC, and 0.87 (range, 0.82-0.92) for TPAA. (Table 1) Measures of inter-observer agreement was substantial across all three reviewers for all three methods of measuring with an average of 0.72 (range, 0.70-0.83) for ATC, 0.74 (range, 0.68-0.83) for PTC, and 0.74 (range, 0.68-0.84) for TPAA (Table 1).Conclusion:In accordance with prior reports, the ATC measurement yields larger values and PTC smaller values when measuring posterior tibial slope. The three different methods of measuring demonstrated almost perfect agreement for intra-rater reliability and substantial agreement for inter-rater reliability. There was no difference in reliability across the three different measurement methods. Thus, despite the transforming anatomy during skeletal maturation, the posterior tibial slope can be reliability measured in the skeletally immature population using plain lateral radiographs and any of the three described methods- ATC, PTC, or TPAA.Figure 1.

The Infantile Hemangioma Referral Score: A Validated Tool for Physicians.

Infantile hemangiomas (IHs) are common; some cases require timely referral and treatment to prevent complications. We developed and validated a reliable instrument for timely and adequate referral of patients with IH to experts by nonexpert primary physicians. In this multicenter, cross-sectional, observational study, we used a 3-stage process: (1) development of the Infantile Hemangioma Referral Score (IHReS) tool by IH experts who selected a representative set of 42 IH cases comprising images and a short clinical history, (2) definition of the gold standard for the 42 cases by a second independent committee of IH experts, and (3) IHReS validation by nonexpert primary physicians using the 42 gold standard cases. A total of 60 primary physicians from 7 different countries evaluated the 42 gold standard cases (without reference to the IHReS tool); 45 primary physicians evaluated these cases using the IHReS questionnaire, and 44 completed retesting using the instrument. IHReS had a sensitivity of 96.9% (95% confidence interval 96.1%-97.8%) and a specificity of 55.0% (95% confidence interval 51.0%-59.0%). The positive predictive value and negative predictive value were 40.5% and 98.3%, respectively. Validation by experts and primary physicians revealed substantial agreement for interrater reliability and intrarater repeatability. IHReS, a 2-part algorithm with a total of 12 questions, is an easy-to-use tool for primary physicians for the purpose of facilitating correct and timely referral of patients with IH. IHReS may help practitioners in their decision to refer patients to expert centers.

Preoperative functional MRI use in neurooncology patients: a clinician survey.

Functional MRI (fMRI) is increasingly being investigated for use in neurosurgical patient care. In the current study, the authors characterize the clinical use of fMRI by surveying neurosurgeons' use of and attitudes toward fMRI as a surgical planning tool in neurooncology patients. A survey was developed to inquire about clinicians' use of and experiences with preoperative fMRI in the neurooncology patient population, including example case images. The survey was distributed to all neurosurgical departments with a residency program in the US. After excluding incomplete surveys and responders that do not use fMRI (n = 11), 50 complete responses were included in the final analysis. Responders were predominantly from academic programs (88%), with 20 years or more in practice (40%), with a main area of practice in neurooncology (48%) and treating an adult population (90%). All 50 responders currently use fMRI in neurooncology patients, mostly for low- (94%) and high-grade glioma (82%). The leading decision factors for ordering fMRI were location of mass in dominant hemisphere, location in a functional area, motor symptoms, and aphasia. Across 10 cases, language fMRI yielded the highest interrater reliability agreement (Fleiss' kappa 0.437). The most common reasons for ordering fMRI were to identify language laterality, plan extent of resection, and discuss neurological risks with patients. Clinicians reported that fMRI results were not obtained when ordered a median 10% of the time and were suboptimal a median 27% of the time. Of responders, 70% reported that they had ever resected an fMRI-positive functional site, of whom 77% did so because the site was "cleared" by cortical stimulation. Responders reported disagreement between fMRI and awake surgery 30% of the time. Overall, 98% of responders reported that if results of fMRI and intraoperative mapping disagreed, they would rely on intraoperative mapping. Although fMRI is increasingly being adopted as a practical preoperative planning tool for brain tumor resection, there remains a substantial degree of discrepancy with regard to its current use and presumed utility. There is a need for further research to evaluate the use of preoperative fMRI in neurooncology patients. As fMRI continues to gain prominence, it will be important for clinicians to collectively share best practices and develop guidelines for the use of fMRI in the preoperative planning phase of brain tumor patients.

Operation of the Basketball Jump Shooting Accuracy Test: Intra- and inter-rater reliability of scoring procedures and floor and ceiling effects for test performance

The purpose of this study was to determine the intra- and inter-rater reliability of scoring procedures used in the newly developed Basketball Jump Shooting Accuracy Test (BJSAT) and assess for floor and ceiling effects in test performance. Thirty-one semi-professional basketball athletes completed four trials of the BJSAT. The BJSAT contains one jump shot at eight different locations, equally distributed across two- and three-point shots. Intra-rater reliability was determined by assessing the level of agreement between scores live in-person and watching captured video by the same assessor. Inter-rater reliability was determined by examining the level of agreement between two assessors who separately scored the BJSAT while watching captured video. Descriptive statistics and Cohen’s kappa (κ) were calculated to quantify the intra- and inter-rater reliability of the BJSAT. Floor and ceiling effects in scoring outcomes were analyzed to evaluate the suitability of the BJSAT. Significance for the study was set at p <0.05. Intra-rater reliability demonstrated an almost perfect (κ = 0.85, p <0.01) agreement between scores (12.6 ± 2.5 vs. 13.1 ± 2.8). The agreement for inter-rater reliability was rated as substantial (12.3 ± 2.5 vs. 13.5 ± 2.9, κ = 0.70, p <0.01). Floor and ceiling effects were absent in the BJSAT indicating the assessment is suitable for semi-professional basketball athletes. The BJSAT is an assessment where one or multiple assessors can reliably score shooting performance for functions including player monitoring, to assess the efficacy of interventions aimed at improving skills and to assist with team selection across the season.

Scrum Language Use in a Software Engineering Firm: An Exploratory Study

Background: Although agile and Scrum have been important frameworks in software engineering for over a decade, little research has explored how teams use Scrum language within their sprints. Literature review: Most explorations of Scrum communication have been collected through self-reported means. These studies are inherently unable to explore how Scrum teams use Scrum-centric language in their meetings in ways that adhere or run counter to standard Scrum practice. Research questions: 1. In what ways is Scrum reflected in the language used by team members in various sprint meetings? 2. What associations exist between the job title of team members and their use of Scrum language? 3. What does a discourse analysis reveal about the ways in which this team uses language to value and discount Scrum? Research methodology: For three sprints over 10 weeks, I recorded meetings of 27 Scrum team members. I transcribed these meetings, developed a codebook for assigning Scrum language categories, conducted an interrater reliability agreement on the data, completed a correspondence analysis on how Scrum language associates with meeting types and job titles, and conducted a discourse analysis to determine in what ways these teams value and discount Scrum. Results/discussion: Scrum language was found in all recorded meetings across all three sprints, with much language found in the planning meetings. Few associations existed between Scrum language and job title, suggesting that Scrum at this engineering firm is an egalitarian process. In addition, the discourse analysis revealed that this engineering firm valued User Story and Sprint Execution language while discounting Capacity and Story Pointing language. Conclusions: Although this group broadly adheres to Scrum practices about 68% of the time, this study finds that several current standard components of Scrum are routinely discounted. This exploratory study suggests that more research into the in-situ use of Scrum language in engineering workplaces is necessary to better inform engineering professionals about the communicative expectations of Scrum and to better enable engineering communication educators to prepare future engineers for Scrum realities.

Inter-rater reliability of medication error classification in a voluntary patient safety incident reporting system HaiPro in Finland.

Medication errors are common in healthcare. Medication error reporting systems can be established for learning from medication errors and risk prone processes, and their data can be analysed and used for improving medication processes in healthcare organisations. However, data reliability testing is crucial to avoid biases in data interpretation and misleading findings informing patient safety improvement. To assess the inter-rater reliability of medication error classifications in a voluntary patient safety incident reporting system (HaiPro) widely used in Finland, and to explore reported medication errors and their contributing factors. The data consisted of medication errors (n = 32 592), including near misses, reported by 36 Finnish healthcare organisations in 2007-2009. The reliability of the original classifications was tested by an independent researcher reclassifying a random sample of errors (1%, n = 288) based on narratives. The inter-rater reliability of agreement (κ) of the classifications was calculated to describe the degree of conformity between the researcher and the original data classifiers. Descriptive statistics were used to describe the medication errors. The inter-rater reliability between the researcher and the original data classifiers was acceptable (κ ≥ 0.41) in 11 of 42 (26%) medication error classes. Thus, these errors could be pooled from different healthcare units for the exploration of medication errors at the level of all reporting organisations. Contributing factors were identified in 48% (n = 137) of the medication error narratives in the random sample (n = 288). The most commonly reported errors were dispensing errors (34%, n = 10 906), administration errors 25% (n = 7972), and documentation errors 17% (n = 5641). The data classified by different classifiers can be pooled for some of the medication error classes. Consistency of the classification and the quality of narratives need improvement, as well as reporting and classification of contributing factors to provide high quality information on medication errors.

An articulatory-based disyllabic and trisyllabic Non-Word Repetition test: reliability and validity in Italian 3- to 7-year-old children

ABSTRACTThe contribution of the phonological working memory to the Non-Word Repetition (NWR) task is well established, but growing evidence also suggests a valuable underlying role of oro-motor abilities. Assuming that NWR involves output implementation mediated by the speech motor system, the study aimed to develop a novel Italian NWR task of disyllabic and trisyllabic items. The task, for the first time, was composed using each Italian speech sound.The study also aimed to investigate the reliability proprieties of the task (test–retest, intra-rater, inter-rater), internal consistency, concurrent and construct validity, and to collect normative data for pre-school children. An observational cross-sectional study was conducted and 375 Italian-speaking typically developing children aged 3.0–6.11 years were assessed with the novel test. Two raters assessed inter-rater reliability in a random sub-sample; test–retest reliability was examined through a repeated administration of the task two weeks apart; intra-rater reliability was investigated by two evaluations of the same audio-recordings. To evaluate concurrent validity children were also assessed through a pre-existing NWR test, and to estimate construct validity, scores from children of different age groups were compared.Results indicated excellent test–retest, intra-rater, inter-rater reliability agreement, high internal consistency and good concurrent validity. Normative data suggested that successful performance on NWR increases with age and declines with an increase in stimuli length. Despite the low working memory load in the disyllabic and trisyllabic items, a saturation of the task was not observed for any age group.The novel NWR test is a reliable and valid instrument, and it can be applied in clinical practice to assess how the child retains, plans and articulates a phonological and motor programme of unknown verbal material.

The Development And Testing Of A Direct Observation Protocol For Children’S Free-play Activity

Traditional real-time direct observation (DO) systems have been used for decades to assess children’s free-living physical activity (PA). Using video-taped DO would overcome several methodological issues and allow for more precise assessments of behaviors. PURPOSE: To develop and test a novel video-based DO system for children’s free-play activity. METHODS: Following iterative DO system development (The Observer XT, Noldus), 28 children (age=8.4±1.5 years) participated in a 30-minute indoor free-play session. The participants were recorded using a GoPro camera and wore an accelerometer on the hip (AG-H) and non-dominant wrist (AG-W). Researchers coded videos for the main Whole-Body Movement and four modifiers: 1) Locomotion, 2) Limb Movement, 3) Activity Type, and 4) MET value. For intrarater reliability, percent agreement was calculated from six randomly selected videos, using duplicate entries by an expert coder one-week apart. For inter-rater reliability, three videos were used to calculate percent agreement between entries from trained, novice coders (n=6) and the expert coder. To assess construct validity, time spent in activity intensity categories from expert-coded DO MET values were compared with accelerometer estimates using Wilcoxon Rank-Sum tests. RESULTS: Percent agreement for intra-rater reliability was above 80% except for Locomotion (47%; video 4, 26%; video 3) and Limb Movement, and MET value (19%, 78%, respectively; video 3). Across all variables, percent agreement for inter-rater reliability ranged widely from 12%-96%, 0-100%, and 36%-97% for videos 1, 2 and 3, respectively. Mean estimated time spent in PA intensity categories from AG-H overestimated sedentary (SED; p=0.008), moderate (MPA; p<0.001), and moderate-to-vigorous PA (MVPA; p=0.017) and underestimated light, (LPA; p< 0.001). The AG-W underestimated SED (p=0.03) and LPA (p<0.001) but overestimated MPA (p<0.001) and MVPA (p<0.001). CONCLUSIONS: The current DO system is feasible for observing detailed changes in children’s free-play activity. However, refinement to the system must be made to improve reliability before it is adopted as a criterion measure for free-play activity in children. Supported by: University of Massachusetts Amherst Commonwealth Honors College

Reliability of the Dutch Cleft Speech Evaluation Test and Conversion to the Proposed Universal Scale.

The Dutch cleft speech evaluation test (DCSET) has been implemented by the speech-language pathologists nationwide in the Netherlands since 2003, but the inter- and intrarater reliability was unknown. Two speech-language pathologists experienced in evaluating cleft speech assessed audio recordings of 20 children with cleft speech using the DCSET, and reassessed the recordings 2 weeks later. Intra- and interrater reliability was calculated, but found to be unacceptable after the first phase of this study using audio recordings. Following consensus training and some modifications in the scoring scales, the study was repeated using video recordings of 20 different children with cleft speech. Results from the second phase of this study using standardized video recordings showed fair, moderate, and good reliability on different subsets of the DCSET. Intrarater reliability (Kappa 0.59-1.00) was greater than interrater reliability (Kappa 0.33-0.79). Interrater reliability agreement was good (Kappa 0.63-0.79) for consonant production errors and speech understandability and acceptability, moderate (Kappa 0.59) for the resonance of the nasal passage, and fair (Kappa 0.33-0.37) for the resonance of the mixed and denasal passages. Subsequently, an algorithm was made to convert the DCSET scales to universal scales for international comparison of cleft speech as suggested by Henningsson et al in 2008.

Simulated patient training: Using inter-rater reliability to evaluate simulated patient consistency in nursing education

BackgroundSimulated patients (SPs) are frequently used for training nursing students in communication skills. An acknowledged benefit of using SPs is the opportunity to provide a standardized approach by which participants can demonstrate and develop communication skills. However, relatively little evidence is available on how to best facilitate and evaluate the reliability and accuracy of SPs' performances. AimThe aim of this study is to investigate the effectiveness of an evidenced based SP training framework to ensure standardization of SPs. The training framework was employed to improve inter-rater reliability of SPs. MethodsA quasi-experimental study was employed to assess SP post-training understanding of simulation scenario parameters using inter-rater reliability agreement indices. Two phases of data collection took place. Initially a trial phase including audio-visual (AV) recordings of two undergraduate nursing students completing a simulation scenario is rated by eight SPs using the Interpersonal Communication Assessments Scale (ICAS) and Quality of Discharge Teaching Scale (QDTS). In phase 2, eight SP raters and four nursing faculty raters independently evaluated students' (N=42) communication practices using the QDTS. ResultsIntraclass correlation coefficients (ICC) were >0.80 for both stages of the study in clinical communication skills. ConclusionThe results support the premise that if trained appropriately, SPs have a high degree of reliability and validity to both facilitate and evaluate student performance in nurse education.

Testing the reliability and validity of a Transition Map for older adults in long term care settings

Mapping individual patterns of decline in older adults may aid coordinating long term aged care. This study developed a new scale (Transition Maps) to summarise the overall care pathway for long term aged care residents, in a simplified manner incorporating mapping concepts. Transition Maps were developed using mixed methods in two phases, and based on expert opinion, literature review, and input from aged care health professionals. Four professions (primary physician, nurse, allied health, lifestyle services) generated 147 Transition Maps for 38 residents living in a long term care. Preliminary construct validity and inter-rated reliability were evaluated. Results showed that Inter-rater reliability of agreement with the overall care pathway for each resident was kappa = 0.492. Consensus was lowest between nurse care managers and primary physicians (kappa = 0.384), and highest between nurse managers and Lifestyle Services (kappa = 0.77). Preliminary testing of the Transition Map scale provides initial support of construct validity and inter-rater reliability and provides some evidence that Transition Maps can improve the coordination of long term aged care.

Proficiency in identifying, managing and communicating medical errors: feasibility and validity study assessing two core competencies.

BackgroundCommunication skills and professionalism are two competencies in graduate medical education that are challenging to evaluate. We aimed to develop, test and validate a de novo instrument to evaluate these two competencies.MethodsUsing an Objective Standardized Clinical Examination (OSCE) based on a medication error scenario, we developed an assessment instrument that focuses on distinctive domains [context of discussion, communication and detection of error, management of error, empathy, use of electronic medical record (EMR) and electronic medical information resources (EMIR), and global rating]. The aim was to test feasibility, acceptability, and reliability of the method.ResultsFaculty and standardized patients (SPs) evaluated 56 trainees using the instrument. The inter-rater reliability of agreement between faculty was substantial (Fleiss k = 0.71) and intraclass correlation efficient was excellent (ICC = 0.80). The measured agreement between faculty and SPs evaluation of resident was lower (Fleiss k = 0.36). The instrument showed good conformity (ICC = 0.74). The majority of the trainees (75 %) had satisfactory or higher performance in all six assessed domains and 86 % found the OSCE to be realistic. Sixty percent reported not receiving feedback on EMR use and asked for subsequent training.ConclusionAn OSCE-based instrument using a medical error scenario can be used to assess competency in professionalism, communication, using EMRs and managing medical errors.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-016-0755-5) contains supplementary material, which is available to authorized users.

A Training Program Using an Audience Response System to Calibrate Dental Faculty Members Assessing Student Clinical Competence

The aim of this study was to assess the effectiveness of calibration training of departmental faculty and competency graders using an audience response system on operative dentistry concepts across 12 months. The training sessions were designed to further solidify the process and equilibration of clinical opinions among faculty members and provide a more calibrated grading assessment during patient care for student performance feedback. Four (quarterly) calibration sessions occurred over 12 months in 2015. The first session was considered the baseline (control value) for this study. Pre- and post-calibration interrater agreement was assessed. Additionally, a pre and post assessment with ten Likert-scale questions was used to measure students' perceptions of instructional consistency. The results showed that a statistically significant increase in conceptual knowledge scores occurred for both departmental faculty members and competency graders across each of the four sessions (one-factor ANOVA; p<0.05). Interrater reliability agreement also significantly improved for both department faculty members and competency graders' clinical assessments over 12 months of implementation (Cohen's Kappa; p<0.05). There was a statistically significant increase in positive student perceptions on all ten questions (dependent t-test; p<0.05). Implementation of an audience response system for departmental and competency graders was found to be effective in facilitating a discussion forum, calibrating clinical assessments, and improving student perceptions. The positive results from this study support the value of dental schools' introducing faculty development programs to ensure consistent instruction for assessing dental student competence.

Self-administered version of the Fabry-associated pain questionnaire for adult patients.

BackgroundFabry-associated pain may be the first symptom of Fabry disease (FD) and presents with a unique phenotype including mostly acral burning triggerable pain attacks, evoked pain, pain crises, and permanent pain. We recently developed and validated the first Fabry Pain Questionnaire (FPQ) for adult patients. Here we report on the validation of the self-administered version of the FPQ that no longer requires a face-to-face interview but can be filled in by the patients themselves allowing more flexible data collection.MethodsAt our Würzburg Fabry Center for Interdisciplinary Treatment, Germany, we have developed the self-administered version of the FPQ by adapting the questionnaire to a self-report version. To do this, consecutive Fabry patients with current or past pain history (n = 56) were first interviewed face-to-face. Two weeks later patients’ self-reported questionnaire results were collected by mail (n = 55). We validated the self-administered version of the FPQ by assessing the inter-rater reliability agreement of scores obtained by supervised administration and self-administration of the FPQ.ResultsThe FPQ contains 15 questions on the different pain phenotypes, on pain development during life with and without therapy, and on impairment due to pain. Statistical analysis showed that the majority of questions were answered in high agreement in both sessions with a mean AC1-statistic of 0.857 for 55 nominal-scaled items and a mean ICC of 0.587 for 9 scores.ConclusionsThis self-administered version of the first pain questionnaire for adult Fabry patients is a useful tool to assess Fabry-associated pain without a time-consuming face-to-face interview but via a self-reporting survey allowing more flexible usage.Electronic supplementary materialThe online version of this article (doi:10.1186/s13023-015-0325-7) contains supplementary material, which is available to authorized users.

A standardised tool for assessing needs in forensic psychiatric population: clinical validation of the Italian CANFOR, staff version.

The Camberwell Assessment of Need - Forensic Version (CANFOR) is a standardised assessment tool specifically designed to assess needs for care in forensic psychiatric populations. The original English version of the instrument has shown good psychometric properties. The aim of this study was to validate the Italian version of the CANFOR-staff tool. After translation and back-translation, the Italian CANFOR tool was administered to a sample of 50 forensic psychiatric patients. Convergent validity was tested using the Brief Psychiatric Rating Scale (BPRS) and the Global Assessment of Functioning (GAF) by applying Kendall's tau-b. Inter-rater and test-retest reliabilities were measured by ICCs for need dimensions (total and unmet) and Cohen's kappa coefficients for individual need items. Regarding convergent validity, a higher number of needs (total and unmet) were associated with more severe psychiatric symptoms (BPRS). Higher numbers of unmet needs were also associated with lower levels of global functioning (GAF). ICCs for total and unmet needs scores indicated a good level of agreement for inter-rater reliability and a very good level for test-retest, respectively. Regarding the specific items, inter-rater Cohen's kappa was high (moderate to very good agreement) for 18 items in relation to the presence of a need and for 15 items in the rating of an unmet need, whereas Cohen's kappa for test-retest reliability was very high for all the items in the presence of a need and high for 18 of the unmet need domains. The Italian version of CANFOR has adequate psychometric properties. It can be considered a promising instrument for the assessment of needs of forensic psychiatric patients.

Validated Assessment Tools for Reconstructing Facial Defects in Simulation

Objectives: 1) Create a 3D functional organosilicate model for assessing facial reconstruction that is valid and has high inter-rater reliability. 2) Perform a prospective blinded study using 3D and 2-dimensional (2D) models for evaluating facial reconstruction. Methods: A 3D organosilicate model of a human adult face was built with a corresponding 2D paper model. Surgical subspecialty trainees from six institutions prospectively drew a z-plasty on the paper model and the organosilicate model. Next, the subjects each performed a z-plasty on the organosilicate model with surgical instruments and suture. Three blinded reviewers rated two drawings and one scar revision per subject. Results: Forty-three trainees encompassing 3 PGY1, 10 PGY2, 13 PGY3, 11 PGY4, 3 PGY5, and 3 PGY6 (fellows) completed the study. Subjects were placed into three groups, novices (PGY1+PGY2), intermediates (PGY3+PGY4), and experts (PGY5+PGY6). Experts performed better than intermediates and intermediates better than novices for performing the z-plasty and drawing on the organosilicate model. This statistically significant difference between groups was determined by one-way ANOVA (p =&lt;.05) and Tukey post-hoc test p=&lt;.05 and showed substantial inter-rater reliability agreement among the three raters with an average Cohen’s Kappa=0.76. On the 2D paper model the experts performed better than novices (p=&lt;.05); however, there was no difference between experts and intermediates (p=0.98). Conclusions: We have created a 3D organosilicate model for facial reconstruction that is valid and shows high inter-rater reliability. The 3D model was able to detect a difference between experts and intermediates where the 2D model could not.

Cross-cultural adaptation and reliability evaluation of Iranian version of Functional Assessment Measure in spinal cord injury patients

Background and purposeThe aim of this study was to translate and validate the Iranian version of the Functional Assessment Measure (FAM) among patients with spinal cord injury (SCI). Material and methodsTwo hundred patients with SCI participated in this study. A convenience sampling approach was used for selection of the patients. The FAM was translated into Persian language and then independently translated back into English. The reliability of the FAM was assessed by test-retest methods with a 14-day interval for interrater reliability. Intraclass correlation coefficient (ICCs) was calculated and interrater reliability and intrarater reliability were assessed. ResultsThe mean age of the patients was 35.7 years (SD, 7.2) and 86% of patients were male. The Cronbach alpha coefficient for both raters was above 0.70. Intrarater reliability of the Iranian version of the FAM ranged from good to excellent agreement. The highest level of intrarater reliability was observed for Community mobility (ICC = 0.93). There was good to excellent agreement for interrater reliability of the FAM. The FAM could differentiate between subgroups of patients based on the level of injury but not for the time elapsed from the injury. ConclusionsBased on the results of this study, the Iranian version of the FAM (FAM-Ir) was highly valid and reliable for evaluation of functional ability in patients with SCI. Therefore, we would suggest that the FAM could also be used as an assessment tool for SCI patients.

The Quantification of Violence Scale: a Simple Method of Recording Significant Violence

Although there are many rating scales recording the incidence and intensity of violence there are none that are specifically concerned with the measurement and assessment of severe violence. To develop a scale sensitive to variation centred on severe violence, establish its normative values, test its feasibility, and assess its reliability and validity in different populations. The Quantification of Violence Scale (QOVS) was developed in two stages. First, a list of 30 commonly experienced violent episodes in clinical psychiatric practice were evaluated and tested by weighting each episode by severity. Second, a numerical scale used to record the severity of the episode according to its degree of planning, intent and consequences. Violent episodes in two clinical populations were compared using the Modified Overt Aggression Scale (MOAS) and the preliminary version of the QOVS over periods up to 18 months, following which the numerical scale was developed. Good (0.60-0.74) to Excellent (> 0.75) test-retest and inter-rater reliability agreement was obtained with both forms of the scale (intra-class correlations of 0.75 and 0.69 respectively), and similar agreement with MOAS scores was reached (0.67) in clinical populations. The scale was quick and easy to use in practice, and a score defining severe violence (9 on the numerical scale and 16 on the matched scale) was determined. The QOVS, in its two forms, is a useful measure of recording significant violence in clinical and forensic practice.

Validation and reliability of the Modified Elderly Mobility Scale

Objective: To establish concurrent validity, interrater and test–retest reliability of the Modified Elderly Mobility Scale (MEMS). Methods: Ninety elderly patients were scored on the MEMS. To establish concurrent validity, 75 patients MEMS scores were compared to Functional Independence Measure (FIM) scores using Spearman's correlation. Videotaped patient performances were used to establish interrater and test–retest reliability using percentage absolute agreement and intraclass correlation coefficients (ICCs). Results: The total MEMS score demonstrated a significant association with the motor (r = 0.725) and total FIM scores (r = 0.718). Absolute agreement for interrater reliability was greater than 93% for all test items, with 97 and 98% for the two new measures, respectively. Test–retest reliability demonstrated similar high levels of absolute agreement and had ICCs ranging from 0.870 to 1.0. Conclusions: The MEMS is a quick, valid and reliable test of motor function of elderly patients with a spread of functional levels.