Inter-rater Reliability Analysis Research Articles

Using FDA reports to inform a classification for health information technology safety problems

To expand an emerging classification for problems with health information technology (HIT) using reports submitted to the US Food and Drug Administration Manufacturer and User Facility Device Experience (MAUDE) database. HIT events submitted to MAUDE were retrieved using a standardized search strategy. Using an emerging classification with 32 categories of HIT problems, a subset of relevant events were iteratively analyzed to identify new categories. Two coders then independently classified the remaining events into one or more categories. Free-text descriptions were analyzed to identify the consequences of events. Descriptive statistics by number of reported problems per category and by consequence; inter-rater reliability analysis using the κ statistic for the major categories and consequences. A search of 899 768 reports from January 2008 to July 2010 yielded 1100 reports about HIT. After removing duplicate and unrelated reports, 678 reports describing 436 events remained. The authors identified four new categories to describe problems with software functionality, system configuration, interface with devices, and network configuration; the authors' classification with 32 categories of HIT problems was expanded by the addition of these four categories. Examination of the 436 events revealed 712 problems, 96% were machine-related, and 4% were problems at the human-computer interface. Almost half (46%) of the events related to hazardous circumstances. Of the 46 events (11%) associated with patient harm, four deaths were linked to HIT problems (0.9% of 436 events). Only 0.1% of the MAUDE reports searched were related to HIT. Nevertheless, Food and Drug Administration reports did prove to be a useful new source of information about the nature of software problems and their safety implications with potential to inform strategies for safe design and implementation.

Content validity and inter-rater reliability of a checklist to assess the ergonomic practice of computer professionals

Objectives: The objective of this study is to establish the Content validity and inter-rater reliability of an instrument (Checklist) to assess ergonomic practice of Computer professionals using a rigorous Judgment-quantification process and Intra-class correlation coefficient respectively. Methods and Measures: The Draft Checklist composed of 43 items related to Working Postures, Seating, Keyboard/Mouse, Monitor, Table and Accessories and finally Rest breaks and Exercises. A panel of 9 experts validated the Draft Ergonomic Practice Checklist. The Content Validity Index (CVI) was calculated after receiving all correspondence regarding content validity for each item of the checklist. Finally, a Focus group was held to evaluate the instrument for overall comprehensiveness. A total of 20 computer professionals both males and females of mean age of 33.8 years participated in the inter-rater reliability study. The raters of this study were 2 Physiotherapy faculties with mean clinical and teaching experience of 7.5 years. Both the raters assessed the ergonomic practice of the computer professionals using the checklist. Observations were recorded independently and not discussed among the raters. Analysis: The Content Validity Index (CVI) was calculated by tallying the results of the experts based on the degree to which the experts agree on the relevance and clarity of the items. Intraclass correlation coefficient was used to assess the inter-rater reliability of the ergonomic practice checklist based on the observations recorded independently by the raters. Results: Results from the panel of experts yielded a 0.98 overall Content validity index. Inter-rater reliability analysis revealed that r value was above 0.87 for all the sections of the checklist thus being in the excellent range. Conclusion: The process used to determine Content validity and the inter-rater reliability proved to offer structure and consistency to the instrument development. High CVI scores were generated for the items judged relevant to the content domain as well as for the overall instrument. The checklist demonstrated an excellent reliability. The results support the Content validity and the inter-rater reliability of this Checklist as a tool to assess the Ergonomic Practice of Computer Professionals.

Qualitative evaluation of a positive youth development course in a university setting in Hong Kong

Qualitative evaluation was carried out to understand the perceptions of the students taking a course entitled “Tomorrow’s Leaders” at The Hong Kong Polytechnic University. Based on the reflections of 189 students, results showed that most informants used positive descriptors to describe the course. When the informants were invited to name a metaphor that could stand for the course, the related metaphors were basically positive in nature. Intra- and inter-rater reliability analyses revealed that the coding of the positive or negative nature of the responses was reliable. The present study provides qualitative evaluation findings supporting the effectiveness of this course in promoting holistic development in Chinese university students in Hong Kong.

Reliability of Scapular Classification in Examination of Professional Baseball Players

Clinically evaluating the scapulothoracic joint is challenging. To identify scapular dyskinesis, clinicians typically observe scapular motion and congruence during self-directed upper extremity movements. However, it is unclear whether this method is reliable. We therefore determined the interrater reliability of a scapular classification system in the examination of professional baseball players. Seventy-one healthy uninjured professional baseball players between the ages of 18 and 32 years volunteered to participate. We used a digital video camera to film five repetitions of scapular plane elevation while holding a 2-pound weight. Four examiners then independently classified the motions on video into one of four types. Interrater reliability analysis using the kappa (k) statistic was performed for: (1) classifying each scapula into one of the four types; (2) classifying each scapula as being abnormal (Types I-III) or normal (Type IV); and (3) classifying both scapula as both being symmetric (both normal or both abnormal) or asymmetric (one normal, one abnormal). We found low reliability for all analyses. In classifying each scapula as one of the four types, reliability was k = 0.245 for the left limb and k = 0.186 for the right limb. When considering the dichotomous classifications (abnormal versus normal), reliability was k = 0.264 for left and k = 0.157 for right. For bilateral symmetry/asymmetric, reliability was k = 0.084. We found low reliability of visual observation and classification of scapular movement. Current evaluation strategies for evaluating subtle scapular abnormalities are limited. Level III, diagnostic study. See Guidelines for Authors for a complete description of levels of evidence.

Comparison of the planimetry and point-counting methods for the assessment of the size of the mandible cysts on orthopantomograms

Objective: The purpose of this study is to compare the computer-assisted planimetry and point-counting methods in evaluating the sizes of the mandibular cysts with respect to their agreement and the time required to analyze. Study Design: The surface areas of 46 mandibular cyst lesions on orthopantomograms were estimated using the point-counting and computer-assisted planimetry methods. Three observers evaluated the outlined areas twice, using the point-counting (PC) and computer-assisted planimetry (CAP) methods with an interval of two weeks. In the planimetry technique, digitalized images and ImageJ software were used to measure the surface area of the half mandibles and cysts. The grids were superimposed over the same images and the number of points hitting the interested structures was counted for the point-counting technique. The projection area fraction (PAF) of the cysts within the mandible was estimated using the obtained values by means of the two techniques. Intraclass correlation coefficient was used to assess the level of agreement between the two methods. Inter-rater reliability analysis using the Kappa statistic was performed to determine consistency among raters. Results: CAP and PC techniques showed consistent intra-observer values in all observers. Intraclass correlation between CAP and PC measurements of first, second and third observers were found to be 0.9986, 0.9988 and 0.9994 respectively. The durations of PC technique was 32% higher than the CAP technique. Conclusion: PC and CAP methods were seemed to show sufficient agreement to be used interchangeably. The main disadvantage of the PC analysis is it takes more time than CAP method. Key words:Orthopantomogram, mandible, cyst, point-counting, planimetry.

Prevalence of the metabolic syndrome in rural India—a disparity in definitions

Although the metabolic syndrome is a known predictor of coronary heart disease and type-2 diabetes mellitus, it has no agreed definition. The concordance of its various definitions has been studied for a limited number of populations and there are few studies on the rural populations in India. The present study was done to determine the prevalence of metabolic syndrome in a rural population of South India and to evaluate the concordance of the modified National Cholesterol Education Programme-Adult Treatment Panel III definition and International Diabetes Federation definition for the diagnosis of Metabolic Syndrome. Anthropometric and biochemical parameters (fasting blood glucose and lipid profile) and blood pressure were measured using standard procedures. The prevalence of metabolic syndrome was calculated using the two sets of criteria and compared for their concordance. Descriptive statistics was used to analyze age and risk factors of metabolic syndrome. The chi-square test was applied to compare the prevalence of metabolic syndrome obtained from the two criteria (significant if p < 0.05). An inter-rater reliability analysis using the kappa statistic was performed to determine consistency between the two sets of criteria in diagnosing the metabolic syndrome. The modified National Cholesterol Education Programme-Adult Treatment Panel III definition and International Diabetes Federation definition for metabolic syndrome identified overall age-adjusted prevalence of 17.8% and 20.5% respectively, which were not significantly different. Kappa statistics revealed only moderate agreement of 0.44 between the two sets of criteria. The impact of economic development and preponderance of genetic factors is increasing the prevalence of metabolic syndrome in rural India. It is important to determine which definition of the metabolic syndrome best predicts coronary heart disease and type 2 diabetes in this population in order to formulate effective public health policy.

Convergence spasm in conversion disorders: prevalence in psychogenic and other movement disorders compared with controls

BackgroundConvergence spasm refers to transient ocular convergence, miosis and accommodation associated with disconjugate gaze mimicking abducens palsy. While it may be a manifestation of brainstem pathology, this sign is often...

Evaluation of Ability for Basic Movement Scale for Children Type T (ABMS-CT) for disabled children

BackgroundThe objective of this pilot study was to test the validity and reliability of a new scale, the Ability for Basic Movement Scale for Children Type T (ABMS-CT). MethodsForty-nine pediatric patients with disabilities (aged 1.00–15.17years; 29 males and 20 females) participated in this prospective study. To prove the validity and reliability of the ABMS-CT, subjects were administered the ABMS-CT by two physicians. In addition to the ABMS-CT score, data on age, diagnosis, and results of the Functional Independence Measure for Children (WeeFIM) were recorded. ResultsSpearman’s rank correlation coefficient analysis showed that the ability to perform basic movements according to the individual scores for each item on the ABMS-CT and the total scores of the ABMS-CT correlated significantly with the total scores of the motor and cognitive WeeFIM, respectively (r=0.753–0.892, p=0.0001). The five items on the ABMS-CT had appropriate internal consistency (Cronbach’s α=0.966). Inter-rater reliability analysis indicated that the “oral and facial area”, “hands and fingers”, “one leg”, “both legs”, and “stairs” items on the ABMS-CT had almost perfect reliability (κ=0.854–0.925). ConclusionThis study provides evidence for the validity and reliability of the ABMS-CT with regard to assessment of the functional ability for complex movements in disabled pediatric patients even if they can walk independently.

Contrast-Enhanced Sonographic Characterization of the Vascularity of the Repaired Rotator Cuff

To characterize the distribution of vascularity of the postoperative rotator cuff tendon using a maximum intensity projection technique after contrast-enhanced sonography. We retrospectively evaluated image data on 23 patients (11 male and 12 female) with intact rotator cuff repairs who had previously undergone contrast-enhanced sonography of their shoulders using lipid microspheres before and after a standardized exercise protocol. The patients were on average 3 months out from their surgery. Using offline image analysis software, a maximum intensity projection image was obtained for each patient, reflecting the regional vascular distribution within the repair and adjacent soft tissue. Subjective analysis was performed in 4 regions of interest: peribursal, articular medial, articular lateral, and suture anchor, independently by 2 musculoskeletal radiologists using a semiquantitative scale ranging from 0 to 4 for each region (0, no enhancement; 1, 1%-25% enhancement; 2, 26%-50%; 3, 51%-75%; and 4, 76%-100%). A combined vascularity score (0-8) was produced for each region and formed the basis for the subjective analysis. Using a Mann-Whitney nonparametric test, the data showed significantly higher regional enhancement in the peribursal and suture anchor regions compared to the tendon (P < .001). Exercise resulted in a statistically significant increase in the extent of enhancement in all regions (P < .002). Inter-rater reliability analysis using a weighted κ statistic showed strong agreement (0.63-0.64) for the suture anchor site and moderate agreement for the others (peribursal, 0.35-0.39; articular medial, 0.45-0.55; and articular lateral 0.32-0.33). The maximum intensity projection technique after contrast-enhanced sonography provides a topographic map of rotator cuff vascularity; the latter has been implicated as an important factor in promoting bone-tendon healing. Approximately 3 months after rotator cuff repair, the suture anchor and peribursal regions showed the most robust vascularity. Maximum intensity projection imaging further establishes that there is a global increase in vascular response at the repair site after exercise.

Ego Identity Status of Medical Students in Clerkship

Background: Medical students encounter a variety of experiences that have an impact on their emerging professional identity. Clerkship, in particular, presents opportunities for students to consider their career options and decide upon a career path. The process of developing their professional identity begins well before clerkship, however. Anecdotal evidence suggests that interests in medicine begin as early as childhood. This study retrospectively examines the decision-making process clerks make in choosing medicine as a career.Methods: A total of 76 clerks (36 male, 34 female, 6 not reported) responded to four open-ended and two follow-up questions that measure career interests and pursuits. Questions addressed when and how students developed interests in medicine and alternate careers before beginning medical school. An additional eight closed questions drawn from the Ego Status Extended Objective Measure of Ego Identity Status II (EOM-EIS-II) were administered. Content analyses and inter-rater reliability analyses were conducted to classify students according to Marcia’s1 four ego identity statuses.Results: Having obtained high inter-rater consistency (Cohen’s Kappa coefficient of 0.92), responses to the open-ended questions resulted in the classification of three identity statuses. In total, 49.3% of students were in the ‘achieved’ (high exploration and commitment to choices) status and 48.1% were in the ‘foreclosed’ (low exploration but high commitment to choices) status. A small percentage (1.3%) of students were in the ‘moratorium’category (high exploration but low commitment to choices), while none of the students were in the ‘diffused’ (low exploration and low commitment to choices) category.Conclusions: With approximately half of the students demonstrating a ‘foreclosed’ status, this study reveals that despite exposure to a variety of careers when attending university, only half of the students had seriously pursued a career outside of medicine. The majority of students, moreover, developed an interest in medicine before adulthood, and did so independently from parental influence.

Breast density: Computerized analysis on digitized mammograms

Mammographically dense breast tissue is related to a higher risk of breast cancer. We aim to evaluate a computerized system, assess whether it can provide an accurate and objective estimation of the breast density and if it can accurately classify the mammograms according to the ACR/BIRADS system. We retrospectively reviewed the mediolateral oblique (MLO) and cranial-caudal (CC) views of 83 normal mammograms and classified them, both manually and with the use of computerized texture analysis (CTA), according to their density. We grouped the mammograms either into two (ACR 1–2, ACR 3–4) or four categories (ACR 1 to 4). An inter-rater reliability analysis was performed using the kappa statistic to determine consistency among the radiologist and the CTA. The best matching was observed for the MLO view when the classification involved 2 groups (94%). The equivalent matching for the CC view was 92.8%. When we used all 4 ACR categories the matching was lower: i.e. 84.3% for the MLO view and 79.5% for the CC view. For older patients (>50 years old) the best matching was for the MLO views while for the younger patients equal matching was observed for both views. Overall, substantial to almost perfect agreement was observed between the two methods of assessment. CTA is a reliable and accurate form of computerized assisted diagnosis. If a single view is to be used, it should be the MLO view since the addition of CC view does not seem improve the sensitivity of the method.

Myositis disease activity tool in Chinese patients with polymyositis/dermatomyositis

To test the reliability, discriminating validity and clinical value of myositis disease activity tool in Chinese patients with polymyositis/dermatomyositis (PM/DM). Fifty-four PM/DM patients (PM, n = 18; DM, n = 36) at our hospital between January 2009 and June 2010 were enrolled into this prospective study. The disease activity was assessed by the MDAAT (myositis disease activity assessment tool). The reliability and discriminating validity were assessed with test-retest reliability, interrater reliability and t-test analysis. The reliability analysis of MDAAT indicated 8 items (constitutional, cutaneous, skeletal, gastrointestinal, pulmonary, cardiovascular, muscular & global) had excellent reliability, the ICC (intraclass correlation coefficient) was 0.72 (95%CI: 0.57, 0.83), 0.88 (95%CI: 0.79, 0.93), 0.83 (95%CI: 0.72, 0.90), 0.62 (95%CI: 0.42, 0.76), 0.81 (95%CI: 0.70, 0.89), 0.59 (95%CI: 0.38, 0.74), 0.85 (95%CI: 0.75, 0.91) and 0.84 (95%CI: 0.73, 0.90) respectively. MDAAT had excellent discrimination validity in all items except for cardiovascular ones. With excellent reliability and discriminating validity, MDAAT may be used to assess the disease activity and therapeutic effects of Chinese PM/DM patients.

Measurement of Activated Coagulation Time in Children: Evaluation of the Blood-Saving Kaolin i-STAT Activated Coagulation Time Technique in Pediatric Cardiac Anesthesia

To compare the activated coagulation times (ACTs) measured with the blood-saving kaolin i-STAT 1 ACT technique (Abbott Point of Care Inc, Princeton, NJ) with ACTs obtained from the widely used ACTR II device (Medtronic, Inc, Minneapolis, MN) in children undergoing cardiac surgery. A prospective, observational single-center study. Forty-four pediatric cardiac surgery patients. Surgery was performed with cardiopulmonary bypass (CPB) necessitating heparinization. ACTs measured on the i-STAT 1 device (2 × 95 μL) were compared with those obtained from the Medtronic ACTR II device (2 × 0.5 mL). Blood samples were drawn before, during, and after heparinization for CPB and paired for statistical analysis. The 2 techniques were compared using simple and multiregression analyses and the Bland-Altman method. In total, 179 intrarater and 142 interrater data pairs were analyzed. The intrarater reliability of the 2 devices was good, with a mean bias and limits of agreement of +2.0 and -55.5/+59.5 seconds for the Medtronic ACTR II and +0.5 and -59.9/+60.9 seconds for the i-STAT 1. An interrater reliability analysis of the mean of simultaneously measured ACT of the Medtronic ACTR II and both i-STAT 1 devices yielded a mean bias of -5.3 seconds and limits of agreement of -210.1/+199.5 seconds. A comparison of the higher of the paired ACT values from both devices showed similar results. After the removal of heparin, the i-STAT 1's ACT values became significantly lower than those measured on the Medtronic ACTR II (p < 0.001). Simple and multiregression analyses revealed that base excess independently influenced the mean bias of the ACT values from the Medtronic ACTR II (p = 0.037) and i-STAT 1 devices (p = 0.036). The kaolin i-STAT 1 ACT technique agreed well with the Medtronic ACTR II technique during the nonheparinized phase that preceded CPB. The overall agreement between the ACT obtained from the 2 devices was poor. The routine use of i-STAT 1 measured ACT values cannot be recommended as a reliable alternative to the Medtronic ACTR II.

Interrater Reliability and Time Measurement Validity of Speed–Agility Field Tests in Adolescents

The aim of this study was to examine the interrater reliability (trained vs. untrained raters) and criterion-related validity (manual vs. automatic timing) of the 4 × 10-m shuttle run and 30-m running speed tests (times measured). The study comprised 85 adolescents (38 girls) aged 13.0-16.9 years from the Healthy Lifestyle in Europe by Nutrition in Adolescence study. The time required to complete the 4 × 10-m shuttle run and 30-m running tests was simultaneously measured (a) manually with a stopwatch by both trained and untrained raters (for interrater reliability analysis), and (b) by using photoelectric cells (for validity analysis). Systematic error, random error, and heteroscedasticity were studied with repeated-measured analysis of variance and Bland-Altman plots. The systematic error for untrained vs. trained raters and the untrained raters vs. photoelectric cells were in all cases ∼0.1 seconds (p < 0.01), that is, untrained raters recorded higher times. No systematic error was found between trained raters and photoelectric cells (p > 0.05). No heteroscedasticity was shown in any case (p > 0.05). The findings indicate that manual measurements by a trained rater, using a stopwatch, seem to be a valid method to assess speed and agility fitness testing in adolescents. Researchers must be trained to minimize the measurement error.

Local adaptation and evaluation of a falls risk prevention approach in acute hospitals

To determine whether locally adapting a falls risk factor assessment tool results in an instrument with clinimetric properties sufficient to support an acute hospital's falls prevention program. Prospective cohort study of predictive validity and observational investigation of intra- and inter-rater reliability. Acute wards in two large hospitals in Melbourne, Australia. One hundred and thirty acute hospital inpatients participated in the predictive accuracy evaluation, with 25 and 35 inpatients used for the intra-rater and inter-rater reliability analyses, respectively. To develop a falls risk screen and assessment instrument through local adaptation of an existing tool. Clinimetric property analysis of new instrument (Western Health Falls Risk Assessment, WHeFRA) and comparison with 'gold standard tool' (STRATIFY). Fallers, falls and falls per 1000 bed days. Sensitivity (Sens), specificity (Spec), Youden Index (YI) and these three statistics based on event rate of falls (Sens(ER), Spec(ER) and YI(ER)), were calculated to determine predictive accuracy. Reliability was determined using intraclass correlation coefficient (ICC), weighted kappa and signed rank test. Seven participants (5.4%) fell, with 14 falls (fall rate: 10.7 falls per 1000 patient bed days). The WHeFRA instrument was significantly more accurate at predicting fallers and the rate of falls than the STRATIFY. Intra-rater reliability ICC (95% confidence intervals) for WHeFRA screen was 0.94 (0.86-0.97) and inter-rater reliability was 0.78 (0.61-0.88). Local adaptation of an existing tool resulted in an instrument with favorable clinimetric properties and may be a viable procedure for facilitating falls prevention program development and implementation in acute hospital settings.

Construction and Validation of an Observational Scale of Neighborhood Characteristics

This paper reports the development and validation of the Neighborhood Observation Scale, a 41 item measure of neighborhood physical appearance, social appearance, safety, and amenities. Three independent ratings were collected on each of 244 neighborhoods in 132 census block groups in five South Carolina counties, for a total of 732 observations. Inter-rater reliability analysis showed 92.6% agreement among at least two raters and an intraclass correlation of .54, indicating acceptable reliability. Exploratory factor analysis extracted eight factors for neighborhood physical appearance, school/park/public space physical appearance, neighborhood social appearance, indicated social engagement, observed social engagement, park/public space social engagement, resident watchfulness, and neighborhood safety risk. A follow-up confirmatory factor analysis supported the eight factor structure. Reliability analysis produced alpha coefficients ranging from .55 to .94 for the eight factors, indicating reliability for use as sub-scales. Convergent validity was determined by generally acceptable correlations of the factor sub-scales with measures of neighborhood distress, rates of child maltreatment and ICD-9-CM coded child injuries, and resident perceptions of neighboring, collective efficacy, reciprocal helping, and children’s safety in the home. Implications for research and policy and program development are discussed.

A Meta‐Analysis of the relationship between the Rorschach Ego Impairment Index (EII) and psychiatric severity

The present study examined the relationship between the Rorschach Ego Impairment Index (EII) and psychiatric severity. Search procedures yielded 13 independent samples (total N = 1402, average n = 108, standard deviation = 90) for inclusion in the meta-analysis. Inter-rater reliability analyses demonstrated that coding of effect sizes and moderator variables was completed with good to excellent reliability. Results indicated that higher EII scores were associated with greater psychiatric severity, with an overall weighted effect size of r = 0.29, p = 0.000002 (95% confidence interval = 0.17-0.40), supporting the EII's validity as a measure of psychological impairment. Publication bias analyses did not indicate any significant cause for concern regarding the results. The data were demonstrably heterogeneous (Q = 56.82, p = 0.0000001), and results of post-hoc tests indicated that effect sizes with dependent variables obtained via researcher ratings were significantly larger than any of the following: effect sizes with dependent variables obtained via clinician ratings, informant ratings, information about level of treatment or placement status or self-report ratings (p's = 0.0005, 0.003, <0.001, <0.001, respectively). In addition, there was a trend for effect sizes based on performance-based measures to be larger than those based on information about level of treatment or placement status (p = 0.098) as well as those based on self-report measures (p = 0.076). Other moderator analyses were non-significant (p 's > 0.10). Copyright © 2010 John Wiley & Sons, Ltd. Key Practitioner Message: • The Rorschach Ego Impairment Index (EII) demonstrated validity in measuring psychiatric severity across a range of normative, outpatient, residential, and inpatient samples. • The degree of the EII's validity in assessing psychiatric severity compared favorably to the overall validity of the Rorschach and the MMPI. • The EII appears to be most valid in capturing psychiatric severity as measured by researcher ratings of social competency or estimated ego impairment.

A Study on the Writing Performance Assessment of Inter-rater Reliability Analysis

A Study on the Writing Performance Assessment of Inter-rater Reliability Analysis

Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis

Background Many Electronic Health Record (EHR) systems fail to provide user-friendly interfaces due to the lack of systematic consideration of human-centered computing issues. Such interfaces can be improved to provide easy to use, easy to learn, and error-resistant EHR systems to the users. Objective To evaluate the usability of an EHR system and suggest areas of improvement in the user interface. Methods The user interface of the AHLTA (Armed Forces Health Longitudinal Technology Application) was analyzed using the Cognitive Task Analysis (CTA) method called GOMS (Goals, Operators, Methods, and Selection rules) and an associated technique called KLM (Keystroke Level Model). The GOMS method was used to evaluate the AHLTA user interface by classifying each step of a given task into Mental (Internal) or Physical (External) operators. This analysis was performed by two analysts independently and the inter-rater reliability was computed to verify the reliability of the GOMS method. Further evaluation was performed using KLM to estimate the execution time required to perform the given task through application of its standard set of operators. Results The results are based on the analysis of 14 prototypical tasks performed by AHLTA users. The results show that on average a user needs to go through 106 steps to complete a task. To perform all 14 tasks, they would spend about 22 min (independent of system response time) for data entry, of which 11 min are spent on more effortful mental operators. The inter-rater reliability analysis performed for all 14 tasks was 0.8 (kappa), indicating good reliability of the method. Conclusion This paper empirically reveals and identifies the following finding related to the performance of AHLTA: (1) large number of average total steps to complete common tasks, (2) high average execution time and (3) large percentage of mental operators. The user interface can be improved by reducing (a) the total number of steps and (b) the percentage of mental effort, required for the tasks.

VERITAS‐Pro: a new measure of adherence to prophylactic regimens in haemophilia

Prophylactic treatment is recommended for severe haemophilia. Non-adherence to a prophylactic regimen can limit treatment effectiveness and compromise outcomes. The aim of this study is to validate a new prophylactic treatment adherence scale entitled Validated Hemophilia Regimen Treatment Adherence Scale--Prophylaxis (VERITAS-Pro), a self-/parent-report questionnaire consisting of 24 questions on six (four-item) subscales (Time, Dose, Plan, Remember, Skip, Communicate) that takes approximately 10 min to complete and is currently available in English only. Participants were recruited to complete the VERITAS-Pro for validation and reliability analysis; and observers were recruited for inter-rater reliability analysis. Validation measures included subjective adherence ratings from participants and providers and the total number of recommended infusions administered as obtained from infusion logs. Data were evaluated for the entire sample and for parent-report and self-report subsamples. The study sample included 67 males, 53 (79.1%) diagnosed with severe FVIII deficiency. Internal consistency for the total VERITAS-Pro score and all subscales was good to excellent; test-retest reliability correlations were very strong. Validation measures were strongly correlated with VERITAS-Pro scores. The VERITAS-Pro is a reliable and valid measure of adherence to prophylactic treatment of haemophilia. The VERITAS-Pro has greater utility than a global or informal rating of adherence because it represents a quantified and validated measure of adherence from the patient's perspective and it divides adherence into specific areas, allowing insight into particular issues underlying non-adherence. This tool may increase sensitivity to adherence problems and allow more targeted interventions to enhance adherence.