High Levels Of Inter-rater Reliability Research Articles

LATINO HOSPICE PATIENTS’ INVOLVEMENT IN CARE DECISIONS AND PLANNING: FACTORS THAT SHAPE PARTICIPATION PREFERENCES

Patient participation in care decisions is a primary tenet of patient-centered care and has been emphasized in policies and programs shaping clinical practice. Patient self-determination and involvement is associated with improved outcomes, including patient satisfaction and compliance. However, studies exploring Latino patients’ care involvement preferences find a cultural preference for limited autonomy and shared or family-based decision-making. This study describes terminally-ill older Latinos and their proxies’ preferences regarding involvement in decision-making and care planning following a referral to hospice care. Semi-structured interviews were conducted with 13 hospice-enrolled Latinos 65 or older, or their proxies. The sample is predominantly female (54% of all patients; 80% of proxies) and Mexican (77%). The average age of all patients was 82 years, and of proxies 54. Patients interviewed had an average of 7 years of education and proxies an average of 11 years. Patients were receiving care in a variety of settings, including inpatient hospice units, nursing homes, and community. Interviews were transcribed verbatim in the original language and coded thematically using an inductive approach. A second coder was used to code a Spanish and an English transcript (15% of the data), and a high level of interrater reliability was obtained (k = 0.85). Findings elucidate the paths through which culture, religion, and trust in provider shape preferences for limited autonomy and planning, confirming previous findings. However, this was not replaced by a preference for family-based decision-making. Findings highlight the importance of Latinos having access to culturally-competent primary care providers and fostering lasting relationships.

A Two‐Stage Method for Classroom Assessments of Essay Writing

AbstractTo capitalize on professional expertise in educational assessment, it is desirable to develop and test methods of rater‐mediated assessment that enable classroom teachers to make reliable and informative judgments. Accordingly, this article investigates the reliability of a two‐stage method used by classroom teachers to assess primary school students’ persuasive writing. Stage 1 involves pairwise comparisons and stage 2 involves rating against calibrated exemplars from stage 1 plus performance descriptors. A high level of interrater reliability among teachers was obtained. This is consistent with previous evidence that the two‐stage method is a viable classroom assessment method without extensive training and moderation. Implications for assessment practices in education are discussed with a focus on the widely expressed desire to value professional expertise.

Reliability and diagnostic accuracy of cervicothoracic differentiation testing and regional unloading for identifying improvement after thoracic manipulation in individuals with neck pain

Background & purposeThe cervicothoracic differentiation test (CTDT), cervical and thoracic unloading are used clinically to guide treatment. This study sought to determine the reliability and diagnostic accuracy of these tests. MethodsA prospective diagnostic accuracy study was performed at two outpatient clinics and one university research center. A convenience sample of 48 individuals with neck pain was recruited. Cervical and thoracic unloading tests and CTDT were performed with symptom relief considered a positive test. Pain was assessed using a visual analog pain scale (VAS) at rest and during provocative movements. The reference standard was pain relief following thoracic manipulation. Change in pain was used to identify improvement at the MCID (15 mm) and 50% improvement thresholds. ResultsAll three tests demonstrated high levels of inter-rater reliability, K = 0.90[0.77–1.00]. Of 48 individuals who completed the study, 39 (81.3%) were improved ≥ MCID; compared to 34 (70.8%) at the 50% threshold. As a single test, the CTDT yielded the strongest diagnostic utility (at MCID threshold) based on ROC curve: AUC 0.791 s.e. 0.078; with high specificity (0.89[51.75–99.72]); LR+ 6.23 [0.97–40]; LR- 0.35 [0.20–0.58]; and PPV 96.43. Unloading tests demonstrated high sensitivity, but poor specificity and likelihood ratios. Composite tests improved specificity, but with lower accuracy and minimal changes in ROC area compared to the CTDT in isolation. ConclusionsThe CTDT is a specific test with significant diagnostic utility to identify individuals who will experience immediate pain relief following thoracic manipulation. The CTDT should be considered during the clinical decision making process when treating individuals with neck pain.

Lessons learned from measuring fidelity with the Motivational Interviewing Treatment Integrity code (MITI 4)

Background and aimsThe Motivational Interviewing Treatment Integrity code (MITI) measures fidelity to, and the quality of, Motivational Interviewing (MI), and can also be used when MI is combined with other treatment methods. The current study presents a fidelity measurement with the MITI 4.2.1, in both Motivational Enhancement Therapy sessions and the combined Community Reinforcement Approach-Senior (CRAS). MethodThe MITI 4.2.1 was used to evaluate treatment sessions provided in the Elderly Study, a multi-national randomized trial evaluating treatment for alcohol use disorders in the elderly. Following expert recommendations, training was conducted at two international sites as well as at the Danish site. Twenty percent of the sessions at the Danish study site were rated. Twelve percent were multiply rated by all raters. Interrater reliability was assessed by the Intraclass Correlations Coefficient (ICC). ResultsMean ICC of the 52 sessions rated by all raters was 0.78 (95% CI: 0.70; 0.86). The rare measures confront and emphasize autonomy, and the global measure softening sustain talk only reached fair levels of ICC, while the remaining measures were good or excellent. In the sessions of MI combined with other treatment approaches in the CRAS, the MITI 4.2.1 has a similar reliability as in MET sessions only, except for the measure persuade with permission. ConclusionThe MITI 4.2.1 is a reliable instrument for measuring fidelity to Motivational Interviewing elements, also in the context of Community Reinforcement Approach Senior. However, in softening sustain talk, the rare measures, and persuade with permission it has proved more difficult to reach high levels of interrater reliability.

The QUEST for quality online health information: validation of a short quantitative tool

BackgroundOnline health information is unregulated and can be of highly variable quality. There is currently no singular quantitative tool that has undergone a validation process, can be used for a broad range of health information, and strikes a balance between ease of use, concision and comprehensiveness. To address this gap, we developed the QUality Evaluation Scoring Tool (QUEST). Here we report on the analysis of the reliability and validity of the QUEST in assessing the quality of online health information.MethodsThe QUEST and three existing tools designed to measure the quality of online health information were applied to two randomized samples of articles containing information about the treatment (n = 16) and prevention (n = 29) of Alzheimer disease as a sample health condition. Inter-rater reliability was assessed using a weighted Cohen’s kappa (κ) for each item of the QUEST. To compare the quality scores generated by each pair of tools, convergent validity was measured using Kendall’s tau (τ) ranked correlation.ResultsThe QUEST demonstrated high levels of inter-rater reliability for the seven quality items included in the tool (κ ranging from 0.7387 to 1.0, P < .05). The tool was also found to demonstrate high convergent validity. For both treatment- and prevention-related articles, all six pairs of tests exhibited a strong correlation between the tools (τ ranging from 0.41 to 0.65, P < .05).ConclusionsOur findings support the QUEST as a reliable and valid tool to evaluate online articles about health. Results provide evidence that the QUEST integrates the strengths of existing tools and evaluates quality with equal efficacy using a concise, seven-item questionnaire. The QUEST can serve as a rapid, effective, and accessible method of appraising the quality of online health information for researchers and clinicians alike.

The Usability of Unmanned Aerial Vehicles (UAVs) for Pedestrian Observation

The monitoring of pedestrian activity is challenging, primarily because its traffic levels are typically lower and more variable than those of motorized vehicles. Compared with other on-the-ground observation tools, unmanned aerial vehicles (UAVs) could be suitable for counting and mapping pedestrians in a reliable and efficient way. Thus, this study establishes and tests a new method of pedestrian observation using UAVs. The results show that UAV observations demonstrate high levels of interrater reliability (intraclass correlation coefficient = 0.99) and equivalence reliability (Cronbach’s α = .97 with on-the-ground counts and .73 with Google Street View). Practical implications of the new tool are discussed.

Ultrasound measures of muscle thickness may be superior to strength testing in adults with knee osteoarthritis: a cross-sectional study

BackgroundEvaluation of muscle strength as performed routinely with a dynamometer may be limited by important factors such as pain during muscle contraction. Few studies have compared formal strength testing with ultrasound to measure muscle bulk in adults with knee osteoarthritis (OA).MethodsWe investigated the muscle bulk of lower limb muscles in adults with knee OA using quantitative ultrasound. We analyzed the relationship between patient reported function and the muscle bulk of hip adductors, hip abductors, knee extensors and ankle plantarflexors. We further correlated muscle bulk measures with joint torques calculated with a hand held dynamometer. We hypothesized that ultrasound muscle bulk would have high levels of interrater reliability and correlate more strongly with pain and function than strength measured by a dynamometer. 23 subjects with unilateral symptomatic knee OA completed baseline questionnaires including the Western Ontario and McMaster Universities Arthritis Index (WOMAC) and Lower Extremity Activity Scale. Joint torque was measured with a dynamometer and muscle bulk was assessed with ultrasound.ResultsHigher ultrasound measured muscle bulk was correlated with less pain in all muscle groups. When comparing muscle bulk and torque measures, ultrasound-measured muscle bulk of the quadriceps was more strongly correlated with measures of pain and function than quadriceps isometric strength measured with a dynamometer.ConclusionsUltrasound is a feasible method to assess muscle bulk of lower limb muscles in adults with knee OA, with high levels of interrater reliability, and correlates negatively with patient reported function. Compared with use of a hand held dynamometer to measure muscle function, ultrasound may be a superior modality.

Training Coding Specialists for the Future: Methods and Materials for the Beta Version of ICD-11

IntroductionIn June 2018, the World Health Organization (WHO) will release the 11th Version of International Classification of Diseases (ICD-11). New training methods and materials are required. As a WHO Collaborating Center, with Canadian Institute for Health Information (CIHI) members, we trained 6 coding professionals for testing ICD-11 coding processes.&#x0D; Objectives and ApproachThe objective was to achieve a high level of inter-rater reliability using ICD-11 for acute care chart coding. We used Adult Learning principles with CIHI members and 6 certified coding specialists to co-create presentations, practice materials, and decision trees to teach knowledge and skill with ICD-11 tooling and content. Training involved 14 hours of interactive learning plus additional practice hours. A bank of questions and coding scenarios tested knowledge and application of ICD-11 terminology and principles. Coding was undertaken on a set of 3000 randomly selected inpatient Calgary hospital discharges as part of a large CIHR funded ICD-11 field trial.&#x0D; ResultsThe coding team achieved an average score of 84% on the ICD-11 coding quiz and 0.65 (0.33 -1.0) agreement on parent code of main condition for the coding quiz scenarios. 60 inpatient charts were coded by more than one coder to test inter-rater reliability. Agreement was ≧ 0.80 for the majority of parent codes for main condition. Coding differences may be due to unfamiliar code choices or training gaps. New code descriptions in ICD-11 enhance code selection. Challenges included training while codes were being built in the ICD-11 browser, and minimal coding rules or standards.&#x0D; Conclusion/ImplicationsRecommendations include more code descriptions in the browser and rules in a reference guide, teaching from simple to complex conditions, and multiple scenarios with ‘gold standard’ codes for practice. Reference Guide, Coding Tool, and Browser recommendations have been shared with members of the WHO Morbidity and Quality &amp; Safety Advisory groups.

It's not that Difficult: An Interrater Reliability Study of the DSM–5 Section III Alternative Model for Personality Disorders

ABSTRACTThe Diagnostic and Statistical Manual of Mental Disorders (5th ed. [DSM–5]; American Psychiatric Association, 2013) Section III Alternative Model for Personality Disorders (AMPD) represents a novel approach to the diagnosis of personality disorder (PD). In this model, PD diagnosis requires evaluation of level of impairment in personality functioning (Criterion A) and characterization by pathological traits (Criterion B). Questions about clinical utility, complexity, and difficulty in learning and using the AMPD have been expressed in recent scholarly literature. We examined the learnability, interrater reliability, and clinical utility of the AMPD using a vignette methodology and graduate student raters. Results showed that student clinicians can learn Criterion A of the AMPD to a high level of interrater reliability and agreement with expert ratings. Interrater reliability of the 25 trait facets of the AMPD varied but showed overall acceptable levels of agreement. Examination of severity indexes of PD impairment showed the level of personality functioning (LPF) added information beyond that of global assessment of functioning (GAF). Clinical utility ratings were generally strong. The satisfactory interrater reliability of components of the AMPD indicates the model, including the LPF, is very learnable.

Using adaptive comparative judgment for student formative feedback and learning during a middle school design project

While design-based pedagogies have increasingly been emphasized, the assessment of design projects remains difficult due to the large number of potentially “correct” solutions. Adaptive comparative judgment (ACJ), an approach based on assessors/judges working through a series of paired comparisons and selecting the better of two items, has demonstrated high levels of inter-rater reliability with design projects. Efforts towards using ACJ for assessing design have largely centered on summative assessment. However, evidence suggests that ACJ may be a powerful tool for formative assessment and design learning when undertaken by students. Therefore, this study investigated middle school students participated in ACJ at the midpoint and conclusion of a design project, both receiving and providing feedback to/from their peers through the ACJ process. Findings demonstrated promise for using ACJ, as a formative assessment and feedback tool, to improve student learning and achievement.

Correlates of post‐traumatic growth following childhood and adolescent cancer: A systematic review and meta‐analysis

A growing number of children and adolescents are experiencing and surviving cancer. This review aims to identify the demographic, medical, and psychosocial correlates of perceived post-traumatic growth in individuals of any age who were affected by paediatric cancer. Findings will highlight protective factors that may facilitate post-traumatic growth, allowing for directed social support, intervention, and follow-up care. A systematic search based on the key concepts "post-traumatic growth," "neoplasms," and "paediatric" retrieved 905 records from online databases: Embase, Ovid MEDLINE, PILOTS: Published International Literature on Traumatic Stress, PsycINFO, and Web of Science. Eligible studies were appraised as excellent quality with a high level of interrater reliability. The results of 18 studies were synthesised. After the removal of outliers, post-traumatic growth shared small, negative associations with time since diagnosis (r=-0.14) and time since treatment completion (r=-0.19), and small, positive associations with age at diagnosis (r=0.20), age at survey (r=0.17), post-traumatic stress symptoms (r=0.11), and social support (r=0.25). Post-traumatic growth was positively and moderately associated with optimism (r=0.31). Several findings were consistent with a comparable meta-analysis in adult oncology populations. Targeted social support, clinical intervention, and education may facilitate post-traumatic growth. Longitudinal research in individuals affected by childhood and adolescent cancer would allow an examination of the effects of predictive variables on post-traumatic growth over time.

Grounding lexical diversity in human judgments

The present study discusses the relevance of measures of lexical diversity (LD) to the assessment of learner corpora. It also argues that existing measures of LD, many of which have become specialized for use with language corpora, are fundamentally measures of lexical repetition, are based on an etic perspective of language, and lack construct validity. The proposed solution draws from Zipf’s (1935) emic perspective of language, which views LD as a matter of perception, but which also assumes that competent speakers of a common language share similar perceptions. The present study tests whether this is true and specifically whether untrained human raters will show high levels of inter-rater reliability in their judgments of the levels of LD found in 60 texts extracted from a corpus of narratives written in English by a mix of language learners and native speakers. The results confirm Zipf’s assertion, but also indicate that a relatively large number of motivated raters are needed to demonstrate this tendency. The remainder of the study discusses the implications these results have for the development of an automated measure of LD to be used with learner corpora. The proposed method begins with human judgments of a representative subsample of a corpus, proceeds to a statistical model of objective measures that accurately predicts the human judgments, and ends with a multidimensional, corpus-specific automated measure that outputs reliable estimates of how a reliable group of human judges would rate the levels of LD in the texts of that corpus.

The usability of unmanned aerial vehicles (UAVs) for measuring park-based physical activity

While various observation techniques have been developed to measure park use or park-based physical activity, no study has used unmanned aerial vehicles (UAVs) to do so. Thus, this study develops a new observation method that uses UAVs to survey park-based physical activity. This study tests the inter-rater reliability and criterion validity of the UAV-using observation method in comparison to an existing on-the-ground observation tool in five diverse urban parks in Salt Lake City, Utah. With a systematic observation tool, SOPARC (System for Observing Play and Recreation in Communities), this study finds that the UAV observations show a high level of inter-rater reliability (ICC = 0.99 for a total number of users). In addition, compared to the results of on-the-ground observations using SOPARC, those of UAV observations demonstrate validity (ICC = 0.98 for a total number of users). Compared to existing methods, the UAV observation tool, covering larger target areas, is suitable for counting park users in a more reliable and efficient way and mapping their use patterns; however, the tool is weaker at collecting detailed user information and surveying under poor weather conditions. Thus, the UAV method could complement existing tools. Finally, this study suggests practical implications of the UAV observation method.

Development and Validation of the Cochlear Implant Surgical Competency Assessment Instrument.

We present a new instrument for evaluation of cochlear implant (CI) surgical skills and review its validation process. An instrument to assess CI surgical competency incorporated results of structured surveys of comprehensiveness sent to 30 international CI experts and US trainees. One-hundred evaluations of 28 residents, fellows, and practicing CI surgeons were completed. Surgical skills were evaluated by four experienced neurotologists (two raters per subject) using two temporal bones per subject. A training session was completed by 24 subjects between the first and second procedure. Comparison of two blinded rater's scores per subject provided information on interrater reliability. Correlation of competency scores with degree of training and with improvement after a training session provided information on construct validity. High levels of interrater reliability were confirmed by using the intraclass correlation coefficient. Construct validity was demonstrated by correlation of higher performance scores with increasing years of training, board certification, and fellowship training. Construct validity is also supported by improvement in scores after a CI training session as well as by acceptability surveys. Data indicate that this instrument is an objective, accurate, and dependable procedure-specific instrument for evaluating CI surgical competency. The cochlear implant surgical competency assessment (CI-SCA) can be used to establish CI surgical competency, identify surgical skills that require remediation and demonstrate progress during training.

Systematic Conceptualization of Person Centered Medicine and Development and Validation of a Person-centered Care Index

Background: A world-wide movement is emerging for person centered medicine and healthcare collaboratively developed by the International College of Person Centered Medicine and a number of global institutions through annual Geneva Conferences and International Congresses as well as scholarly publications. While frequently quoted definitional note have been advanced through this maturational process, efforts are needed to conceptualize and measure systematically person-centered care.Objectives: These included the elucidation of the core concepts of person centered medicine and healthcare, the design of a prototype measuring instrument, and the study of its metric structure, acceptability, reliability and validity.Methods: The following methodological approaches were employed: A systematic review of the literature, consultation exercises with broad international panels composed of health professionals and representatives of patient and family organizations, and quantitative and qualitative data analyses.Results: The following key concepts underlying person centered medicine were elucidated: 1) Ethical Commitment, 2) Cultural Sensitivity, 3) Holistic scope, 4) Relational Focus, 5) Individualized Care, 6) Common Ground for Collaborative Diagnosis and Care, 7) People-centered Systems of Care, and 8) Person-centered Education and Research. On this basis, a Person-centered Care Index was developed composed of 8 broad items and 33 sub-items, each measured on a 4-point scale. The study of its metric structure revealed high Cronbach internal consistency (0.95), scale unidimensionality through factor analysis (69 % of the variance accounted for by the first factor), and interesting inter-correlations such as the sub-item attaining the highest correlation with the global average score being “fulfillment of the person’s life project” (0.88). Validation studies in California, London and Lucknow (India) showed quite high levels of inter-rater reliability (above 0.80 intra-class correlations for most items) and substantial content validity.Discussion: The elucidated core concepts of person centered medicine appear to be consistent with those of international studies on the bases of person- and people-centeredness in primary care and on research and implementation of person centered care. The concepts are also consistent with the key domains of person-centered diagnostic approaches. Further validation studies with larger samples in diverse settings and cultures seem to be warranted.Conclusions. The emerging core concepts of person centered medicine appear to be solid. The Person-centered Care Index built on such concepts appears to have suitable metrics and promising acceptability, reliability and content validity. Further empirical research is recommended.

Interrater reliability of Violence Risk Appraisal Guide scores provided in Canadian criminal proceedings.

Published research suggests that most violence risk assessment tools have relatively high levels of interrater reliability, but recent evidence of inconsistent scores among forensic examiners in adversarial settings raises concerns about the "field reliability" of such measures. This study specifically examined the reliability of Violence Risk Appraisal Guide (VRAG) scores in Canadian criminal cases identified in the legal database, LexisNexis. Over 250 reported cases were located that made mention of the VRAG, with 42 of these cases containing 2 or more scores that could be submitted to interrater reliability analyses. Overall, scores were skewed toward higher risk categories. The intraclass correlation (ICCA1) was .66, with pairs of forensic examiners placing defendants into the same VRAG risk "bin" in 68% of the cases. For categorical risk statements (i.e., low, moderate, high), examiners provided converging assessment results in most instances (86%). In terms of potential predictors of rater disagreement, there was no evidence for adversarial allegiance in our sample. Rater disagreement in the scoring of 1 VRAG item (Psychopathy Checklist-Revised; Hare, 2003), however, strongly predicted rater disagreement in the scoring of the VRAG (r = .58). (PsycINFO Database Record

MRI measures of fat infiltration in the lower extremities following motor incomplete spinal cord injury: reliability and potential implications for muscle activation.

Muscle fat infiltration (MFI) is an expected consequence of incomplete spinal cord injury (iSCI). The MFI magnitude may have clinical value in determining functional recovery. However, there is a lack of understanding of how MFI relates to the volitional muscle activity in people with motor incomplete spinal cord injury (iSCI). Five iSCI and 5 uninjured age-matched control subjects participated in the study. In this preliminary study, we established the reliability of MFI quantification of select lower extremity muscles across different raters. Secondly, we assessed the magnitude and distribution of MFI in the lower legs of iSCI and uninjured control participants. Thirdly, we explored the relationship between MFI in the plantar flexor muscles and the ability to volitionally activate these muscles. High levels of inter-rater reliability were observed. The iSCI group had significantly elevated and a vastly different MFI distribution in the lower leg muscles compared to healthy controls. MFI was negatively correlated with volitional activation in iSCI. Our preliminary results sanction the importance of lower extremity MFI quantification as a potential measure in determining the functional outcomes in iSCI, and the subsequent pathological sequelae.

Protocol for the Evaluation of Speech Intelligibility in Dysarthrias: Evidence of Reliability and Validity

Background/Aims: Studies about the reliability and validity of intelligibility measures are fundamental to direct their use and interpretation in both clinical and research settings. The present study assessed inter-rater reliability, internal consistency and criterion validity of scores for the sentence and word subtests of the Protocol for the Evaluation of Speech Intelligibility (PESI). Methods: This study included 180 adult volunteers who were divided into three groups: 30 speakers with dysarthria, 30 control speakers and 120 listeners. Intelligibility scores were calculated according to the percentage of words correctly transcribed in each subtest: words in sentences and single words. Results: The intelligibility measurements exhibited a high level of inter-rater reliability for sentences [intraclass correlation coefficient (ICC) = 0.94] and for single words (ICC = 0.96). The items from the single words subtest of the PESI revealed high internal consistency (Cronbach's alpha = 0.93). The measurement of intelligibility of sentences and single words had a high discriminatory power (0.82 and 0.95, respectively), with high sensitivity and specificity, particularly for scores in the single words subtest. Conclusion: This study provides preliminary evidence of the reliability and validity of the PESI.

A Prototype Public Speaking Skills Assessment: An Evaluation of Human‐Scoring Quality

The purpose of this paper is to summarize the evaluation of human‐scoring quality for an assessment of public speaking skills. Videotaped performances given by 17 speakers on 4 tasks were scored by expert and nonexpert raters who had extensive experience scoring performance‐based and constructed‐response assessments. The Public Speaking Competence Rubric was used to score the speeches. Across all of the dimensions of presentation competence, interrater reliability between expert and nonexpert raters ranged between .23 and .71. The dimensions of public speaking competence associated with the lowest interrater reliability were effectual persuasion and word choice (.41 and .23, respectively). Even expert raters, individuals with a background in teaching and evaluating oral communication, had difficulty agreeing with one another on those dimensions. Low‐inference dimensions such as visual aids and vocal expression were associated with much higher levels of interrater reliability, .65 and .75, respectively. The holistic score was associated with an interrater reliability of .63. These results point to the need for a significant investment in task, rubric, and training development for the public speaking competence assessment before it can be used for large‐scale assessment purposes.

Review and Evaluation of Mindfulness-Based iPhone Apps.

BackgroundThere is growing evidence for the positive impact of mindfulness on wellbeing. Mindfulness-based mobile apps may have potential as an alternative delivery medium for training. While there are hundreds of such apps, there is little information on their quality.ObjectiveThis study aimed to conduct a systematic review of mindfulness-based iPhone mobile apps and to evaluate their quality using a recently-developed expert rating scale, the Mobile Application Rating Scale (MARS). It also aimed to describe features of selected high-quality mindfulness apps.MethodsA search for “mindfulness” was conducted in iTunes and Google Apps Marketplace. Apps that provided mindfulness training and education were included. Those containing only reminders, timers or guided meditation tracks were excluded. An expert rater reviewed and rated app quality using the MARS engagement, functionality, visual aesthetics, information quality and subjective quality subscales. A second rater provided MARS ratings on 30% of the apps for inter-rater reliability purposes.ResultsThe “mindfulness” search identified 700 apps. However, 94 were duplicates, 6 were not accessible and 40 were not in English. Of the remaining 560, 23 apps met inclusion criteria and were reviewed. The median MARS score was 3.2 (out of 5.0), which exceeded the minimum acceptable score (3.0). The Headspace app had the highest average score (4.0), followed by Smiling Mind (3.7), iMindfulness (3.5) and Mindfulness Daily (3.5). There was a high level of inter-rater reliability between the two MARS raters.ConclusionsThough many apps claim to be mindfulness-related, most were guided meditation apps, timers, or reminders. Very few had high ratings on the MARS subscales of visual aesthetics, engagement, functionality or information quality. Little evidence is available on the efficacy of the apps in developing mindfulness.