Average Interrater Reliability Research Articles

Reliability of pediatric Rome IV criteria for the diagnosis of disorders of gut-brain interaction.

The diagnosis of disorders of gut-brain interaction (DGBI) in children is exclusively based on clinical criteria called the Rome criteria. The inter-rater reliability (IRR) measures how well two raters agree with a diagnosis using the same diagnostic tool. Previous versions of the Rome criteria showed only fair to moderate IRR. There have been no studies assessing the IRR of the current edition of the pediatric Rome criteria (Rome IV). This study sought to investigate the IRR of the pediatric Rome IV criteria and compare its reliability with the previous versions of the Rome criteria. We hypothesized that changes made to Rome IV would result in higher IRR than previous versions. This study used the same methodology as the previous studies on Rome II and III, including identical clinical vignettes, number of raters, and levels of expertise. Participants included 10 pediatric gastroenterology fellows and 10 pediatric gastroenterology specialists. IRR was assessed using the percentage of agreement and Cohen's kappa coefficient to account for possible agreement by chance. The average IRR percentage of agreement using the Rome IV criteria was 55% for pediatric gastroenterologists and 48.5% for fellows, indicating moderate agreement (k = 0.54 for specialists, k = 0.47 for fellows). The results demonstrated higher percentages of agreement and kappa coefficients compared to the Rome II and III criteria. The findings demonstrate improved reliability in Rome IV compared to Rome II and III, suggesting that the changes incorporated into the Rome IV criteria have enhanced diagnostic consistency. Despite the advancements, the reliability is still moderate, indicating the need for further refinement of future versions of the Rome criteria.

Reliability of landmark identification for analysis of the temporomandibular joint in real-time MRI

BackgroundReal-time magnetic resonance imaging (rtMRI) is essential for diagnosing and comprehending temporomandibular joint (TMJ) movements. Current methods for tracking and analysis require manual landmark placement on each acquisition frame. Therefore, our study aimed to assess the inter- and intra-rater reliability of placing cephalometric landmarks in frames from a dynamic real-time TMJ MRI.Material and methodsFour real-time MRIs of the right TMJ were taken during mandibular movement at ten frames per second. Seven dentists identified ten landmarks on two frames (intercuspal position—ICP—and maximum mouth opening—MMO) twice at a two-week interval, yielding 112 tracings. Six typical cephalometric measurements (angles and distances) were derived from these landmarks. The reliabilities of landmarks and measurements were evaluated using distance-based (dbICC), linear mixed effect model intraclass correlation (lmeICC), and standard ICC.ResultsThe average inter-rater reliability for the landmarks stood at 0.92 (dbICC) and 0.93 (lmeICC). The intra-rater reliability scores were 0.97 and 0.98. Over 80% of the landmarks showed an ICC greater than 0.98 (inter-rater) and over 0.99 (intra-rater). The lowest landmark ICC was observed for the orbitale and the oblique ridge of the mandibular ramus. However, the cephalometric angle and distance measurements derived from these landmarks showed only moderate to good reliability, whereas the reliability in the frames with ICP was better than those with MMO. Measurements performed in the ICP frame were more reliable than measurements in the MMO frame.ConclusionWhile dentists reliably localize isolated landmarks in real-time MRIs, the cephalometric measurements derived from them remain inconsistent. The better results in ICP than MMO are probably due to a more familiar jaw position. The higher error rate of the TMJ measurements in MMO could be associated with a lack of training in real-time MRI analysis in dentistry.

The development and preliminary evaluation of the Genetic Counseling Skills Checklist.

Genetic counseling (GC) relies on communication to help people understand and adapt to genetic contributions of disease, and there is need for a practical and reliable method of comprehensively documenting GC communication skills without intensive coding. To this end, we created a novel process measure called the Genetic Counseling Skills Checklist (GCSC), utilizing previously validated measures, communication/counseling frameworks, and prior research findings. A multistage iterative process was used to develop, evaluate, and modify the GCSC to improve its clarity, usability, and content validity. To assess interrater reliability, randomly assigned, untrained individuals (i.e., coders) used the GCSC version 3 to code multiple simulated GC sessions. Average measures intraclass correlation coefficients (ICCs) were calculated for each of the 8 GCSC process categories using one-way, random effects models. After relatively minor modifications to the GCSC, two pairs of experienced coders used GCSC version 4 to independently code additional GC sessions and Cohen's Kappa coefficients (κ) were calculated to assess interrater reliability for each process category. The GCSC contains five to eight items within each category and demonstrates good content validity given its ability to capture nearly all GC skills that genetic counselors reported using in a prior qualitative study. Interrater reliability of GCSC version 3 among coders with limited experience was moderate or good for 6 out of the 8 process categories as evidenced by ICCs ranging from 0.55 to 0.86. Average interrater reliability of GCSC version 4 among one pair of experienced coders was strong for all eight process categories (κ ranging from 0.82 to 0.94); among the second pair of experienced coders scores were strong for six categories (κ ranging from 0.80 to 0.87) and moderate for two categories (κ of 0.77 and 0.78). The results suggest the need for training and experience to assure adequate interrater reliability across GCSC coders. Future work is needed to create a formalized training program for coders, complete a larger study to further validate the measure, and use the GCSC to document variability in skills used across providers and sessions.

Interventions for Health Promotion and Obesity Prevention for Children And Adolescents With Developmental Disabilities (DD)

<h3>Objective(s)</h3> To evaluate efficacy of health promotion and obesity prevention interventions among children and adolescents with DD. <h3>Data Sources</h3> Databases searched: PubMed, CINAHL, PsycINFO, Cochrane Library, Psychology and Behavioral Sciences Collection, Health Source: Nursing/Academic, Web of Science, Embase, and Scopus. Search terms: child*, adolesc*, pediatric*, disability, intellectual*, impair*, development*, autis*, down syndrome*, cognitive*; health promotion, healthy lifestyle, healthy eating, obesity intervention, obesity prevention, physical exercise; BMI, eating behaviors, physical in/activity, dietary intake, and sedentary behaviors. <h3>Study Selection</h3> Inclusion criteria: (1) children and adolescents (6-17 years) with DD (2) interventions targeting obesity outcomes within a child's natural environment; (3) peer-reviewed, experimental or quasi-experimental studies; (4) quantitative or mixed methods approach; (5) reported primary outcomes for the child; (6) published between 2010 – 2021. Reviewers worked in pairs to independently screen articles. Author and journal information was blinded. Selection was based on consensus. Average interrater reliability across screeners was high (92.7%). Of 21,889 unique records, 21 studies met selection criteria. <h3>Data Extraction</h3> Data Extraction was conducted by six research team members who worked in three pairs using a standardized form adapted from existing sources. We adapted Reichow's (2011) evaluative method for rating the rigor of each study and determining the overall level of evidence for each intervention. Interrater reliability for rating of rigor was 88%. Rating disagreements within pairs were resolved through discussion. <h3>Data Synthesis</h3> Interventions were categorized into five categories including: aerobic and strength training, sport-based physical activity, aquatic exercise, active video gaming, and diet and lifestyle interventions. None of the intervention categories met the recommended threshold for ‘established' EBP. Aquatic exercise programs and active videogaming programs approached Reichow's (2011) cut off for ‘probable EBP'. Diet and lifestyle interventions, aerobic and strength training exercise programs, and sports-based physical activity programs warranted an overall rating of ‘not an EBP'. <h3>Conclusions</h3> Interventions such as sports-based exercise programs and active video gaming showed promise, but more research is needed to confirm their effectiveness. Future research must focus on theoretically grounded and rigorous intervention studies with diverse samples. <h3>Author(s) Disclosures</h3> None

Development and Validation of Comic-based Learning Module in Physics

Providing instructional materials that support the diverse learning styles and needs of 21st-century learners is becoming a trend in today's online learning. Thus, this descriptive developmental research design aimed to develop Comic-based Learning Modules (CLM) as instructional material envisioned to meet these learners' needs for a better understanding of Physics concepts. This study employed the ADDIE model for the instructional material development design. The initial developed CLM was revalidated following the comments and suggestions of the validators before distribution for online learning. Four experts validated the CLM in terms of content/lessons, illustrations, additional features, language, layout, and overall presentation using a researcher-made criterion-based reference evaluation tool. The validators rated the developed instructional material featuring three CLM as "Highly Acceptable" (4.66). The value of the average inter-rater reliability coefficient kappa of the ratings of the validators is interpreted as "Substantial Agreement" (0.63). CLM was then implemented as instructional material and found to have high usability as rated by the students. Also, the CLM drew positive feedback for online learning use because of its exciting nature and uniqueness.

Comparing human coding to two natural language processing algorithms in aspirations of people affected by Duchenne Muscular Dystrophy

Qualitative methods can enhance our understanding of constructs that have not been well portrayed and enable nuanced depiction of experience from study participants who have not been broadly studied. However, qualitative data require time and effort to train raters to achieve validity and reliability. This study compares recent advances in Natural Language Processing (NLP) models with human coding. This web-based study (N=1,253; 3,046 free-text entries, averaging 64 characters per entry) included people with Duchenne Muscular Dystrophy (DMD), their siblings, and a representative comparison group. Human raters (n=6) were trained over multiple sessions in content analysis as per a comprehensive codebook. Three prompts addressed distinct aspects of participants’ aspirations. Unsupervised NLP was implemented using Latent Dirichlet Allocation (LDA), which extracts latent topics across all the free-text entries. Supervised NLP was done using a Bidirectional Encoder Representations from Transformers (BERT) model, which requires training the algorithm to recognize relevant human-coded themes across free-text entries. We compared the human-, LDA-, and BERT-coded themes. Study sample contained 286 people with DMD, 355 DMD siblings, and 997 comparison participants, age 8-69. Human coders generated 95 codes across the three prompts and had an average inter-rater reliability (Fleiss’s kappa) of 0.77, with minimal rater-effect (pseudo R2=4%). Compared to human coders, LDA does not yield easily interpretable themes. BERT correctly classified only 61-70% of the validation set. LDA and BERT required technical expertise to program and took approximately 1.15 minutes per open-text entry, compared to 1.18 minutes for human raters including training time. LDA and BERT provide potentially viable approaches to analyzing large-scale qualitative data, but both have limitations. When text entries are short, LDA yields latent topics that are hard to interpret. BERT accurately identified only about two thirds of new statements. Humans provided reliable and cost-effective coding in the web-based context. The upfront training enables BERT to process enormous quantities of text data in future work, which should examine NLP’s predictive accuracy given different quantities of training data.

Presbylarynx: validation of a classification based on morphological characteristics.

Validation of a classification to stratify presbylarynx into three types. Evaluate the reproducibility of this classification. 30 video recordings of rigid laryngostroboscopies were presented to 20 otorhinolaryngologists from five tertiary hospitals: 10 residents and 10 specialists. Specialists with different differentiations were included. Among 1200 evaluations performed (30 video recordings × 20 raters, twice), average intra-rater agreement rate was 0.847 (p < 0.001). The average inter-rater reliability was 0.67 ± 0.179 on the first evaluation, and 0.691 ± 0.131 on the second evaluation. This study presented the validation of a classification that allows stratifying laryngeal endoscopic signs in elderly patients with presbyphonia. We believe that this classification will enrich the diagnostic protocol of the aging voice and will improve the documentation of treatment outcomes.

Reliability assessment and validation of the post-acne hyperpigmentation index (PAHPI) in a population from Sub-Saharan Africa in Senegal

BackgroundA post-acne hyperpigmentation index (PAHPI) has been developed in the United States to better compare therapeutic modalities. Our aim in this study was to validate the PAHPI score in patients with skin type VI from sub-Saharan Africa. Patients and methodsThe study was conducted in Dakar, Senegal. Twenty-one patients with Fitzpatrick skin type VI, aged 17 to 55 years, presenting hyperpigmentation secondary to acne were included. Ongoing use of skin bleaching products or acne treatments was allowed. Four trained dermatologists rated the patients using the PAHPI. A narrow-band reflectance spectrophotometer (Mexameter MX-18, Cologne, Germany) was used to measure the degree of pigmentation of involved and adjacent skin on 6 representative facial lesions. ResultsThe average inter-rater reliability (weighted Kappa) showed substantial agreement for intensity (0.67), moderate agreement for number (0.53) and fair agreement for lesion size (0.28). Inter-rater reliability for the total PAHPI was excellent for both day 1 and day 2 (interclass correlation coefficient of 0.87 and 0.85, respectively; P<0.0001). Intra-rater reliability for total PAHPI ranged from 0.83 to 0.93 (P<0.0001). PAHPI scoring thus demonstrated excellent reliability both between and within raters. The association was moderate to substantial for all raters on both days (range for rho on day 1: 0.531 to 0.815; range for rho on day 2: 0.448 0.762). The correlations between the Mexameter (Courage and Khazaka) measurements and PAHPI scores showed moderate to substantial agreement. ConclusionAlthough tested primarily in African American women to date, PAHPI is also valid for patients from sub-Saharan Africa.

Validity and Inter-rater Reliability of the Scoring Rubrics for the Science Teacher TPACK Test Instrument

This study examined the content validity of the scoring rubric instrument for measuring science teachers’ TPACK and the inter-rater reliability in using the instrument. This research was conducted as part of research and development which has been designed for the development of instruments for measuring teacher knowledge. The analysis carried out was a qualitative analysis based on triangulation of the three validators’ validation results and quantitative analysis for inter-rater reliability based on the Intraclass Correlation Coefficient (ICC) obtained for each question. The validation involved three science education experts from the university to assess the suitability of the scoring rubrics in the technological pedagogical content knowledge (TPACK) framework. Inter-rater reliability examining involved 100 participants who answered 15 questions on the instrument and three experienced raters to assess the participants' answers. The validation results showed that the instrument content was valid for measuring the knowledge tested and had very high inter-rater reliability coefficient for all items The validation results show that qualitatively the contents of the instrument are valid for measuring the knowledge being tested and had an average inter-rater reliability coefficient of 0.94 (very high).

Measuring behavior change technique delivery and receipt in physical activity behavioral interventions.

Research on physical activity behavioral support has mainly focused on measuring the absence or presence of behavior change techniques (BCTs) delivered by a counselor. We present a method to measure BCT delivery and receipt in physical activity behavioral support interventions. The method was developed and tested using transcripts from behavior change counseling sessions delivered as part of a theory- and evidence-based physical activity intervention for adults with disabilities. Using existing methods, a new method was developed to code counselor and clients' verbal statements (BCTs and other statements). Two coders independently coded 30 transcripts of audio-recorded counseling sessions. Interrater reliability was assessed using percentage agreement and Prevalence Adjusted Bias Adjusted Kappa (PABAK). Forty-eight codes were developed for counselor statements (35 BCT delivery and 13 other statements) and 46 codes for client statements (34 BCT receipt and 12 other statements). The average interrater reliability was considered nearly perfect for the counselor statements (84% agreement; PABAK = .98) and client statements (86% agreement; PABAK = .98). The BCT delivered and received were most frequently related to Goals and Planning (counselor:36%, client:35%), Support Strategies (counselor:21%, client:17%) and Self Belief (counselor:13%, client:24%). This study presents a reliable coding method to measure BCT delivery and receipt in physical activity behavioral support interventions. The method can be used to enhance intervention fidelity assessment and study interactions between counselors and clients with and without disabilities. Measuring and evaluating BCT delivery and receipt can provide new insights into what types of behavioral support work best under which circumstances. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Avoiding the Deep Plantar Arterial Arch in Transmetatarsal Amputations: A Cadaver Study.

The deep plantar arterial arch (DPAA) is formed by an anastomosis between the deep plantar artery and the lateral plantar artery. The potential risk of injury to the DPAA is concerning when performing transmetatarsal amputations, and care must be taken to preserve the anatomy. We sought to determine the positional anatomy of the DPAA based on anatomical landmarks that could be easily identified and palpated during transmetatarsal amputation. In an effort to improve our understanding of the positional relationship of the DPAA to the distal metatarsal parabola, dissections were performed on 45 cadaveric feet to measure the location of the DPAA with respect to the distal metatarsal epiphyses. Images of the dissected specimens were digitally acquired and saved for measurement using in-house-written software. The mean, SD, SEM, and 95% confidence interval were calculated for all of the measurement parameters and are reported on pooled data and by sex. An independent-samples t test was used to assess for sex differences. Interrater reliability of the measurements was estimated using the intraclass correlation coefficient. The origin of the DPAA was located a mean ± SD of 35.6 ± 3.9 mm (95% confidence interval, 34.5-36.8 mm) proximal to the perpendicular line connecting the first and fifth metatarsal heads. The average interrater reliability across all of the measurements was 0.921. This study provides the positional relationship of the DPAA with respect to the distal metatarsal parabola. This method is easily reproducible and may assist the foot and ankle surgeon with surgical planning and approach when performing partial pedal amputation.

Determination of Interrater Reliability of a Universal Evaluator Rubric to Assess Student Pharmacist Communication Skills

Objective. To evaluate the interrater reliability of a universal evaluator rubric used to assess student pharmacist communication skills during patient education sessions.Methods. Six US schools and colleges of pharmacy each submitted 10 student videos of a simulated community pharmacy patient education session and recruited two raters in each of the five rater groups (faculty, standardized patients, postgraduate year one residents, student pharmacists, and pharmacy preceptors). Raters used a rubric containing 20 items and a global assessment to evaluate student communication of 12 videos. Agreement was computed for individual items and overall rubric score within each rater group, and for each item across all rater groups. Average overall rubric agreement scores were compared between rater groups. Agreement coefficient scores were categorized as no to minimal, weak, moderate, strong, or almost perfect agreement.Results. Fifty-five raters representing five rater groups and six pharmacy schools evaluated student communication. Item agreement analysis for all raters revealed five items with no to minimal or weak agreement, 10 items with moderate agreement, one item with strong agreement, and five items with almost perfect agreement. Overall average agreement across all rater groups was 0.73 (95% CI, 0.66-0.81). The preceptor rater group exhibited the lowest agreement score of 0.68 (95% CI, 0.58-0.78), which significantly deviated from the overall average.Conclusion. While strong or almost perfect agreement scores were not observed for all rubric items, overall average interrater reliability results support the use of this rubric in a variety of raters to assess student pharmacist communication skills during patient education sessions.

The Outcomes for Human Trafficking Instrument: Validity and Reliability Testing

ABSTRACT This study supports evidence-based services for victims of HT by developing and testing an outcome instrument designed to assess changes in HT program participant status on key outcome categories. Study methods included conducting an environmental scan; expert panels of service providers, survivors, and researchers; instrument scoring and cognitive interviews with HT service providers. Hypothetical case vignettes were then used to assess instrument reliability and concurrent validity – concurrent validity - first among program directors, then among case managers, and finally, comparing case managers to program directors. Average interrater reliability across items within outcome categories among program directors was excellent (ICC > 0.90) for most outcome categories, indicating consensus ratings could serve as a gold standard for assessing the performance of less seasoned HT service providers using the instrument. Case managers had mostly moderate (ICC > 0.50) to good (ICC > 0.75) average interrater reliability across items within outcome categories among themselves, and fair (> 0.40) to good (> 0.75) average agreement across items within outcome categories with the program directors. The resulting excel-based instrument is free; user-friendly; applicable to diverse settings; relevant to victims of sex and labor trafficking, whether domestic or foreign born, adults or minors; and supports data management, visualization, export, and analysis.

Analysing EHR navigation patterns and digital workflows among physicians during ICU pre-rounds.

Background:Some physicians in intensive care units (ICUs) report that electronic health records (EHRs) can be cumbersome and disruptive to workflow. There are significant gaps in our understanding of the physician–EHR interaction.Objective:To better understand how clinicians use the EHR for chart review during ICU pre-rounds through the characterisation and description of screen navigation pathways and workflow patterns.Method:We conducted a live, direct observational study of six physician trainees performing electronic chart review during daily pre-rounds in the 30-bed medical ICU at a large academic medical centre in the Southeastern United States. A tailored checklist was used by observers for data collection.Results:We observed 52 distinct live patient chart review encounters, capturing a total of 2.7 hours of pre-rounding chart review activity by six individual physicians. Physicians reviewed an average of 8.7 patients (range = 5–12), spending a mean of 3:05 minutes per patient (range = 1:34–5:18). On average, physicians visited 6.3 (±3.1) total EHR screens per patient (range = 1–16). Four unique screens were viewed most commonly, accounting for over half (52.7%) of all screen visits: results review (17.9%), summary/overview (13.0%), flowsheet (12.7%), and the chart review tab (9.1%). Navigation pathways were highly variable, but several common screen transition patterns emerged across users. Average interrater reliability for the paired EHR observation was 80.0%.Conclusion:We observed the physician–EHR interaction during ICU pre-rounds to be brief and highly focused. Although we observed a high degree of “information sprawl” in physicians’ digital navigation, we also identified common launch points for electronic chart review, key high-traffic screens and common screen transition patterns.Implications:From the study findings, we suggest recommendations towards improved EHR design.

Construction and Validation of a Video Coding Tool for an Intervention to Improve Parental Feeding

Abstract Objectives Home-based interventions have used video-recorded meals to assess feeding practices, yet no studies have used videos to directly provide feedback on the practices observed. This study describes the development and initial validation of a video coding tool to assess feeding practices observed in video-recorded family meals in order to provide feedback to caregivers. Methods The tool with operational definitions was developed based on the previous literature and other tools that capture caregiver feeding practices. To assess face validity, a sample of child feeding experts (n = 6) reviewed the tool and completed an 8-item online survey. Usability and content were assessed on a scale of 0–100, with 100 representing high usability and importance, respectively. The tool was modified based on expert feedback and used by trained research assistants to code 10 video-recorded family meals. Inter-rater reliability (IRR) was calculated and 3 videos were then randomly selected and coded by each research assistant at two different time points to assess test re-test reliability. Results Expert ratings of tool usability (81.83 ± 11.67) and content (87.67 ± 13.98) included feedback regarding the need to expand the operational definitions to better code practices, in particular with regards to pressure and encouragement. They also suggested merging practices, such as nutrition education and reasoning, which may be difficult to discern in an observational setting. Average IRR was 86.4% with pressure (75.1%) and encouragement (66.9%) having the lowest rates of agreement between coders. Limited choices (100%) and restriction (95.9%) had the highest rates of agreement. For test re-test reliability, the average agreement between the two timepoints was 80.0%. Conclusions Following the feedback from experts, the face validity of the developed tool improved, and the inter-rater reliability and test re-test reliability of the tool was acceptable. Future studies should focus on the expansion of operational definitions and training efforts to further improve inter-rater and test re-test reliabilities. Funding Sources This work was supported by the National Institutes on Health National Heart, Lung and Blood Institute [Tovar/R34HL 140,229].

Development and Reliability of the Comprehensive Crisis Plan Checklist, 2nd Edition

AbstractAs the research base for school crisis intervention and prevention expands, the need for well‐developed tools to assess school readiness in the event of a crisis increases. This paper describes how the Comprehensive Crisis Plan Checklist (CCPC) was updated to reflect advances in crisis management and crisis planning. An extensive literature search and pilot study were used to refine existing items and create new items for the checklist. The Comprehensive Crisis Plan Checklist, 2nd Edition (CCPC‐2) has 102 items separated into three sections: prevention, intervention, and postvention. The CCPC‐2 can be used by crisis teams to create new crisis plans or evaluate existing ones. Users are encouraged to carefully consider the inclusion of all items and articulate why individual items are not necessarily based on their specific needs. The CCPC‐2 was given to 10 pairs of raters to evaluate school‐based crisis plans; average interrater reliability was 89.04%. Discussion focuses on item analysis and how to use the checklist within a school setting.

Automated analysis of cognitive presence in MOOC discussions

The Community of Inquiry (CoI) framework [1] has been broadly used to analyse learning experience in online discussion forums for two decades. Cognitive presence, which is a primary dimension of the CoI framework, manifests the reflection of (re)constructing knowledge and problem-solving processes in the learning experience [2]. Researchers doing text analysis using machine learning techniques are making promising contributions to analysing phases of cognitive presence automatically [3]–[5] in online discussions. However, most studies of automated cognitive analysis focus on improving the accuracy and reliability of the classifiers. They ignored that another purpose of applying machine learning techniques in educational research should be to pinpoint research bias that scholars neither intended to nor can have found without computer support. This session will present the example of ‘research bias’ discovered from both manual and automated classification of cognitive phases, provoking scholars to rethink and improve the conflicting part in the taxonomies of cognitive presence under MOOC context.&#x0D; &#x0D; The manual-classification rubric that used to label discussion messages of a target MOOC combines Garrison, Anderson and Archer’s [2] scheme with Park’s [6] revised version. The rubric describes four phases of cognitive presence (i.e. triggering event, exploration, integration and resolution), and indicators of each phase in online discussions. We reported the average inter-rater reliability between two human raters achieved 95.4% agreement (N = 1002) with a Cohen’s weighted kappa of 0.96. Interestingly, we found the average inter-rater reliability decreased to 80.1% after increasing the size of data samples (N = 1918) and the number of human raters to three. After training the automated classifiers to predict phases of cognitive presence, the confusion matrix implies that most of the disagreements between computer raters occurred between adjacent phases of cognitive presence. The disagreements between human raters also have the same problems. We assume the additional categories may exist between cognitive phases in such MOOC discussion messages. These details will be discussed during the presentation.&#x0D; &#x0D; References&#x0D; [1] D. Garrison, T. Anderson, and W. Archer, “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education,” Internet High. Educ., vol. 2, no. 2, pp. 87–105, 1999.&#x0D; [2] D. Garrison, T. Anderson, and W. Archer, “Critical thinking, cognitive presence, and computer conferencing in distance education,” Am. J. Distance Educ., vol. 15, no. 1, pp. 7–23, 2001.&#x0D; [3] V. Kovanović, S. Joksimović, D. Gašević, and M. Hatala, “Automated cognitive presence detection in online discussion transcripts,” in Automated cognitive presence detection in online discussion transcripts’ CEUR Workshop Proceedings (vol. 1137), 2014.&#x0D; [4] V. Kovanović et al., “Towards automated content analysis of discussion transcripts,” Proc. Sixth Int. Conf. Learn. Anal. Knowl. - LAK ’16, pp. 15–24, 2016.&#x0D; [5] E. Farrow, J. Moore, and D. Gasevic, “Analysing discussion forum data: a replication study avoiding data contamination,” 9th Int. Learn. Anal. Knowl. Conf., no. March, 2019.&#x0D; [6] C. Park, “Replicating the Use of a Cognitive Presence Measurement Tool,” J. Interact. Online Learn., vol. 8, no. 2, pp. 140–155, 2009.

How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis

Assessments of psychotherapeutic competencies play a crucial role in research and training. However, research on the reliability and validity of such assessments is sparse. This study aimed to provide an overview of the current evidence and to provide an average interrater reliability (IRR) of psychotherapeutic competence ratings. A systematic review was conducted, and 20 studies reported in 32 publications were collected. These 20 studies were included in a narrative synthesis, and 20 coefficients were entered into the meta-analysis. Most primary studies referred to cognitive-behavioral therapies and the treatment of depression, used the Cognitive Therapy Scale, based ratings on videos, and trained the raters. Our meta-analysis revealed a pooled ICC of 0.82, but at the same time severe heterogeneity. The evidence map highlighted a variety of variables related to competence assessments. Further aspects influencing the reliability of competence ratings and regarding the considerable heterogeneity are discussed in detail throughout the manuscript.

Location of the Deep Plantar Artery: A Cadaveric Study.

The deep plantar (D-PL) artery originates from the dorsalis pedis artery in the proximal first intermetatarsal space, an area where many procedures are performed to address deformity, traumatic injury, and infection. The potential risk of injury to the D-PL artery is concerning. The D-PL artery provides vascular contribution to the base of the first metatarsal and forms the D-PL arterial arch with the lateral plantar artery. In an effort to improve our understanding of the positional relationship of the D-PL artery to the first metatarsal, dissections were performed on 43 embalmed cadaver feet to measure the location of the D-PL artery with respect to the base of the first metatarsal. Digital images of the dissected specimens were acquired and saved for measurement using in-house software. Means, standard deviations, and 95% confidence intervals (CIs) were calculated for all of the measurement parameters. We found that the origin of the D-PL artery was located at a mean ± SD of 11.5 ± 3.9 mm (95% CI, 4.5-24.7 mm) distal to the first metatarsal base and 18.6% ± 6.5% (95% CI, 8.1%-43.4%) of length in reference to the proximal base. The average interrater reliability across all of the measurements was 0.945. This study helps clarify the anatomical location of the D-PL artery by providing parameters to aid the surgeon when performing procedures in the proximal first intermetatarsal space. Care must be taken when performing procedures in the region to avoid unintended vascular injury to the D-PL artery.

Assessing treatment integrity in personalized CBT: the inventory of therapeutic interventions and skills

ABSTRACTThe third wave of cognitive behavioral therapy (CBT) has increased the heterogeneity of today’s CBT practice, while developments in patient-focused research are paving the road to the empirical personalization of CBT. This paper presents the development and psychometric properties of a therapy video rating instrument, which was designed to adequately assess the treatment integrity (adherence and competence) of modern, personalized CBT. The Inventory of Therapeutic Interventions and Skills (ITIS) was developed based on two existing CBT adherence and competence scales and augmented with third wave content and overarching therapeutic strategies. The instrument was then applied by graduate students and post-graduate clinicians to rate N = 185 therapy videos from N = 70 patients treated at a university outpatient clinic. Descriptive results, inter-rater reliability, item structure, and associations with session outcome and alliance were examined. Average inter-rater reliability was excellent for Interventions items and good for Skills items. Intercorrelations were low between Interventions items, but higher and significant between Skills items, which loaded on a single factor. Several ITIS items were shown to be predictive of session outcome and alliance, even after controlling for the nested data structure. Implications of these results for future research and clinical training are discussed.