Analysis of the reliability of the evaluation and scoringprocess for abstract submissions for the “JournéesFrancophones de la Kinésithérapie 2023”

Abstract

Background: The French Society of Physiotherapy (SFP) organises biennial conferences known as the "JournéesFrancophones de Kinésithérapie" (JFK) since 2007. Abstracts are submitted and are evaluated for acceptance by twoindependent reviewers from the SFP using a predefined rating checklist. However, the reliability of this process has neverbeen evaluated.Objective: This study aims to assess the inter-rater reliability and internal consistency of the JFK submissionrating process conducted by the scientific committee.Methods: Blind reviewers evaluated each submission in pairs using astandardized 47-item rating checklist, categorised into five domains including background, method, results and relevance tophysiotherapy. Reliability was assessed using an Intraclass Correlation Coefficient (ICC) and Cohen’s Kappa. Agreement wasassessed using the Standard Error of Measurement (SEM), Coefficient of Variation, Bland-Altman analysis, and percentageof agreement. Internal consistency was assessed using Cronbach’s Alpha.Results: 36 reviewers assessed a total of 217abstracts. The reliability, measured by ICC, was poor (0.39 [CI95% = 0.30; 0.49]) as was the agreement; SEM = 3.08 andCoefficient of Variation = 23.1%. All individual checklist items had a Cohen’s kappa coeff below 0.6. All but one domainhad a Cronbach Alpha above 0.7, indicating good consistency. However, five domains had a Cronbach Alpha above 0.9,suggesting redundancy.Conclusion: The JFK 2023 submission rating process displayed poor reliability. These findings canguide improvements in creating the JFK 2025 checklist. This study may help future scientific committees to enhance theirevaluation process.