Abstract
We hypothesized that crowdsourcing assessments could be applied to the Postureteroscopic Lesion Scale (PULS) for ureteral injury. At a single institution, we prospectively digitally recorded 14 ureters at the terminal portion of standard ureteroscopic procedures. Each recording was reviewed by 10 global experts to determine a mean PULS score. Following training, the Crowd-Sourced Assessment of Technical Skills, C-SATS® (C-SATS, Inc., Seattle, WA) platform was used to obtain crowd-based reviews. The mean crowd PULS scores was determined using the linear mixed-effects (LME) model. The intraclass correlation coefficient (ICC) was calculated to measure the agreement among experts. Spearman's rank correlation (rho) was used to quantify the strength of the relationship between the crowd LME mean and the experts. Ten expert's reviews and 2100 layman reviews were obtained in 21 days and 49 hours, respectively. The ICC for the 10 experts was 0.68 (95% confidence interval 0.49, 0.86). When the expert mean PULS was <1, the crowd scored those recordings at 1 or greater. The highest scored recording by the experts was a 3.2, which the crowd scored at 2.25. The correlation between the crowd LME means and expert means across all videos was 0.70 (p = 0.0056) indicative of moderately strong agreement. In this initial application of crowd-sourced evaluation of ureteral injury, there was a moderately strong correlation between crowd and expert ratings. Refinement of the training, through exposure to the nuances of ureteral injuries, in particular for PULS <1 or ≥3, may lead to better crowd/expert correlation. Compared to expert review, crowd data can be collected with much greater efficiency.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
