Methods of Analyzing eContour's Engagement: Interrupted Time Series Analysis Versus Pre/Post Analysis

Abstract

Background eContour is an evidence-based interactive contouring atlas. As of November 2022 there are 36,541 registered users. This study compares research methodology when assessing eContour's engagement, specifically a single arm interrupted time series analysis (ITSA) vs. traditional pre/post comparison analysis. The ITSA method is chosen because of its robust ability to account for trend, seasonality and autocorrelation bias. Methods Monthly case views data was extracted through mixpanel analytics software (Jan 2019 - Oct 2022). A single arm interrupted time series analysis (ITSA) with multiple intervention time points is a quasi-experimental design. We defined two time points to offer insight into trends: start of the pandemic (March 2020) and the e-Contour/MIM/ACRO collaboration in March 2022. These time points were selected given the potential to impact engagement as represented through the number of case views/month. Results of ITSA were compared to the commonly used pre/post analysis. In this traditional measure, we used Wilcoxon rank sum test to compare monthly eContour case views before and after each respective time points (March 2020 & 2022). Results The y-intercept (y 0) of the ITSA model estimates engagement during the pre-period at 13,467 views/month. The model predicts a shift in the y-intercept (y ∆) of 2,974 views/month as the immediate impact, although the estimate is not statistically significant (p<0.24). Over time, however, the pandemic has a sustained impact, with an estimated slope coefficient of 1,084 views/month (p<0.001). The immediate impact of eContour/MIM/ACRO collaboration yielded no immediate positive outcome in engagement - a decrease in 2373 case views/month (p=0.4760). In addition, the long-term impact continued to exhibit suboptimal outcomes on engagement (p=0.1091), with a decrease in 971 case views/month. Using traditional pre/posttest analysis, the pandemic has demonstrated an increase from 13,504 to 27,540 case views/month (p<0.01). A similar increase in engagement was observed around the ACRO 2022 time point (23,999 to 40,802; p<0.01). Discussion This study demonstrates the importance of selecting the appropriate methodology to assess engagement. Both the ITSA and pre/post methodologies demonstrated increased trends in engagement post-pandemic, whereas the ITSA did not demonstrate increased engagement post ACRO 2022. The major limitation of our current ITSA is the shorter number of time-points post ACRO 2022. Longer follow-up is required to observe a potential sustained increase in engagement. eContour is an evidence-based interactive contouring atlas. As of November 2022 there are 36,541 registered users. This study compares research methodology when assessing eContour's engagement, specifically a single arm interrupted time series analysis (ITSA) vs. traditional pre/post comparison analysis. The ITSA method is chosen because of its robust ability to account for trend, seasonality and autocorrelation bias. Monthly case views data was extracted through mixpanel analytics software (Jan 2019 - Oct 2022). A single arm interrupted time series analysis (ITSA) with multiple intervention time points is a quasi-experimental design. We defined two time points to offer insight into trends: start of the pandemic (March 2020) and the e-Contour/MIM/ACRO collaboration in March 2022. These time points were selected given the potential to impact engagement as represented through the number of case views/month. Results of ITSA were compared to the commonly used pre/post analysis. In this traditional measure, we used Wilcoxon rank sum test to compare monthly eContour case views before and after each respective time points (March 2020 & 2022). The y-intercept (y 0) of the ITSA model estimates engagement during the pre-period at 13,467 views/month. The model predicts a shift in the y-intercept (y ∆) of 2,974 views/month as the immediate impact, although the estimate is not statistically significant (p<0.24). Over time, however, the pandemic has a sustained impact, with an estimated slope coefficient of 1,084 views/month (p<0.001). The immediate impact of eContour/MIM/ACRO collaboration yielded no immediate positive outcome in engagement - a decrease in 2373 case views/month (p=0.4760). In addition, the long-term impact continued to exhibit suboptimal outcomes on engagement (p=0.1091), with a decrease in 971 case views/month. Using traditional pre/posttest analysis, the pandemic has demonstrated an increase from 13,504 to 27,540 case views/month (p<0.01). A similar increase in engagement was observed around the ACRO 2022 time point (23,999 to 40,802; p<0.01). This study demonstrates the importance of selecting the appropriate methodology to assess engagement. Both the ITSA and pre/post methodologies demonstrated increased trends in engagement post-pandemic, whereas the ITSA did not demonstrate increased engagement post ACRO 2022. The major limitation of our current ITSA is the shorter number of time-points post ACRO 2022. Longer follow-up is required to observe a potential sustained increase in engagement.