340 The Accuracy of Predictive Analytics in Forecasting Emergency Department Volume Pre- and Post-COVID Pandemic

Abstract

BackgroundForecasting emergency department (ED) volume is a critical piece of determining staffing needs. Methods of doing so have historically relied on prior volumes and subjective predictions of future volumes. Objective methods of volume prediction, such as triple exponential smoothing (TES), are more accurate. However, highly variable periods brought on by times of extreme uncertainty, such as the COVID pandemic, raise questions about predictive capacity.Study ObjectivesWe sought to determine which method of triple exponential smoothing was most accurate in predicting pre-pandemic and post-COVID (after the onset of COVID in February, 2020) ED volume, the effects of the pandemic on their accuracy, and whether such models could regain the pre-pandemic accuracy after the disruptive influence of COVID began.MethodsFour methods of monthly volume forecasting were compared: simple exponential smoothing with a 24-month run-up (SES), Microsoft Excel’s AAA version of the exponential smoothing (ES) algorithm with a 24-month run-up, Holt-Winter TES using 12 and 24-month run-up. Accuracy was assessed using mean absolute percentage error (MAPE). We examined data from four emergency departments between March 2018 and March 2022, 3 adult emergency departments and 1 pediatric emergency department, with total pre-COVID annual census >250,000 patients.ResultsIn the 24-months pre-COVID, the overall average MAPE across 4 TES methods was 2.49% ± 0.53%, while the overall average MAPE for the 24 months after the onset of the COVID pandemic was 12.56% ± 5.11%. Among all EDs, the 24-month Holt-Winter had the greatest accuracy pre-COVID, 1.58% ± 0.35% and post-COVID, 8.26% ± 3.40%. Using an acceptable standard of error limit of 1 standard deviation above the upper range pre-COVID MAPE (3.02%), the 24-month Holt-Winter TES was accurate 38.5% of the time post-COVID. There was no indication that the model’s accuracy improved with time post-COVID. The pediatric ED was consistently less accurate than the adult EDs, pre-COVID (5.72% ± 3.1%) and post- COVID (19.47% ± 15.73%). Using all 4 EDs, the 24-month Holt-Winter TES model’s accuracy decreased from 2.35% ± 0.47% for pre-COVID to 10.09% ± 0.62% post-COVID. (See Table 1) An example of the forecasting models is shown in Figure 1 for one adult ED site.ConclusionsTriple exponential smoothing represents an improvement in the accuracy of ED volume prediction. However, the COVID pandemic has significantly upset this balance, resulting in accuracy levels that are 4-5 times lower than they once were. Post-COVID, even the most accurate TES method was only able to meet pre-COVID predictive accuracy levels approximately 1/3rd of the time and it’s forecasting ability didn’t improve over time. Hospital and ED operations leadership need to take this into account when forecasting budgetary needs. Future work confirming this decrease in forecasting accuracy, nationally and regionally, is needed.View Large Image Figure ViewerDownload Hi-res image Download (PPT)No, authors do not have interests to disclose BackgroundForecasting emergency department (ED) volume is a critical piece of determining staffing needs. Methods of doing so have historically relied on prior volumes and subjective predictions of future volumes. Objective methods of volume prediction, such as triple exponential smoothing (TES), are more accurate. However, highly variable periods brought on by times of extreme uncertainty, such as the COVID pandemic, raise questions about predictive capacity. Forecasting emergency department (ED) volume is a critical piece of determining staffing needs. Methods of doing so have historically relied on prior volumes and subjective predictions of future volumes. Objective methods of volume prediction, such as triple exponential smoothing (TES), are more accurate. However, highly variable periods brought on by times of extreme uncertainty, such as the COVID pandemic, raise questions about predictive capacity. Study ObjectivesWe sought to determine which method of triple exponential smoothing was most accurate in predicting pre-pandemic and post-COVID (after the onset of COVID in February, 2020) ED volume, the effects of the pandemic on their accuracy, and whether such models could regain the pre-pandemic accuracy after the disruptive influence of COVID began. We sought to determine which method of triple exponential smoothing was most accurate in predicting pre-pandemic and post-COVID (after the onset of COVID in February, 2020) ED volume, the effects of the pandemic on their accuracy, and whether such models could regain the pre-pandemic accuracy after the disruptive influence of COVID began. MethodsFour methods of monthly volume forecasting were compared: simple exponential smoothing with a 24-month run-up (SES), Microsoft Excel’s AAA version of the exponential smoothing (ES) algorithm with a 24-month run-up, Holt-Winter TES using 12 and 24-month run-up. Accuracy was assessed using mean absolute percentage error (MAPE). We examined data from four emergency departments between March 2018 and March 2022, 3 adult emergency departments and 1 pediatric emergency department, with total pre-COVID annual census >250,000 patients. Four methods of monthly volume forecasting were compared: simple exponential smoothing with a 24-month run-up (SES), Microsoft Excel’s AAA version of the exponential smoothing (ES) algorithm with a 24-month run-up, Holt-Winter TES using 12 and 24-month run-up. Accuracy was assessed using mean absolute percentage error (MAPE). We examined data from four emergency departments between March 2018 and March 2022, 3 adult emergency departments and 1 pediatric emergency department, with total pre-COVID annual census >250,000 patients. ResultsIn the 24-months pre-COVID, the overall average MAPE across 4 TES methods was 2.49% ± 0.53%, while the overall average MAPE for the 24 months after the onset of the COVID pandemic was 12.56% ± 5.11%. Among all EDs, the 24-month Holt-Winter had the greatest accuracy pre-COVID, 1.58% ± 0.35% and post-COVID, 8.26% ± 3.40%. Using an acceptable standard of error limit of 1 standard deviation above the upper range pre-COVID MAPE (3.02%), the 24-month Holt-Winter TES was accurate 38.5% of the time post-COVID. There was no indication that the model’s accuracy improved with time post-COVID. The pediatric ED was consistently less accurate than the adult EDs, pre-COVID (5.72% ± 3.1%) and post- COVID (19.47% ± 15.73%). Using all 4 EDs, the 24-month Holt-Winter TES model’s accuracy decreased from 2.35% ± 0.47% for pre-COVID to 10.09% ± 0.62% post-COVID. (See Table 1) An example of the forecasting models is shown in Figure 1 for one adult ED site. In the 24-months pre-COVID, the overall average MAPE across 4 TES methods was 2.49% ± 0.53%, while the overall average MAPE for the 24 months after the onset of the COVID pandemic was 12.56% ± 5.11%. Among all EDs, the 24-month Holt-Winter had the greatest accuracy pre-COVID, 1.58% ± 0.35% and post-COVID, 8.26% ± 3.40%. Using an acceptable standard of error limit of 1 standard deviation above the upper range pre-COVID MAPE (3.02%), the 24-month Holt-Winter TES was accurate 38.5% of the time post-COVID. There was no indication that the model’s accuracy improved with time post-COVID. The pediatric ED was consistently less accurate than the adult EDs, pre-COVID (5.72% ± 3.1%) and post- COVID (19.47% ± 15.73%). Using all 4 EDs, the 24-month Holt-Winter TES model’s accuracy decreased from 2.35% ± 0.47% for pre-COVID to 10.09% ± 0.62% post-COVID. (See Table 1) An example of the forecasting models is shown in Figure 1 for one adult ED site. ConclusionsTriple exponential smoothing represents an improvement in the accuracy of ED volume prediction. However, the COVID pandemic has significantly upset this balance, resulting in accuracy levels that are 4-5 times lower than they once were. Post-COVID, even the most accurate TES method was only able to meet pre-COVID predictive accuracy levels approximately 1/3rd of the time and it’s forecasting ability didn’t improve over time. Hospital and ED operations leadership need to take this into account when forecasting budgetary needs. Future work confirming this decrease in forecasting accuracy, nationally and regionally, is needed.No, authors do not have interests to disclose Triple exponential smoothing represents an improvement in the accuracy of ED volume prediction. However, the COVID pandemic has significantly upset this balance, resulting in accuracy levels that are 4-5 times lower than they once were. Post-COVID, even the most accurate TES method was only able to meet pre-COVID predictive accuracy levels approximately 1/3rd of the time and it’s forecasting ability didn’t improve over time. Hospital and ED operations leadership need to take this into account when forecasting budgetary needs. Future work confirming this decrease in forecasting accuracy, nationally and regionally, is needed.