Surface ozone has become one of the most important air pollutants in recent years, posing a huge threaten to human health. The accurate prediction of ozone pollution, especially when ozone concentration exceeds permissible standard, has become an important research topic in the field of air quality management, where data-driven methods are widely used. Among these methods, the long short-term memory neural network (LSTM) has received significant attention in ozone forecasting for its ability in modeling long-term time dependence. However, from the time series perspective, short-term nonstationary characteristics resulted from periodic trends of ozone data have not been sufficiently considered by LSTM. To address this issue, we propose a novel deep learning-based prediction method named multi-order difference embedded LSTM (MDELSTM). Through statistical analysis of ozone time series, a multi-order difference strategy is employed to extract the periodic information. By embedding it into LSTM layers, a brand-new deep learning prediction model with the ability to extract both long-term stationary information and short-term periodic information is constructed for ozone forecasting. The proposed method is verified through the Los Angeles air quality dataset between 2013 and 2017. The prediction accuracy of ozone concentration over the next hour is improved compared to other state-or-the-art methods in terms of mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R2). To further illustrate the ability of the proposed method in predicting high-level ozone concentration, new evaluation indicators oriented on situations where the ozone concentration exceeds permissible standard are proposed. The results demonstrate its application prospect in hourly ozone pollution prediction and prevention.