Accurate prediction of short-term inbound passenger flow at high-speed railway stations is of great significance for the refined operation of stations, the formulation of emergency plans, and the provision of intelligent services. The arrival of passengers traveling on the same train at the same station shows a similar pattern, which is called the departure passenger arrival pattern (DPAP). The short-term inbound passenger flow at the station is composed of the short-term inbound passenger flow of all waiting trains within the same period. Inspired by this, this paper develops an ensemble prediction model based on the time series decomposition modeling strategy to introduce the DPAP to the short-term inbound passenger flow prediction at stations. Firstly, we propose a new framework for studying the DPAP to calculate the fitted station short-term inbound passenger flow, which is only affected by the DPAP. During this process, we find that 7minutes is the optimal time granularity. Secondly, based on the singular spectrum analysis, we prove that the DPAP is the determining factor affecting the station short-term inbound passenger flow. Finally, we propose an ensemble prediction model that considers the DPAP to achieve short-term inbound passenger flow prediction at stations. The model consists of two parts: the deterministic and stochastic components prediction, where the former is predicted by the fitted station short-term inbound passenger flow, and the latter is achieved by the combination of historical stochastic components and weather type with the help of the Seq2Seq model based on time attention mechanism. Using real inbound passenger flow data, we compare the proposed model with 13 benchmark models and the results show that under different training and prediction steps, our model achieves optimal prediction performance, whether in all-day period and the busiest period of the station. Through further ablation experiments, it has been proven that the introduction of the DPAP effectively improves the prediction accuracy. Our model can provide scientific support for the intelligent operation of stations and the refined management of passenger flow.