Understanding the complex correlation between the built environment and subway passenger flow can provide unique insights for the development of transportation operations and urban coordination policies. Few studies have systematically analyzed the rationality of selecting built environment variables and further explored the non-linear relationships. In this study, we integrated various sources of built environmental factors and developed an interpretable machine learning analysis framework using backward elimination extreme gradient boosting and SHapley Additive exPlanations (SHAP) values analysis (BE-XGBoost-SHAP). The framework was validated by analyzing passenger flows during the morning peak, non-peak, and evening peak periods at the station level. The research results indicate that there are significant differences between built environment factors and the time-varying passenger flow. Land use characteristics significantly dominate across all three temporal periods. The importance of other variable types in relation to passenger flows varies significantly across the three time periods. It is worth noting that the relationships between all variables and passenger flow at different time periods are non-linear, with the majority displaying threshold effects. Compared with the gradient boosting decision tree (GBDT) and ordinary least squares (OLS) models, the proposed interpretive framework performs better as regards R-square, root mean square error (RMSE), and mean absolute error (MAE) metrics. This study offers valuable insights, elucidating the pivotal land use attributes that notably affect passenger flow, the significance of varied built environment factors across distinct time spans, and the acknowledgment of non-linearities and threshold effects within these relationships. These findings are imperative for urban planning and the enhancement of station area design.