This study investigates the compression deformation behavior of an Mg-Gd-Y-Nd-Zr alloy at temperatures ranging from 293 K to 573 K and strain rates ranging from 293K to 573K and strain rates ranging from 1000 s−1 to 2100 s−1 using a split Hopkinson pressure bar. A modified Johnson-Cook (JC) constitutive model and a backpropagation neural network (BPNN) model based on the improved whale optimization algorithm (IWOA) are established. Four statistical metrics, including correlation coefficient (R), mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE), are employed to evaluate the predictive accuracy of the two models. The findings indicate that the flow stress of the alloy is sensitive to strain, strain rate, and temperature. Increasing strain and strain rate or decreasing deformation temperature results in higher flow stress. Error calculations revealed that the modified Johnson-Cook constitutive model has an R of 0.98731, a MAPE of 7.3653%, a MAE of 19.6305 MPa, and a RMSE of 26.5704 MPa. In contrast, the model established using the IWOA-BPNN has an R of 0.99996, a MAPE of 0.58894%, a MAE of 1.2671 MPa, and a RMSE of 1.7709 MPa. The IWOA-BPNN model demonstrates higher accuracy and accurately predicts the flow stress of the alloy.