Prediction of structural responses is essential for the analysis of structural behaviour subjected to dynamic loads. Existing approaches are limited in different ways. Experimental tests are expensive due to the requirement of intensive labour hours and specialised equipment. Numerical models can provide high-fidelity and labour-effective simulation of impact tests after proper validation, but it comes with high computational costs, which prohibit the usage of numerical methods for intensive and large-scale simulations in a design office. Data-driven machine learning approaches are also applied to structural response predictions, but they often exploit direct input–output mapping schemes that predict static field variables without capturing the dynamic response process. To close these gaps, we propose a novel machine learning approach based on graph neural networks (GNN) for full-field structural dynamics prediction. Our approach adopts a discretised representation of structures with an iterative rollout prediction scheme, and therefore it can simulate comprehensive spatiotemporal structural dynamics, providing the full potential for structural dynamics analysis. With several benchmark tests, it is demonstrated that our approach can generate accurate predictions of related field variables, e.g., displacement, strain, and stress, for different structures with a wide range of input parameters, such as structure geometry, impact speed and location. Additional interpolation and extrapolation tests are also conducted to show that our approach enjoys inherent generalisability and can produce satisfactory prediction even when all inputs are sampled outside of the training distribution. Our approach is also efficient and runs an order of magnitude faster than the commonly used numerical competitors. As the first attempt of using GNN for structural dynamics prediction and the result is promising, it is believed that GNN is well-suited for effective and efficient prediction of dynamic structural responses.