Abstract Missing data is a significant challenge in medical research. In longitudinal studies of Alzheimer’s disease (AD) where structural magnetic resonance imaging (MRI) is collected from individuals at multiple time points, participants may miss a study visit or drop out. Additionally, technical issues such as participant motion in the scanner may result in unusable imaging data at designated visits. Such missing data may hinder the development of high-quality imaging-based biomarkers. To address the problem of missing MRI data in studies of AD, we introduced a novel 3D diffusion model specifically designed for imputing missing structural MRI (Recovery of Missing Neuroimaging using Diffusion models (ReMiND)). The model generates a whole-brain image conditional on a single structural MRI observed at a past visit or conditional on one past and one future observed structural MRI relative to the missing observation. The performance of models was compared with two alternative imputation approaches: forward filling and image generation using variational autoencoders. Experimental results show that our method can generate 3D structural MRI with high similarity to ground-truth images at designated visits. Furthermore, images generated using ReMiND show relatively lower differences in volume estimation between the imputed and observed images compared to images generated by forward filling or autoencoders. Additionally, ReMiND provides more accurate estimated rates of atrophy over time in important anatomical brain regions than the two comparator methods. Our 3D diffusion model can impute missing structural MRI data at a single designated visit and outperforms alternative methods for imputing whole-brain images that are missing from longitudinal trajectories.