Exercise is associated with lower prevalence and severity of pain, and is widely recommended for pain management. However, the mechanisms the exercise effect on pain remain unclear. In this study, we examined the association of exercise with pain and aimed to identify its neurobiological mediators. We utilized a baseline data of a clinical trial for people with low back pain. Participants reported pain intensity and exercise habit, as well as pain-related psychological and emotional assessments. We also obtained brain imaging data using a resting-state functional MRI and performed mediation analyses to identify brain regions mediating the exercise effect on pain. Forty-five people with low back pain (mean pain intensity = 59.6 and mean duration = 9.9 weeks) were included in this study. Participants with an exercise habit (n = 29) showed significant less pain compared to those without an exercise habit (n = 16). Mediation analysis using resting-state functional connectivity identified the left thalamus, right amygdala, and medial prefrontal cortex as statistical mediators of the exercise effect on pain (indirect effect = −0.460, 95% confidence interval = −0.767 to −0.153). In conclusion, our findings suggest that brain function of the specific regions is probably a neuro-mechanism of exercise alleviating pain.