Obstructive sleep apnea (OSA) is associated with cognitive impairment and brain structural alterations, but longitudinal outcomes are understudied. To examine the associations of OSA with cognition and white matter (WM) integrity over a 4-year period. This prospective cohort study was conducted in a community-based adult population among participants who had both baseline (2011-2014) and 4-year follow-up (2015-2018) polysomnography, diffusion tensor imaging, and cognitive assessment data. Participants with neurological disorders, anomalous findings on brain magnetic resonance imaging, or inadequate quality of the evaluations were excluded. Data were analyzed from March to November 2021. Participants were categorized depending on the presence vs absence of OSA at baseline and follow-up polysomnographic analysis. The main outcomes were proportional changes over a 4-year period in neuropsychological performance and WM integrity. The neuropsychological assessment battery included verbal and visual memory, verbal fluency, Digit Symbol-coding, Trail Making Test-A, and Stroop Test. WM integrity was assessed by fractional anisotropy, axial, and radial diffusivity. To examine interactions with age and sex, participants were subgrouped by age older than 60 years vs 60 years or younger and men vs women. A total of 1998 individuals were assessed for eligibility, and 888 were excluded based on exclusion criteria, leaving 1110 participants (mean [SD] age, 58.0 [6.0] years; 517 [46.6%] men) for analysis, including 458 participants grouped as OSA-free, 72 participants with resolved OSA, 163 participants with incident OSA, and 417 participants with persistent OSA. Incident OSA was associated with altered WM integrity and with concomitant changes in sustained attention compared with participants without OSA (eg, change in Digit Symbol-coding test score, -3.2% [95% CI, -5.2% to -1.2%]). Participants with resolved OSA showed better visual recall at the follow-up (change in Visual Reproduction-immediate recall test, 17.5% [95% CI, 8.9% to 26.1%]; change in Visual Reproduction-delayed recall test, 33.1% [95% CI, 11.3% to 54.9%]), with concordant changes in diffusion parameters at the relevant anatomic areas. In the older group only (age >60 years), persistent OSA was associated with altered WM integrity and cognition (eg, Visual Reproduction-recognition test: β = -24.2 [95% CI, -40.7 to -7.7]). Sex also was associated with modifying the association of OSA with WM integrity of the left posterior internal capsule, the left genu of corpus callosum, and the right middle cerebellar peduncle only in men and with cognition only in women (eg, Visual Reproduction-immediate recall test: β = 33.4 [95% CI, 19.1 to 47.7]). These findings suggest that dynamic changes in OSA status were significantly associated with WM integrity and cognition, which varied by age and sex. It is possible that adequate interventions for OSA could better preserve brain health in middle to late adulthood.