The microstructural characteristics of white and gray matter in mild cognitive impairment (MCI) and the early-stage of Alzheimer's disease (AD) remain unclear. This study aimed to systematically identify the microstructural damages of MCI/AD in studies using neurite orientation dispersion and density imaging (NODDI), and explore their correlations with cognitive performance. Multiple databases were searched for eligible studies. The 10 eligible NODDI studies were finally included. Patients with MCI/AD showed overall significant reductions in neurite density index (NDI) of specific white matter structures in bilateral hemispheres (left hemisphere: -0.40 [-0.53, -0.27], P < 0.001; right: -0.33 [-0.47, -0.19], P < 0.001), involving the bilateral superior longitudinal fasciculus (SLF), uncinate fasciculus (UF), the left posterior thalamic radiation (PTR), and the left cingulum. White matter regions exhibited significant increased orientation dispersion index (ODI) (left: 0.25 [0.02, 0.48], P < 0.05; right: 0.27 [0.07, 0.46], P < 0.05), including the left cingulum, the right UF, and the bilateral parahippocampalcingulum (PHC), and PTR. Additionally, the ODI of gray matter showed significant reduction in bilateral hippocampi (left: -0.97 [-1.42, -0.51], P < 0.001; right: -0.90 [-1.35, -0.45], P < 0.001). The cognitive performance in MCI/AD was significantly associated with NDI (r = 0.50, P < 0.001). Our findings highlight the microstructural changes in MCI/AD were characterized by decreased fiber orientation dispersion in the hippocampus, and decreased neurite density and increased fiber orientation dispersion in specific white matter tracts, including the cingulum, UF, and PTR. Moreover, the decreased NDI may indicate the declined cognitive level of MCI/AD patients.