This study aims to examine whether hypovitaminosis D was associated with cognitive impairment among chronic kidney patients with different level of albuminuria. This population-based cross-sectional study was conducted on elderly (over 60 years old) with urine albumin to creatinine ratio (UACR) ≥ 30 mg/g from 2011 to 2014 in the US National Health and Nutrition Examination Survey (NHANES). Cognitive function was assessed by the Consortium to Establish a Registry for Alzheimer’s Disease Word List Learning (CERAD). Subjects were divided into 2 groups according to the absence or presence of cognitive impairment and a propensity score matching (PSM) was further conducted. The association was assessed with Spearman correlation and logistic regression analysis. The positive association of 25-hydroxyvitamin D3 (25(OH)D3) and cognitive score was presented. PSM analysis revealed that a higher level of 25(OH)D3 correlated to a better cognitive function in CKD patients with albuminuria, especially in patients with 30 mg/g ≤ UACR < 300 mg/g. This study indicated that a low 25(OH)D3 level was associated with poor cognitive performance, especially in patients with microalbuminuria. Thus, early diagnosis of vitamin D insufficiency and an effective intervention might be a useful therapeutic strategy to prevent cognitive decline in patients with the progression of renal dysfunction.