Objectives To examine whether neighborhood and household disorder matter for cognitive functioning among middle-aged and older adults and whether the disorder-cognition link is moderated by social network resources. Method Data are drawn from National Social Life, Health, and Aging Project (NSHAP) Wave 2 (N = 3198). Both neighborhood and household were considered as key residential contexts that shape one’s social life and health. Exposure to neighborhood and household disorder was measured using interviewer assessments of signs of disorder and decay, including the presence of disrepair, trash, noise, and unpleasant smells such as air pollution, in the buildings and streets in which the respondent lives. Cognitive function was measured using the survey-adapted Montreal Cognitive Assessment (MoCA-SA). Network size, network range, and frequency of interaction among network members were assessed as moderators. Results Neighborhood and household disorder were independently associated with cognitive function. However, disorder in the household appeared to have more direct associations with cognitive function than did the neighborhood when both were present. The association between household disorder and cognitive function was mitigated by network size, such that poor housing conditions were associated with lower cognitive function only for those with small social networks. Conclusion This study suggests a larger network may play a role in minimizing the negative influence of household disorder on cognitive function for middle-aged and older adults. Social policy and intervention aimed at promoting network ties may help reduce further disparities in cognitive function, especially for those vulnerable groups living in a poor-quality household.