Adverse perceptions of neighborhood safety, aesthetics and quality including access to resources can induce stress and may make individuals more sensitive to cardiopulmonary effects of air pollution exposure. Few studies have examined neighborhood perceptions as important and modifiable non-chemical stressors of the built environment that may exacerbate effects of air pollution on cardiopulmonary health outcomes, particularly among general population based cohorts. This study examined associations between low-level chronic exposure to fine particulate matter (PM2.5) and cardiopulmonary health, and the potential mediating or modifying effects of adverse neighborhood perceptions. Using data from the Survey of the Health of Wisconsin (SHOW), 2230 non-asthmatic adults age 21–74 were included in the analyses. The overall goals of this study were to assess if individuals who experience stress from neighborhood environments in which they live were more sensitive to low levels of fine particular matter (PM2.5 μg/m3). Demographic predictors of air pollution exposure included younger age, non-White race, lower education and middle class income. After adjustments, objective lung function measures (FEV1 and FEV1 to FVC ratio) were the only cardiopulmonary health indicators significantly associated with chronic three-year annual averages of PM2.5. Among all non-asthmatics, a ten unit increase in estimated three year annual average PM2.5 exposure was significantly associated with lower forced expiratory volume (L) in one second FEV1 (β = −0.40 μg/L; 95% CI −0.45, −0.06). Among all individuals, adverse perceptions of the neighborhood built environment did not appear to statistically moderate or mediate associations. However, stratified analysis did reveal significant associations between PM2.5 and lung function (FEV1) only among individuals with negative perceptions and increased reports of neighborhood stressors. These findings included individuals who felt their neighborhoods were poorly maintained (β = −0.82; 95% CI −1.35, −0.28), experienced stress from crime (β = −0.45; 95% CI −0.94, 0.04) or reported neighborhood is not well maintained (β = −1.13, CI −2.04, −0.24). These significant associations were similar for FEV1 to FVC ratio. Multi-pronged approaches addressing both neighborhood built environment aesthetics and air pollution regulation may be necessary to protect vulnerable and susceptible individuals and reduce persistent inequalities.