Abstract Study Objectives Few studies of middle-aged and older adults have examined the association between age and sleep using objective sleep measures. We examined these associations in adults aged ≥40 years using wrist actigraphy, and investigated whether these associations differed by sex and race. Methods Participants were 468 cognitively normal adults aged ≥40 years enrolled in the Baltimore Longitudinal Study of Aging who completed wrist actigraphy. We used Generalized Least Squares Models to examine the associations of age with actigraphic sleep parameters, including total sleep time (TST), sleep efficiency, sleep onset latency, and wake after sleep onset (WASO). We conducted interaction and stratification analyses to test whether cross-sectional age-sleep associations were modified by sex and race. Results In analyses adjusting for sex, body mass index, and individual medical conditions, older age was associated with longer TST from ages 40–70 that plateaued after age 70. Older age also was associated with lower sleep efficiency, longer sleep onset latency, and greater WASO. In men only, after age 70, older age was associated with shorter TST, lower sleep efficiency, longer onset latency, and greater WASO. However, we did not observe any significant interactions of race with age. Conclusions Older age was associated with longer TST from ages 40 to 70 and with poorer sleep quality after age 40, and these relationships might vary by sex. Future studies with larger sample sizes are needed to investigate mechanisms that may account for sex differences in the observed age-sleep associations.