It remains uncertain whether long-term use of benzodiazepines is associated with age-related cognitive decline, and if cognitive ability in early life is the driver of any association. This study examines the association of cognitive ability in young adulthood with later use of benzodiazepines and explores whether the use of benzodiazepines during adult life is associated with cognitive decline in late midlife. The study samples include cognitive tests on the Børge Priens Prøve (BPP) from the conscription board examination (age 19 years) from 335 513 men born 1949-1961 and data from re-examinations of 5183 men 44 years later. Cognitive decline was defined as the difference between scores at the conscription board and the re-examination. Information on purchases of benzodiazepines was obtained from the Danish National Prescription Registry, 1995-2022. Associations were analysed using Cox proportional hazards and linear regression. In total, 120 911 (36%) men purchased benzodiazepines during a follow-up of 20 years. Lower cognitive scores in young adulthood were associated with a higher risk of initiating benzodiazepines (hazard ratio [95% CI] = 0.71[0.68-0.75]). Men with the highest cumulative use of benzodiazepines had larger cognitive decline (β-coefficient [95% CI] = -1.66 [-2.09 to -1.23] BPP scores) compared with never users. Current benzodiazepine users showed a larger cognitive decline than never users (β-coefficient [95% CI] = -2.42[-3.18 to -1.66] BPP scores) and this partially explained the above association. These estimates for cognitive decline were relatively small and may lack clinical relevance. Low cognitive ability increases the risk of benzodiazepine use in adulthood and cognitive decline is more pronounced in those with the highest benzodiazepine use compared with never-use, but the difference lacks clinical significance.