Auditory temporal processing declines with age, leading to potential deleterious effects on communication. In young normal-hearing listeners, perceptual rate discrimination is rate limited around 300 Hz. It is not known whether this rate limitation is similar in older listeners with hearing loss. The purpose of this study was to investigate age- and hearing-loss-related rate limitations on perceptual rate discrimination, and age- and hearing-loss-related effects on neural representation of these stimuli. Younger normal-hearing, older normal-hearing, and older hearing-impaired listeners performed a pulse-rate discrimination task at rates of 100, 200, 300, and 400 Hz. Neural phase locking was assessed using the auditory steady-state response. Finally, a battery of non-auditory cognitive tests was administered. Younger listeners had better rate discrimination, higher phase locking, and higher cognitive scores compared to both groups of older listeners. Aging, but not hearing loss, diminished neural-rate encoding and perceptual performance; however, there was no relationship between the perceptual and neural measures. Higher cognitive scores were correlated with improved perceptual performance, but not with neural phase locking. This study shows that aging, rather than hearing loss, may be a stronger contributor to poorer temporal processing, and cognitive factors such as processing speed and inhibitory control may be related to these declines.