This study was designed to examine age effects on various auditory perceptual skills using a large group of listeners (155 adults, 121 aged 60–88 years and 34 aged 18–30 years), while controlling for the factors of hearing loss and working memory (WM). All subjects completed 3 measures of WM, 7 psychoacoustic tasks (24 conditions) and a hearing assessment. Psychophysical measures were selected to tap phenomena thought to be mediated by higher-level auditory function and included modulation detection, modulation detection interference, informational masking (IM), masking level difference (MLD), anisochrony detection, harmonic mistuning, and stream segregation. Principal-components analysis (PCA) was applied to each psychoacoustic test. For 6 of the 7 tasks, a single component represented performance across the multiple stimulus conditions well, whereas the modulation-detection interference (MDI) task required two components to do so. The effect of age was analyzed using a general linear model applied to each psychoacoustic component. Once hearing loss and WM were accounted for as covariates in the analyses, estimated marginal mean thresholds were lower for older adults on tasks based on temporal processing. When evaluated separately, hearing loss led to poorer performance on roughly 1/2 the tasks and declines in WM accounted for poorer performance on 6 of the 8 psychoacoustic components. These results make clear the need to interpret age-group differences in performance on psychoacoustic tasks in light of cognitive declines commonly associated with aging, and point to hearing loss and cognitive declines as negatively influencing auditory perceptual skills.