Response speeds in simple decision-making tasks begin to decline from early and middle adulthood. However, response times are not pure measures of mental speed but instead represent the sum of multiple processes. Here we apply a Bayesian diffusion model to extract interpretable cognitive components from raw response time data. We apply our model to cross-sectional data from 1.2 million participants to examine age differences in cognitive parameters. To efficiently parse this large dataset, we apply a Bayesian inference method for efficient parameter estimation using specialized neural networks. Our results indicate that response time slowing begins as early as age 20, but this slowing was attributable to increases in decision caution and to slower non-decisional processes, rather than to differences in mental speed. Slowing of mental speed was observed only after approximately age 60. Our research thus challenges widespread beliefs about the relationship between age and mental speed.