An increase in number of neurons is presumed to underlie the enhancement of cognitive abilities in brain evolution. The evolution of human cognition is then expected to have accompanied a prolongation of net neural-processing time due to the accumulation of processing time of individual neurons over an expanded number of neurons. Here, we confirmed this prediction and quantified the amount of prolongation in vivo, using noninvasive measurements of brain responses to sounds in unanesthetized human and nonhuman primates. Latencies of the N1 component of auditory-evoked potentials recorded from the scalp were approximately 40, 50, 60, and 100 ms for the common marmoset, rhesus monkey, chimpanzee, and human, respectively. Importantly, the prominent increase in human N1 latency could not be explained by the physical lengthening of the auditory pathway, and therefore reflected an extended dwell time for auditory cortical processing. A longer time window for auditory cortical processing is advantageous for analyzing time-varying acoustic stimuli, such as those important for speech perception. A novel hypothesis concerning human brain evolution then emerges: the increase in cortical neuronal number widened the timescale of sensory cortical processing, the benefits of which outweighed the disadvantage of slow cognition and reaction.