Duration perception can be heavily distorted owing to repetitive exposure to a relatively long or short sensory event, often causing a duration aftereffect. Here, we used a novel procedure to show that adaptations to both single and average durations produced the duration aftereffect. Participants completed a duration reproduction task (Experiment 1) or a duration category rating task (Experiment 2) after long-term adaptations to a stimulus of medium duration and to stimuli of averagely medium duration. We found that adaptations to both single and average durations resulted in duration aftereffects. The simultaneously recorded functional magnetic resonance imaging (fMRI) data revealed that the reduction in neural activity due to long-term adaptation to single duration was observed in the right supramarginal gyrus (SMG) of the parietal lobe, while adaptation to average duration resulted in fMRI adaptations in the left postcentral gyrus (PCG) and middle cingulate gyrus (MCG). At the individual level, the magnitude of the behavioral aftereffect was positively correlated with the magnitude of fMRI adaptation in the right SMG after adaptation to single duration, while there were no significantly positive correlations between the behavioral aftereffect and fMRI adaptations in the left PCG and MCG. These results suggest that there are different neural mechanisms for aftereffects caused by adaptations to single and average durations.