Studies about changes in hippocampal volumes in subjects with bipolar disorder (BD) have been contradictory. Since the number of manic episodes and hospitalization has been associated with brain changes and poor cognitive outcomes among BD patients, we have hypothesized that these variables could clarify this issue. We stratified subjects with BD in early (BD-Early), intermediate (BD-intermediate) and late (BD-Late) stages as a function of number of manic episodes and prior hospitalization. Then, we compared their hippocampal volumes and California Verbal Learning Test-II (CVLT-II) scores with healthy controls (HC) using the general linear model. A total of 173 subjects were included in the study (112 HC, 15 BD-Early, 30 BD-Intermediate, and 16 BD-Late). We found a significant group effect on hippocampus volume (F(3,167) = 3.227, p = 0.024). Post-hoc analysis showed that BD-Late subjects had smaller hippocampus than HC (p = 0.017). BD-Early and BD-Intermediate subjects showed no significant difference in hippocampus volume compared to HC and BD-Late subjects. The CVLT trial 1 to 5 scores were significantly different across the groups (F(3,167) = 6.371, p < 0.001). Post-hoc analysis showed that BD-Intermediate (p = 0.006) and BD-Late (p = 0.017) subjects had worse memory performance during immediate recall than HC, while the performance difference between BD-Early subjects and HC was not significant (p = 0.208). These findings add to the notion that BD is a neuroprogressive disorder with brain changes and cognitive impairment according to prior morbidity (number of manic episodes and hospitalization). Also, they suggest that hippocampus is a brain marker and a potential therapeutic target for patients at late stage.