Olive (Olea europaea L.) is one of the major oil fruit tree crops worldwide. However, the mechanisms underlying olive fruit growth remain poorly understood. Here, we examine questions regarding the interaction of endoreduplication, cell division, and cell expansion with olive fruit growth in relation to the final fruit size by measuring fruit diameter, pericarp thickness, cell area, and ploidy level during fruit ontogeny in three olive cultivars with different fruit sizes. The results demonstrate that differences in the fruit size are related to the maximum growth rate between olive cultivars during early fruit growth, about 50 days post-anthesis (DPA). Differences in fruit weight between olive cultivars were found from 35 DPA, while the distinctive fruit shape became detectable from 21 DPA, even though the increase in pericarp thickness became detectable from 7 DPA in the three cultivars. During early fruit growth, intense mitotic activity appeared during the first 21 DPA in the fruit, whereas the highest cell expansion rates occurred from 28 to 42 DPA during this phase, suggesting that olive fruit cell number is determined from 28 DPA in the three cultivars. Moreover, olive fruit of the large-fruited cultivars was enlarged due to relatively higher cell division and expansion rates compared with the small-fruited cultivar. The ploidy level of olive fruit pericarp between early and late growth was different, but similar among olive cultivars, revealing that ploidy levels are not associated with cell size, in terms of different 8C levels during olive fruit growth. In the three olive cultivars, the maximum endoreduplication level (8C) occurred just before strong cell expansion during early fruit growth in fruit pericarp, whereas the cell expansion during late fruit growth occurred without preceding endoreduplication. We conclude that the basis for fruit size differences between olive cultivars is determined mainly by different cell division and expansion rates during the early fruit growth phase. These data provide new findings on the contribution of fruit ploidy and cell size to fruit size in olive and ultimately on the control of olive fruit development.