Our ability to represent temporal, spatial, and numerical information is critical for understanding the world around us. Given the prominence of quantitative representations in the natural world, numerous cognitive, neurobiological, and developmental models have been proposed as a means of describing how we track quantity. One prominent theory posits that time, space, and number are represented by a common magnitude system, or a common neural locus (i.e., Bonn & Cantlon in Cognitive Neuropsychology, 29(1/2), 149-173, 2012; Cantlon, Platt, & Brannon in Trends in Cognitive Sciences, 13(2), 83-91, 2009; Meck & Church in Animal Behavior Processes, 9(3), 320, 1983; Walsh in Trends in Cognitive Sciences, 7(11), 483-488, 2003). Despite numerous similarities in representations of time, space, and number, an increasing body of literature reveals striking dissociations in how each quantity is processed, particularly later in development. These findings have led many researchers to consider the possibility that separate systems may be responsible for processing each quantity. This review will analyze evidence in favor of a common magnitude system, particularly in infancy, which will be tempered by counter evidence, the majority of which comes from experiments with children and adult participants. After reviewing the current data, we argue that although the common magnitude system may account for quantity representations in infancy, the data do not provide support for this system throughout the life span. We also identify future directions for the field and discuss the likelihood of the developmental divergence model of quantity representation, like that of Newcombe (Ecological Psychology, 2, 147-157, 2014), as a more plausible account of quantity development.