In this work, we present the three-dimensional reconstruction of the subsurface damage (SSD) within the optical components at the level of several microns with a self-referenced spectral domain optical coherence tomography (SDOCT) system, from which the quantitative information, including the maximum depth, the cluster depth, the shape, the size and the damage density, can be acquired. First, to compare the actual maximum depths with the ones computed by the formulas for predicting the maximum depth, the theoretical and empirical formulas proposed so far were summarized. The values of the maximum depths of SSD within eight samples were then measured. It was found that the empirical relationship between the maximum depth and the abrasive size is reliable for the situation where SSD is only generated by the abrasives, and other theoretical and empirical formulas are more suitable for calculating the maximum depth of SSD within optical components with the surface roughness at several microns. For optical components with smooth surfaces, our self- referenced SDOCT system is able to provide actual values of the maximum depths. Second, the three layer structure of the ground sample can be clearly identified from the cross sectional images, which is in agreement with the three layer model. Third, the quantitative information of SSD may provide a new guidance for the study of the laser-induced damage threshold (LIDT), which is an important factor for evaluating the lifetime of optical components and is dependent on the physical properties of material. All these results are very helpful for quantitatively evaluating the quality of the optical elements and suggesting a new standard in which the quality of optical components in the manufacturing process should also be evaluated by these parameters of SSD.