The manufacture of high-precision surfaces is the foundation of building high-performance optical systems. For over 50 years, the tolerance for optical surfaces has been specified by the root-mean-square (rms) or peak-to-valley (PV) value over the entire surface geometry. However, different regions on optical surfaces do not contribute equally to image quality and, thus, can tolerate different levels of errors. A global tolerance described by a single or few parameters cannot precisely provide the manufacturing requirements of each region on the surface, which may result in unnecessary accuracy specifications for surfaces. Furthermore, the components with the same PV or rms figure errors can produce different imaging qualities; however, this difference cannot be distinguished by the conventional figure of merit. To address these problems, a framework that includes a local tolerance model and a quality merit function for optical surfaces is proposed. The local tolerance model can provide an accurate tolerance for each region on the surface so the targeted wave aberration requirements are met during components manufacturing. More importantly, the proposed merit function closely ties the surface figure error to imaging performance, e.g., the findings can explain that the component with lower geometric accuracy may produce better imaging quality. This framework provides new insights into optical design, manufacture, and metrology and especially paves the way for the manufacture of high-precision large-aperture systems.