Hole position accuracy and perpendicularity are significant factors in aerospace drilling. This study presents a novel compensation method with reference holes to reduce the hole position error and initial normal deviation for automated drilling systems. The hole-pose error and related generalized addition and subtraction methods are first defined using quaternions. Subsequently, approaches for accurately measuring the pose of the reference hole and calibrating the coordinate system are proposed. On this basis, the position error and initial normal error are estimated by bilinear interpolation based on the error similarity on the two-dimensional manifold, where the manifold coordinates are obtained by Hermite surface reconstruction. Subsequently, the hole-pose error is compensated for by generalized addition. For the initial normal modification, a compensation strategy utilizing the data of processed adjacent holes for interpolation was adopted. In addition, for situations in which it is inconvenient to arrange real reference holes on the product, a compensation approach employing virtual reference holes was designed. Finally, simulations and experiments are conducted, and the results demonstrate that the position error and initial normal deviation are effectively reduced by at least 42.25% and 74.68% with the proposed method, which further ensures that the indicators meet the requirements of aerospace drilling.