Accuracy of patient setup for accelerated partial breast irradiation (APBI) using intensity modulated radiation therapy (IMRT) is challenging due to the low visibility of the lumpectomy cavity in planar X-ray images and CBCT. Surgical clips, chest wall, or skin surface are typically used as surrogates of the lumpectomy cavity. The correlation of these three surrogates with the lumpectomy cavity needs to be validated.Twenty-five patients (12 left-sided and 13 right-sided) who underwent IMRT-APBI at our institution were included in the study. Daily CT-on-rail images were acquired. Sixteen patients had surgical clips. A total of 125 setup CT images were retrospectively analyzed, comparing the isocenter shifts aligning to the ipsilateral chest wall-lung interface, skin surface, and surgical clips with the isocenter shifts directly aligning to the lumpectomy cavity. The alignment differences (setup errors) in lateral, longitudinal, and vertical directions and 3D vector differences were reported. Two-sided, paired t-test was used for statistical analysis. The correlation of the setup errors with the volume of the ipsilateral breast, the lateral distance of the lumpectomy cavity center to the body midline, the minimum distance from the lumpectomy cavity to the chest wall, and the minimum distance to skin was analyzed using Pearson's correlation.3D vector errors for the surgical clips, chest wall, and skin surface-based alignment were 1.8+/-1.0, 4.7+/-2.9, and 3.6+/-2.1 mm, respectively. Alignment using surgical clips outperformed alignment using the chest wall (P < 0.001) and skin surface (P < 0.001). The alignment using skin surface outperformed the alignment using chest wall (P < 0.001). The 95% confidence interval for the surgical clip-based alignment was (-2.5, 2.9) mm in lateral, (-2.1, 1.6) mm in vertical, and (-2.5, 3.2) mm in longitudinal directions. For chest wall-based alignment, the 95% confidence interval was (-8.3, 5.1) mm in lateral, (-14.4, 6.7) mm in vertical, and (-6.4, 7.7) mm in longitudinal directions. For skin surface-based alignment, the 95% confidence interval was (-7.8, 6.1) mm in lateral, (-3.9, 4.2) mm in vertical, and (-5.8, 5.6) mm in longitudinal directions. To ensure coverage for 99% of the treatment fractions, a uniform setup margin of 5, 15, and 10 mm was needed for the surgical clips, chest wall, and skin surface-based alignment, respectively. Setup error using chest wall alignment has a moderate correlation (R2 = 0.45) with the minimum distance from the lumpectomy cavity to the chest wall. Breast volume, lateral distance to the midline, and minimum distance to skin all had low correlation (R2 < 0.3) with setup errors.For accelerated partial breast irradiation, patient setup is most accurate by using surgical clips with a mean residual error of 1.8 mm due to deformation of the lumpectomy cavity. Using skin surface or chest wall as surrogates for setup requires larger planning margins than the surgical clips.