Temporal contrast degradation due to mid-spatial-frequency error in chirped-pulse amplification stretcher optics is studied. Third-order cross-correlation measurements reveal a temporal peak that appears when using two different mirrors processed by magnetorheological finishing, despite an improvement in rms roughness compared to a third unprocessed mirror. Simulations based on measured power spectral density show how the actual impact on contrast is different from measurements using a typical bandwidth-limited third-order cross correlator. Strategies are proposed to avoid this type of contrast degradation while exploiting computer numerically controlled polishing techniques for enhancement of surface figure and roughness.