This paper investigates the effect of cumulative damage on the energy absorption capability of plain-woven fabric systems. The cumulative damage was generated by creating pre-damaged holes, aiming to simulate the post-impacted sample targets. Ballistic impact tests were performed to characterize the ballistic protection of the pre-damaged and undamaged systems, and FE simulation was used to study the underlying mechanisms of energy dissipation. The results showed that the pre-damaged systems appear to be less protective than the undamaged system in most cases. The ballistic performance of the fabrics is more sensitive to the pre-damaged holes that are located on the primary yarns than that on the secondary yarns. When the primary yarns are broken, the holes are detrimental to the evolution of the transverse deflection but are beneficial in increasing the amount of energy dissipated by frictional sliding during the yarn pull-out process. Angle-plying the constituent layers of multi-ply panels is found to be useful in minimizing property degradation. It was also determined that the angle-plied systems absorb less energy than the aligned system regardless of the existence of the pre-damaged holes.