This study aims to investigate the impact performance of composite panels consisting of plain-woven Kevlar fabric and rubber matrix. A finite element (FE) model in conjunction with experimental tests was developed to simulate the response of neat fabric and composite under impact loading. Each warp and weft yarn of fabric was individually modeled and combined with rubber matrix network to form the composite. To understand the effect of natural rubber on impact resistance of Kevlar/rubber composites, two types of rubber with different formulation were considered and their mechanical properties were obtained by split Hopkinson pressure bar tests and assigned to the model. Numerical results showed good agreement with the experimental data for both neat fabric and composite. It was shown that rubber matrix improves the ballistic performance of Kevlar fabric by keeping composite flexibility. High hardness rubber matrix composite has higher energy absorption capacity compared to the low hardness rubber matrix composite, due to presence of stronger intermolecular chains. Additionally, deformation and damage mechanism of fabric and composite were investigated under impact loading. The results were presented, discussed and commented upon.