Infrared micro-lens arrays (MLAs) are widely used in advanced optical systems due to their advantages such as low focusing depth and high sensitivity. Elliptical vibration cutting (EVC) is a promising approach for the fabrication of MLAs on infrared brittle materials. However, the mechanism of ductile machining of MLAs prepared by EVC has not been fully elucidated so far. In this paper, based on the vibration intermittent cutting characteristics and the transient material removal state, a ductile machining model of MLAs on brittle material by EVC was established. This model effectively calculates the subsurface damage of the machined surface and realizes the prediction of the critical depth for ductile machining of MLAs. Furthermore, the concave micro-lenses were prepared on single crystal silicon by EVC and ordinary cutting (OC) to verify this model. The results demonstrated that EVC could significantly enhance the critical depth by approximately 4.3 times compared to OC. Microstructural surface damage predominantly occurs at the exit side of the tool cutting. This proposed model accurately predicts the actual critical depth, with an average error of about 7.5%. Additionally, elevating the amplitude in the depth of cut direction could increase the critical depth, but a larger amplitude would inhibit the increase of the critical depth. This study contributes to a better understanding of ductile machining of microstructure on brittle materials and facilitates the process optimization of MLAs fabrication using EVC.