Generating realistic three-dimensional (3D) yarn-level models for fabrics has become a significant research topic due to the complexity of fabric structures and the requirement for high-quality models with no interpenetration between yarns. The generation process of idealized geometric models leads to inaccurate descriptions of yarn contact, specifically the interpenetrations and spurious voids between yarns. A geometric optimization procedure was developed to address the defects in the idealized models, involving a series of geometry-driven operations, such as the shrinking and expansion of yarn volume and the straightening of the yarn centerline, to obtain accurate and consistent fabric models for Finite Element Analysis. During the geometric optimization, a method for yarn deformation based on position-based dynamics (PBD) was applied to simulate the real deformation during yarn interweaving, ensuring no interpenetrations between yarns while preserving yarn volume. The accuracy of the model is validated by comparison with the real fabric images. Although the methods involved in the procedure are all geometric, their deformation results have realistic physical effects. In addition, the procedure can be applied to various woven fabric structures.