In this work, the failure behavior of dieless clinching joints for AA5052-H32 aluminum alloy under cross tension and tensile shear loading conditions was investigated by experimental and simulation methods. Firstly, The VUMAT subroutine for the N–H shear-modified GTN model (Proposed by Nahshon and Hutchinson, a modified GTN model considering hole shear effects) was written. Subsequently, the conventional and shear parameters calibration was accomplished. A finite element model (FEM) for the failure process of dieless clinching joints based on the N–H shear-modified GTN microscopic damage model, elimination of large deformation mesh distortion and consideration of the material hardening history was established. In addition, the deformation, damage and destruction processes of the joint under two loading conditions were analyzed. And macro/microfracture analysis of the failed joints was performed to further reveal the failure behavior. The results show that the established FEM of the failure process agrees well with the experimental results and failure modes are consistent. The model accuracy in predicting the tensile and shear loads is 91.9% and 92.7%, respectively. This provides a reference for the process design and joint arrangement of the dieless clinching joints in practical applications.