Fabricated steel structure systems have undergone significant advancements. However, problems associated with exterior wall panel systems have become increasingly prominent, particularly the issue of exterior wall panel shedding. Three full-scale fabricated composite shear wall exterior panel components were designed and tested under quasi-static conditions. The failure sequence and characteristics of the exterior wall panels at different displacement angles, including their out-of-plane displacement and stiffness degradation, were evaluated to assess their seismic and connection performance in different building construction methods. The results indicated that drilling holes near the centre of autoclaved lightweight concrete (ALC) boards improved the seismic performance of wall panels. Long round holes in the ALC boards allowed the dissipation of sliding energy during the initial deformation of wall panels. Furthermore, wall panels with bonded anchorage connections for rockwool boards exhibited better seismic performance than those with pure anchorage connections. To verify the accuracy of the findings, a structural finite element model was established, and the influences of various parameters, such as the position of ALC board connecting bolts, length of ALC board holes, and bonding rate of rockwool boards, on the stress and deformation of the exterior wall panel system were analyzed. Based on the test results and finite element simulation analysis, an anti-shedding connection construction method was proposed.