The semi-rigid connected steel frame has good displacement ductility and energy dissipation capacity, and the interaction between the traditional steel frame and the filled wall is the critical factor affecting its seismic performance. In this paper, for the semi-rigid steel frame, the composite wall panel and the frame are separated by foam concrete mortar, and the effective connection is achieved by the tensioned steel bar. The premature brittle failure of composite wall panels can be prevented using friction energy dissipation between wall panels. By using ABAQUS simulation method, a semi-rigid steel frame composite wall is established. The failure mode, hysteresis curve and skeleton curve of simulation and test are compared and analyzed, and the reliability of the model is proved. The finite element model with the different number of wall panels is established to analyze its influence on the seismic performance of the structure. The results show that the frame structure realizes the effective connection between the composite wall panel and the concrete-filled steel tube frame, which jointly resists the earthquake action and reduces the damage of the earthquake action to the filled wall. With the increase in the number of composite wall panels, the ultimate load decreases gradually. The initial stiffness of the four layers of wall panels is more significant and decreases rapidly. When the wall panel has three or four layers, the energy dissipation capacity of the specimen is the strongest. The two are relatively close, stable at 24.48, and the increase is 5.15% compared with the second layer of the wall panel, and the increase is 12.72% compared with the third layer of the wall panel.