Impact toughness has a strong size effect, which restricts the accurate evaluation of material toughness and safety of structural design. However, our understanding of the internal mechanical mechanism that influences the size effect remains incomplete. To fill this gap, this paper adopts the method of combining experiment, theory, and simulation. From the point of view of the dependence of plastic and fracture behavior of material on stress state, the internal mechanical mechanism of the size effect of impact toughness (αk) was studied. Firstly, the Charpy impact tests of different sizes were carried out on Al 6061. αk was divided into cracking initiation toughness (αini) and cracking growth toughness (αgro). The variation of αini and αgro with specimen size was analyzed. Then, through a series of tests, the influence of stress state on the plastic and fracture behavior of Al 6061 was studied. To simulate the whole process of the Charpy impact test, an uncoupled 3D fracture model was proposed. Subsequently, a stress state dependent yield function, modified Voce hardening law, and the proposed fracture model were used to simulate the Charpy impact tests. The predicted results were in good agreement with the tests. Finally, the internal mechanical mechanisms of the size effects of αini and αgro were analyzed. The results show that: (1) The αini and αgro have significant size effects, and there are significant differences between them. (2) The internal mechanical mechanism of the size effect of αini was revealed; that is, the dependence of the plastic behavior of the material on the stress state is the internal root cause of the dependence of αini on the specimen size. (3) The internal mechanical mechanism of the size effect of αgro was revealed; that is, the dependence of the plastic behavior of the material on the stress state and the fracture behavior of the material under ultra-high stress triaxiality is the internal fundamental reason affecting the size effect of αgro. (4) This study reveals the internal complex and intimate relationships between plastic behavior, fracture behavior, toughness behavior, stress state, and the size effect, providing a valuable reference for accurate and comprehensive understanding, evaluation, and selection of metal materials.