The lug type joint serves as a critical connecting structure in aerospace vehicles, allowing for convenient assembly and disassembly, and transfer the load during service. However, cracks often initiate at the stress concentration location near the lugs hole, posing a significant threat to the safety of the aircraft structure. Therefore, evaluating the stress intensity factors for complex lug structures is of great significance. In this study, a physics knowledge-based neural network method of calculating the stress intensity factors of attachment lugs is proposed. Three types of cracked lug structures are analyzed: through-thickness crack in straight lug, quarter elliptical corner crack in straight lug and quarter elliptical corner crack in tapered lug. Various complex influencing factors in actual situations are also considered, such as the impact of the interference fit between the pin and the lug, as well as different loading directions and crack positions of tapered lugs. The stress intensity factor dataset generated by 3D finite element method, and then the neural network with its powerful learning ability and prior physical knowledge are used to achieve accurate fitting of the data. The results demonstrate that incorporating physical knowledge features significantly improves the model performance of the neural network. This method can be extended to analyze crack in structures with arbitrary geometry and complex loading conditions.