With the increase of software system complexity, a high requirement of the reliability, stability and security of software quality is put forward. At present, artificial intelligence (AI) defect detection adoptes machine learning technology to realize code scanning and semantic analysis on software defects. The traditional machine learning technology for software defect detection is generally based on algorithms such as BP neural network model, Naïve-Bayes model, and fingerprint identification model, etc. Regarding the features of software artificial intelligence (AI) defect detection, this paper proposes a layered detection technology based on software behavior decision tree model. Furthermore, a corresponding test environment is established to make contrast test of previously tested software. The results of the experiment shows that, with the comprehensive consideration of building time cost and false alarm rate and other factors, the artificial intelligence (AI) defect detection technology based on software behavior decision tree model is superior to other technologies.