In order to improve the teaching quality of online education, the prediction method of students' online academic performance has been studied. First, the learning analysis, artificial intelligence (AI) and other related theoretical concepts are analyzed and introduced. Then, the decision tree of single classification algorithm and the random forest (RF) of ensemble learning algorithm are analyzed, and the academic performance prediction model of online education is constructed by RF algorithm. Finally, the data of education platform is used for empirical analysis to verify the reliability and practicability of the academic performance prediction algorithm of online education. The connotation of learning analysis, the role and elements of learning analysis in the learning process are introduced. The algorithm principle of RF and decision tree is analyzed. By using the idea of information entropy and discretization, the continuous variables are processed to improve the fitting degree of the algorithm. The model is evaluated by empirical analysis, and the test accuracy of several different algorithms is compared. It is found that the prediction accuracy of the RF algorithm is more than 90%, which shows that the prediction method can help teachers and students to carry out better teaching and learning activities, so as to better improve students' ability to master knowledge. It is hoped that the result can provide some reference for the management of students' learning behavior and the optimization of teachers' teaching strategies in online learning activities