This study aims to introduce the GLMM tree and show its usefulness in educational research. The GLMM tree model is an extension of MOB to be applied to multilevel data, which detects subgroups with differential effects to estimate the fixed-effect in each subgroup and the random effect by the cluster to which each observation belongs. In this study, we identified the differential effects by detecting subgroups of afterschool programs and private tutoring participation in mathematics achievement of second grade high school students. Using the GLMM trees, students were divided into 16 and 15 subgroups respectively according to the participation of afterschool programs and the private tutoring. Also, cognitive and affective variables such as pre-mathematics achievement, interest in mathematics were selected as nodes. For both treatments, it was confirmed that the fixed-effect was estimated differently for each subgroup. Based on the results, we compared the differential effects of participation in afterschool programs and private tutoring and factors selected as nodes of models, and discussed the potential of the GLMM tree model in educational research.