This study aims to examine the effects of different instructional interventions (i.e., autonomy-supportive, controlling, or a combination thereof) on student motivation and academic research performance. Here, a total of 328 s-year Chinese students majoring in mathematics completed the Academic Motivation Scale (AMS) survey before and after the intervention involving an array of research-related assignments. Learners in the autonomy-supportive group showed a lower level of amotivation and higher intrinsic motivation compared to the pretest; the differences were 2.62 and 11.95 points. Their level of extrinsic motivation remained unchanged. The controlling intervention had the opposite effect, causing the mean extrinsic motivation score to increase by 10.25 points. The hybrid approach showed significant improvement in all three scales: amotivation (−3.82 points), extrinsic motivation (+4.35 points), and intrinsic motivation (+7.13 points). The autonomy-supportive teaching style corresponds to lower scores in mathematical modeling (72.30), linear algebra (74.62), and numerical analysis (77.65). High standard deviations indicate that research activity scores are spread out over a wide range. The controlling instructional approach yielded higher scores (83.95, 81.68, and 81.88 points, respectively) with the smallest standard deviations. Learners in the hybrid group scored the highest (86.98, 82.30, and 82.90 points, respectively), with moderate standard deviations indicating a well-balanced performance. The findings indicate that a hybrid approach, which combines autonomy support with instructor control, is the most effective in enhancing both intrinsic and extrinsic motivation among students, as well as improving their academic performance. The implications for higher education policy include the necessity of developing and implementing hybrid learning models that cater to the individual needs of students while providing the requisite level of support and oversight.