AbstractThis design‐based research project explored how various design features of AR‐based learning environments (ARLE) influence students' mathematics self‐efficacy and learning of kinematics. Specifically, five ARLEs with different design features were developed and implemented with 136 seventh‐grade students in two rounds. Data were gathered from pre‐ and post‐mathematics self‐efficacy tests, interviews, classroom observation records and reflective journals, to identify which design features promote or hinder students' learning and self‐efficacy. Results revealed that adopting a well‐designed ARLE is a viable way to promote students' learning and mathematics self‐efficacy, and the success of ARLE lies in (1) coherent tasks with proper time for self‐exploration, progressive levels, useful problem contexts and a certain degree of open‐endedness, and (2) sufficient technological affordances with strengthened authenticity, clarity and integrated media presentation, as well as engaging interactions with decent flexibility, embodied actions and feedback. These results validated the applicability of the proposed checklist for ARLE designers and practitioners in their design and implementation of ARLEs, within the domain of mathematics, physics and beyond. Practitioner notesWhat is already known about this topic AR‐based learning environments (ARLEs) have demonstrated substantial advantages in STEM education, but ARLEs can be a drawback if they are not adequately designed. ARLE may facilitate students' mathematics self‐efficacy, and relevant design features are underexplored. Design‐based research sustains interventions combining empirical research and theory‐driven design of innovative learning environments, for example, ARLEs. What this paper adds Demonstrates the potential of ARLEs that can be used in classrooms to support STEM learning. Identifies a collection of effective and ineffective design features of ARLEs that promote or hinder students' STEM learning and mathematics self‐efficacy. Tackles the challenges for the design of ARLEs for classrooms by taking AR technological affordances and the pedagogical perspective together. Implications for practice and/or policy Five ARLEs for STEM topics were developed and implemented in secondary schools. Evidence of positive impacts of ARLEs on students' learning and mathematics self‐efficacy. Our research produces useful guidelines for designers and practitioners in their development and implementation of ARLEs in real classrooms.