With the advances of the internet and today's innovation, it has become conceivable to conduct teaching and learning activities remotely through the online platform. Existing research says that student’s attention state and learning result are strongly correlated. However, despite its importance, this can be a challenging task, as students in general taking an online class may be in a variety of different environments and may be multitasking or distracted by other factors. This review paper aims to address these challenges by exploring the opportunities offered by machine learning techniques in attention detection for effective online teaching and learning. By leveraging machine learning algorithms, which can analyze large volumes of data, including eye-tracking, facial expressions, and body movements, we can develop robust models for attention detection in online learning environments. This paper reviews the challenges specific to online learning, such as students' attention deficits and learning styles, and highlights the limitations of current attention detection methods. Furthermore, it provides recommendations to advance attention detection technology, emphasizing the potential of machine learning to enhance attention detection technology for effective online teaching and learning.