Towards Neuroadaptive Personal Learning Environments: Using fNIRS to Detect Changes in Attentional State

Abstract

Event Abstract Back to Event Towards Neuroadaptive Personal Learning Environments: Using fNIRS to Detect Changes in Attentional State Leah Friedman1*, Ruixue Liu1, Aria Kim2, Erin Walker3 and Erin Solovey1 1 Drexel University, College of Computing and Informatics, United States 2 Williams College, United States 3 Arizona State University, School of Computing, Informatics, and Decision Systems Engineering, United States Aims: Current ubiquity of online learning tools has led to an increased interest in developing neuroadaptive personalized learning environments using non-invasive brain imaging tools (Anderson et al., 2009; Heraz and Frasson, 2009; Keating et al., 2016; Liu, Walker, Solovey, 2017). Detecting mind wandering (MW) is one target for assessing the state of a learner. Mind wandering can be detrimental to student learning, where instead of processing external task-related information, students engage in internal non-task thoughts (Smallwood, Fishman, & Schooler, 2007; Mills et al., 2013). Examining direct brain activity during non-response periods can help disentangle focused from mind-wandering states. Functional near-infrared spectroscopy (fNIRS) has increasingly been used to assess cognitive state in real-time during interactive computing tasks (Solovey, et al., 2012, Afergan et al., 2014). fNIRS is also an ideal tool for detecting MW due to the involvement of the frontal cortex in attentional regulation through activation of the default network (Christoff et al., 2009; Gruberger et al., 2011). Prior work by Durantin et al. (2015) indicated significant differences in oxygenated hemoglobin (HbO) between mind wandering and focused states. Closely following the protocol of Durantin et al. (2015), we explore the detection of mind-wandering states from fNIRS data. Preliminary results indicate significant differences in HbO levels over periods of mind wandering as compared to focused states. This work serves as a foundation for modeling attentional state using machine learning tools to enable the creation of better and more personalized technology-based learning experiences. Methods: We conducted a study with 12 participants (8 female) between the ages of 18-41 (Mean: 28, SD: 6.48). A Sustained Attention to Response Task (SART) (Manly et al., 1999) was used to elicit mind wandering while capturing frontal lobe activity with fNIRS. A number (1-9) was presented at the center of a white screen for 0.5 seconds followed by a 1.0 second pause. Participants were instructed to press the spacebar for each stimulus other than the number 3 (target stimulus). Stimuli were presented in 6 blocks, with 10 targets and 190 non-targets presented pseudorandomly in each block such that targets were not presented back to back. Hemodynamic data was collected with an 8-channel fNIRS device manufactured by ISS, Inc., with sensors placed against the forehead and kept in place with headbands that reduced light interference. The SART program sent markers to an iMotions data acquisition platform indicating the stimulus type (target or non-target). Additionally, fNIRS raw data was sent in real-time via TCP/IP to the iMotions platform, which synchronized stimuli with the fNIRS signal. We were interested in examining the behavioral and brain activity differences between correct target responses (refrained from key press, not mind wandering) and incorrect target responses (key press, mind wandering). Raw fNIRS intensity values were converted to changes in oxygenated (HbO) hemoglobin using the modified Beer-Lambert Law (Villringer & Chance, 1997). Analysis was completed in MATLAB. Mean correct response accuracy for target and non-target stimuli was calculated along with 95% confidence intervals. For each target appearance, we extracted fNIRS data from 30 seconds prior to the target and 10 seconds after each target (referred to as target periods). Window length was informed by the work of Durantin et al. (2015). Target periods with significant artifacts were excluded using a threshold of +/- 2.0 □□M. In order to assess change in HbO over time, we used the 5 seconds before each 40-second window as a baseline. We calculated the average value in the baseline window and subtracted that value from the values in the 40-second window. Finally, we calculated the folded average of HbO change across all participants for the correct and incorrect target responses. One participant was excluded from all analysis due to lack of in-range fNIRS data. Results: Across all six blocks, mean accuracy in target stimuli responses was .7041 (SE .0372), with a 95% CI from .6223-.7860 (Fig. 1). This was significantly lower than mean accuracy for non-target stimuli (Mean: .9493, SE: .0226). In fNIRS channels 2 and 3 (located at left anterior prefrontal cortex), and 7 and 8 (located at right anterior prefrontal cortex) (Fig. 2), HbO decreased significantly in the 15 seconds preceding a missed target, and remained relatively constant in all channels during correct target hits (Fig. 3). There were no significant differences in Hb over SART errors vs. no errors. Conclusion: Consistent with prior work, we observed significantly lower response accuracy to target stimuli than non-target stimuli. Periods of poor performance during SART have been associated with mind wandering and have also been associated with activation in the medial prefrontal cortex, measured by fNIRS (Durantin, Dehais, & Delorme, 2015). Consistent with the findings of Durantin et al. (2015), our work shows that there are differences in frontal lobe blood oxygenation patterns between periods of mind wandering and periods of focus. In contrast to previous findings that showed a significant decrease in HbO immediately after the appearance of a target, our results showed a significant decrease in the 10 to 15 seconds before a target. Furthermore, where Durantin et al. noted temporal differences only in one optode near the right medial prefrontal cortex, our preliminary results indicate that frontal lobe areas in both the left and right sides of the anterior prefrontal cortex may have distinct temporal patterns of HbO change during MW, and should be considered when modeling MW using machine learning. This builds a foundation for exploring brain patterns during mind wandering in more realistic and complex tasks during learning. We aim to continue exploring this data through different signal processing methodology, including noise reduction using short separation channels, and plan to correlate behavioral and fNIRS data to subjective responses from the post-experiment survey. Using this behavioral and physiological dataset, we also plan to build and test various machine learning models to differentiate between mind wandering and focused states. Ultimately, we aim to build on these findings and detect attentional state in real time during the use of neuroadaptive personalized learning environments (Keating, et al., 2016; Liu, Walker & Solovey, 2017). Figure 1 Figure 2 Figure 3 References Afergan, D., Peck, E.M., Solovey, E.T., Jenkins, A., Hincks, S.W., Brown, E.T., Chang, R. & Jacob, R.J.K. (2014). Dynamic difficulty using brain metrics of workload. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14, 3797-3806. http://dx.doi.org/10.1145/2556288.2557230 Anderson, J., Betts, S., Ferris, J., & Fincham, J. (2009). Neural imaging to track mental states while using an intelligent tutoring system. PNAS, 107(15), 7018–7023. doi: 10.1073/pnas.1000942107 Christoff, K., Gordon, A.M., Smallwood, J., Smith, R., & Schooler, J.W. (2009). Experience sampling during fMRI reveals default network and executive system contributions to mind wandering. PNAS, 106(21), 8719-8724. doi: 10.1073/pnas.0900234106 Durantin, G., Dehais, F., & Delorme, A. (2015). Characterization of mind wandering using fNIRS. Frontiers in Systems Neuroscience, 9. doi: 10.3389/fnsys.2015.00045. Gruberger, M., Ben-Simon, E., Levkovitz, Y., Zangen, A., & Hendler, T. (2011). Towards a neuroscience of mind-wandering. Frontiers in Human Neuroscience, 5. doi: 10.3389/fnhum.2011.00056 Heraz, A., & Frasson, C. (2009). Predicting Learner Answers Correctness through Brainwaves Assessment and Emotional Dimensions. AIED, 49-56. Keating, S., Walker, E., Motupali, A., & Solovey, E. (2016, May). Toward Real-time Brain Sensing for Learning Assessment: Building a Rich Dataset. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 698-1705. http://dx.doi.org/10.1145/2851581.2892496 Liu, R., Walker, E., & Solovey, E.T. (2017). Toward Neuroadaptive Personal Learning Environments. In Proceedings of 2017 Conference on Neuroadaptive Technology. Manly, T., Robertson, I. H., Galloway, M., & Hawkins, K. (1999). The absent mind: further investigations of sustained attention to response. Neuropsychologia, 37, 661-670. doi: 10.1016/s0028-3932(98)00127-4 Mills, C., D'Mello, S., Lehman, B., Bosch, N., Strain, A., & Graesser, A. (2013). What Makes Learning Fun? Exploring the Influence of Choice and Difficulty on Mind Wandering and Engagement during Learning. Artificial Intelligence in Education, 71-80. Springer Berlin Heidelberg. Smallwood, J., Fishman, D. J., & Schooler, J. W. (2007). Counting the cost of an absent mind: Mind wandering as an underrecognized influence on educational performance. Psychonomic Bulletin & Review, 14(2), 230-236. https://doi.org/10.3758/BF03194057 Solovey, E., Schermerhorn, P., Scheutz, M., Sassaroli, A., Fantini, S. & Jacob, R. (2012). Brainput: Enhancing Interactive Systems with Streaming Fnirs Brain Input. Proc. ACM CHI, 2193-2202. doi: 10.1145/2207676.2208372 Villringer, A. & Chance, B. (1997). Non-Invasive Optical Spectroscopy and Imaging of Human Brain Function. Trends in Neuroscience, 20(10), 435-442. Doi: 10.1016/S0166-2236(97)01132-6 Keywords: mind wandering, fNIRS, sustained attention, SART, personalized learning, Intelligent tutoring systems, neuroadaptive technology Conference: 2nd International Neuroergonomics Conference, Philadelphia, PA, United States, 27 Jun - 29 Jun, 2018. Presentation Type: Oral Presentation Topic: Neuroergonomics Citation: Friedman L, Liu R, Kim A, Walker E and Solovey E (2019). Towards Neuroadaptive Personal Learning Environments: Using fNIRS to Detect Changes in Attentional State. Conference Abstract: 2nd International Neuroergonomics Conference. doi: 10.3389/conf.fnhum.2018.227.00021 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 03 Apr 2018; Published Online: 27 Sep 2019. * Correspondence: Ms. Leah Friedman, Drexel University, College of Computing and Informatics, Philadelphia, PA, 19104, United States, lmf323@drexel.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Leah Friedman Ruixue Liu Aria Kim Erin Walker Erin Solovey Google Leah Friedman Ruixue Liu Aria Kim Erin Walker Erin Solovey Google Scholar Leah Friedman Ruixue Liu Aria Kim Erin Walker Erin Solovey PubMed Leah Friedman Ruixue Liu Aria Kim Erin Walker Erin Solovey Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.