The modality effect reverses in a virtual reality learning environment and influences cognitive load

Abstract

When Virtual Reality (VR) is used to present learning content, the three-dimensional image is often not sufficient and additional text is required, either in the form of visual or auditory text. Such additional instructional text could be particularly beneficial due to the many visual stimuli and the risk of visual working memory overload. Relieving working memory by using both channels in the audio-visual presentation, there might be more cognitive capacity available to invest in germane processes for learning. In addition, the image and the text can be processed simultaneously, supporting deeper learning processes. Based on the modality effect, it is therefore assumed that an audio-visual presentation in VR, compared to a visual-only presentation, can lead to higher learning outcomes (recall, comprehension and transfer). In a between-subjects design we analysed the modality principle in VR with 61 subjects (69% female). We hypothesized that when verbal information in VR is given auditorily instead of visually, it leads to overall higher learning outcomes. For intrinsic cognitive load we assumed no differences, but lower extraneous cognitive load and higher germane cognitive load for the audio-visual condition. However, results show a reverse modality effect. The visual-only condition showed higher learning outcome scores for recall, comprehension and transfer. As expected, the conditions showed no differences with regard to the intrinsic cognitive load. However, we also found no differences in extraneous cognitive load. Contrary to our hypothesis we found higher germane cognitive load for the visual-only condition compared to the audio-visual condition, which may be attributed to repeated reading, reading strategies or other self-regulatory strategies. Future research could focus on the use of strategies or evaluate relevant process data.