When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Background and Aims: Understanding how immersive technologies like AR, VR, and MR can transform education by enabling interactive and experiential learning. By addressing adoption challenges and highlighting successful case studies, it aims to help educators and policymakers effectively integrate these technologies while promoting equitable access and informed decision-making. Thus, this paper aims to explore the role of AR, VR, and MR in enhancing learning experiences. Methodology: The methodology ensures a thorough review by taking a systematic approach to data collection and analysis from various sources, with a focus on recent advances in immersive technologies. By combining qualitative and quantitative analyses, the paper aims to provide a comprehensive overview of how AR, VR, and MR affect educational practices and outcomes. Results: Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) improve education by making difficult concepts more accessible and engaging. AR makes abstract concepts tangible, VR provides immersive experiences for deeper understanding, and MR connects the digital and physical worlds. These technologies work together to create interactive learning environments that meet a variety of learning needs, promote critical thinking, and encourage creativity. Conclusion: Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) all improve education by making complex concepts more understandable and engaging through interactive experiences. These technologies create a dynamic learning environment by combining AR's tangibility, VR's immersion, and MR's integration of the digital and physical worlds.

The purpose of this study is to determine perception of postgraduate Computer Education and Instructional Technologies (CEIT) students regarding the concepts of Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), Augmented Virtuality (AV) and Mirror Reality; and to offer a table that includes differences and similarities between these concepts. This study also aims to determine the likelihood of CEIT postgraduate students for using the said concepts in education. In this context, the frequently used reality concepts in the CEIT field have been examined from the perspective of the participants and in terms of the following traits: frequency of potential use, perceived usefulness, and perceived effectiveness. The phenomenological method was used in this qualitative study. 10 CEIT graduate students have been the participants of this research; with 4 of these pursuing a PhD and 6 pursuing a Master's Degree. 14 open-ended questions related to AR, VR, MR, AV and Mirror Reality concepts were used throughout semi-structured and face-to-face interviews in order to collect data. Findings show that AR and VR are the most familiar concepts. Participants have several misconceptions about the reality concepts but the least amount of misconception was associated with AR and VR. Most of the participants had no idea about MR and none of them had any idea about Mirror Reality. Findings refer that VR is the most frequently used kind of reality owing to the fact that it can be developed and implemented more easily and there are several AR studies because of its current popularity.

The purpose of this research is to study the elements and processes of digital teacher design. Generative AI can be processed visually for navigation in conjunction with visual intelligence through mixed reality technology, thereby adding practical learning and character design for animation purposes. This research focuses on the application of generative AI technology combined with visual processing to develop digital teachers with intelligent visual abilities using mixed reality technology, to enable the creation of more dimensional and interactive practical learning experiences, especially in the field of character design for animation. Using visual intelligence allows learners to learn better using a hands-on approach, which improves learners' design and creative skills in a hybrid reality and digital environment. Education for digital teacher design can use generative AI that can be processed visually for navigation with the aid of visual intelligence. This study was conducted in the form of an observational study supported by a literature review to study the elements and processes for modeling the design of digital teachers. Use questionnaires to assess the modeling process and digital teacher design process. The researcher evaluated the design process using 3 practical teachers, 3 gen-ai experts, 3 visual intelligence experts, and 3 MR technology experts, for a total of 12 people. The evaluation led to an average value of 4.69 ± 0.40, with an average value of 4.69 ± 0.40 (the highest quality). Generative AI that can process visuals for navigation with visual intelligence through mixed reality technology enhances the practical learning of character design for animation, resulting in learners being able to effectively develop important skills and be highly involved in the learning process. The use of mixed reality technology enhances the immersive and engaging learning experience for learners. It is effective and can increase the learning of character design for animation work, which can be applied in teaching and learning management in the future.

Humans love cats, at least half of humans. Quantum cats are even nicer to our minds. Thus, Generative AI (Gen AI) created and animated 3D cats of Schrödinger in mixed reality, which are close to infinite fun. In this project, we bring Schrödinger’s famous thought experiment to life by combining the power of Gen AI, Quantum Computing concepts, and Mixed Reality (MR). The result is an Interactive 3D experience that enables users to observe, influence, and explore the behaviors of AI-generated quantum cats within an immersive environment, promoting education, entertainment, and engagement.The goal is both playful and provocative: to offer a hands-on metaphor for quantum uncertainty and superposition through intuitive interactions. The project also highlights how generative AI accelerates 3D content creation, enabling rapid iteration and rich visual storytelling. By integrating these technologies, we aim to demonstrate a new kind of experiential interface – one that is scientific, artistic, and, unsurprisingly, fun.The demo includes a Gen AI-powered parts with 3D and Animation, Interactivity, Quantum and Spatial computing components. More details are presented below; the system architecture and implementation details will also be included in the final paper. Recent developments in Generative AI promise to accelerate the production speed of nearly everything, including interactive 3D experiences. Practically, Text-to-3D, Image-to-3D, and Multiview Image-to-3D provide production-ready results at least in terms of an edge Mixed Reality device. Furthermore, one can achieve not only the models but also the characters, rig them, and animate them. With the acceptable results in generation and animation, characters can be created faster than achievable by human artists, initially, and then even quicker, with better quality.We conducted a state-of-the-art literature review for 3D character generation. We performed experiments with the most promising tools and models, including Meshy 4 and 5 preview, Luma Genie, Tripo 2.5, Trellis, and Hunyuan 2.0. The generation results were visually acceptable with all the selected 3D models and tools. We identified two models that returned visually acceptable results compatible with the rig and animation pipelines. The models from Meshy and Tripo appeared to be compatible with Mixamo and Animate Anything, respectively. Our Interactive MR-enhanced demo offers a fresh perspective on the foundations of quantum computing, making them easy to grasp and even “touch”. The core idea derives from the iconic Schrödinger’s cat thought experiment with a twist. While the original setup focuses on the principle of superposition and involves a single cat to illustrate the behavior of a quantum system, which can exist in a superposition of multiple states simultaneously, we introduce additional cats to illustrate the concept of quantum entanglement. This phenomenon manifests itself in peculiar long-range correlations that cannot be accounted for classically and results in states of composite quantum systems, which cannot be factored into individual states of the constituting subsystems. The key premise of the experiment remains - the cats are placed into the box with poison vials triggered via a spontaneous mechanism of radioactive decay, introducing a true uncertainty element. As long as the box is closed to the observer, the cats can be viewed as both alive and dead at the same time. Within our virtual environment, additional separating walls can be shifted inside the box, allowing us to “entangle” the cats. Namely, if the cats are located in the same box compartment impacted by the same poison vial, upon inspection, they will both be found either alive or dead. The inspection itself is performed by opening the box and illustrates the measurement in quantum mechanics, which results in the collapse of the superposition to a single definitive state. In the light of quantum computing, cats in the proposed demo represent a system of qubits, while user interaction with cats’ results in unitary transformations applied to the state of this system. This is visualized using a dedicated widget containing a quantum circuit for state preparation. Therein, superposition is generated utilizing Hadamard gates, and the entanglement is introduced through the action of two-qubit CNOT gates. The state evaluation procedure is performed in a simulated manner. To implement the human-computer interaction system, we utilized the Magic Leap Unreal Engine SDK. During the development phase, we experimented with various input methods — both controller-based and hand-tracking — and ultimately chose hand-tracking for its ability to deliver a more intuitive, natural, and immersive user experience.We briefly presented our research and experimental work in Gen AI 3D and the HCI domain, incorporating the flavor of quantum physics.

Considering both the transformative opportunities and challenges presented by generative AI (GenAI) in academic writing, effectively integrating GenAI into the academic setting becomes a significant need requiring prioritization. Yet, there is limited understanding regarding the nature of interactions between different types of students, what behavioral patterns students exhibit during a student-GenAI interaction (SAI) on a given task, and how these different SAI patterns relate to the actual writing task performance. This study, therefore, aimed to identify SAI patterns of academic writing tasks depending on students’ level of AI literacy and examine the differences in academic writing performance between the identified SAI patterns. Drawing from the combination of three data sources, including think-aloud protocols, screen-recordings, and chat histories between 36 Chinese graduate students and a GenAI writing system, epistemic network analysis (ENA) was used to reveal the distinctive SAI patterns of students with different levels of AI literacy. The study found that students with a high level of AI literacy exhibited a collaborative approach to SAI, actively accepting GenAI’s suggestions and engaging GenAI in meta-cognitive-related activities such as planning, whereas students with a low level of AI literacy demonstrated much less interaction with GenAI in completing their writing tasks, instead choosing to ideate and evaluate independently. In addition, the Wilcoxon rank-sum (Mann-Whitney U) test was conducted to assess the writing task performance of the two AI literacy groups. Findings revealed statistical differences in all evaluation rubrics (content, structure/organization, expression). This study offers implications for the design and implementation of GenAI agents in writing tasks and the pedagogy of GenAI-assisted instruction.

The chapter explores the transformative impact of generative AI on higher education, highlighting its potential to revolutionize learning, teaching, and research methods. Key points include: a) Generative AI tools enable personalized, flexible, and interactive learning experiences. b) AI-driven technologies enhance teaching strategies through customized education and real-time lesson modification. c) Digital teaching assistants and AI-powered chat advisors provide scalable student support. d) In research, generative AI simplifies complex tasks, improving efficiency. e) The text suggests frameworks for integrating AI into curricula, emphasizing hybrid instructional methods. f) Ethical considerations, including fairness, data privacy, and addressing biases, are crucial. g) Challenges include academic dishonesty risks and intellectual property issues. h) The importance of developing critical digital literacy skills is emphasized. i) Future directions involve the convergence of AI with augmented and virtual reality for immersive learning experiences.

The study examines the problem of using augmented and virtual reality in the process of blended learning in general secondary education. Analysis of recent research and publications has shown that the use of augmented and virtual reality in the educational process has been considered by scientists. However, the target group in these studies is students of higher education institutions. Most of the works of scientists are devoted to the problem of introducing augmented reality into the traditional educational process. At the same time, the use of augmented and virtual reality technologies in the process of blended learning remains virtually unexplored. The study analyzes the meaning of the concept of "blended learning". The conceptual principles of blended learning are considered. It has been found that scholars differ in their understanding of the concept of "blended learning". Sometimes researchers distinguish between the components of blended learning: full-time and online learning. The study presents the special advantages of blended learning and the taxonomy of blended learning. It was found that there are some difficulties in implementing blended learning. The article outlines the practical use of virtual and augmented reality. The definition of augmented and virtual reality is given. The mixed reality is considered as a separate kind of notion. Separate applications of virtual and augmented reality that can be used in the process of blended learning are considered (MEL Chemistry VR; Anatomyou VR; Google Expeditions; EON-XR). As a result of the study, the authors propose possible ways to use augmented reality in the educational process. The model of using augmented and virtual reality in blended learning in general secondary education institutions was designed. It consists of the following blocks: goal; teacher’s activity; forms of education; teaching methods; teaching aids; organizational forms of education; pupil activity and results. Based on the model, the methodology of using augmented and virtual reality in blended learning in general secondary education was developed. The methodology contains the following components: target component, content component, technological component and resultant component. The methodology is quite universal and can be used for any subject in general secondary education. The types of lessons in which it is expedient to use augmented (AR) and virtual reality(VR) are determined. Recommendations are given at which stage of the lesson it is better to use AR and VR tools (depending on the type of lesson).

In the field of surgery, major changes that have occurred include the advent of minimally invasive surgery and the realization of the importance of the ‘systems’ in the surgical care of the patient (Pierorazio & Allaf, 2009). Challenges in surgical training are two-fold: (i) to train the surgical residents to manage a patient clinically (ii) to train them in operative skills (Singh & Darzi,2013). In Pakistan, another issue with surgical training is
 that we have the shortest duration of surgical training in general surgery of four years only, compared to six to eight years in Europe and America (Zafar & Rana, 2013). Along with it, the smaller number of patients to surgical residents’ ratio is also an issue in surgical training. This warrants formal training outside the operation room. It has been reported by many authors that changes are required in the current surgical training system due to the significant deficiencies in the graduating surgeon (Carlsen et al., 2014; Jarman et al., 2009; Parsons, Blencowe, Hollowood, & Grant, 2011). Considering surgical training, it is imperative that a surgeon is competent in clinical management and operative skills at the end of the surgical training. To achieve this outcome in this challenging scenario, a resident surgeon should be provided with the opportunities of training outside the operation theatre, before s/he can perform procedures on a real patient. The need for this training was felt more when the Institute of Medicine in the USA published a report, ‘To Err is Human’ (Stelfox, Palmisani, Scurlock, Orav, & Bates, 2006), with an aim to reduce medical errors. This is required for better training and objective assessment of the surgical residents. The options for this training include but are not limited to the use of mannequins, virtual patients, virtual simulators, virtual reality, augmented reality, and mixed reality. Simulation is a technique to substitute or add to real experiences with guided ones, often immersive in nature, that reproduce substantial aspects of the real world in a fully interactive way. Mannequins, virtual simulators are in use for a long time now. They are available in low fidelity to high fidelity mannequins and virtual simulators and help residents understand the surgical anatomy, operative site and practice their skills. Virtual patients can be discussed with students in a simple format of the text, pictures, and videos as case files available online, or in the form of customized software applications based on algorithms. In a study done by Courtielle et al, they reported that knowledge retention is increased in residents when it is delivered through virtual patients as compared to lecturing (Courteille et al., 2018).But learning the skills component requires hands-on practice. This gap can be bridged with virtual, augmented, or mixed reality. There are three types of virtual reality (VR) technologies: (i) non-immersive, (ii) semi-immersive, and (iii) fully immersive. Non-immersive (VR) involves the use of software and computers. In semi-immersive and immersive VR, the virtual image is presented through the head-mounted display(HMD), the difference being that in the fully immersive type, the virtual image is completely obscured from the actual world. Using handheld devices with haptic feedback the trainee can perform a procedure in the virtual environment (Douglas, Wilke, Gibson, Petricoin, & Liotta, 2017). Augmented reality (AR) can be divided into complete AR or mixed reality (MR). Through AR and MR, a trainee can see a
 virtual and a real-world image at the same time, making it easy for the supervisor to explain the steps of the surgery. Similar to VR, in AR and MR the user wears an HMD that shows both images. In AR, the virtual image is transparent whereas, in MR, it appears solid (Douglas et al., 2017). Virtual augmented and mixed reality has more potential to train surgeons as they provide fidelity very close to the real situation and require fewer physical resources and space compared to the simulators. But they are costlier, and affordability is an issue. To overcome this, low-cost solutions to virtual reality have been developed. It is high time that we also start thinking on the same lines and develop this means of training our surgeons at an affordable cost.

Recent advancements in artificial intelligence (AI) have shown significant potential in the medical field, although many applications are still in the research phase. This paper provides a comprehensive review of advancements in augmented reality (AR), mixed reality (MR), and virtual reality (VR) for surgical applications from 2019 to 2024 to accelerate the transition of AI from the research to the clinical phase. This paper also provides an overview of proposed databases for further use in extended reality (XR), which includes AR, MR, and VR, as well as a summary of typical research applications involving XR in surgical practices. Additionally, this paper concludes by discussing challenges and proposed solutions for the application of XR in the medical field. Although the areas of focus and specific implementations vary among AR, MR, and VR, current trends in XR focus mainly on reducing workload and minimizing surgical errors through navigation, training, and machine learning-based visualization. Through analyzing these trends, AR and MR have greater advantages for intraoperative surgical functions, whereas VR is limited to preoperative training and surgical preparation. VR faces additional limitations, and its use has been reduced in research since the first applications of XR, which likely suggests the same will happen with further development. Nonetheless, with increased access to technology and the ability to overcome the black box problem, XR’s applications in medical fields and surgery will increase to guarantee further accuracy and precision while reducing risk and workload.

Informed consent in healthcare requires patients to have a sufficient understanding of their upcoming procedure before deciding to proceed. Unfortunately, education prior to a surgical procedure is constrained by barriers including poor health literacy, language barriers, one-sided dialogue during consultations, anxiety, and knowledge retention. Extended reality (XR), which includes virtual reality (VR), augmented reality (AR), and mixed reality (MR) has the potential to improve informed consent processes by creating an immersive, interactive, and multimodal sensory experience that supports patient education. The purpose of the study was to review the extant literature on the effectiveness of XR technology in improving patient education, a vital component of informed consent. We screened fifty-two articles and ten relevant papers from PubMed, Scopus, and Compendex, which were included in the review based on our eligibility criteria. We found that VR and AR proved effective in enhancing patient education in eight studies, and thus improving informed consent processes. MR was not utilized in the studies reviewed. The studies were conducted in several countries and positives findings were reported from a broad range of clinical settings and procedures. Though further investigation is needed, this is a promising finding that may encourage health systems to implement similar interventions prior to procedures. The review also provided an overview of the existing XR technology utilized for patient education such as a downloadable mobile application with a virtual chatbot character, and an environment designed to simulate the MRI patient’s perspective. These applications provide immersive and interactive experiences when paired with a head mounted headset such as Google VR Cardboard. The findings also revealed that XR tools are customizable and can be tailored to specific surgical procedures, which makes the potential of implementation applicable to a broader range of settings.

The current revolution of generative artificial intelligence is transforming global dynamics which is also essential to petroleum engineers for effectively completing technical tasks. Henceforth the main aim of this study is to investigate the application of generative AI techniques for improving the efficiency of petroleum reservoir management. The outcomes of this study will help in developing and implementing generative AI algorithms tailored for reservoir management tasks, including reservoir modeling, production optimization, and decision support. In this study generative AI technique is employed to integrate with augmented reality (AR) to enhance reservoir management. The methodology involves developing a generative AI model to simulate pore-scale fluid flow, validated against experimental data. AR is utilized to visualize and interact with the simulation results in a real-time, immersive environment. The integration process includes data preprocessing, model training, and AR deployment. Performance metrics such as accuracy, computational efficiency, and user interaction quality are evaluated to assess the effectiveness of the proposed approach in transforming traditional reservoir management practices. The developed generative AI model demonstrated high accuracy in simulating pore-scale fluid flow, closely matching experimental data with a correlation coefficient of 0.95. The AR interface provided an intuitive visualization, significantly improving user comprehension and decision-making efficiency. Computational efficiency was enhanced by 40% compared to traditional methods, enabling real-time simulations and interactions. Moreover, it was observed that Users found the AR-driven approach more engaging and easier to understand, with a reported 30% increase in correct decision-making in reservoir management tasks. The integration of generative AI with AR allowed for dynamic adjustments and immediate feedback, which was particularly beneficial in complex scenarios requiring rapid analysis and response. Concludingly, the combination of generative AI and AR offers a transformative approach to reservoir management, enhancing both the accuracy of simulations and the effectiveness of user interactions. This methodology not only improves computational efficiency but also fosters better decision-making through immersive visualization. Future work will focus on refining the AI model and expanding the AR functionalities to cover a broader range of reservoir conditions and management strategies. This study introduces a novel integration of generative AI and augmented reality (AR) for reservoir management, offering a pioneering approach to pore-scale fluid flow simulation. By combining high-accuracy AI-driven simulations with real-time, immersive AR visualizations, this methodology significantly enhances user interaction and decision-making efficiency. This innovative framework transforms traditional practices, providing a more engaging, efficient, and accurate tool for managing complex reservoir systems.

Mixed reality (MR) provides new opportunities for creative and innovative learning. MR supports the merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real-time (MacCallum & Jamieson, 2017). The MR continuum links both virtual and augmented reality, whereby virtual reality (VR) enables learners to be immersed within a completely virtual world, while augmented reality (AR) blend the real and the virtual world. MR embraces the spectrum between the real and the virtual; the mix of the virtual and real worlds may vary depending on the application. The integration of MR into education provides specific affordances which make it specifically unique in supporting learning (Parson & MacCallum, 2020; Bacca, Baldiris, Fabregat, Graf & Kinshuk, 2014). These affordance enable students to support unique opportunities to support learning and develop 21st-century learning capabilities (Schrier, 2006; Bower, Howe, McCredie, Robinson, & Grover, 2014).
 
 In general, most integration of MR in the classroom tend to be focused on students being the consumers of these experiences. However by enabling student to create their own experiences enables a wider range of learning outcomes to be incorporated into the learning experience. By enabling student to be creators and designers of their own MR experiences provides a unique opportunity to integrate learning across the curriculum and supports the develop of computational thinking and stronger digital skills. The integration of student-created artefacts has particularly been shown to provide greater engagement and outcomes for all students (Ananiadou & Claro, 2009).
 
 In the past, the development of student-created MR experiences has been difficult, especially due to the steep learning curve of technology adoption and the overall expense of acquiring the necessary tools to develop these experiences. The recent development of low-cost mobile and online MR tools and technologies have, however, provided new opportunities to provide a scaffolded approach to the development of student-driven artefacts that do not require significant technical ability (MacCallum & Jamieson, 2017). Due to these advances, students can now create their own MR digital experiences which can drive learning across the curriculum.
 
 This presentation explores how teachers at two high schools in NZ have started to explore and integrate MR into their STEAM classes. This presentation draws on the results of a Teaching and Learning Research Initiative (TLRI) project, investigating the experiences and reflections of a group of secondary teachers exploring the use and adoption of mixed reality (augmented and virtual reality) for cross-curricular teaching. The presentation will explore how these teachers have started to engage with MR to support the principles of student-created digital experiences integrated into STEAM domains.

The growing attention towards immersive technologies such as augmented reality (AR), virtual reality (VR), mixed reality (MR), extended reality (XR), and the metaverse are revolutionizing cultural heritage education and tourism. Such technologies offer immersive and interactive experiences that transform the user’s exploration of museums, cultural heritage sites, educational content, and historical landmarks. This article presents a structured framework that addresses the advancement and application of these technologies in cultural heritage education to improve user experience, learning, emotional connection, and motivation. To further explore recent trends, issues, and opportunities, the article offers a comprehensive overview of the impact of state-of-the-art immersive technology on user experience within heritage education environments . The study also outlined standard questionnaires and effective methodologies for user experience evaluations. Furthermore, the article addresses the influence of standards and guidelines recommended by standardized bodies and organizations on XR and metaverse applications. It discussed how these standards and recommendations can play a role in setting protocols that shape the development of immersive heritage education environments. Finally, we introduce an architecture model for XR and metaverse applications that can assess developers, researchers, and stakeholders to enable immersive and interactive educational experiences, bridging geographical and physical barriers. This research is intended to help academic and industry stakeholders understand the integration of digital heritage preservation tools and user experience standards critical to advancing educational engagement in cultural heritage.

Engineering Technology education stands at the precipice of a profound transformation driven by the integration of Generative Artificial Intelligence (Generative AI). Incorporating generative AI into engineering technology education can enhance the learning experience, foster creativity, and prepare students for the increasingly AI-driven field of engineering. It allows students to focus on problem-solving, innovation, and the application of engineering principles, while AI handles routine tasks and provides valuable insights and guidance. However, it's crucial to strike a balance and ensure that students also develop a deep understanding of the fundamental concepts and skills that underlie the technology they are using. This abstract provides an overview of a study that explores the transformative potential and application of Generative AI in engineering technology education. Generative AI refers to a category of AI models and algorithms that have the ability to generate new content that is similar to, or in some cases indistinguishable from, content created by humans. These AI systems are designed to generate data, such as text, images, audio, and more, based on patterns and knowledge they've learned from large datasets during their training. Integrating Generative AI into engineering education can be a valuable way to prepare students for the future and equip them with skills relevant to emerging technologies. This study explores how Generative AI can revolutionize the traditional pedagogical approach by enabling the development of interactive lab experiences, simulations, and practical exercises to integrate and create a greater understanding of AI capabilities. These innovations create authentic learning environments, equipping students with hands-on experience and honing their problem-solving skills. This study also scrutinizes the ethical implications and challenges tied to the incorporation of Generative AI in education. It emphasizes the need for unbiased AI algorithms and responsible usage while calling for comprehensive training and support for instructors in harnessing this innovative technology. In conclusion, this study intends to demonstrate that harnessing Generative AI in engineering technology education has the potential to revolutionize the way students learn in addition to preparing students to leverage these technologies for innovative engineering solutions and equip them with valuable skills that are increasingly in demand in various engineering domains.