Real-time neural network prediction for handling two-hands mutual occlusions

Hands deserve particular attention in virtual reality (VR) applications because they represent our primary means for interacting with the environment. Although marker-based motion capture with inverse kinematics works adequately for full body tracking, it is less reliable for small body parts such as hands and fingers which are often occluded when captured optically, thus leading VR professionals to rely on additional systems (e.g. inertial trackers). We present a machine learning pipeline to track hands and fingers using solely a motion capture system based on cameras and active markers. Our finger animation is performed by a predictive model based on neural networks trained on a movements dataset acquired from several subjects with a complementary capture system. We employ a two-stage pipeline, which first resolves occlusions, and then recovers all joint transformations. We show that our method compares favorably to inverse kinematics by inferring automatically the constraints from the data, provides a natural reconstruction of postures, and handles occlusions better than three proposed baselines.

Hands in virtual reality applications represent our primary means for interacting with the environment. Although marker-based motion capture with inverse kinematics (IK) works for body tracking, it is less reliable for fingers often occluded when captured with cameras. Many computer vision and virtual reality applications circumvent the problem by using an additional system (e.g. inertial trackers). We explore an alternative solution that tracks hands and fingers using solely a motion capture system based on cameras and active markers with machine learning techniques. Our animation of fingers is performed by a predictive model based on neural networks, which is trained on a movements dataset acquired from several subjects with a complementary capture system (inertial). The system is as efficient as a traditional IK algorithm, provides a natural reconstruction of postures, and handles occlusions.

Primary school students often find it difficult to differentiate two dimensional and three-dimensional geometric shapes. Taking advantage of the ability of Virtual Reality (VR) and Augmented Reality (AR) to visualize 3D objects, we evaluate the potential of VR and AR technologies for teaching the lesson of geometric solids to primary school children. To the best of our knowledge there are no previous cases in the literature describing a comparative evaluation of VR and AR technologies in education, and more specifically in the field of mathematics for primary school children. An experimental evaluation was staged to test the following hypothesis: Hypothesis 1: VR and AR applications make the teaching of mathematics more interactive and interesting and they also contribute to a more efficient learning and understanding of mathematical concepts. Hypothesis 2: The use of VR applications is more effective when compared to AR applications for mathematics teaching activities. For the needs of the experimental evaluation, we designed a lesson plan comprised of three activities: Classification of shapes into solid or plane shapes, identification of solid shapes appearing in a typical city environment, and classification of solid shapes. The lesson plan was implemented based on the traditional method that utilizes printed material, three related VR and three AR applications. The developed VR and AR applications for the current research do not require specialized equipment. For the AR applications, the users only need to use their mobile device or tablet and for VR applications they only need to use a mobile phone and low-cost virtual reality glasses. As part of the study 30 fourth, fifth and sixth class primary school students were divided equally into the control group who used the traditional teaching method, and the AR and VR groups who used AR and VR applications respectively. Participants were provided with questionnaires before (pre-test) and after the test (post-test) to measure factors such as user attention, presence, enjoyment, science knowledge, auditory knowledge, and visual knowledge. According to the results, new technologies in education in the form of virtual and augmented reality improve interactivity and student interest in mathematics education, contributing to more efficient learning and understanding of mathematical concepts when compared to traditional teaching methods. No significant difference was observed between virtual and augmented reality technologies with regards to the efficiency of the methods that contribute to the learning of mathematics, suggesting that both virtual and augmented reality display similar potential for educational activities in Mathematics. Based on statistical evidence Hypothesis 1 was accepted and Hypothesis 2 was rejected. The current research is one of the first attempts ever to compare VR and AR technologies for Mathematics teaching activities in primary school. The findings of our research can provide valuable feedback to educators and developers who plan to use or develop VR or AR technologies for educational activities. Given that these days VR and AR applications, like the ones used in the experimental evaluation, do not require highly specialized equipment, the introduction of AR and VR based activities both for in-class and extra curriculum activities provide a promising way for more efficient Mathematics training activities.

This paper includes a research review in five bibliographic databases on using the application of virtual reality (VR) and augmented reality (AR) in physical and occupational therapy (POT). This literature review addresses five research questions and two sub‐research questions. A total of 36 relevant studies were selected in the review based on the defined keywords and inclusion‐exclusion criteria. The primary motivation for using the application of VR and AR in POT is that it is accurate, involves higher patient participation, and requires less therapy recovery time. The standard software tool used is the Unity 3D game engine, and the common device used is the Oculus Rift HMD. Various applications of VR and AR consist of different VR environments and AR contents used in POT. Post‐stroke rehabilitation, rehabilitation exercises, pain management, mental and behavioral disorders, and autism in children are the main aspects addressed through the VR and AR environments. Literature review indicates that questionnaires, interviews, and observation are the primary metrics for measuring therapy's effectiveness. The study's findings show positive results such as reduced treatment time, nervousness, pain, hospitalization period, making therapy enjoyable and encouraging, improved quality of life, and focus on using the application of VR and AR in POT. This review will be relevant to researchers, VR and AR application designers, doctors, and patients using the application of VR and AR in POT. Further research addressing multiple participants with clinical trials, adding new VR environments and AR content in VR and AR applications, including follow‐up sessions, and increasing training sessions while using the application of VR and AR in POT are recommended.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Having a virtual avatar in virtual reality(VR) applications can increase immersion. However, most of the existing VR basic sets only contain limited tracking sensors to provide effective information for full-body motion capture and reconstruction. Therefore, we present a lightweight real-time full-body motion capture and reconstruction prototype system only using head-mounted display and motion controllers. The system constructs HeadIK and HandIK which are optimized based on the human inverse kinematics (IK) algorithm to reconstruct the upper body motion accurately. And we address the animation blending space and FootIK to achieve natural smooth lower body motion that is not neglected by blindfolded users. Meanwhile, the animation retargeting algorithm ensures that our system can adapt to various body shapes. Experiments show that our method can reconstruct the natural movement of the whole body using only consumer-grade VR basic sets. We demonstrate that our system can enhance immersion in the VR experience and can be used in most VR applications.

The rapid development of virtual and augmented reality technologies is currently taking place in almost all spheres of activity. Elements of virtual and augmented reality are used in such areas as education, medicine, transport, gaming, tourism and others. The active spread of these technologies causes the emergence of special competencies in the IT labor market and, as a result, the formation of new professions.Many Russian universities are training students in IT training areas. Specialization in the development of computer games and virtual reality applications has begun recently. The provision of practical classes is accompanied by specific tasks, which gives students the opportunity to improve the use of software and technical devices.The relevance of the research is determined by the current demand for the use of the latest technologies by IT developers in the field of creating computer games. Today, technologies that provide a player’s immersion in virtual reality are becoming more and more popular. One of these technologies is a suit with wearable sensors that track a person’s position in space in real time. However, there are quite a few real described projects in the literature and on the Internet. This study examines the process of developing a task for creating a game application using virtual reality technology: a suit with wearable sensors for teaching students.Materials and methods of research. Timely identification of the needs of the IT market in personnel training allows educational organizations to form new training programs of different levels of training. This approach makes it possible to target the educational and methodological materials being developed to use the latest achievements in the development of the field under study.Using a systematic approach, the study characterizes virtual reality suits and sensors for monitoring the position in the user’s space. Thus, the goal of the task was to ensure the immersiveness and convenience of interaction between the player and the game environment.Based on materials on software, position sensors in space, the approach of pedagogical design was applied and the procedure was formed for a practical task, reflecting the relevant competencies.Results. The study was conducted on the basis in the framework of laboratory and practical work of students, as well as at a real enterprise. Training in the new profile of the direction of training “Applied informatics” is fully equipped with all the latest technologies in this field. As a result of the work, the content of the practical task was developed.Real development of virtual and augmented reality applications is conducted jointly with students. Almost all projects used a suit with body sensors.Conclusion. Our study examines in detail the process of developing an application using a suit with wearable sensors for further training of students. Based on the results, work can be carried out on real projects for any field. Based on the research materials, it is planned to issue a textbook for students with the profile of developing computer games and virtual / augmented reality applications.

Application of virtual and augmented reality has been popular in various areas such as education, entertainment, training, communication, design, therapy, and more. In education, particularly, virtual reality (VR) has been widely used by researchers or educators to create interactive instructions for interesting problem in alternatively real-like environments. Educators have recognized the potentials of VR to educational instructions in traditional classrooms where the environment of instructions would be limited to two-dimensional space and third-person’s view to a problem. Application of VR can allow students to be placed themselves into a virtual environment of interest with more immersive perception of problem conditions and application of instructions. In the literature, a considerable number of studies have applied digitalization technologies including virtual and augmented reality in design of process of workstations, manufacturing workplaces, and maintenance activities. In general, most of studies evaluated potentials of application of virtual reality from the perspective of fidelity, usability, and cost, but not from the perspective of effectiveness of instructions. In traditional instructions of manual lifting analysis, workplace or task conditions for manual lifting are usually given by statements, sometimes with pictorial descriptions. From the perspective of learners who need to investigate and analyze lifting tasks, it would be necessary to convert two-dimensional descriptions of task conditions into three-dimensional spaces for better understanding of tasks. However, the conversion would be limited with individual’s capability of imaginary perception. Providing virtual task environment for lifting task conditions to learners would be very helpful to understand underlying conditions for lifting task. The objective of this study was to design a teaching aid to improve instructions in virtual reality environments and to compare effectiveness of instructions between traditional instructions and virtual reality instructions. One exemplary lifting task published by Centers for Disease Control and Prevention was designed by using the Unity software. This example may be used by other educators to develop their own virtual instructions for different task analyses.

Applications of virtual and augmented reality in infectious disease epidemics with a focus on the COVID-19 outbreak

Robot-Human Mapping Model Learning for Robotic Imitation using Deep Learning and Virtual Reality

BackgroundVirtual reality is the science of creating a virtual environment for the assessment of various anatomical regions of the body for the diagnosis, planning and surgical training. Augmented reality is the superimposition of a 3D real environment specific to individual patient onto the surgical filed using semi-transparent glasses to augment the virtual scene.. The aim of this study is to provide an over view of the literature on the application of virtual and augmented reality in oral & maxillofacial surgery.MethodsWe reviewed the literature and the existing database using Ovid MEDLINE search, Cochran Library and PubMed. All the studies in the English literature in the last 10 years, from 2009 to 2019 were included.ResultsWe identified 101 articles related the broad application of virtual reality in oral & maxillofacial surgery. These included the following: Eight systematic reviews, 4 expert reviews, 9 case reports, 5 retrospective surveys, 2 historical perspectives, 13 manuscripts on virtual education and training, 5 on haptic technology, 4 on augmented reality, 10 on image fusion, 41 articles on the prediction planning for orthognathic surgery and maxillofacial reconstruction. Dental implantology and orthognathic surgery are the most frequent applications of virtual reality and augmented reality. Virtual planning improved the accuracy of inserting dental implants using either a statistic guidance or dynamic navigation. In orthognathic surgery, prediction planning and intraoperative navigation are the main applications of virtual reality. Virtual reality has been utilised to improve the delivery of education and the quality of training in oral & maxillofacial surgery by creating a virtual environment of the surgical procedure. Haptic feedback provided an additional immersive reality to improve manual dexterity and improve clinical training.ConclusionVirtual and augmented reality have contributed to the planning of maxillofacial procedures and surgery training. Few articles highlighted the importance of this technology in improving the quality of patients’ care. There are limited prospective randomized studies comparing the impact of virtual reality with the standard methods in delivering oral surgery education.

Precise preoperative anatomical visualization and understanding of an intracranial aneurysm (IA) are fundamental for surgical planning and increased intraoperative confidence. Application of virtual reality (VR) and mixed reality (MR), thus three-dimensional (3D) visualization of IAs could be significant in surgical planning. Authors provide an up-to-date overview of VR and MR applied to IA surgery, with specific focus on tailoring of the surgical treatment. A systematic analysis of the literature was performed in accordance with the PRISMA guidelines. Pubmed, and Embase were searched to identify studies reporting use of MR and VR 3D visualization in IA surgery during the last 25 years. Type and number of IAs, category of input scan, visualization techniques (screen, glasses or head set), inclusion of haptic feedback, tested population (residents, fellows, attending neurosurgeons), and aim of the study (surgical planning/rehearsal, neurosurgical training, methodological validation) were noted. Twenty-eight studies were included. Eighteen studies (64.3%) applied VR, and 10 (35.7%) used MR. A positive impact on surgical planning was documented by 19 studies (67.9%): 17 studies (60.7%) chose the tailoring of the surgical approach as primary outcome of the analysis. A more precise anatomical visualization and understanding with VR and MR was endorsed by all included studies (100%). Application of VR and MR to perioperative 3D visualization of IAs allowed an improved understanding of the patient-specific anatomy and surgical preparation. This review describes a tendency to utilize mostly VR-platforms, with the primary goals of a more accurate anatomical understanding, surgical planning and rehearsal.

Virtual reality (VR) and augmented reality (AR) have been explored to be an effective alternative to integrating mental health intervention proposals, particularly in eldercare. The objective is to map the usability and applications of VR and AR technologies in interventions for the elderly population. The main areas of interventions in AR and VR applied to the elderly are stimulation and cognitive rehabilitation, physical rehabilitation, treatment of mental diseases, and promotion of quality of life. Despite the need for further studies, VR and AR have strong adherence among the elderly and demonstrate promising potential in interventions that seek to promote mental health and improve the quality of life.

Computational thinking (CT) skills are increasingly important in education to prepare students for the challenges of the digital age. Augmented Reality (AR) and Virtual Reality (VR) have been introduced as immersive technologies that have the potential to enhance CT skills through more interactive learning experiences. However, there is still a gap in understanding the effectiveness of these technologies in supporting the development of CT, particularly in different levels of education and disciplines. Although several studies have highlighted the benefits of AR and VR in education, no systematic review integrates these findings to identify advantages, challenges, and opportunities for further implementation. Therefore, this study conducted a systematic review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines by analyzing 25 empirical studies (AR=17, VR=8) obtained from the Scopus database (2008-2024). The analysis addresses four key research questions: (1) the current state of AR/VR in CT development, (2) their advantages, (3) implementation challenges, and (4) future research directions. The results show that AR is more widespread than VR at various levels of education, with dominance in higher education followed by secondary and primary schools. Computer science is the main field of application of AR and VR, while AR is also widely applied in mathematics to increase interest and problem-solving. A total of 11 studies reported significant impacts of these technologies on CT, with AR being superior in increasing student motivation and engagement, as well as aiding in problem-solving and debugging. In contrast, VR provides a more immersive learning experience by strengthening concept understanding, especially in programming and recursion. However, several obstacles in the application of AR and VR, such as hardware limitations, costs, and user skills, affect the effectiveness of these technologies in the learning environment. This study also identified potential future research, including the exploration of VR in primary and kindergarten education, the application of VR in non-computer science fields, and the efficient use of these technologies in supporting the CT process. This study provides more precise insights into the optimal ways of utilizing AR and VR in developing CT skills. It is a reference for educators, policymakers, and researchers in supporting CT learning.

Virtual reality stroke rehabilitation - hype or hope?