Augmented Reality Framework Research Articles

Sharing Augmented Reality between a Patient and a Clinician for Assessment and Rehabilitation in Daily Living Activities

In rehabilitation settings that exploit Mixed Reality, a clinician risks losing empathy with the patient by being immersed in different worlds, either real and/or virtual. While the patient perceives the rehabilitation stimuli in a mixed real–virtual world, the physician is only immersed in the real part. While in rehabilitation, this may cause the impossibility for the clinician to intervene, in skill assessment, this may cause difficulty in evaluation. To overcome the above limitation, we propose an innovative Augmented Reality (AR) framework for rehabilitation and skill assessment in clinical settings. Data acquired by a distributed sensor network are used to feed a “shared AR” environment so that both therapists and end-users can effectively operate/perceive it, taking into account the specific interface requirements for each user category: (1) for patients, simplicity, immersiveness, engagement and focus on the task; (2) for clinicians/therapists, contextualization and natural interaction with the whole set of data that is linked with the users’ performances in real-time. This framework has a strong potential in Occupational Therapy (OT) but also in physical, psychological, and neurological rehabilitation. Hybrid real and virtual environments may be quickly developed and personalized to match end users’ abilities and emotional and physiological states and evaluate nearly all relevant performances, thus augmenting the clinical eye of the therapist and the clinician-patient empathy. In this paper, we describe a practical exploitation of the proposed framework in OT: setting-up the table for eating. Both a therapist and a user wear Microsoft HoloLens 2. First, the therapist sets up the table with virtual furniture. Next, the user places the corresponding real objects (also in shape) to match them as closely as possible to the corresponding virtual ones. The therapist’s view is augmented during the test with motion, balance, and physiological estimated cues. Once the training is completed, he automatically perceives deviations in the position and attitude of each object and the elapsed time. We used a camera-based localization algorithm achieving a level of accuracy of 5 mm with a confidence level of 95% for position and 1° for rotation. The framework was designed and tested in collaboration with clinical experts of Villa Rosa rehabilitation hospital in Pergine (Italy), involving both a set of patients and healthy users to demonstrate the effectiveness of the designed architecture and the significance of the analyzed parameters between healthy users and patients.

An Indoor Location-Based Augmented Reality Framework.

This paper proposes an indoor location-based augmented reality framework (ILARF) for the development of indoor augmented-reality (AR) systems. ILARF integrates an indoor localization unit (ILU), a secure context-aware message exchange unit (SCAMEU), and an AR visualization and interaction unit (ARVIU). The ILU runs on a mobile device such as a smartphone and utilizes visible markers (e.g., images and text), invisible markers (e.g., Wi-Fi, Bluetooth Low Energy, and NFC signals), and device sensors (e.g., accelerometers, gyroscopes, and magnetometers) to determine the device location and direction. The SCAMEU utilizes a message queuing telemetry transport (MQTT) server to exchange ambient sensor data (e.g., temperature, light, and humidity readings) and user data (e.g., user location and user speed) for context-awareness. The unit also employs a web server to manage user profiles and settings. The ARVIU uses AR creation tools to handle user interaction and display context-aware information in appropriate areas of the device's screen. One prototype AR app for use in gyms, Gym Augmented Reality (GAR), was developed based on ILARF. Users can register their profiles and configure settings when using GAR to visit a gym. Then, GAR can help users locate appropriate gym equipment based on their workout programs or favorite exercise specified in their profiles. GAR provides instructions on how to properly use the gym equipment and also makes it possible for gym users to socialize with each other, which may motivate them to go to the gym regularly. GAR is compared with other related AR systems. The comparison shows that GAR is superior to others by virtue of its use of ILARF; specifically, it provides more information, such as user location and direction, and has more desirable properties, such as secure communication and a 3D graphical user interface.

A Multifactor Comparative Assessment of Augmented Reality Frameworks in Diverse Computing Settings

Research and development on different augmented reality (AR) frameworks have come a long way when it comes to image tracking, object tracking, plane tracking and light estimation. However, there might be trade-offs and varying results obtained from different AR frameworks, depending on the use cases, and this is critical for consideration during immersive application development. Besides the current literature effort, this research proposes a multifactor comparative analysis of two core AR frameworks, which aims to analyze and evaluate ARKit and ARCore in diverse computing settings. This research developed a structural application which evaluated three major test parameters across ten devices spanning ARKit and ARCore. The first parameter relates to evaluating AR measurements using four different distance criteria. The second parameter evaluated resource utilization, relating to the central processing unit (CPU) and random access memory (RAM), while the last parameter evaluated plane detection based on light estimation. Findings conclude that ARKit is the preferable AR framework for AR measurement accuracy and reliability within the tested distance criteria. ARCore is the most optimized AR framework in terms of RAM utilization. Regarding plane detection based on light estimation, ARCore is the preferable choice under low lighting conditions, however, ARKit is the most suitable AR framework under adequate ambient lighting conditions. The findings of this research could guide future prototyping and immersive mobile application development within the context of the parameters used.

Development of AR Experiment on Electric-Thermal Effect by Open Framework with Simulation-Based Asset and User-Defined Input

Augmented reality (AR), an advanced information and communication technology, has gained widespread concerns in teaching physics due to its possibility for reducing cost of equipment, enhancing comprehension of abstract concepts, and overcoming tendency of diminishing interests of students. Although the fruitful applications have been created frequently with the aid of AR framework using standard devices, for example, head-mounted display, there is less likely for the image detection-based framework to integrate the virtual assets with the real objects with ability of supporting inputs of the invisible elements and thus makes it difficult to maintain evolutions of the augmented information over the physical quantity such as current in the AR-based physical experiment. In this paper, an open AR framework with simulation-based assets triggered by user-defined inputs is proposed to allow physics teachers to create their own AR teaching materials. Considering the teaching of electric-thermal effects by exploring the thermoelectric cooler-based thermal management of power device, furthermore, the proposed framework is illustrated from the perspective of the three paramount tasks: (1) a simple and accurate generation of assets from the finite element simulation; (2) a low-cost and convenient measurement of physical quantities by the microcomputer unit with Bluetooth connectivity; and (3) a real-time and changeable controlment of simulation-based assets by the measured data using the thread-based Python script. Experimental results show that students are excited in the AR application interacted with the real operation of current, and moreover, physic teachers are easy to design and deploy the simulation-based framework with user-defined inputs to create their own AR learning materials for sparking growing interests of students.

Cloud-based cross-platform collaborative augmented reality in flutter

Augmented Reality (AR) as a technology in the business area is utilized in new frontiers such as collaborative real-time experiences and cloud-based solutions. However, there is still a strong tendency towards game engines, which hinders widespread adoption for businesses. We present a collaborative AR framework (Flutter plugin) aimed at lowering the entry barriers and operating expenses of AR applications. A cross-platform and cloud-based solution combined with a web-based content management system (cloud) is a powerful tool for non-technical staff to take over operational tasks such as providing 3D models or moderating community annotations. To achieve cross-platform support, the AR Flutter plugin builds upon ARCore (Android) and ARKit (iOS) and unifies the two frameworks using an abstraction layer written in Dart. In this extensive description we present an in-depth summary of the concepts to realize the framework and prove its performance being on the same level as the native AR frameworks. This includes application-level metrics like CPU and RAM consumption and tracking-level qualities such as keyframes per second used by the underlying SLAM algorithm, detected feature points, and area of tracked planes. Our contribution closes a gap in today’s technological landscape by providing an AR framework with the familiar development process of cross-platform apps. Building upon on a content management system (cloud) and AR can be a game changer to achieve business objectives, while being not restrained to stand-alone single-purpose apps. This will trigger a potential paradigm shift for previously complex-to-realize applications relying on AR, e.g., in production and planning. The AR Flutter plugin is fully open-source, the code can be found at: https://github.com/CariusLars/ar_flutter_plugin.

Edge AR X5: An Edge-Assisted Multi-User Collaborative Framework for Mobile Web Augmented Reality in 5G and Beyond

Multi-user mobile Augmented Reality (AR) has been successfully used in various fields as a novel visual interaction technology. But current mainstream wearable device-based and app-based solutions are still facing cross-platform, real-time communication, and intensive computing requirements. Mobile Web technology is envisioned to be a promising supporting technology for cross-platform application of mobile AR especially in 5G networks, which provide pervasive communication and computing resources thereby forming a formidable framework for the practical application of multi-user mobile Web AR. However, the problem of how to use these new techniques properly to achieve efficient communication and computing collaboration is obviously paramount in order for multi-user mobile Web AR to be realized in 5G networks. In this article, we propose the first edge-assisted multi-user collaborative framework for mobile Web AR in the 5G era. First, we propose a heuristic mechanism BA-CPP for efficient communication planning, which allows multi-user interaction synchronization to be achieved. Second, we introduce a motion-aware key frame selection mechanism called Mo-KFP to optimize the computational efficiency of the edge system, and simultaneously alleviate the initialization problem by collaborating with nearby mobile devices using the Device-to-Device (D2D) communication technique. Experiments are conducted in a real-world 5G network, and the results demonstrate the superiority of our proposed collaborative framework.

Exploiting Augmented Reality and Computer Vision for Healthcare Education: The Case of Pharmaceutical Substances Visualization and Information Retrieval.

Augmented Reality (AR) is already used as the primary visualization and user interaction tool in several scientific and business areas. At the same time new AR technologies and frameworks considerably facilitate both the development of innovative applications and also their wide adoption in different domains of everyday life. In the area of healthcare AR solutions make use of mobile or wearable devices and glasses to support, among others, education and healthcare professionals training. The aim of this paper is to present a prototype mHealth app for education, which uses AR and computer vision technologies for pharmaceutical substances recognition on drug packaging. The conceptual design of the system includes three main components which are responsible for a) Text recognition, b) Drug identification and c) AR operations for interactivity. The prototype application is available in Android or iOS platforms and has been evaluated in real-world scenarios. Camera and screen of the mobile phones fulfill the text recognition and AR operations, which eliminates the need for special equipment, while PubChem and 3D Model databases provide assets required for the drug identification and AR visualizations. The results highlight the value of AR for educational purposes, especially when combined with advanced image recognition technologies to build interactive AR encyclopedias.

Blending Real and Virtual Objects in Augmented Reality Environments

Although Augmented Reality applications are becoming increasingly popular, the lack of visual realism in rendering still remains an open problem due to its computational cost. Physically-based algorithms can generate renderings with a high degree of photorealism, and they are becoming popular after the recent development of hardware accelerators. This work shows how to integrate Augmented Reality frameworks with ray tracing frameworks to create scenes with high-quality, real-time reflections and refractions, with emphasis on the blending of virtual objects to the real environment. To support the interaction between real and virtual elements, a textured cube using images from the real environment must be provided. Our framework does not add processing overhead to the application when comparing the use of the proposed middleware to the use of ray tracing frameworks alone. We will show that with our approach, photorealistic augmented reality rendering can be achieved in real time without the use of any special equipment.

EdgeXAR: A 6-DoF Camera Multi-target Interaction Framework for MAR with User-friendly Latency Compensation

The computational capabilities of recent mobile devices enable the processing of natural features for Augmented Reality (AR), but the scalability is still limited by the devices' computation power and available resources. In this paper, we propose EdgeXAR, a mobile AR framework that utilizes the advantages of edge computing through task offloading to support flexible camera-based AR interaction. We propose a hybrid tracking system for mobile devices that provides lightweight tracking with 6 Degrees of Freedom and hides the offloading latency from users' perception. A practical, reliable and unreliable communication mechanism is used to achieve fast response and consistency of crucial information. We also propose a multi-object image retrieval pipeline that executes fast and accurate image recognition tasks on the cloud and edge servers. Extensive experiments are carried out to evaluate the performance of EdgeXAR by building mobile AR apps upon it. Regarding the Quality of Experience (QoE), the mobile AR apps powered by EdgeXAR framework run on average at the speed of 30 frames per second with precise tracking of only 1-2 pixel errors and accurate image recognition of at least 97% accuracy. As compared to Vuforia, one of the leading commercial AR frameworks, EdgeXAR transmits 87% less data while providing a stable 30FPS performance and reducing the offloading latency by 50 to 70% depending on the transmission medium. Our work facilitates the large-scale deployment of AR as the next generation of ubiquitous interfaces.

Developing an Augmented Reality Immersive Learning Design (AILeaD) Framework: A Fuzzy Delphi Approach

The use of augmented reality as a teaching aid among teachers is becoming more widespread. So, it is a must to create a holistic framework of augmented reality in order to enhance the teaching and learning context. Therefore, the purpose of this study is to develop an augmented reality immersive learning design (AILeaD) framework by using the Fuzzy Delphi Technique. The objective of this research is to gain experts’ consensus in building and identifying constructs and elements as well as their place in the AILeaD framework. The Fuzzy Delphi Technique is used as a systematic method to evaluate, verify as well as to make a decision in modifying all needed constructs and elements in order to create the AILeaD framework through the consensus of the experts. The research questionnaire is divided into three sections which are technology skills, instructional designs and type of augmented reality applications in a total of 36 items and using the Likert-7point scale. The sample of this research consists of 10 experts from educational technology, teaching technology, augmented reality and model development. The finding showed that the experts agreed and approved all of the constructs and elements where the three conditions were fulfilled which are the item value is less than or same with the threshold value (d) < 0.2, experts' consensus percentage exceeded 75% and defuzzification value per item exceeded α-cut = 0.5. This study could contribute to the educational policy that focuses on the development of teaching quality with the improvement of existing skills among educators through the innovation and utilization of technology in the teaching and learning context. This is in line with Malaysia Education Blueprint known as Shift 7 (2013 – 2025) where it leverages the utilization of ICT to enhance the quality of education in Malaysia. The implication of this study is to make the AILeaD framework as a guideline for teachers, lecturers and instructional designers in producing the teaching aids in accordance with augmented reality and taking Gagne Nine Events, Cognitive Theory of Multimedia Learning dan Revised Bloom Taxonomy into account.

COMPARISON OF SELECTED AUGMENTED REALITY FRAMEWORKS FOR INTEGRATION IN GEOGRAPHIC CITIZEN SCIENCE PROJECTS

Abstract. Augmented reality (AR) offers functionalities that can be beneficial for citizen science (CS) projects. Especially location-based approaches have potential for geographically oriented CS projects, as objects can be placed based on geographic coordinates. Since choosing a suitable AR framework for integration into a CS project can be challenging, this paper gives an overview of the common AR frameworks and takes a closer look at three selected ones that are particularly suitable for CS projects (AR.js, AR Foundation, and ViroReact). Prototypes were implemented for the selected frameworks to investigate which framework is best suited for specific use cases. Marker-based tracking approaches, image recognition and location-based placement were considered. The results show that the framework AR.js is particularly suitable for marker-based tracking with very simple markers and therefore represents a good entry point for CS projects to integrate initial AR functionality into their project. AR Foundation and ViroReact, on the other hand, are faster and more reliable with the detection of more complex markers. The location-based approach can be implemented with all three frameworks, but the precision of the placement of the objects strongly depends on the accuracy of the sensors of the mobile device.

Secure Collaborative Augmented Reality Framework for Biomedical Informatics.

Augmented reality is currently of interest in biomedical health informatics. At the same time, several challenges have appeared, in particular with the rapid progress of smart sensor technologies, and medical artificial intelligence. This yields the necessity of new needs in biomedical health informatics. Collaborative learning and privacy are just some of the challenges of augmented reality technology in biomedical health informatics. This paper introduces a novel secure collaborative augmented reality framework for biomedical health informatics-based applications. Distributed deep learning is performed across a multi-agent system platform. The privacy strategy is then developed for ensuring better communications of the different intelligent agents in the system. In this research work, a system of multiple agents is created for the simulation of the collective behaviours of the smart components of biomedical health informatics. Augmented reality is also incorporated for better visualization of medical patterns. A novel privacy strategy based on blockchain is investigated for ensuring the confidentiality of the learning process. Experiments are conducted on real use cases of the biomedical segmentation process. Our strong experimental analysis reveals the strength of the proposed framework when directly compared to state-of-the-art biomedical health informatics solutions.

Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training

Deep learning (DL) algorithms have achieved significantly high performance in object detection tasks. At the same time, augmented reality (AR) techniques are transforming the ways that we work and connect with people. With the increasing popularity of online and hybrid learning, we propose a new framework for improving students’ learning experiences with electrical engineering lab equipment by incorporating the abovementioned technologies. The DL powered automatic object detection component integrated into the AR application is designed to recognize equipment such as multimeter, oscilloscope, wave generator, and power supply. A deep neural network model, namely MobileNet-SSD v2, is implemented for equipment detection using TensorFlow’s object detection API. When a piece of equipment is detected, the corresponding AR-based tutorial will be displayed on the screen. The mean average precision (mAP) of the developed equipment detection model is 81.4%, while the average recall of the model is 85.3%. Furthermore, to demonstrate practical application of the proposed framework, we develop a multimeter tutorial where virtual models are superimposed on real multimeters. The tutorial includes images and web links as well to help users learn more effectively. The Unity3D game engine is used as the primary development tool for this tutorial to integrate DL and AR frameworks and create immersive scenarios. The proposed framework can be a useful foundation for AR and machine-learning-based frameworks for industrial and educational training.

AIGuide: Augmented Reality Hand Guidance in a Visual Prosthetic

Locating and grasping objects is a critical task in people’s daily lives. For people with visual impairments, this task can be a daily struggle. The support of augmented reality frameworks in smartphones can overcome the limitations of current object detection applications designed for people with visual impairments. We present AIGuide, a self-contained smartphone application that leverages augmented reality technology to help users locate and pick up objects around them. We conducted a user study to investigate the effectiveness of AIGuide in a visual prosthetic for providing guidance; compare it to other assistive technology form factors; investigate the use of multimodal feedback, and provide feedback about the overall experience. We gathered performance data and participants’ reactions and analyzed videos to understand users’ interactions with the nonvisual smartphone user interface. Our results show that AIGuide is a promising technology to help people with visual impairments locate and acquire objects in their daily routine. The benefits of AIGuide may be enhanced with appropriate interaction design.

Augmented reality for human–swarm interaction in a swarm-robotic chemistry simulation

We present a novel augmented reality (AR) framework to show relevant information about swarm dynamics to a user in the absence of markers by using blinking frequency to distinguish between groups in the swarm. In order to distinguish between groups, clusters of the same group are identified by blinking at a specific time interval that is distinct from the time interval at which their neighbors blink. The problem is thus to find blinking sequences that are distinct for each group with respect to the group’s neighbors. Selecting an appropriate sequence is an instance of the distributed graph coloring problem, which can be solved in $$O(\log (n))$$ time with n being the number of robots involved. We demonstrate our approach using a swarm chemistry simulation in which robots simulate individual atoms that form molecules following the rules of chemistry. An AR display is then used to display information about the internal state of individual swarm members as well as their topological relationship, corresponding to molecular bonds in a context that uses robot swarms to teach basic chemistry concepts.

Augmented-reality-assisted timber drilling with smart retrofitted tools

An ordinary electric drill was integrated into a context-aware augmented reality (AR) framework to assist in timber-drilling tasks. This study is a preliminary evaluation to detail technical challenges, potential bottlenecks, and accuracy of the proposed object- and tool-aware AR-assisted fabrication systems. In the designed workflow, computer vision tools and sensors are used to implement an inside-out tracking technique for retrofitted drills based on a reverse engineering approach. The approach allows workers to perform complex drilling angle operations according to computer-processed feedback instead of drawing, marking, or jigs. First, the developed methodology was presented, and its various devices and functioning phases were detailed. Second, this first proof of concept was evaluated by experimentally scanning produced drillings and comparing the discrepancies with their respective three-dimensional execution models. This study outlined improvements in the proposed tool-aware fabrication process and clarified the potential role of augmented carpentry in the digital fabrication landscape.

What is XR? Towards a Framework for Augmented and Virtual Reality

Augmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and professionals have been inconsistent in their use of these terms. This has led to conceptual confusion and unclear demarcations. Inspired by prior research and qualitative insights from XR professionals, we discuss the meaning and definitions of various terms and organize them in our proposed framework. As a result, we conclude that (1) XR should not be used to connote extended reality, but as a more open approach where the X implies the unknown variable: xReality; (2) AR and VR have fundamental differences and thus should be treated as different experiences; (3) AR experiences can be described on a continuum ranging from assisted reality to mixed reality (based on the level of local presence); and (4), VR experiences can be conceptualized on a telepresence-continuum ranging from atomistic to holistic VR.

Using augmented reality for shopping: a framework for AR induced consumer behavior, literature review and future agenda

PurposeA current technological trend, which has gained even more traction recently due to the COVID-19 pandemic, is the use of augmented reality (AR) in shopping environments. AR is addressing contemporary challenges rooted in online shopping (e.g. in terms of experientiality and try-on) and is fundamentally reshaping consumers' experiences. The purpose of this study is to provide a synthesized and structured overview of the state-of-the-art research focused on AR shopping.Design/methodology/approachThe authors conduct a systematic literature review of the empirical academic corpus focused on shopping via AR technology.FindingsThe review reveals the diverse psychological (cognitive, affective, and social) as well as behavioral outcomes related to the use of AR in the shopping context. The authors integrate the results into a framework for AR induced consumer behavior in shopping, thereby providing an important overview of the dynamics in AR-related shopping and the factors influencing the adoption of the technology by consumers. Specifically, the authors encountered that the technological abilities of AR (e.g. in terms of interactivity, vividness, informativeness, etc.) are a source for enhanced utilitarian and hedonic shopping experiences that can support intentions to purchase a product, reuse an AR app, or recommend it to others. Importantly, our review reveals the demand for several avenues for future research.Originality/valueThe authors provide an overview and synthesis of how and where AR is employed in shopping contexts, what theories and technological characteristics of AR are commonly analyzed, and what psychological and behavioral outcomes AR has been found to evoke. Based on our findings, the authors derive a framework that illustrates the dynamics in AR shopping and give an in-depth discourse on 13 future research agenda points related to thematic, theoretical, methodological, and technological matters.

Development of an augmented reality-based process management system: The case of a natural gas power plant

Since the beginning of the Industry 4.0 era, Augmented Reality (AR) has gained significant popularity. Especially in production industries, AR has proven itself as an innovative technology renovating traditional production activities, making operators more productive and helping companies to make savings in different expense items. Despite these findings, its adoption rate is surprisingly low especially in production industries, due to various organizational and technical limitations. Various AR platforms have been proposed to eliminate this gap, however, there is still not a widely accepted framework for such a tool. This research presents the reasons behind the low adoption rate of AR in production industries, and analyzes the existing AR frameworks. Based on the findings from these analyses and a conducted field study, a cloud-based AR framework, which provides tools for creating AR applications without any coding and features for managing, monitoring and improving industrial processes is proposed. The design and development phases are presented together with the evaluation of the platform in a real-world industrial scenario.

Correction Method of Visual Errors in Pole Vault Take-off Action Images

Pole Vault is one of the most difficult and complicated sport in athletic and tracks events; it is high jumping action with vaulting pole stick. In the study of Pole Vault technology, the rapid advancement of Augmented Reality (AR) technology has considerable significance for the visual error correction of the pole vault take-off action images in the simulated world. The key objective is to evaluate the correction algorithm of the visual errors of take-off action images of pole vault sport. This study analyzes the AR framework, features, and pole vault, as well as province methods for correcting visual defects. Pole vault take-off action images have been simulated with AR computation technique, and the AR images of continues pole vault made. This paper primarily provides an overview of augmented reality (AR) technology, including its properties and benefits, as well as a few visual error correcting methods. AR discusses how to choose an aerobics instructor who can help you maintain an aerobics regimen on a regular basis. Augmented reality has been used to model these pictures, and AR images of aerobics have been created. The visual error correction of the longitudinal direction, transverse direction, and height direction of AR simulated continue pole vault images with Scale Invariant Feature Transform (SIFT) algorithm, Mean Absolute Deviation (MAD) algorithm, Normalize Cross-Correlation (NCC) algorithm, and Sobel Edge Detection (SED) algorithm is proposed in this study. This revealed that poor performance in the pole vault sport is due in large part to a lack of a well-planned structure for implementing a corrective mechanism for visual mistake correction in take-off action images. The statistical result of perfect pole vault technique dissected and described, including the best manage, takeoff, drive and swing, and bar separation. Along with technical work, a pole vaulters training regimen should involve sprinting, weight lifting, and gymnastics as well.