Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach.

The education system evolves and transforms towards interactive and immersive learning tools in this digital age. Augmented reality has also evolved as a ubiquitous, robust, and effective technology for providing innovative educational tools. In engineering education, many abstract concepts require technological intervention for conceptual understanding and better instructional content. While learning through the immersive tools, system usability has great importance in terms of effectiveness, efficiency, and satisfaction. Effectiveness refers to users' accuracy and completeness in achieving defined goals; efficiency relates to expended resources about the precision and completeness with which users achieve their objectives; satisfaction deals with a positive attitude towards using the product. If the system fails to provide good usability, it may cause adverse effects such as increasing stress, lacking necessary features, increasing the users' cognitive load, and negatively impacting the student's motivation. In this study, two mobile augmented reality (MAR) applications were developed as an instructional tool to teach the students about Karnaugh maps in the digital electronics course. The first application is a Keypad-based MAR application that uses a keypad matrix for user interaction and the second application is a Marker-based MAR application that uses multiple markers to solve K-Map for producing an optimum solution of the given problem. An experimental study was conducted to determine the student's opinion of the developed MAR applications. The study was designed to determine the system usability of the two MAR applications using the System Usability Score (SUS) and Handheld Augmented Reality Usability Score (HARUS) models. 90 engineering students participated in the study, and they were randomly divided into two different groups: keypad-based group and Marker-based group. The keypad-based group included 47 students who had hands-on experience with a keypad-based MAR application, whereas the marker-based group included 43 students who had hands-on experience with multiple marker-based MAR applications. The experimental outcomes indicated that the keypad-based MAR application has better SUS and HARUS scores than the marker-based MAR application which suggests that the keypad-based MAR application has provided better user interaction.

Mobile Augmented Reality (AR) applications allow the user to interact with virtual objects positioned within the real world via a smart phone, tablet or smart glasses. As the popularity of these applications grows, recent researchers have identified several security and privacy issues pertaining to the collection and storage of sensitive data from device sensors. Location-based AR applications typically not only collect user location data, but transmit it to a remote server in order to download nearby virtual content. In this paper we show that the pattern of network traffic generated by this process alone can be used to infer the user's location. We demonstrate a side-channel attack against a widely available Mobile AR application inspired by Website Fingerprinting methods. Through the strategic placement of virtual content and prerecording of the network traffic produced by interacting with this content, we are able to identify the location of a user within the target area with an accuracy of 94%. This finding reveals a previously unexplored vulnerability in the implementation of Mobile AR applications and we offer several recommendations to mitigate this threat.

Augmented reality (AR) has evolved from research projects into mainstream applications that cover diverse fields, such as entertainment, health, business, tourism and education. In particular, AR games, such as Pokémon Go, have contributed to introducing the AR technology to the general public. The proliferation of modern smartphones and tablets with large screens, cameras, and high processing power has ushered in mobile AR applications that can provide context-sensitive content to users whilst freeing them to explore the context. To avoid ambiguity, I define mobile AR as a type of AR where a mobile device (smartphone or tablet) is used to display and interact with virtual content that is overlaid on top of a real-time camera feed of the real world. Beyond being mere entertainment, AR and games have been shown to possess significant affordances for learning. Although previous research has done a decent job of reviewing research on educational AR applications, I identified a need for a comprehensive review on research related to educational mobile AR games (EMARGs). This paper explored the research landscape on EMARGs over the period 2012–2017 through a systematic literature review complemented by two case studies in which the author participated. After a comprehensive literature search and filtering, I analyzed 31 EMARGs from the perspectives of technology, pedagogy, and gaming. Moreover, I presented an analysis of 26 AR platforms that can be used to create mobile AR applications. I then discussed the results in depth and synthesized my interpretations into 13 guidelines for future EMARG developers.

Abstract: This repeated measure experiment study aimed to explore LINUS students or Low Achiever (LA) students’ improvement in learning English language vocabularies using Mobile Augmented Visual Reality (MAVR). It specifically tried to analyse the effect of MAVR on the English vocabulary development of 45 primary school students from the LINUS programme in Selangor. In this study, they were provided access to an AVR-game based apps in learning English vocabularies with mobile computing devices. Vocabulary size was later measured using the British Picture Vocabulary Scale II and analysed with repeated measures ANOVA. Analysis showed that there was an increase in the scores and the differences between the levels of the within-subjects factor was significant. This implies that Mobile Augmented Visual Reality (MAVR) materials can be used as an interactive tool for LA learners in learning a language. Findings highlight the role of teachers in the implementation process and point out possible directions for more effective application of MAVR in this field.
 
 Keywords: Augmented Reality (AR), LINUS students, Mobile Augmented Reality (MAVR), Visual Reality (VR), Vocabulary.

The recent advancements in mobile technologies have enabled the widespread adoption of augmented reality (AR) to enrich cultural heritage (CH) digital experiences. Mobile AR leverages visual recognition capabilities and sensor data to superimpose digital elements into the user’s view of their surroundings. The pervasive nature of AR serves several purposes in CH: visitor guidance, 3D reconstruction, educational experiences, and mobile location-based games. While most literature reviews on AR in CH focus on technological aspects such as tracking algorithms and software frameworks, there has been little exploration of the expressive affordances of AR for the delivery of meaningful interactions. This paper (based on the PRISMA guidelines) considers 64 selected publications, published from 2016 to 2023, that present mobile AR applications in CH, with the aim of identifying and analyzing the (mobile) AR (interaction) design patterns that have so far been discussed sporadically in the literature. We identify sixteen (16) main UX design patterns, as well as eight (8) patterns with a single occurrence in the paper corpus, that have been employed—sometimes in combination—to address recurring design problems or contexts, e.g., user navigation, representing the past, uncovering hidden elements, etc. We analyze each AR design pattern by providing a title, a working definition, principal use cases, and abstract illustrations that indicate the main concept and its workings (where applicable) and explanation with respect to examples from the paper corpus. We discuss the AR design patterns in terms of a few broader design and development concerns, including the AR recognition approach, content production and development requirements, and affordances for storytelling, as well as possible contexts and experiences, including indoor/outdoor settings, location-based experiences, mobile guides, and mobile games. We envisage that this work will thoroughly inform AR designers and developers abot the current state of the art and the possibilities and affordances of mobile AR design patterns with respect to particular CH contexts.

This paper explores the affordances of augmented reality content in a mobile augmented reality application. A user study was conducted by performing a multi-camera video recording of seven think aloud sessions. The think aloud sessions consisted of individual users performing tasks, exploring and experiencing a mobile augmented reality (MAR) application we developed for the iOS platform named ARad. We discuss the instrumental affordances we observed when users interacted with augmented reality content, as well as more complex affordances rising from conventions from media content, AR and the traditional WIMP paradigm. We find that remediation of traditional newspaper content through the MAR medium can provide engaging, pleasing and exciting user experiences. However, the some of the content still suffers from being shoveled onto the MAR platform without adapting it properly. Finally, we discuss what content was most successfully mediated to the user and how the content impacts the user experience.

Augmented reality (AR) aims to combine real and virtual worlds, offering real-time 3D interactions between users and virtual information. Existed AR systems detect specific markers in environments and augmented them with virtual contents such as 3D objects and videos. With the explosive growth of powerful, less expensive mobile devices, AR systems are extended to mobile devices. However, they fail to achieve the accurate and real-time integration of the real world and virtual contents because of the insufficient detection accuracy to objects in real world and computational resource shortage. In this paper, we propose a hybrid AR framework that integrates cloud computing, 5G communication and deep learning technology to achieve accurate tracking and ultra-low latency mobile AR applications. Our AR framework contains three blocks, an image capturing block, a deep learning based objects tracking block and a WebGL based rendering block. We use cloud computing technology to deploy them on mobile devices and a cloud server separately according to calculation amounts and use 5G to achieve communication between the cloud server and mobiles devices. The experiment results show that the accuracy and running speed of our framework can meet the requirements of most mobile AR applications.

In this paper, we design a mobile augmented reality (AR) system called TweetGlue, which overlays text messages (i.e., tweets) that are posted to a local social networking service onto live view images from cameras in mobile/wearable devices. By displaying the tweets at the current positions of the users who posted them, it supports social interaction between the users. Accurate pose tracking of mobile devices is an essential building block of such mobile AR applications. While visual features that are extracted from scenes are commonly used for vision-based pose estimation, it would not be suitable for such a mobile social AR application because the features may be often occluded by human bodies nearby. To cope with the problem, we leverage an external pedestrian tracking system using a small number of laser-based distance measurement sensors (i.e., LRS sensors) to utilize the surrounding human bodies as virtual markers for pose estimation. The mobile devices periodically analyze images from the embedded camera sensor to estimate relative positions of pedestrians in the images. By matching the estimated relative positions with accurate human location measurements by the LRS sensors, the system robustly identifies location and horizontal orientation of the devices. Through simulation experiments, we show that the TweetGlue system can accurately identify pose of mobile devices in 83% of the simulated scenarios.

This case study outlines the development and utilization of a Mobile Augmented Reality (MAR) application to teach an asynchronous online lesson on e-commerce. The MAR technology was leveraged primarily to enhance the presentation of online learning materials, add interactivity to the learning process, and enable students to access the lesson from anywhere via their ubiquitous mobile devices. Two groups, comprising a total of 105 business students, participated in the online lesson during the COVID pandemic. Students' experiences of the implementation were captured through an online survey to find out how the MAR technology was perceived, and what were the benefits and challenges of using the technology. This study contributes to the research on the use of augmented reality in education and offers practical recommendations for teachers to consider when designing and implementing MAR online lessons, including the importance of learner-centered design, careful guidance on technology use, and encouragement of student interaction. The study concludes that augmented reality is a useful tool for improving learning materials and asynchronous online learning practices and MAR applications can be effective for learning with compact materials and micro-credentials.

The flipped classroom model enables students to acquire fundamental knowledge outside of class time; thus, include reading materials or video lectures. This will be able to free up class time for knowledge application, student engagement in active learning, and higher-order thinking. The flipped classroom model is increasingly being adopted in competency-based medical education. However, the potential of flipped classrooms for enhancing medical education has not yet been proven and it poses a major challenge to students who have not mastered self-regulated learning strategies. Thus, they may not be able to understand the information presented in the course materials or to strategically use learning resources outside of class. In this project, we have created a mobile augmented reality (AR) application for students studying anatomy to help them grasp the idea of the AR flipped classroom. Students were instructed to use the same three mobile augmented reality (MAR) applications with different design and approach. The three types of MAR applications are — those with markers and notes on printed notebooks (BARA1), those with markers and notes are on a website (BARA2), and those with all notes in the MAR application and the marker are a tangible 3D object (BARA3). From the series of experiment, we found out that most students favour the BARA3, meanwhile the medical lecturers essentially prefer the BARA1. In this research three things have been considered: interface design, the usefulness of Augmented Reality (AR), technical problems with the use of AR applications, and educators training on the use of AR applications. Without a well-designed interface and guidance for the students, AR technology can be too complicated to use, especially for those who are not familiar with the technology.

There are two problems in the existing mobile augmented reality (MAR) system, one is the difficulty in developing and the other is the lack of versatile AR observer. To address them, a generic MAR framework was proposed in the paper, which contains three components: a versatile observer which is run on smart mobile device to see the AR effect produced by MAR application, an MAR server that provides network and data service for the MAR application, and an MAR application customizer which is used by developers to tailor their desired applications. The Vuforia SDK is used to implement the observer, and the XML technology is applied to achieve the goal of customizing MAR application. The generic framework enables developers to do code-free development, and provides the convenience of observing different MAR applications by one MAR observer. The experimental results show that this framework reduces the difficulty and time in developing AR application and makes it easy to observing AR effects by users.

Advancements in Augmented Reality (AR) technologies and processing power of mobile devices have created a surge in the number of mobile AR applications. Nevertheless, many AR applications have adopted surface gestures as the default method for interaction with virtual content. In this paper, we investigate two gesture modalities, surface and motion, for operating mobile AR applications. In order to identify optimal gestures for various interactions, we conducted an elicitation study with 21 participants for 12 tasks, which yielded a total of 504 gestures. We classified and illustrated the two sets of gestures, and compared them in terms of goodness, ease of use, and engagement. The elicitation process yielded two separate sets of user-defined gestures; legacy surface gestures, which were familiar and easy to use by the participants, and motion gestures, which had better engagement. From the interaction patterns of this second set of gestures, we propose a new interaction class called TMR (Touch-Move-Release), which defines for mobile AR.

Purpose: This research aims to improve the User Experience (UX) of Augmented Reality (AR) mobile applications for MSMEs and buyers through the Design Thinking method. This research solves the problem of suboptimal UX in AR-based mobile applications. This study hypothesizes that the application of Design Thinking can result in significant improvements in the UX of AR mobile applications, which is evidenced by an increase in heuristic evaluation scores. Methods: The Design Thinking approach (Empathize, Define, Ideate, Prototype, Test) is implemented. Data were collected through interviews, observations, and heuristic evaluation questionnaires. Result: Initial heuristic testing showed several usability problems in the developed AR mobile applications, such as Help and Documentation (H10), Recognition Rather than Recall (H6), and Error Prevention (H5). After the application of the Design Thinking method and design iteration, the heuristic testing showed that the results of the evaluation comparison before and after the improvement showed a high effectiveness of the corrective actions taken, with an average decrease in severity score of 37% based on the Nielsen scale (0–4), indicating that the most critical and major issues were successfully reduced to cosmetic or minor levels. Novelty: This research contributes in the form of a practical framework to improve the UX of AR mobile applications for MSMEs and buyers by utilizing the Design Thinking method. The results of this research can be a reference for developers in designing user-friendly AR mobile applications.

Augmented Reality (AR) is the combination of real-world and virtual content where it allows users to interact with virtual objects in real-time. In order to achieve optimal performance in mobile AR applications, choosing a suitable tracking algorithm is an important issue as the smartphone has its own limited processing capabilities and memories when compared to the PC platform. Tracking in AR consists of four important components; detector, descriptor, matcher and pose estimator. The robustness of mobile AR application in illumination is an issue that has not been resolved. The descriptor is one of the important components as it will affect the overall performance of mobile AR applications. In order to create an efficient and robust mobile AR application, the descriptor must be efficient which means that it must be able to function in a short time and robust to changes in illumination. FREAK (Fast Retina Keypoint) has been proposed as the most appropriate descriptor for mobile AR applications based on previous research. FREAK can function in the shortest time and most robust to changes in scale and rotation. Although FREAK can function in the shortest time and robust to changes but the color information that will affect the illumination invariance of a descriptor had been ignored by FREAK. This study has developed a FREAK descriptor using RGB color space to improve the robustness of illumination invariance. A vertical concatenation technique has been used to combine all the key points extracted. Evaluation for the robustness of descriptors in terms of illumination changes has been done using the ALOI data set. The accuracy of the RGB FREAK descriptor has increased in illumination invariance (light color changes and lighting arrangement changes) compared to the original FREAK descriptor.

Augmented reality technology adoption: Case of a mobile application in Turkey