Reality Interface Research Articles

Learning on mobile augmented reality trails of integrity and ethics

At the heart of university education, there must be an emphasis on students developing academic integrity and ethics (AIE), which is essential for their personal development and future professional careers. This paper reports on a project which employs an augmented reality (AR) interface accessed on mobile devices to bring AIE scenarios alive for students in everyday campus contexts. Mobile learning paths called ‘Trails of Integrity and Ethics’ (TIEs) have been created on Hong Kong university campuses, with students walking through study locations where ethical dilemmas might arise, and using an AR app to learn about, consider and respond to a range of problematic scenarios. In addition, subject-specific TIEs have been developed in which students face ethical dilemmas specific to their disciplines, and are tasked with responding according to professional norms and standards.After the first 2 years of this 4-year funded project, more than 1000 students have participated in the TIEs. Analysis of data from their mobile device clickstreams, pre- and post-trail reflective texts and user experience surveys has led to encouraging initial findings. There is some early evidence suggesting that the mobile AR trails have helped students to become more active and engaged in their learning of abstract conceptual knowledge about AIE, and that their perspectives on AIE have changed as they have begun to link ethical dilemmas on the TIEs with their everyday realities.

An Integrated Haptic System combining VR, a Markerless Motion Capture System & Tactile Actuators

In the industrial environments, it is common that robotic or remote interaction with both rigid objects and soft or deformable objects is required. However, it is usual in such an environment that only one mode of manipulation is used, and that little or no distinction is made between rigid or deformable objects. The ability to “feel” or touch an object easy a naturalistic way to determine what type of object is being manipulated. By feeling an object appropriate manipulation techniques can be applied. A novel Virtual Reality (VR) interface is presented that incorporates tactile feedback in order to “feel” objects being manipulated. Incorporation of an important extra “sense” into such a system allows far more nuanced and dexterous interaction to occur in manufacturing environments that may be “messy”, have imprecisely located objects or that have a range of different materials present.

Intuitive robot programming through environment perception, augmented reality simulation and automated program verification

The increasing complexity of products and machines as well as short production cycles with small lot sizes present great challenges to production industry. Both, the programming of industrial robots in online mode using hand-held control devices or in offline mode using text-based programming requires specific knowledge of robotics and manufacturer-dependent robot control systems. In particular for small and medium-sized enterprises the machine control software needs to be easy, intuitive and usable without time-consuming learning steps, even for employees with no in-depth knowledge of information technology. To simplify the programming of application programs for industrial robots, we extended a cloud-based, task-oriented robot control system with environment perception and plausibility check functions. For the environment perception a depth camera and pointcloud processing hardware were installed. We detect objects located in the robot’s workspace by pointcloud processing with ROS and the PCL and add them to the augmented reality user interface of the robot control. The combination of process knowledge from task-oriented application programming and information about available workpieces from automated image processing enables a plausibility check and verification of the robot program before execution. After a robot program has been approved by the plausibility check, it is tested in an augmented reality simulation for collisions with the detected objects before deployment to the physical robot hardware. Experiments were carried out to evaluate the effectiveness of the developed extensions and confirmed their functionality.

Assistive grasping with an augmented reality user interface

Assisting impaired individuals with robotic devices is an emerging and potentially transformative technology. This paper describes the design of an assistive robotic grasping system that allows impaired individuals to interact with the system in a human-in-the-loop manner, including the use of a novel cranio-facial electromyography input device. The system uses an augmented reality interface that allows users to plan grasps online that match their task-oriented intents. The system uses grasp quality measurements that generate more robust grasps by considering the local geometry of the object and the effect of uncertainty during grasp acquisition. This interface is validated by testing with real users, both healthy and impaired. This work forms the foundation for a flexible, fully featured human-in-the-loop system that allows users to grasp known and unknown objects in cluttered spaces using novel, practical human–robot interaction paradigms that have the potential to bring human-in-the-loop assistive devices out of the research environment and into the lives of those that need them.

ÌNIN Holographic Evolving: Toward a Civilization of Symphony

Until we make linear questions, we will receive linear answers. And when will we be able to see the nature of all that exists?This research proposes a new paradigm not based on new orders of ideas, but on essential human faculties and specifically on still unexplored qualities of intuition. Just as human beings started a new interrelation with reality within the shift from pre-linguistic to linguistic era, so today we can start to interface reality through an unexplored intuitive faculty, within the shift from logic thinking into an ‘analogic-intuitive’ consciousness.

An Intelligent Boxing Application through Augmented Reality for two users – Human Computer Interaction attempt.

Background/Objectives: The technological achievements seen in this modern world are very intuitive. One such intuition has resulted in this research. All of us must have seen the movie IRON MAN and the gestures the hero performs are all simply superb and it has been the motivation for us to proceed with this application that we created. Methods/Statistical analysis: Augmented Reality is a technology which enriches the real world with digital information. An Augmented Reality (AR) Interface superimposes digital objects or interactive computer graphics, on to the real world dynamically. Since its introduction, the technology has been capable of presenting possibilities that have been challenging for other technologies to offer and meet. Nevertheless, AR environments has been largely limited to simple browsing or simply viewing of virtual information registered to the real world. In a few more years, AR will definitely change the way individuals view the world. Findings : We focus at making an interactive gesture tracking system using augmented reality that duplicates the real world environment by blending virtual elements into the real world. This idea is then taken to build an application in AR, through which two users can in parallel use the Boxing application using the punch gestures. The application would let the winner announced through finding out the amount of pressure applied for knocking out the boxing bag. We have discussed in this paper the core elements required to develop an application in augmented reality, which holds good for boxing application we developed as well. The tools required are, leap motion device and unity software. A detailed form of creating an application with the output generated is given. AR is going to be the future and the software giants are all investing dollars into it. So this innovation is a step ahead in AR research which can be further extended to many other applications.

Accessible Human-Robot Interaction for Telepresence Robots: A Case Study

AbstractThe quality of life of people with special needs, such as residents of healthcare facilities, may be improved through operating social telepresence robots that provide the ability to participate in remote activities with friends or family. However, to date, such platforms do not exist for this population.Methodology: Our research utilized an iterative, bottomup, user-centered approach, drawing upon our assistive robotics experiences. Based on the findings of our formative user studies, we developed an augmented reality user interface for our social telepresence robot. Our user interface focuses primarily on the human-human interaction and communication through video, providing support for semi-autonomous navigation. We conducted a case study (n=4) with our target population in which the robot was used to visit a remote art gallery.Results: All of the participants were able to operate the robot to explore the gallery, form opinions about the exhibits, and engage in conversation.Significance: This case study demonstrates that people from our target population can successfully engage in the active role of operating a telepresence robot.

Construction of virtual patient model for maxillofacial palpation training system

Palpation has been considered as one of the important diagnostic skills for doctors, which is conducted on a patient to identify the size, the consistency, and the location of a subcutaneous tumor. Accurate palpation skill often detects the cause of disease early. Even in the field of dentistry, palpation has been considered as the important diagnostic skill for dentists. By the way, any practical training of palpation against a patient did not implemented in the dental education because the patient safety has a priority more than its educational effect. Then we propose a training system for maxillofacial palpation by using virtual reality and haptic interface. The details of modeling a virtual patient model for the training system are shown in this paper, and the experimental results are also described.

A Dynamic Blocks Platform Based on Projective Augmented Reality and Tangible Interfaces for Educational Activities

This paper describes a dynamic blocks platform, called ARBlocks, which is based on projective augmented reality and tangible user interfaces aiming early childhood educational activities development. This tool was conceived specifically for children, applying design techniques to determinate the best shape, material and typography for young students. The content is displayed by projectors, which exhibit the required information only on the blocks surface using an automatic projector calibration technique. The blocks are tracked through a frame marker using the moving edges approach with multiple hypotheses to improve robustness. ARBlocks was evaluated from three different perspectives: computational, educational and user experience. Technical results show that the platform achieves a real time frame rate and an accurate projector calibration, as well as precisely displays information over the blocks. Additionally, in the educational evaluation the teachers interviewed asserted that the ARBlocks has a great potential and can be a very useful tool to be used in classrooms. Regarding the user experience, both teachers and children were excited in using this system continuously.

Evaluation of CAD Model Manipulation in Desktop and Multimodal Immersive Interface

In this paper is presented an evaluation study regarding the manipulation of 3D CAD models using multimodal Virtual Reality (VR) interface and traditional desktop environment. A multimodal interface based on VR technologies that offers direct access to the space model by using several sensorial channels (visual, tactile, verbal) is described. Users can manipulate 3D objects using 6DOF tracking device, gestures and visualize the design status in an immersive CAVE - like system. The results of the evaluation study illustrate that using VR technologies as an alternative to replace WIMP CAD software interface is viable and brings more advantages.

Lung segmentation refinement based on optimal surface finding utilizing a hybrid desktop/virtual reality user interface

Recently, the optimal surface finding (OSF) and layered optimal graph image segmentation of multiple objects and surfaces (LOGISMOS) approaches have been reported with applications to medical image segmentation tasks. While providing high levels of performance, these approaches may locally fail in the presence of pathology or other local challenges. Due to the image data variability, finding a suitable cost function that would be applicable to all image locations may not be feasible.This paper presents a new interactive refinement approach for correcting local segmentation errors in the automated OSF-based segmentation. A hybrid desktop/virtual reality user interface was developed for efficient interaction with the segmentations utilizing state-of-the-art stereoscopic visualization technology and advanced interaction techniques. The user interface allows a natural and interactive manipulation of 3-D surfaces. The approach was evaluated on 30 test cases from 18 CT lung datasets, which showed local segmentation errors after employing an automated OSF-based lung segmentation. The performed experiments exhibited significant increase in performance in terms of mean absolute surface distance errors (2.54±0.75mm prior to refinement vs. 1.11±0.43mm post-refinement, p≪0.001). Speed of the interactions is one of the most important aspects leading to the acceptance or rejection of the approach by users expecting real-time interaction experience. The average algorithm computing time per refinement iteration was 150ms, and the average total user interaction time required for reaching complete operator satisfaction was about 2min per case. This time was mostly spent on human-controlled manipulation of the object to identify whether additional refinement was necessary and to approve the final segmentation result. The reported principle is generally applicable to segmentation problems beyond lung segmentation in CT scans as long as the underlying segmentation utilizes the OSF framework. The two reported segmentation refinement tools were optimized for lung segmentation and might need some adaptation for other application domains.

The Human Interface Technology Laboratory New Zealand (HIT Lab NZ)

The Human Interface Technology Laboratory New Zealand (HIT Lab NZ) is the leading research laboratory in New Zealand for Augmented Reality (AR) and advanced user interface research, and one of the largest AR research groups in the world. This paper provides an introduction to the HIT Lab NZ, an overview of its history and a review of past and current and future research projects.

A CORBA-Based Control Architecture for Real-Time Teleoperation Tasks in a Developmental Humanoid Robot

This paper presents the development of new Humanoid Robot Control Architecture (HRCA) platform based on Common Object Request Broker Architecture (CORBA) in a developmental biped humanoid robot for real-time teleoperation tasks. The objective is to make the control platform open for collaborative teleoperation research in humanoid robotics via the internet. Meanwhile, to generate optimal trajectory generation in bipedal walk, we proposed a real time generation of optimal gait by using Genetic Algorithms (GA) to minimize the energy for humanoid robot gait. In addition, we proposed simplification of kinematical solutions to generate controlled trajectories of humanoid robot legs in teleoperation tasks. The proposed control systems and strategies was evaluated in teleoperation experiments between Australia and Japan using humanoid robot Bonten-Maru. Additionally, we have developed a user-friendly Virtual Reality (VR) user interface that is composed of ultrasonic 3D mouse system and a Head Mounted Display (HMD) for working coexistence of human and humanoid robot in teleoperation tasks. The teleoperation experiments show good performance of the proposed system and control, and also verified the good performance for working coexistence of human and humanoid robot.

Towards a Predictive Mixed Reality User Interface for Mobile Robot Teleoperation

Lack of situation awareness significantly decreases the performance in missions where a mobile robot is operated by a human from remote. The user interface is a key influencing element for situation awareness of the human operator. The information from the remote site is limited to what the robot's sensors can provide. In addition, this information is in general only delivered with a certain - maybe varying - communication delay. Predictive displays provide a promising approach to cope with these problems. In order to increase the situation awareness for the human operator in teleoperation scenarios predictive user interfaces can be used to achieve an artificial excoentric view.This work presents an approach how a predictive mixed reality user interface can be realized with the help of motion control theory. The human operator commands a virtual robot projected into the camera image delivered to the human operator from the physical robot. Hereby a trajectory for the real physical robot is generated, which is executed by the physical robot after a certain time. Combined with mixed reality technologies an artificial, exocentric view of the mobile robot is achieved which leads to a short time predictive user interface for mobile-robot teleoperation.

Impact of Spatial Reference Frames on Human Performance in Virtual Reality User Interfaces

The design of virtual reality user interfaces (VRUI) is still an open field of research and development. One category of VRUI is the 3D floating menus that can be manipulated by users in free space. These menus can contain various controls such as buttons, sliders, and text. This article presents an experimental study that aims at testing the impact of five spatial reference frames on human performance with VRUI. Fifteen subjects participated in the study. Wearing a head-mounted display (HMD) and tracking hardware, they were asked to move a slider on a floating menu in a 3D virtual space. The menus were completely virtual such that no physical surface was providing haptic feedback to the user. Main results show that the reference frame has a determinant impact on task execution time, positional accuracy and user preference. We also found a link between the number of degrees of freedom shared between these reference frame and user performance. Results also show that user preference can greatly differ from experimental results obtained by measuring task completion times and positional accuracy.

Augmented reality user interface for an atomic force microscope-based nanorobotic system

A real-time augmented reality (AR) user interface for nanoscale interaction and manipulation applications using an atomic force microscope (AFM) is presented. Nanoscale three-dimensional (3-D) topography and force information sensed by an AFM probe are fed back to a user through a simulated AR system. The sample surface is modeled with a B-spline-based geometry model, upon which a collision detection algorithm determines whether and how the spherical AFM tip penetrates the surface. Based on these results, the induced surface deformations are simulated using continuum micro/nanoforce and Maugis-Dugdale elastic contact mechanics models, and 3-D decoupled force feedback information is obtained in real time. The simulated information is then blended in real time with the force measurements of the AFM in an AR human machine interface, comprising a computer graphics environment and a haptic interface. Accuracy, usability, and reliability of the proposed AR user interface is tested by experiments for three tasks: positioning the AFM probe tip close to a surface, just in contact with a surface, or below a surface by elastically indenting. Results of these tests showed the performance of the proposed user interface. This user interface would be critical for many nanorobotic applications in biotechnology, nanodevice prototyping, and nanotechnology education

BRIDGING THE GAPS: HYBRID TRACKING FOR ADAPTIVE MOBILE AUGMENTED REALITY

Tracking accuracy in a location-aware mobile system can change dynamically as a function of the user's location and other variables specific to the tracking technologies used. This is especially problematic for mobile augmented reality systems, which ideally require extremely precise position tracking for the user's head, but which may not always be able to achieve that level of accuracy. While it is possible to ignore variable positional accuracy in an augmented reality user interface, this can make for a confusing system; for example, when accuracy is low, virtual objects that are nominally registered with real ones may be too far off to be of use. To address this problem, we describe an experimental mobile augmented reality system that: (1) employs multiple position-tracking technologies, including ones that apply heuristics based on environmental knowledge; (2) coordinates these concurrently monitored tracking systems; and (3) automatically adapts the user interface to varying degrees of confidence in tracking accuracy. We share our experiences with managing these multiple tracking technologies, employing various techniques to facilitate smooth and reasonable “hand-offs” between the cooperating systems. We present these results in the context of an intelligent navigational guidance system that helps users to orient themselves in an unfamiliar environment, using path planning to guide them toward destinations they choose, and sometimes towards ones the system infers as equally relevant.

Communication Behaviors in Colocated Collaborative AR Interfaces

The authors present an analysis of communication behavior in face-to-face collaboration using a multi-user augmented reality (AR) interface. 2 experiments were conducted. In the 1st experiment, collaboration with AR technology was compared with more traditional unmediated and screen-based collaboration. In the 2nd experiment, the authors compared collaboration with 3 different AR displays. Several measures were used to analyze communication behavior, and the authors found that users exhibited many of the same behaviors in a collaborative AR interface as in face-to-face unmediated collaboration. However, user communication behavior changed with the type of AR display used. The authors describe implications of these results for the design of collaborative AR interfaces and directions for future research.

Moving Eye - Interactive Telepresence Over Internet with a Ball Shaped Mobile Robot

This paper describes a robotic and automation system presently under development for studying home automation and indoor robotic technologies as well as new service scenarios based on these technologies. The system consists of remote users, a home automation server, home automation equipment and a ball shaped mobile robot called Rollo. A remote user connects to the server from the Internet to interact with the robot by using an augmented reality user interface. The Rollo robot, its behaviour and functions in the system are discussed- Utilizing the system for both educational purposes and to develop a remote home aid system is discussed.

Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system

We describe an experimental mobile augmented reality system (MARS) testbed that employs different user interfaces to allow outdoor and indoor users to access and manage information that is spatially registered with the real world. Outdoor users can experience spatialized multimedia presentations that are presented on a head-tracked, see-through, head-worn display used in conjunction with a hand-held pen-based computer. Indoor users can get an overview of the outdoor scene and communicate with outdoor users through a desktop user interface or a head- and hand-tracked immersive augmented reality user interface.