Natural User Interaction Research Articles

Towards the improvement of the cognitive, motoric and academic skills of students with special educational needs using Kinect learning games

There is an ongoing interest in developing motion Kinect-based learning games that are in accordance with intervention protocols for helping children with special educational needs. These games, which offer natural user interaction modalities, seem to be very beneficial for this population because of the combination of physical activities with cognitive-training tasks. There is still an open research issue about how to integrate such games into schools and how to organize systematic evaluation studies for showing their added value. This paper presents the positive findings of a pilot research study in inclusive classroom settings that were carried out at two primary schools with 20 children who have special educational needs and who used the Kinems suite that contains movement-based educational games for such children. Analysis of data gathered via pre- and post-test questionnaires, interviews and an in-depth study of kinetic and learning analytics showed that these games have a positive impact on childrens academic performance and improvement of their cognitive, motor and academic skills.

Characterizing Natural User Interface with Wearable Smart Watches

Background The emergence of new interaction paradigms makes the use of technology inrealizing the users’ natural ways of exploring the real world the ultimate goal of designers today.Research on interactive and immersive technologies for user interface design is still a challenging chore for engineers and scientists when it comes to designing natural interaction for wearable smart devices. To address the challenge, our study aims to develop guidelines for design practitioners in designing wearable smart watches that could offer natural user experiences. Methods To better understand natural user experiences with smart watches, an extensive literature review was conducted. A quantitative survey with 80 participants was conducted, of which the focus was on the expected functions of smart watches. Based on the survey results, we selected eight participants in terms of technology familiarity. To achieve the objectives of our research, three studies were conducted: a design workshop (Study 1), a cultural probe (Study 2), and a focus group interview (Study 3). The design workshop was created to figure out the needs and wishes people have forsmart watches. In the cultural probe, the focus was on figuring out natural interactions with smart watches. Finally, the focus group interview aimed to gain more insights from the results of the cultural probe in terms of natural user interaction with particular functions. Results To address the needs and wishes of the users toward wearable smartwatches, we made a subdivision into three categories, such as functions, input measures, and notification (feedback) methods.According to the results, participants wanted weather notification, health monitoring, and identification as expected functions. Regarding the methodof input, voice command and touch screen were preferred. In order to get feedback, most of the participantswanted vibrations, particularly as a reaction tocompleting the commands or inputs. There was also a suggestion to customize their smart watch. For example, users can select the functions and build their own command system, and even choose the notificationmethods. Considering natural user interface with respect to functions (weather, answering a call, navigation, health monitoring, taking a picture and messaging), specific natural user interfaces were mentioned for particular functions. Conclusions Throughout the study, people’s needs and wishes and their perceptions about natural interaction were identified and the characteristics of natural user interfacesweredetermined. Based on the results, tenperceptions were specifically defined to provide a better understanding of smart watches in terms of natural interaction: user affinity of form, awareness by familiarity, reality correspondence, behavioral extension, purpose orientation, easiness of performance, timeliness, routine acceptance, generality, and rule of thumb. In addition to that, natural user interfaces were categorized into five groups: user familiarity, realistic interaction, accomplishment assistance, contextual appropriateness, and social awareness. In this study,we tried to identify what constitutes anatural interaction and how it should be created. The limitations and further study are discussed at the end.

Assessment of a Microsoft Kinect-based 3D scanning system for taking body segment girth measurements: a comparison to ISAK and ISO standards

ABSTRACTUse of anthropometric data to infer sporting performance is increasing in popularity, particularly within elite sport programmes. Measurement typically follows standards set by the International Society for the Advancement of Kinanthropometry (ISAK). However, such techniques are time consuming, which reduces their practicality. Schranz et al. recently suggested 3D body scanners could replace current measurement techniques; however, current systems are costly. Recent interest in natural user interaction has led to a range of low-cost depth cameras capable of producing 3D body scans, from which anthropometrics can be calculated. A scanning system comprising 4 depth cameras was used to scan 4 cylinders, representative of the body segments. Girth measurements were calculated from the 3D scans and compared to gold standard measurements. Requirements of a Level 1 ISAK practitioner were met in all 4 cylinders, and ISO standards for scan-derived girth measurements were met in the 2 larger cylinders only. A fixed measurement bias was identified that could be corrected with a simple offset factor. Further work is required to determine comparable performance across a wider range of measurements performed upon living participants. Nevertheless, findings of the study suggest such a system offers many advantages over current techniques, having a range of potential applications.

Multi-modal natural interaction in game design: a comparative analysis of player experience in a large scale role-playing game

Previous work on player experience research has focused on identifying the major factors involving content creation and interaction. This has encouraged a large investment in new types of physical interaction artefacts (e.g. Wiimote™, Rock Band™, Kinect™). However, these artefacts still require custom interaction schemes to be developed for them, which critically limits the number of commercial videogames and multimedia applications that can benefit from those. Moreover, there is currently no agreement as to which factors better describe the impact that natural and complex multi-modal user interaction schemes have on users’ experiences—a gap in part created by the limitations in adapting this type of interaction to existing software. Thus, this paper presents a generic middleware framework for multi-modal natural interfaces which enables game-independent data acquisition that encourages further advancement on this domain. Furthermore, our framework can then redefine the interaction scheme of any software tool by mapping body poses and voice commands to traditional input means (keyboard and mouse). We have focused on digital games, where the use of physical interaction artefacts has become mainstream. The validation methods for this tool consisted of a series of increasing difficulty stress tests, with a total of 25 participants. Also, a pilot study was conducted on a further 16 subjects which demonstrated mainly positive impact of natural interfaces on player’s experience. The results supporting this were acquired when subjects played a complex commercial role-playing game whose mechanics were adapted using our framework; statistical tests on the obtained Fun ratings, along with subjective participant opinions indicate that this kind of natural interaction indeed has a significant impact on player’s experience and enjoyment. However, different impact patterns emerge from this analysis, which seem to fit with standing theories of player experience and immersion.

User-designed movement interactions. An exploratory study for natural interactions

The context of this paper is movement and gesture-based interaction for natural user interaction (NUI) design. Naturalness is not an intrinsic but a perceived quality; and although movement seems an ideal candidate for NUIs, it bears its own limitations. Although experience can be drawn from other interaction styles (e.g. desktop graphical interfaces, touch), we advocate that movement - based natural interaction should also be tackled from a fresh perspective. Many existing works are technology - driven; our approach focuses on users’ definitions of interactions, leaving the technical issues for later works. In this paper, we present an experiment that identifies a set of user-defined gestures for a set of core tasks in freehand movement interaction using a large screen display. The experiment is divided into two steps, for exploration and validation. The exploratory step uses a bottom-up approach. We use the context of a large screen television. Subjects were asked to spontaneously perform gestures to achieve a set of typical tasks (e.g. pointing, selecting, engaging/disengaging) in a wizard-of-Oz setup. We isolated the favored interactions for such tasks. The validation step uses a top-down approach. We used a motion capture device to produce skeleton videos of the gestures retained from step one. An online questionnaire allowed validating the first findings and assessing how the suggested interactions can be perceived. From those experiment we drew five natural, typical interactions corresponding to five tasks, that are heavily preferred by subjects and which form good candidates as controls in future interaction designs.

Natural User Interfaces to Teach Math on Higher Education

A common and known problem among students on higher education (in general throughout the scholar life) is the difficulty of learning math. Nowadays the teaching methods for learning math remain relatively the same. Though, with the emerging of new technologies like portable devices (smart phones and tablets) and movement interaction devices (Nintendo WiiMote, Microsoft Kinect and PlayStation Move), learners and teachers have big interest on new ways of interacting through the teaching-learning process. In this sense the Natural User Interactions (NUIs) offer a great potential to facilitate new ways of computer enhanced learning, these have the potential to enhance classroom interactions, by increasing learners participation, facilitating the teachers’ presentations and creating opportunities for discussion. We present a system that combines gestural and touch interactions to support teaching math across multiple personal devices and public displays to enhance and support math education for college students. In a formative usability study, learners and teachers were positive about the interaction design and the learning possibilities for math education. Thus, this created good intentions in the users of continuing using it.

Interaction Design for Mobile Visual Search

Mobile devices are becoming ubiquitous. People take pictures via their phone cameras to explore the world on the go. In many cases, they are concerned with the picture-related information. Understanding user intent conveyed by those pictures therefore becomes important. Existing mobile applications employ visual search to connect the captured picture with the physical world. However, they only achieve limited success due to the ambiguity nature of user intent in the picture-one picture usually contains multiple objects. By taking advantage of multitouch interactions on mobile devices, this paper presents a prototype of interactive mobile visual search, named TapTell, to help users formulate their visual intent more conveniently. This kind of search leverages limited yet natural user interactions on the phone to achieve more effective visual search while maintaining a satisfying user experience. We make three contributions in this work. First, we conduct a focus study on the usage patterns and concerned factors for mobile visual search, which in turn leads to the interactive design of expressing visual intent by gesture. Second, we introduce four modes of gesture-based interactions (crop, line, lasso, and tap) and develop a mobile prototype. Third, we perform an in-depth usability evaluation on these different modes, which demonstrates the advantage of interactions and shows that lasso is the most natural and effective interaction mode. We show that TapTell provides a natural user experience to use phone camera and gesture to explore the world. Based on the observation and conclusion, we also suggest some design principles for interactive mobile visual search in the future.

Toward Nonconventional Human–Machine Interfaces for Supervisory Plant Process Monitoring

This paper describes a novel methodology for designing alternative human-machine interfaces (HMI) dealing with the industrial process control visualization and plant monitoring. The system is based on a multiagent approach in order to allow visualizations using nonconventional display devices (e.g., power-wall or table) combined with the natural user interaction (NUI) paradigm. This type of HMI solution could form an optional extension to current systems employed in a plant monitoring. Namely the utilization of a virtual reality creates new opportunities for the HMI system use cases, where an enhanced visualization and interaction can improve decision making strategies in complex processes. An immersive plant personnel training or realistic process visualizations are examples where the usage of the third dimension can be helpful. Nevertheless, the development of nonconventional HMI increases the complexity of the industrial information system environment. The question is how to adapt the system design for the nonconventional HMI, in a contrast with the conventional solutions. We applied the multiagent approach, which leads to a more robust and less-coupled component structure. An experimental prototype was implemented on the top of the proposed methodology. A simulation of an absorption refrigeration process was used as a process model on top of which the prototype was designed. Interoperability was gained via the automation standard of OPC UA. Furthermore, in order to optimize our NUI agent, a user testing application was developed for the evaluation of exploratory interaction tasks in a power-wall display scenario. The proposed framework provides fundamental guidelines for designing and developing a new generation of HMI systems.

Interactive Multimodal Visual Search on Mobile Device

This paper describes a novel multimodal interactive image search system on mobile devices. The system, the Joint search with ImaGe, Speech, And Word Plus (JIGSAW ${+}$ ), takes full advantage of the multimodal input and natural user interactions of mobile devices. It is designed for users who already have pictures in their minds but have no precise descriptions or names to address them. By describing it using speech and then refining the recognized query by interactively composing a visual query using exemplary images, the user can easily find the desired images through a few natural multimodal interactions with his/her mobile device. Compared with our previous work JIGSAW, the algorithm has been significantly improved in three aspects: 1) segmentation-based image representation is adopted to remove the artificial block partitions; 2) relative position checking replaces the fixed position penalty; and 3) inverted index is constructed instead of brute force matching. The proposed JIGSAW ${+}$ is able to achieve 5% gain in terms of search performance and is ten times faster.

An interdisciplinary methodology for designing and implementing educational tools for children and youth with special needs

This paper describes our research project where we are working in bringing closer new advanced technologies, like Augmented Reality (AR), Virtual Reality (VR) or Natural User Interaction (NUI), to children and youth with special needs. With this aim, we have created an interdisciplinary workgroup consisting of a sustainable platform for the design, development and evaluation of educational tools. After experimental sessions with a group of children testing our prototypes, we are defining a working methodology to carry out our project. We discuss our current progress, open questions, and benefits for the accessibility field.

Feature, design intention and constraint preservation for direct modeling of 3D freeform surfaces

Direct modeling has recently emerged as a suitable approach for 3D free-form shape modeling in industrial design. It has several advantages over the conventional, parametric modeling techniques, including natural user interactions, as well as the underlying, automatic feature-preserving shape deformation algorithms. However, current direct modeling packages still lack several capabilities critical for product design, such as managing aesthetic design intentions, and enforcing dimensional, geometric constraints.In this paper, we describe a novel 3D surface editing system capable of jointly accommodating aesthetic design intentions expressed in the form of surface painting and color-coded annotations, as well as engineering constraints expressed as dimensions. The proposed system is built upon differential coordinates and constrained least squares, and is intended for conceptual design that involves frequent shape tuning and explorations. We also provide an extensive review of the state-of-the-art direct modeling approaches for 3D mesh-based, freeform surfaces, with an emphasis on the two broad categories of shape deformation algorithms developed in the relevant field of geometric modeling.

Browsing museum image collections on a multi-touch table

In this paper we present a multi-touch tabletop system for browsing image databases, conceived for museums and art gallery exhibitions. The system exploits an innovative image browsing paradigm and image retrieval functionalities to perform natural and intuitive user interaction: users can explore the image database by handling digital pictures by means of touch gestures or by the use of a predetermined set of physical objects; once one of these objects is placed on the table, it is automatically recognized and the associated function is triggered. The set of objects and the function associations can be dynamically configured. An innovative feature of our application is that users can interactively create and manipulate image clusters where images are grouped according to their pictorial similarity. This is achieved by placing one or more specific tangible objects on the table surface. The system has been evaluated on a collection of photos organized in groups according to the UNESCO picture categories. The usability tests, performed considering different user categories, show that users consider the application to be attractive and interesting.

Immersive manipulation of virtual objects through glove-based hand gesture interaction

Immersive visualisation is increasingly being used for comprehensive and rapid analysis of objects in 3D and object dynamic behaviour in 4D. Challenges are therefore presented to provide natural user interaction to enable effortless virtual object manipulation. Presented in this paper is the development and evaluation of an immersive human–computer interaction system based on stereoscopic viewing and natural hand gestures. For the development, it is based on the integration of a back-projection stereoscopic system for object and hand display, a hybrid inertial and ultrasonic tracking system to provide the absolute positions and orientations of the user’s head and hands, as well as a pair of high degrees-of-freedom data gloves to provide the relative positions and orientations of digit joints and tips on both hands. For the evaluation, it is based on a two-object scene with a virtual cube and a CT (computed tomography) volume created for demonstration of real-time immersive object manipulation. The system is shown to provide a correct user view of objects and hands in 3D with depth, as well as to enable a user to use a number of simple hand gestures to perform basic object manipulation tasks involving selection, release, translation, rotation and scaling. Also included in the evaluation are some quantitative tests of the system performance in terms of speed and latency.

Supporting agile development by facilitating natural user interaction with executable formal specifications

Agile development benefits from fast feedback from various stakeholders. If implemented in a suitable way, formal methods can enhance the agile development process. With an executable formal specification, it is possible to analyse and simulate the behaviour of the target system before it is being built. However, for the users' and developers' natural participation in the development process, it is necessary to use a real end-user interface and bind it to the execution environment being used in the simulations and animations. This requires, though, that the execution model used to simulate the specification is appropriately changed to facilitate the use of these user interfaces. The authors present a formal and flexible method to facilitate natural user interaction with executable specifications through end-user interfaces. This can be considered as an agile and formal way which provides continuous testable software components through frequently communicated stakeholders' views. The method is based on a modification of the execution of the DisCo system, a software for creating and animating formal specifications. Usage of the method is demonstrated through an exploratory study of a game application.

An automated online crisis dispatcher

During crisis events, human operators in a crisis centre will be overloaded with a large amount of information. The stress of dealing with crisis situations can have a significant impact on the certainty of the information. The need for a system that is able to handle information calls automatically may then become apparent. Our research is about a dialogue system that can play as a crisis hotline dispatcher. The dialogue system offers a natural user interaction through its ability to start a user-friendly dialogue taking care of the content, context and user's emotion. It retrieves information about crisis situations from users while controlling the communication flow. The system is able to recognise the emotion loading of the user's linguistic content. The recogniser uses a database that contains selected keywords on a 2D 'arousal' and 'valence' scale. Its output includes an indication of the urgency of the information regarding the crisis.

Virtual hands and virtual reality multimodal platform to design safer industrial systems

To face with the competitiveness in product design, industrials come up with the solution to use virtual reality (VR) techniques. Coupled with a dynamic simulation, those techniques lead to natural user interactions with virtual environments (VE). Our research focuses on how to model the hands of the operator because they allow him to interact with the environment. In this paper, we address the problem of VR applications to design for a better integration of safety and health requirements. After reviewing the industrial applications using VR, we present our virtual hands which are coupled with a virtual press-brake by using a system of motion capture and a force feedback device. Thus, the operator can interact in real-time with the VE. Our simulation tool is also interfaced to a product model that allows configuring the machine itself. As a result, we are able to estimate the risk level of this machine tool.

A Framework for Virtual Reality with Tangible Augmented Reality-Based User Interface

In this paper, we propose a framework for virtual reality, I2-NEXT, which enables users to interact with virtual objects by tangible objects in immersive networked virtual environment. The primary goal of this framework is to support rapid development of immersive and interactive virtual reality systems as well as various types of user interfaces. The proposed framework consists of user interface for interactions, immersive virtual environment, and networking interface. In this framework, we adopt several design patterns to guarantee that either developers or users (artists) can easily implement their VR applications without strong knowledge of VR techniques such as programming, libraries etc. One of the key features of this framework is the presence of the device module which supports a natural user interaction in a virtual environment. For example, the proposed framework provides users with tangible objects so that the users are able to manipulate virtual objects by touching real objects. The proposed framework also supports large scale stereoscopic display through clustering technique. To realize the effectiveness of the proposed framework, we have been developing an application for digital heritage reconstruction. Having been through development of the system, we believe that virtual reality technology is one of the promising technologies which enable users to experience realities in a digital space. Detailed explanations of each component and system architecture are presented.

A Semantic Space Partitioning Approach to Virtual Camera Composition

In this paper, we present a semantic space partitioning (SSP) approach to the virtual camera composition problem. Virtual camera composition (VCC) consists in positioning a camera in a virtual world, such that the resulting image satises a set of visual cinematographic properties. Whereas most related works concentrate on numerically computing a unique camera position satisfying the problem, we offer to isolate identical possible solutions in 3D volumes with respect to their visual properties, and to propose them to the user. We introduce the notion of semantic volumes as an extension of visual aspects to characterize, compute and manipulate distinct solution sets. Our approach relies on (1) a space partitioning process derived from a study of possible camera locations w.r.t. to the objects in the scene and (2) local search numerical techniques to compute good representatives of each volume. This work is motivated by the lack of VCC tools in 3D software and the will to integrate cinematographic semantics in the description, solving and interaction processes. Experimental results illustrate the suitability of our approach for identifying and providing distinct solution sets. Furthermore, the exploitation of the semantic volumes lays the groundwork for natural and efcient user interaction by providing knowledge and reasoning on possible classes of solutions.

Facial expressive robotic head system for human-robot communication and its application in home environment

This paper describes a robotic-head system as a multimodal communication device for human-robot interaction, and the system's potential application in home environments. Most robotic systems for natural user interaction have facial expressions, since facial expressiveness is regarded as a key component to developing personal attachment along with prosodic expressiveness. In the first part of the paper is the description of our robotic head system Character Robot Face (CRF). A deformation approach and a parametric normalization scheme are proposed to produce facial expressions of nonhuman face models with high recognition rates. In the second half of the paper, CRF is endowed with artificial emotions and assigned tasks conceivable in home environments. A coordination mechanism between the robot's mood (an activated emotion) and its task is proposed so that the robot can, by referring to the emotion-task history, select a task depending on its current mood if there is no explicit task command from the user. When the robot performs a task, a particular emotion value gets boosted according to the same emotion-task history so that the emotion is more likely to be activated.