Complex Interaction Tasks Research Articles

Spatial Augmented Reality for Expanding the Reach of Individuals with Tremor beyond Their Physical Limits.

Tremor is a prevalent neurological disorder characterized by involuntary shaking or trembling of body parts. This condition impairs fine motor skills and hand coordination to varying degrees and can even affect overall body mobility. As a result, tremors severely disrupt the daily lives and work of those affected, significantly limiting their physical activity space. This study developed an innovative spatial augmented reality (SAR) system aimed at assisting individuals with tremor disorders to overcome their physical limitations and expand their range of activities. The system integrates eye-tracking and Internet of Things (IoT) technologies, enabling users to smoothly control objects in the real world through eye movements. It uses a virtual stabilization algorithm for stable interaction with objects in the real environment. The study comprehensively evaluated the system's performance through three experiments: (1) assessing the effectiveness of the virtual stabilization algorithm in enhancing the system's ability to assist individuals with tremors in stable and efficient interaction with remote objects, (2) evaluating the system's fluidity and stability in performing complex interactive tasks, and (3) investigating the precision and efficiency of the system in remote interactions within complex physical environments. The results demonstrated that the system significantly improves the stability and efficiency of interactions between individuals with tremor and remote objects, reduces operational errors, and enhances the accuracy and communication efficiency of interactions.

Pushing with Soft Robotic Arms via Deep Reinforcement Learning

Soft robots can adaptively interact with unstructured environments. However, nonlinear soft material properties challenge modeling and control. Learning‐based controllers that leverage efficient mechanical models are promising for solving complex interaction tasks. This article develops a closed‐loop pose/force controller for a dexterous soft manipulator enabling dynamic pushing tasks using deep reinforcement learning. Force tests investigate the mechanical properties of a soft robot module, resulting in orthogonal forces of N. Then, the policy is trained in simulation leveraging a dynamic Cosserat rod model of the soft robot. Domain randomization mitigate the sim‐to‐real gap while careful reward engineering induced pose and force control even without explicit force inputs. Despite the approximate simulation, the sim‐to‐real transfer achieved an average reaching distance of mm (), an average orientation error of rad () and applied pushing forces up to N. Such performance is reasonable for the intended assistive tasks of the manipulator. The experiments uncovered that the soft robot interacting with the environment exhibited torsional and counter‐balancing movements. Although not explicitly enforced, they emerged from the mechanical intelligence of the manipulator. The results demonstrate the potential of soft robotic manipulation via reinforcement learning.

SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation Gesture

Today, touchscreens stand as the most prevalent input devices of mobile computing devices (smartphones, tablets, smartwatches). Yet, compared with desktop or laptop computers, the limited shortcut keys and physical buttons on touchscreen devices, coupled with the fat finger problem, often lead to slower and more error-prone input and navigation, especially when dealing with text editing and other complex interaction tasks. We introduce an innovative gesture set based on finger rotations in the yaw, pitch, and roll directions on a touchscreen, diverging significantly from traditional two-dimensional interactions and promising to expand the gesture library. Despite active research in estimation of finger angles, however, the previous work faces substantial challenges, including significant estimation errors and unstable sequential outputs. Variability in user behavior further complicates the isolation of movements to a single rotational axis, leading to accidental disturbances and screen coordinate shifts that interfere with the existing sliding gestures. Consequently, the direct application of finger angle estimation algorithms for recognizing three-dimensional rotational gestures is impractical. SwivelTouch leverages the analysis of finger movement characteristics on the touchscreen captured through original capacitive image sequences, which aims to rapidly and accurately identify these advanced 3D gestures, clearly differentiating them from conventional touch interactions like tapping and sliding, thus enhancing user interaction with touch devices and meanwhile compatible with existing 2D gestures. User study further confirms that the implementation of SwivelTouch significantly enhances the efficiency of text editing on smartphones.

Interactive learning for multi-finger dexterous hand: A model-free hierarchical deep reinforcement learning approach

When a multi-fingered dexterous hand interacts with the external environment, it encounters various challenges, including the utilization of complex control techniques and the intricate coordination of finger motion sequences. Previous studies have primarily concentrated on investigating the interaction between multi-fingered dexterous hands and external objects, usually using model-based control or model-free reinforcement learning techniques. However, during practical implementation, various constraining factors are encountered, such as intricate modeling and limited interaction capabilities. In practical scenarios, the utilization of multi-fingered dexterous hands is imperative for the swift and efficient execution of a wide range of interactive tasks, including but not limited to throwing a ball and playing rock-paper-scissors. These tasks require skilled manual dexterity to demonstrate both precise control and quick responsiveness. To tackle this issue, we propose a hierarchical control approach for multi-fingered dexterous hands with interactive functionalities, utilizing model-free deep reinforcement learning. The complex interaction task is decomposed into simple sub-tasks using hierarchical strategy and action primitive decomposition, which effectively reduces the complexity of the action space, and achieves the motion planning and end finger trajectory control of dexterous hand. In a simulated environment, the aforementioned method has successfully executed interactive tasks, including ball throwing and playing rock-paper-scissors. It achieved a maximum normalized reward of 0.83 and an 84 % success rate. These results are noteworthy in terms of both control accuracy and response speed. This study offers novel insights into the effective resolution of the intricate challenges associated with interactions involving multi-fingered dexterous hands and human-computer interaction.

A novel visual tracking scheme for unstructured indoor environments

In the ever-expanding sphere of assistive robotics, the pressing need for advanced methods capable of accurately tracking individuals within unstructured indoor settings has been magnified. This research endeavours to devise a realtime visual tracking mechanism that encapsulates high performance attributes while maintaining minimal computational requirements. Inspired by the neural processes of the human brain’s visual information handling, our innovative algorithm employs a pattern image, serving as an ephemeral memory, which facilitates the identification of motion within images. This tracking paradigm was subjected to rigorous testing on a Nao humanoid robot, demonstrating noteworthy outcomes in controlled laboratory conditions. The algorithm exhibited a remarkably low false detection rate, less than 4%, and target losses were recorded in merely 12% of instances, thus attesting to its successful operation. Moreover, the algorithm’s capacity to accurately estimate the direct distance to the target further substantiated its high efficacy. These compelling findings serve as a substantial contribution to assistive robotics. The proficient visual tracking methodology proposed herein holds the potential to markedly amplify the competencies of robots operating in dynamic, unstructured indoor settings, and set the foundation for a higher degree of complex interactive tasks.

Dynamic fluctuations in foreign language enjoyment during cognitively simple and complex interactive speaking tasks

Despite evidence on the interaction between cognitive individual differences (IDs) and task complexity, our knowledge of how affective IDs, such as foreign language enjoyment (FLE), interact with task complexity and other factors is limited. Since tasks and activities were found by Dewaele and MacIntyre (2014) to be most relevant to FLE, and since task complexity might interact with learners’ perceptions of task difficulty, it is important to investigate how task complexity impacts FLE changes. Informed by the complex dynamic systems theory, this study employed a mixed-methods multiple case study design to study patterns and causes of high and low FLE arousals. The participants were four pairs of Taiwanese high-intermediate EFL university students who were engaged in simple or complex storytelling tasks with speech acts of refusals. The speakers’ interactions were triangulated with an individual learner’s rating of FLE on a per-second scale and stimulated recalls. Results revealed idiosyncratic patterns of FLE fluctuations of peer interlocutors and a high degree of overlap in sources of low and high FLE in both groups. Speakers reported high FLE as a result of interesting storylines inherent in task design and created by peers, the use of picture prompts, peer collaboration, and task performance. Performance problems, failure to retrieve appropriate vocabulary, task design, and lack of ideas led to low FLE arousals. The findings suggest that task complexity combined with other task-induced, social, and individual factors to affect the fluctuations of FLE. Implications for task design and oral communication instruction to promote FLE are discussed.

After a Decade of Teleimpedance: A Survey

Despite the significant progress made in making robots more intelligent and autonomous, today, teleoperation remains a dominant robot control paradigm for the execution of complex and highly unpredictable tasks. Attempts have been made to make teleoperation systems stable, easy to use, and efficient in terms of physical interactions between the follower remote robot and the environment. In particular, the emergence of torque-controlled robots has permitted to regulate the interaction forces from a distance through direct force or impedance control, enabling them to engage in complex interaction tasks. Exploiting this feature, the concept of teleimpedance control was introduced as an alternative method to bilateral force-reflecting teleoperation. The aim was to create a feed-froward yet contact-efficient teleoperation by enriching the leader commands with desired impedance profiles while executing a task. Since then, the teleimpedance concept has found its way into a wide range of interface and controller designs, as well as application domains. Accordingly, after a decade of research progress, this survey aims to provide: first, a convenient introduction of the concept to new researchers in the field, second, consolidate the existing state-of-the-art for active researchers, third, and discuss the pros and cons of different methods in terms of interface and force feedback to provide guidelines for different applications and future developments.

A Survey of Wheeled Mobile Manipulation: A Decision-Making Perspective

Abstract Mobile manipulators that combine base mobility with the dexterity of an articulated manipulator have gained popularity in numerous applications ranging from manufacturing and infrastructure inspection to domestic service. Deployments span a range of interaction tasks with the operational environment comprising minimal interaction tasks such as inspection and complex interaction tasks such as logistics resupply and assembly. This flexibility, offered by the redundancy, needs to be carefully orchestrated to realize enhanced performance. Thus, advanced decision-support methodologies and frameworks are crucial for successful mobile manipulation in (semi-) autonomous and teleoperation contexts. Given the enormous scope of the literature, we restrict our attention to decision-support frameworks specifically in the context of wheeled mobile manipulation. Hence, here, we present a classification of wheeled mobile manipulation literature while accounting for its diversity. The intertwining of the deployment tasks, application arenas, and decision-making methodologies are discussed with an eye for future avenues for research.

Light‐field head‐mounted displays reduce the visual effort: A user study

AbstractHead‐mounted displays (HMDs) allow the visualization of virtual content and the change of view perspectives in a virtual reality (VR). Besides entertainment purposes, such displays also find application in augmented reality, VR training, or tele‐robotic systems. The quality of visual feedback plays a key role for the interaction performance in such setups. In the last years, high‐end computers and displays led to the reduction of simulator sickness regarding nausea symptoms, while new visualization technologies are required to further reduce oculomotor and disorientation symptoms. The so‐called vergence–accommodation conflict (VAC) in standard stereoscopic displays prevents intense use of 3D displays, so far. The VAC describes the visual mismatch between the projected stereoscopic 3D image and the optical distance to the HMD screen. This conflict can be solved by using displays with correct focal distance. The light‐field HMD of this study provides a close‐to‐continuous depth and high image resolution enabling a highly natural visualization. This paper presents the first user‐study on the visual comfort of light‐field displays with a close‐to‐market HMD based on complex interaction tasks. The results provide first evidence that the light‐field technology brings clear benefits to the user in terms of physical use comfort, workload, and depth matching performance.

Effects of task complexity on L2 suggestions

Abstract This exploratory study aims at determining whether increased task complexity affects performance of second language (L2) pragmatics. 34 Spanish learners of English as a foreign language (EFL) undertook simple and complex interactive tasks targeting the speech act of suggesting. Although previous analyses of learner-learner interaction have demonstrated that increasing the cognitive load of a task may affect the number of speech acts in conversation, not enough evidence to support this premise was found. In addition, the assessment of suggestions as provided by native speakers of American English shows that increased task complexity along number of elements, social distance, and degree of imposition is likely to promote accuracy and complexity of pragmatic moves without making L2 learners trade off either accuracy or complexity. Pedagogical implications in the fields of L2 pragmatics and task-based language teaching (TBLT) are further discussed.

Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning.

Accurately modelling user behaviour has the potential to significantly improve the quality of human-computer interaction. Traditionally, these models are carefully hand-crafted to approximate specific aspects of well-documented user behaviour. This limits their availability in virtual and augmented reality where user behaviour is often not yet well understood. Recent efforts have demonstrated that reinforcement learning can approximate human behaviour during simple goal-oriented reaching tasks. We build on these efforts and demonstrate that reinforcement learning can also approximate user behaviour in a complex mid-air interaction task: typing on a virtual keyboard. We present the first reinforcement learning-based user model for mid-air and surface-aligned typing on a virtual keyboard. Our model is shown to replicate high-level human typing behaviour. We demonstrate that this approach may be used to augment or replace human testing during the validation and development of virtual keyboards.

A constraint-based programming approach for robotic assembly skills implementation

• A constraint-based programming formalism for the task-level implementation of robotic assembly skills that allows the skill developer to integrate force control requirements within the skill specification, independently of the specific robotic platform and assembly application. • Reactive force controlled motions are generated in real-time based on force-related objectives and constraints. • The proposed framework opens up the possibility for a non-expert factory worker to intuitively program complex assembly task by simple concatenation of assembly skills. • Demonstrations of a complete assembly process performed with the ABB YuMi dualarm robot in a sensorless configuration. The features of modern collaborative robots, mainly their kinematic redundancy combined with the light-weight structure, can be fully exploited in parts assembly. Traditional robot-level paradigm to robot programming, that requires to explicitly specify the motion of the robot and allows to use contact forces for motion supervision only, cannot be easily applied to complex interaction tasks, such as robotic assembly. Instead, by shifting paradigm to skill-based programming, it is possible to specify force control actions at task level and inherently provide compliant capabilities, without the need to specify the motions of the robot. To this end, this paper presents a constraint-based programming framework for the implementation of assembly skills for light-weight redundant robots, enabling a reactive generation of motion trajectories based on force control requirements. The effectiveness of the proposed approach is experimentally validated on a bimanual assembly use case performed with the ABB YuMi dual-arm robot, requiring a peg-in-hole insertion and a cap-rotation task. Estimation of the interaction force/torque additionally enables the execution of the assembly operation without the need for exteroceptive sensors.

Variable impedance interaction and demonstration interface design based on measurement of arm muscle co-activation for demonstration learning

Learning from Demonstration (LfD) is a good way for robots to learn some complex tasks from human tutors, but the ordinary demonstration mode of LfD is unsuitable for learning interaction tasks that require the robot to run under the variable impedance control mode. In this study, we develop a variable impedance physical human–robot interaction (pHRI) interface and LfD mode for human motion skill transport based on surface electromyography (sEMG) signal and kinesthetic teaching. The pHRI interface and LfD mode developed in this study have the following advantages: 1) they can provide intuitive and pleasing variable impedance interaction effect for kinesthetic teaching on the basis of human arm stiffness estimation using sEMG signal. 2) They can simultaneously provide the expected motion data (expected end-effector trajectory and impedance gain scheduling) of tutors for demonstration learning, and these motion data can be used for learning complex interaction tasks under robot variable impedance control mode. 3) Kinesthetic teaching can provide tutor haptic feedback, which conforms to the human–human teaching mode, thereby reducing professional knowledge requirements and energy consumption of the tutor and improving teaching effect and efficiency. The pHRI interface and LfD mode are experimentally tested by two interaction and demonstration experiments. Results show that the proposed interface and mode have better dynamic interaction characteristic and performance than the constant impedance gain control LfD mode. Moreover, the tutor can be rapidly familiarized with this interface and mode and finish the demonstration with low energy consumption.

Empowering Technology Enabled Care Using <italic>IoT</italic> and Smart Devices: A Review

Designing power aware wearable devices is the main key in building compact size autonomous smart devices to successfully connect health Internet of things solutions. With their ability to perform tasks ranging from simple self-monitoring to complex interactive tasks, these devices hold great promises in providing a large scale cost effective solution to the challenges facing nowadays healthcare systems. Despite the advances in sensing and hardware design, there still remain several technical challenges facing the research community to build devices that meet the computational requirements with a self-powered capability. Overcoming these challenges require major improvements in all the building blocks of wearable devices including sensors, power management, signal processing, computing architectures, and communication. This paper surveys some of the past milestones related to these subsystems and discusses promising research directions addressing their limitations.

Modeling and Predicting Mobile Phone Touchscreen Transcription Typing Using an Integrated Cognitive Architecture

ABSTRACTModeling typing performance has values in both the theory and design practice of human–computer interaction. Previous models have simulated desktop keyboard transcription typing performance; however, as the increasing prevalence of smartphones, new models are needed to account for mobile phone touchscreen typing. In the current study, we built a model for mobile phone touchscreen typing in an integrated cognitive architecture and tested the model by comparing simulation results with human results. The results showed that the model could simulate and predict interkey time performance in both number typing (Experiment 1) and sentence typing (Experiment 2) tasks. The model produced results similar to the human data and captured the effects of digit/letter position and interkey distance on interkey time. The current work demonstrated the predictive power of the model without adjusting any parameters to fit human data. The results from this study provide new insights into the mechanism of mobile typing performance and support future work simulating and predicting detailed human performance in more complex mobile interaction tasks.

Towards an automatic user profiling system for online information sites

Purpose – The purpose of this paper is to identify demographic differences based on how users interact with web applications. The research is needed to develop future systems able to adapt the representation of online information to the user’s specific needs and preferences improving its usability. The following question guides this quest: is there a direct relationship between age and/or gender and interaction? Design/methodology/approach – GOMS (goals, operators, methods, and selection rules) analysis was used to reduce complex interaction tasks into basic operators like pointing, dragging, typing, etc. An experiment was designed to analyse the user performance in the use of these operators through five complex tasks: point-and-click, drag-and-drop, text selection, text edition and menu selection. The sample comprises 592 individuals which took part in the experiment. The performance was analysed using multivariate regression analysis. User laterality and the the user experience were used as control variables. Findings – The factors studied are significant enough to support user classification. The analysis evidenced that men performed significantly better than women when executing interaction pointing and dragging GOMS’s operators, but no significant differences arose with regard to the performance in the typing operators. Older users performed worse in all the interaction tasks. No significant performance differences were detected between left and right-handed users. Research limitations/implications – The study pretends to lay the ground for developing artificial intelligence-based classification systems (e.g. neural networks, decision trees, etc.) able to detect significant differences in user performance, classifying users according to their age, gender and laterality. Practical implications – This user profiling would drive the organisation, selection and representation of the online information according to the specific preferences and needs of each user. This would allow the design of new personalisation algorithms able to perform dynamic adaptation of user interfaces in order to improve the usability of online information systems. Originality/value – This work extends previous research on user performance under a new approach and improved accuracy. First, it relies on the combined and simultaneous analysis of ageing and gender and the use of user laterality and experience as control variables. Second, the use of the GOMS analysis allowed the design of tests that closely resemble the user interaction in online information systems. Third, the size of the sample used in this analysis is much bigger than those used in previous works, allowing a more thorough data analysis which includes the estimation of an advanced model which is quantile regression.

Improved depth perception with three-dimensional auxiliary display and computer generated three-dimensional panoramic overviews in robot-assisted laparoscopy.

In comparison to open surgery, endoscopic surgery offers impaired depth perception and narrower field-of-view. To improve depth perception, the Da Vinci robot offers three-dimensional (3-D) video on the console for the surgeon but not for assistants, although both must collaborate. We improved the shared perception of the whole surgical team by connecting live 3-D monitors to all three available Da Vinci generations, probed user experience after two years by questionnaire, and compared time measurements of a predefined complex interaction task performed with a 3-D monitor versus two-dimensional. Additionally, we investigated whether the complex mental task of reconstructing a 3-D overview from an endoscopic video can be performed by a computer and shared among users. During the study, 925 robot-assisted laparoscopic procedures were performed in three hospitals, including prostatectomies, cystectomies, and nephrectomies. Thirty-one users participated in our questionnaire. Eighty-four percent preferred 3-D monitors and 100% reported spatial-perception improvement. All participating urologists indicated quicker performance of tasks requiring delicate collaboration (e.g., clip placement) when assistants used 3-D monitors. Eighteen users participated in a timing experiment during a delicate cooperation task in vitro. Teamwork was significantly (40%) faster with the 3-D monitor. Computer-generated 3-D reconstructions from recordings offered very wide interactive panoramas with educational value, although the present embodiment is vulnerable to movement artifacts.

Exploring Large Virtual Environments by Thoughts Using a Brain–Computer Interface Based on Motor Imagery and High-Level Commands

Brain–computer interfaces (BCI) are interaction devices that enable users to send commands to a computer by using brain activity only. In this paper, we propose a new interaction technique to enable users to perform complex interaction tasks and to navigate within large virtual environments (VE) by using only a BCI based on imagined movements (motor imagery). This technique enables the user to send high-level mental commands, leaving the application in charge of most of the complex and tedious details of the interaction task. More precisely, it is based on points of interest and enables subjects to send only a few commands to the application in order to navigate from one point of interest to the other. Interestingly enough, the points of interest for a given VE can be generated automatically thanks to the processing of this VE geometry. As the navigation between two points of interest is also automatic, the proposed technique can be used to navigate efficiently by thoughts within any VE. The input of this interaction technique is a newly-designed self-paced BCI which enables the user to send three different commands based on motor imagery. This BCI is based on a fuzzy inference system with reject options. In order to evaluate the efficiency of the proposed interaction technique, we compared it with the state of the art method during a task of virtual museum exploration. The state of the art method uses low-level commands, which means that each mental state of the user is associated with a simple command such as turning left or moving forward in the VE. In contrast, our method based on high-level commands enables the user to simply select its destination, leaving the application performing the necessary movements to reach this destination. Our results showed that with our interaction technique, users can navigate within a virtual museum almost twice as fast as with low-level commands, and with nearly half the commands, meaning with less stress and more comfort for the user. This suggests that our technique enables efficient use of the limited capacity of current motor imagery-based BCI in order to perform complex interaction tasks in VE, opening the way to promising new applications.

Augmented Multisensory Interface Design (AMID): A Human-Centric Approach to Unisensory and Multisensory Augmented Reality Design

This study demonstrates that the use of augmented reality interfaces, designed based on a theoretically driven human-centered framework, can significantly improve task performance. The domain of unmanned ground vehicle landmine/unexploded ordnance detection and disposal was selected because it is a real-world example of a complex human-robot interaction task. Participants in this study were assigned to one of seven experimental conditions. A new method (the augmented multisensory interface design, or AMID, method) was used to identify two unisensory and one multisensory performance-enhancing augmented reality (AR) interface solutions, along with three less appropriate interface solutions. All participants were trained in the operation of the AR devices they were assigned and also in telerobotic landmine detection operations. After completing the training, participants were given the task of clearing a minefield. Results show that participants in the AMID-compliant solution groups significantly outperformed the control group and their non-AMID-compliant group counterparts in terms of the numbers of landmines correctly identified. The AMID-compliant multisensory condition was found to result in significantly increased landmine detection, increased sensitivity, and decreased task time.

Long-term practice effects on a new skilled motor learning: an electrophysiological study

Cortical functions concerned with the execution of skilled movements can be studied through complex interactive tasks. Skilled performance task (SPT) offers the greatest deal of information about the electrophysiological components reflecting pre-programming, execution of the movement and control of the results. Overall, these components are indicated as ‘movement-related brain macropotentials’ (MRBMs). Among them, Bereitschaftspotential (BP) reflects cerebral processes related to the preparation of movement and skilled performance positivity (SPP) reflects control processes on the result of performance. There is some evidence supporting a training effect on MRBMs, but less clear is whether long-term practice of a skilled activity could modify learning strategies of a new skilled task. We recorded MRBMs in subjects trained for a long time to perform a highly skilful athletic activity, i.e. gun shooting, and in a group of control subjects without any former experience in skilled motor activities. Our findings demonstrated the existence of a relationship between pre-programming and performance control, as suggested by decrease of BP amplitude and increase of SPP amplitude in presence of high levels of performance. Long-term practice seems to develop better control models on performance, that reduce the need of a high mental effort in pre-programming a skilled action.