Robot Interface Research Articles

ARMStick - An Intuitive Therapist Interface for Upper-Limb Rehabilitation Robots.

Currently, therapists struggle with interaction of rehabilitation robots due to non-intuitive interfaces. Therefore their acceptance of these robots are limited. This paper presents the development of ARMStick, a lightweight and small robotic interface in the shape of a human arm with 4 actuated and 3 unactuated joints, to facilitate the interaction between therapists and rehabilitation robots. It allows therapists to intuitively perceive joint-dependent data as recorded by rehabilitation robots, and teach poses and trajectories to individualize therapy to the patient. It's range of motion (RoM) covers the RoM of a healthy human. The device's measuring accuracy of CI 95% $ \lt \pm 0.322^{\circ}$ and movement accuracy of CI 70% $ \lt \pm 5.23^{\circ}$ lie within the confidence interval of average visual perception. A demonstration of the device to 5 therapists indicated that it could indeed improve efficiency and efficacy bottlenecks in current robot-assisted therapy. Comparison of ARMStick to two visual user interfaces showed a decrease in mean adaptation time from 15s to 5s for three arm configurations presented to the therapists.

Development of a hyper CLS data-based robotic interface for automation of production-line tasks using an articulated robot arm

Development of a hyper CLS data-based robotic interface for automation of production-line tasks using an articulated robot arm

Evaluation of a Portable fMRI Compatible Robotic Wrist Interface.

This paper presents evaluation of a portable fMRI compatible haptic interface to study the brain correlates of sensorimotor control during wrist motion. The interface is actuated by a shielded DC motor located more than 2 m away from the 3T MR scanner's bore. The achievable wrist torque of the interface is up to 2 Nm, and the interface provides sufficient bandwidth for human motor control experiments. Ergonomic and fMRI compatibility testing with a 3T MR scanner showed that the interface is MR safe, compatible with a strong static magnetic field and radio frequency emission, and its operation does not affect the quality of the acquired images. Clinical Relevance- We present and evaluate an fMRI compatible robotic interface to study human wrist joint motor function.

Human–robot interface design – the ‘Robot with a Tablet’ or ‘Robot only’, which one is better?

ABSTRACT The new era of integrating robots with artificial intelligence and the Internet of Things has provided huge potential for service robots to play important roles in our daily lives. Therefore, it is interesting to investigate whether a robot should have a separate device to incorporate digital content or use its face to toggle between facial expressions and digital content. From the human–robot interface (HRI) design perspective, a robot with a tablet obviously is closer to a real human teacher using a tablet while teaching students in a classroom. However, limited research has been done to explore such HRI design and its impacts on human behaviours. A robot-facilitated storytelling learning system was developed in this study to evaluate the impact of HRI design on college students’ learning behaviours in two different conditions: ‘Robot with Tablet’ or ‘Robot only’. The results revealed that both learning conditions of HRI design benefited college students from knowledge acquisition of vocabulary and comprehension of story content. Furthermore, ‘Robot only’ which is more cost-effective was found to be as good as that of ‘Robot with Tablet’ for the students in terms of knowledge acquisition, cognitive load and learning fatigue.

Low-intensity near-infrared light-tunable gold nanorod-incorporated hydrogel actuator system for remotely controlled human–robot interface applications

Light-responsive soft actuators have recently drawn attention as the need for lightweight remotely controlled actuator systems with high environmental adaptability and safe human–robot interaction interface characteristics has increased. In this study, we focused on a near-infrared (NIR) light-tunable hydrogel soft actuator system capable of fast and precisely controllable mechanical actuation even under the irradiation of NIR light with low intensity. We designed a bilayer type hydrogel soft actuator consisting of a poly(N-isopropylacrylamide) (PNIPAAm) active layer containing gold nanorods (AuNRs) as a photothermal agent and poly(acrylamide) (PAAm) passive layer. AuNRs with an aspect ratio of 4.12 showed the largest longitudinal plasmon resonance absorption peak in the NIR 880 nm range, which was used as a photothermal agent. The AuNR-incorporated PNIPAAm/PAAm hydrogel with a bilayer structure (PNIPAAm-AuNR/PAAm) exhibited more than 70% gel fraction and temperature-sensitive swelling behaviors. In particular, rod-shaped PNIPAAm-AuNR/PAAm hydrogels showed rapid bending deformation by a photothermally induced phase transition of PNIPAAm upon 880 nm laser irradiation, but a PNIPAAm/PAAm bilayer hydrogel without AuNRs did not show a light-induced bending movement. The NIR-responsive bending actuation of the hydrogel can be precisely controlled by cross-linking density, content of photothermal agent, NIR laser intensity, and thickness ratio of the active/passive layer. It was observed that the full bending deformations of the hydrogel into a ring shape through an arc occurred within a minute under NIR irradiation with a lower intensity of less than 1.0 W/cm2. Hydrogels showed a cell viability of more than 95% in the biocompatibility test, indicating no significant cytotoxicity. Therefore, these PNIPAAm-AuNR/PAAm hydrogels are promising soft actuator materials that can be controlled by low-intensity NIR irradiation for remotely controlled human–robot interaction interface applications.

An efficient hand gesture recognition based on optimal deep embedded hybrid convolutional neural network‐long short term memory network model

AbstractHand gestures are the nonverbal communication done by individuals who cannot represent their thoughts in form of words. It is mainly used during human‐computer interaction (HCI), deaf and mute people interaction, and other robotic interface applications. Gesture recognition is a field of computer science mainly focused on improving the HCI via touch screens, cameras, and kinetic devices. The state‐of‐art systems mainly used computer vision‐based techniques that utilize both the motion sensor and camera to capture the hand gestures in real‐time and interprets them via the usage of the machine learning algorithms. Conventional machine learning algorithms often suffer from the different complexities present in the visible hand gesture images such as skin color, distance, light, hand direction, position, and background. In this article, an adaptive weighted multi‐scale resolution (AWMSR) network with a deep embedded hybrid convolutional neural network and long short term memory network (hybrid CNN‐LSTM) is proposed for identifying the different hand gesture signs with higher recognition accuracy. The proposed methodology is formulated using three steps: input preprocessing, feature extraction, and classification. To improve the complex visual effects present in the input images, a histogram equalization technique is used which improves the size of the gray level pixel in the image and also their occurrence probability. The multi‐block local binary pattern (MB‐LBP) algorithm is employed for feature extraction which extracts the crucial features present in the image such as hand shape structure feature, curvature feature, and invariant movements. The AWMSR with the deep embedded hybrid CNN–LSTM network is applied in the two‐benchmark datasets namely Jochen Triesch static hand posture and NUS hand posture dataset‐II to detect its stability in identifying different hand gestures. The weight function of the deep embedded CNN‐LSTM architecture is optimized using the puzzle optimization algorithm. The efficiency of the proposed methodology is verified in terms of different performance evaluation metrics such as accuracy, loss, confusion matrix, Intersection over the union, and execution time. The proposed methodology offers recognition accuracy of 97.86% and 98.32% for both datasets.

P.122 Unbiased whole brain circuit interrogation reveals neurons restoring walking after spinal cord injury

Background: There is presently no cure for locomotor deficits after spinal cord injury (SCI). Very few therapies effectively target the brain due to poor understanding of the brain’s role post-SCI. Newly developed tissue clearing techniques have permitted unbiased three-dimensional circuit analysis, opening new opportunities for SCI-related brain interrogation. Methods: We established a novel brain interrogation pipeline by optimizing mouse brain clearing, imaging, and atlas registration. We leveraged a spontaneous recovery lateral hemisection model to analyze whole brain cell activity and connectivity with the lumbar cord using cFos immunolabelling and virus-mediated projection tracing. We identified a functionally and anatomically dynamic region correlating with recovery and interrogated its locomotor role with optogenetics. We assessed deep brain electrical stimulation (DBS) of this region in a more clinically relevant rat contusion SCI using an established bipedal robotic interface. Results: We unexpectedly uncovered the lateral hypothalamus (LH) to functionally and anatomically correlate with recovery. LHVglut2 optogenetic stimulation significantly augmented locomotor function. LH DBS in rats acutely robustly augmented bipedal locomotion post-SCI. Conclusions: This is the first demonstration of the LH’s role in locomotion post-SCI and is a novel DBS target that robustly augmented locomotor function, dependent on LH glutamatergic cells. LH DBS may be a promising intervention in humans.

Analysis of a User Interface Based on Multimodal Interaction to Control a Robotic Arm for EOD Applications

A global human–robot interface that meets the needs of Technical Explosive Ordnance Disposal Specialists (TEDAX) for the manipulation of a robotic arm is of utmost importance to make the task of handling explosives safer, more intuitive and also provide high usability and efficiency. This paper aims to evaluate the performance of a multimodal system for a robotic arm that is based on Natural User Interface (NUI) and Graphical User Interface (GUI). The mentioned interfaces are compared to determine the best configuration for the control of the robotic arm in Explosive Ordnance Disposal (EOD) applications and to improve the user experience of TEDAX agents. Tests were conducted with the support of police agents Explosive Ordnance Disposal Unit-Arequipa (UDEX-AQP), who evaluated the developed interfaces to find a more intuitive system that generates the least stress load to the operator, resulting that our proposed multimodal interface presents better results compared to traditional interfaces. The evaluation of the laboratory experiences was based on measuring the workload and usability of each interface evaluated.

Multimodal Data-Driven Robot Control for Human–Robot Collaborative Assembly

Abstract In human–robot collaborative assembly, leveraging multimodal commands for intuitive robot control remains a challenge from command translation to efficient collaborative operations. This article investigates multimodal data-driven robot control for human–robot collaborative assembly. Leveraging function blocks, a programming-free human–robot interface is designed to fuse multimodal human commands that accurately trigger defined robot control modalities. Deep learning is explored to develop a command classification system for low-latency and high-accuracy robot control, in which a spatial-temporal graph convolutional network is developed for a reliable and accurate translation of brainwave command phrases into robot commands. Then, multimodal data-driven high-level robot control during assembly is facilitated by the use of event-driven function blocks. The high-level commands serve as triggering events to algorithms execution of fine robot manipulation and assembly feature-based collaborative assembly. Finally, a partial car engine assembly deployed to a robot team is chosen as a case study to demonstrate the effectiveness of the developed system.

AUTOMATION OF AN ASSEMBLY SYSTEM USING A ROBOTIC ARM WITH PREDETERMINED TIME METHOD

ABSTRACT: In this study, the automation of an assembly process using a collaborative robotic arm (Cobot) was analyzed. The manual process is the assembly of a Lego® bricks structure and was analyzed using the technique of predetermined movements MTM-1. Moves were defined and processing time was set. The results obtained show that the automated process can increase production capacity by up to 40% using the same personnel as the manual process, but in different shifts. Likewise, defects due to a bad union between the edges of the bricks, which is the main problem in the manual process, are avoided. Finally, the automation process was detailed by associating the manual movements with the Cobot® commands in the simulation environment of the PolyScope Robot User Interface®. What has application to automate other types of manufacturing sequences and manual assemblies. Keywords: Automatization, Robotic arm; Methods-Time Measurement; Cobot®; Lego® bricks.

Closed-Loop Robotic Arm Manipulation Based on Mixed Reality

Robotic manipulators have become part of manufacturing systems in recent decades. However, in the realm of Industry 4.0, a new type of manufacturing cell has been introduced—the so-called collaborative manufacturing cell. In such collaborative environments, communication between a human operator and robotic manipulators must be flawless, so that smooth collaboration, i.e., human safety, is ensured constantly. Therefore, engineers have focused on the development of suitable human–robot interfaces (HRI) in order to tackle this issue. This research work proposes a closed-loop framework for the human–robot interface based on the utilization of digital technologies, such as Mixed Reality (MR). Concretely, the framework can be realized as a methodology for the remote and safe manipulation of the robotic arm in near real-time, while, simultaneously, safety zones are displayed in the field of view of the shop-floor technician. The method is based on the creation of a Digital Twin of the robotic arm and the setup of a suitable communication framework for continuous and seamless communication between the user interface, the physical robot, and the Digital Twin. The development of the method is based on the utilization of a ROS (Robot Operating System) for the modelling of the Digital Twin, a Cloud database for data handling, and Mixed Reality (MR) for the Human–Machine Interface (HMI). The developed MR application is tested in a laboratory-based machine shop, incorporating collaborative cells.

200 Unbiased Interrogation of Whole Brain Circuits Identifies Neurons That Restore Walking After Spinal Cord Injury

INTRODUCTION: Currently, there is no cure for locomotor deficits after spinal cord injury (SCI). Despite advances in spinal cord modulation with epidural electrical stimulation, there remains a paucity of therapies targeting the brain due to a poor understanding of the brain’s role post-SCI. Recently developed tissue clearing and light sheet imaging techniques have permitted unbiased three-dimensional interrogation of whole brain circuits, which has opened unexplored avenues for SCI-related brain interrogation. METHODS: We established a novel brain interrogation pipeline for SCI by optimizing mouse whole brain clearing, imaging, and atlas registration after a spontaneous recovery lateral hemisection model. We examined both whole brain cell activity and connectivity with the lumbar cord using cFos immunolabelling and virus-mediated projection tracing, respectively, to identify a functionally and anatomically dynamic region correlating with recovery. We interrogated the locomotor role of this region with optogenetics and chemogenetics. In a more clinically relevant rat contusion SCI, we assessed the translatability of deep brain electrical stimulation (DBS) of this region by leveraging an established bipedal robotic interface and rehabilitation paradigm. RESULTS: We unexpectedly uncovered the lateral hypothalamus (LH) to be a functionally and anatomically dynamic region post-SCI correlating with recovery. Optogenetic stimulation of mouse LHVglut2 neurons significantly augmented locomotor function, which required medullary reticular formation (MRF) Vglut2-positive neurons. In line with previously demonstrated sparing of MRF-to-spinal cord projections after contusion SCI, we found that LH DBS in rats post-contusion SCI could acutely robustly augment bipedal rat locomotor function. CONCLUSION: This is the first demonstration of the LH's role in locomotion post-SCI. The LH is a novel DBS target that robustly augmented locomotor function, dependent on LH and brainstem glutamatergic cells.

Novel technique for control of industrial robots with wearable and contactless technologies

Wearable and non-contact sensor technologies have spread to many diverse areas with their ease of use and application, and the variety of applications has been increasing rapidly thanks to technological developments in the recent years. This work ensures the usability of wearable and non-contact sensor technologies in various industrial fields. In this study, it is aimed that operators can easily control industrial robots in real-time without the need for a specialized programming environment. For this purpose, a new human–robot interface, in which industrial robots can be controlled by hand movements, has been created. In this interface, Leap Motion Controller (LMC), which uses an IR Camera for the control of robot arm positioning and motion trajectories, and Myo armband, which detects wrist and hand movements with EMG sensors, is used to control of robot end effector (tool) actions. By means of both devices, robot movements are controlled by industrial communication after hand movements, classification and data processing stages. The data processing, control and positioning results of the platform developed in the article are presented in comparison with similar studies in the literature. As a result, it has been observed that the developed platform works with high accuracy. EMG classification success was determined as 96.9% and hand tracking error between 2.45–2.85 mm. Thanks to this new platform, various robotic processes such as pick-place, object movement, painting applications, etc. will be carried out easily and flexibly without the need for programming.

HP03: Abnormal Dynamic Neuronal Connectivity in children with dystonia

Dystonia is a network disorder resulting from dysfunction of, or abnormal connectivity within, the cortico-basal ganglia-cerebellar network. Despite the dynamic nature of the disorder, stimulus-related changes in connectivity have not been studied. We investigate modulation of neuronal connectivity by a proprioceptive stimulus in dystonia. Methods: Sixteen young people with dystonia and eight controls participated (age 6-20years, mean 12.5). A robotic wrist interface delivered passive wrist extension movements, producing a brief stretch of the right wrist flexors. Scalp EEG (10-20 system) was sampled at 2500Hz. Wrist position was monitored and movement onset synchronised with EEG. Data were segmented into epochs (1 second pre- and 3.5 seconds post-stimulus). Up to 160 epochs were averaged per subject to produce a Stretch Evoked Potential (StretchEP). Event-related network dynamics were estimated using the imaginary part of Wavelet Transform Coherency, excluding volume conduction, with bootstrapping to test for significance against random coupling between brain areas. The overall engagement of cortical areas into short-lived stimulus-related networks (global microscale nodal strength - GMNS) was estimated for each subject. Results: Clear StretchEPs were evoked over contralateral sensorimotor cortex. Individual dynamic connectivity maps revealed particularly strong event-related connectivity in the theta (4-8Hz) band in dystonia. At group level, theta band GMNS was significantly higher in dystonia than controls (p=0.045). The spatial extent of the networks was also stronger in dystonia than controls (non-significant trend). Conclusion: Young people with dystonia show an exaggerated dynamic network response to proprioceptive stimuli, displaying excessive, widespread theta-band synchronisation and over-recruitment across the sensorimotor network.

OC17: Event-related modulation of sensorimotor cortex alpha/mu activity is abnormal in children with dystonia and dystonic cerebral palsy

Sensorimotor processing is abnormal in adult genetic/idiopathic dystonia, but sparsely studied in acquired or childhood dystonia. We investigate sensorimotor processing by measuring event-related synchronisation (ERS) and desynchronization (ERD) in response to a proprioceptive stimulus in children with dystonia/dystonic cerebral palsy (CP). Methods: 30 young people with dystonia (20 genetic/idiopathic; 10 dystonic CP) and 22 controls participated, age 5-21years (mean 12.7years). Proprioceptive stimuli, comprising a brief stretch of the wrist flexors, were delivered to each hand using a robotic wrist interface. Scalp EEG was sampled (2500Hz). Wrist position was monitored and movement onset synchronised with EEG. Up to 160 epochs (1s pre- and 3.5s post-stimulus) were averaged per subject to produce a Stretch Evoked Potential. Time-frequency analyses were performed using continuous Morlet wavelet transforms (1Hz bins from 5-40Hz, 8 wavelet cycles). Relative changes in post-stimulus power with respect to pre-stimulus/baseline, were calculated for the alpha/mu (8-12Hz) band. Results: Controls showed a clear developmental profile in event-related spectral changes, with a prominent early alpha/mu ERD (0.5-1s post-stimulus) and later alpha/mu ERS (1.5–2.5s post-stimulus) over contralateral sensorimotor cortex. Control and dystonia groups were compared using age as a covariate. Alpha ERD was significantly smaller in dystonia for both the dominant (ANCOVA F(2,47)=4.45 p=0.017) and non-dominant hand (ANCOVA F(2,42)=9.397 p<0.001). Alpha ERS was significantly smaller in dystonia for the dominant hand (ANCOVA F(2,47)=7.786 p=0.001). Findings were similar for genetic/idiopathic dystonia and dystonic CP. Conclusions: The impaired alpha/mu modulation indicates a developmental abnormality of sensorimotor processing which is common to genetic/idiopathic dystonia and dystonic CP.

Optimization of a Soft Pressure Sensor in Terms of the Molecular Weight of the Ferroelectric‐Polymer Sensing Layer

AbstractTactile sensing is considered essential for effective object handling by robots. To this end, flexible smart materials are being extensively researched for application as soft pressure sensors. This paper proposes a soft pressure sensor featuring a ferroelectric‐polymer sensing layer for dynamic pressure detection during robotic grasping operations. The solution‐processed sensing layer is optimized in terms of the molecular weight of the ferroelectric polymer and the processing solvent. The proposed sensor exhibits high sensitivity to pressure with minimal response delay. Furthermore, these sensors successfully sensitize a robotic hand, enabling the real‐time detection of pressure signals during object handling, thus demonstrating the potential of the proposed sensor for novel robotic interface applications such as biomimetic electronic skin.

ANALOG-1 ISS – The first part of an analogue mission to guide ESA’s robotic moon exploration efforts

Abstract The European Space Agency’s ANALOG-1 experiment is the culmination of 12 distinct METERON experiments carried out since 2011. These all address aspects of teleoperating a robotic asset from an orbital platform, i.e., technical implementation, user interfaces, autonomy and operations. The ANALOG-1 technology demonstration and operations concept experiment is based upon the surface mission scenario segment of the notional EL3 sample return mission. This segment focuses on the control of a lunar surface robotic asset from the Earth and from the Lunar Gateway. The experiment is taking place in two parts, with the first successfully completed from the ISS in November 2019. It assessed the effectiveness of a state-of-the-art robotic control interface to control a complex mobile robot from orbit, as well as evaluating the scientific interactions, during robotic-assisted geology exploration, between crew in orbit and scientists on the ground. Luca Parmitano operated the robot while he was on the ISS. For this experiment, a complex control station had been installed on the ISS. The experiment demonstrated the advantage of having an immersive control station and high level of robotic dexterity, with Luca finishing all his assigned and secondary geology targets ahead of time.

Mixed Reality Human–Robot Interface With Adaptive Communications Congestion Control for the Teleoperation of Mobile Redundant Manipulators in Hazardous Environments

Robotic interventions with redundant mobile manipulators pose a challenge for telerobotics in hazardous environments, such as underwater, underground, nuclear facilities, particle accelerators, aerial or space. Communication issues can lead to critical consequences, such as imprecise manipulation resulting in collisions, breakdowns and mission failures. The research presented in this paper was driven by the needs of a real robotic intervention scenario in the Large Hadron Collider (LHC) at the European Organization for Nuclear Research (CERN). The goal of the work was to develop a framework for network optimisation in order to help facilitate Mixed Reality techniques such as 3D collision detection and avoidance, trajectories planning, real-time control, and automatized target approach. The teleoperator was provided with immersive interactions while preserving precise positioning of the robot. These techniques had to be adapted to delays, bandwidth limitation and their volatility in the 4G shared network of the real underground particle accelerator environment. The novel application-layer congestion control with automatic settings was applied for video and point cloud feedback. Twelve automatic setting modes were proposed with algorithms based on the camera frame rate, resolution, point cloud subsampling, network round-trip time and throughput to bandwidth ratio. Each mode was thoroughly characterized to present its specific use-case scenarios and the improvements it brings to the adaptive camera feedback control in teleoperation. Finally, the framework was presented according to which designers can optimize their Human-Robot Interfaces and sensor feedback depending on the network characteristics and task.

A Lightweight Accessible Wearable Robotic Interface for Bimanual Haptic Manipulations.

Wearable devices with bimanual force feedback enable natural and cooperative manipulations within an unrestricted space. Weight and cost have a great influence on the potential applications of a haptic device. This paper presents a wearable robotic interface with bimanual force feedback that has considerably reduced weight and cost. To make the reaction force less perceivable than the interaction force, a waist-worn scheme is adopted. The interface mainly consists of a belt, a fastening tape, two serial robotic arms, and two electronics units and batteries. The robotic arms located on both sides of the belt are capable of 3-DoF position tracking and force feedback for each hand. The whole interface is lightweight (only 2.4 kg) and accessible. Furthermore, it is also easy to wear and the operator can wear it only by putting the belt on the waist and fastening the tape, reducing his/her dependency on additional assistance. The interface is optimized to obtain desirable force output and a dexterous workspace without singularity. To evaluate its performance in bimanual cooperative manipulations, an experiment in the virtual environment was conducted. The experimental results showed the subjects had more efficient and stable cooperative manipulations with bimanual force feedback than without force feedback.

Direction Estimation for Multiple Pedestrians and Response Control of Interface Robot

Direction Estimation for Multiple Pedestrians and Response Control of Interface Robot