Interaction Force Estimation Research Articles

Learning Based Estimation of Tool-Tissue Interaction Forces for Stationary and Moving Environments

Learning Based Estimation of Tool-Tissue Interaction Forces for Stationary and Moving Environments

A membership-function-based broad learning system for human-robot interaction force estimation under drawing task.

Estimating interaction force is of great significance in the field of human-robot interaction (HRI) thanks to its guarantee of interaction safety. To this end, this paper proposes a novel estimation method by leveraging broad learning system (BLS) and human surface electromyography (sEMG) signals. Since the previous sEMG may also contain valuable information of human muscle force, it would cause the estimation to be incomplete and abate the estimation accuracy in the case of neglecting the previous sEMG. To remedy this thorn, a new linear membership function is first developed to calculate contributions of sEMG at different sampling times in the proposed method. Subsequently, the contribution values calculated by the membership function are integrated with features of sEMG to be considered as the input layer of BLS. For extensive studies, five different features extracted from sEMG signals and their combination are explored to estimate the interaction force by the proposed method. Lastly, the performance of the proposed method is compared with those of three well-known methods through experimental test regarding the drawing task. The experimental results confirm that combining the time domain (TD) with frequency domain (FD) features of sEMG can enhance the estimation quality. Moreover, the proposed method outperforms its contenders with respect to estimation accuracy.

Human–Exoskeleton Interaction Force Estimation in Indego Exoskeleton

Accurate interaction force estimation can play an important role in optimizing human–robot interaction in an exoskeleton. In this work, we propose a novel approach for the system identification of exoskeleton dynamics in the presence of interaction forces as a whole multibody system without imposing any constraints on the exoskeleton dynamics. We hung the exoskeleton through a linear spring and excited the exoskeleton joints with chirp commands while measuring the exoskeleton–environment interaction force. Several structures of neural networks were trained to model the exoskeleton passive dynamics and estimate the interaction force. Our testing results indicated that a deep neural network with 250 neurons and 10 time–delays could obtain a sufficiently accurate estimation of the interaction force, resulting in an RMSE of 1.23 on Z–normalized applied torques and an adjusted R2 of 0.89.

Fixed‐time observer‐based controller for the human–robot collaboration with interaction force estimation

Fixed‐time observer‐based controller for the human–robot collaboration with interaction force estimation

A CoppeliaSim Dynamic Simulator for the Da Vinci Research Kit

The design of a physics-based dynamic simulator of a robot requires to properly integrate the robot kinematic and dynamic properties in a virtual environment. Naturally, the closer is the integrated information to the real robot properties, the more accurate the simulator predicts the real robot behaviour. A reliable robot simulator is a valuable asset for developing new research ideas; its use dramatically reduces the costs and it is available to all researchers. This letter presents a dynamic simulator of the da Vinci Research Kit (dVRK) patient-side manipulator (PSM). The kinematic and dynamic properties of the simulator rely on the parameters identified in Wang et al. With respect to the kinematic simulator previously developed by some of the authors, this work: (i) redefines the kinematic architecture and the actuation model by modeling the double parallelogram and the counterweight mechanism, to reflect the structure of the real robot; (ii) integrates the identified dynamic parameters in the simulation model. The obtained simulator enables the design and validation of control strategies relying on the robot dynamic model, including interaction force estimation and control, that are fundamental to guarantee safety in many surgical tasks.

SEMG-Upper Limb Interaction Force Estimation Framework Based on Residual Network and Bidirectional Long Short-Term Memory Network

It is of great significance to estimate the interaction force of upper limbs accurately for improving the control performance of human–computer interaction. However, due to the randomness of the input biological signals and the influence of environmental interference, the interaction force is difficult to estimate using the current methods. Therefore, based on the advantages of the Residual Network (ResNet) and Bidirectional Long Short-Term Memory Network (BiLSTM) model, this paper proposes an end-to-end regression model that integrates ResNet and BiLSTM with an attention mechanism. This model is more suitable for time series sEMG signals. Moreover, it improves the feature extraction ability of the signal and improves the accuracy of interaction force estimation. Experimental results show that this method can automatically extract effective features without professional knowledge. In addition, our method is superior to existing methods in estimation accuracy and generalization ability.

Toward Sensorless Interaction Force Estimation for Industrial Robots Using High-Order Finite-Time Observers

This article proposes a novel interaction force estimation scheme capable of estimating the interaction forces for industrial robots with high precision in the absence of force sensors. Specifically, we first establish the dynamic model of a class of industrial robots via model identification methods. Furthermore, a high-order finite-time observer (HOFTO) is established to estimate the interaction forces, where HOFTO serves as an interaction force observer and is designed to accurately estimate external time-varying interaction forces. In addition, we strictly analyze the stability of our approach in two different cases, endowing HOFTO with reliable estimation abilities in broader applications. Subsequently, we integrate the estimated force information into the applications of force control frameworks (e.g., collision detection and impedance control). Several evaluations on a real 6-axes robot demonstrate the effectiveness of the proposed approach.

VDC-based admittance control of multi-DOF manipulators considering joint flexibility via hierarchical control framework

Multi-degree-of-freedom (Multi-DOF) manipulators have shown a high potential for enhancing the flexibility and performance of robotic manipulations. However, the presence of unknown disturbance, including uncertain dynamics and external forces/torques, makes the control of a multi-DOF manipulator rather complicated, and the stability of the robotic system is hard to be guaranteed. In this paper, a virtual decomposition control (VDC)-based admittance control approach for multi-DOF manipulators has been proposed considering joint flexibility via hierarchical control framework. The joint flexibility is solved by a separate adaptive controller different from the manipulator’s links. The high-level admittance controller is built upon a low-level VDC-based inner control loop, which can deal with the complicated system dynamics (including the joint friction and joint flexibility) and modeling uncertainty. The external force/torque (F/T) is estimated with a generalized momentum-based interaction force estimation technique; thereby avoiding the cost of wrist F/T sensors. The robotic system’s stability has been guaranteed in both free-space motions and constrained motions using the specific features of VDC (proof of each subsystem’s virtual stability). The advantages and effectiveness of the proposed method in tuning the robot–environment dynamic behavior are demonstrated through experiments.

Visual Interaction Force Estimation Based on Time-Sensitive Dual-Resolution Learning Network

Haptic force feedback is an important perception method for humans to understand the surrounding environment. It can estimate tactile force in real time and provide appropriate feedback. It has important research value for robot-assisted minimally invasive surgery, interactive tactile robots, and other application fields. However, most of the existing noncontact visual power estimation methods are implemented using traditional machine learning or 2D/3D CNN combined with LSTM. Such methods are difficult to fully extract the contextual spatiotemporal interaction semantic information of consecutive multiple frames of images, and their performance is limited. To this end, this paper proposes a time-sensitive dual-resolution learning network-based force estimation model to achieve accurate noncontact visual force prediction. First, we perform continuous frame normalization processing on the robot running the video captured by the camera and use the hybrid data augmentation to improve the data diversity; secondly, a deep semantic interaction model is constructed based on the time-sensitive dual-resolution learning network, which is used to automatically extract the deep spatiotemporal semantic interaction information of continuous multiframe images; finally, we construct a simplified prediction model to realize the efficient estimation of interaction force. The results based on the large-scale robot hand interaction dataset show that our method can estimate the interaction force of the robot hand more accurately and faster. The average prediction MSE reaches 0.0009 N, R 2 reaches 0.9833, and the average inference time for a single image is 6.5532 ms; in addition, our method has good prediction generalization performance under different environments and parameter settings.

Analytical Estimation of Interaction Force and its Application Point for Collaborative Robots

The paper deals with parameters estimation in the human-robot collaboration scenario. It presents an analytical algorithm for computing of the interaction force and its application point using measurement data obtained from the internal joint torque sensors only. The proposed algorithm is based on a specific extension of static equilibrium equations allowing to find the desired interaction force and action line. Further, this general solution is combined with geometric constraints describing manipulator surfaces and corresponding friction cones. Particular attention is paid to singular cases arising when the interaction force action line intersects one or several joint sensor axes. The developed technique was carefully evaluated via the simulation study.

Simultaneous estimation of joint angle and interaction force towards sEMG-driven human-robot interaction during constrained tasks

Human has excellent motor capability and performance in conducting various manipulation tasks. During some tasks such as tightening/loosening a screw with a screwdriver, the motion is accompanied by force exertion to the environment (that is, constrained motion). To obtain natural human-robot interaction (HRI) as human interacts/collaborates with the environment, decoding the human’s movement intention in a way of motion and interaction force is meaningful for robots to carry out such constrained tasks. This paper proposes a long shortterm memory (LSTM) -based decoding method for the simultaneous estimation of human motion and interactive force from muscle activities represented by surface electromyography (sEMG) signals. The sEMG recorded from the muscles of forearm is used to decode human’s movement intention. In order to extract smooth features from non-stationary sEMG signals, Bayesian filter is applied instead of traditional time-domain feature extraction method. From the real-time experiments on eight subjects, the LSTM-based decoding method represents high accuracy of motion estimation (91.7%) and force estimation (96.1%) despite of the existence of muscle coupling and non-stationary mapping between muscle activities and motion/interaction force during such constrained tasks. It indicates that the estimated motion and interaction force can be further applied for HRI in accomplishing constrained tasks.

Toward Vision-Based High Sampling Interaction Force Estimation With Master Position and Orientation for Teleoperation

In this study, a vision-based high sampling rate interaction force estimation method is proposed for teleoperation systems that uses master position and orientation information without using physical sensors such as force/torque (F/T) or tactile sensors. The proposed method uses red-green-blue (RGB) images, six-axis robot pose and motor current data, gripper position and current data, as well as master position and orientation information as inputs without requiring force sensors. To estimate the interaction forces, a deep neural network composed of densely connected convolutional network (DenseNet) and long short-term memory (LSTM) is proposed. The database was created by operators using grip and picking motions to interact with 10 objects over a teleoperation system. In addition, we compared the proposed method with different deep learning networks that used different sets of inputs. The results show that the proposed model can estimate 1 kHz interaction force based on 60 Hz images and 1 kHz master inputs. Moreover, the results indicate that the master position and orientation information are useful in estimating the interaction force at a high sampling rate through the result of the change in the network input.

Sensorless environment stiffness and interaction force estimation for impedance control tuning in robotized interaction tasks

Industrial robots are increasingly used to perform tasks requiring an interaction with the surrounding environment (e.g., assembly tasks). Such environments are usually (partially) unknown to the robot, requiring the implemented controllers to suitably react to the established interaction. Standard controllers require force/torque measurements to close the loop. However, most of the industrial manipulators do not have embedded force/torque sensor(s) and such integration results in additional costs and implementation effort. To extend the use of compliant controllers to sensorless interaction control, a model-based methodology is presented in this paper. Relying on sensorless Cartesian impedance control, two Extended Kalman Filters (EKF) are proposed: an EKF for interaction force estimation and an EKF for environment stiffness estimation. Exploiting such estimations, a control architecture is proposed to implement a sensorless force loop (exploiting the provided estimated force) with adaptive Cartesian impedance control and coupling dynamics compensation (exploiting the provided estimated environment stiffness). The described approach has been validated in both simulations and experiments. A Franka EMIKA panda robot has been used. A probing task involving different materials (i.e., with different - unknown - stiffness properties) has been considered to show the capabilities of the developed EKFs (able to converge with limited errors) and control tuning (preserving stability). Additionally, a polishing-like task and an assembly task have been implemented to show the achieved performance of the proposed methodology.

Vision-Based Suture Tensile Force Estimation in Robotic Surgery.

Compared to laparoscopy, robotics-assisted minimally invasive surgery has the problem of an absence of force feedback, which is important to prevent a breakage of the suture. To overcome this problem, surgeons infer the suture force from their proprioception and 2D image by comparing them to the training experience. Based on this idea, a deep-learning-based method using a single image and robot position to estimate the tensile force of the sutures without a force sensor is proposed. A neural network structure with a modified Inception Resnet-V2 and Long Short Term Memory (LSTM) networks is used to estimate the suture pulling force. The feasibility of proposed network is verified using the generated DB, recording the interaction under the condition of two different artificial skins and two different situations (in vivo and in vitro) at 13 viewing angles of the images by changing the tool positions collected from the master-slave robotic system. From the evaluation conducted to show the feasibility of the interaction force estimation, the proposed learning models successfully estimated the tensile force at 10 unseen viewing angles during training.

Application of Free-Surface Immersed-Boundary Lattice Boltzmann Method to Waves Acting on Coastal Structures

AbstractA free-surface immersed-boundary lattice Boltzmann method for wave–structure interaction and hydrodynamic force estimation is introduced as a proposed model with example applications. The s...

A sensorless interaction forces estimator for bilateral teleoperation system based on online sparse Gaussian process regression

This paper focuses on sensorless estimation of the interaction forces between the slave manipulator and its surrounding environment in bilateral teleoperation system. The proposed estimator is based on online sparse Gaussian process regression (OSGPR) and it does not need the use of commercially available force sensors. Therefore, the proposed estimator can overcome the shortcomings associated with the employment of force sensors. Through the adoption of machine learning technique, the proposed estimator can accurately estimate the interaction forces even when there are parametric uncertainties and unmodeled disturbances in the dynamic model of the slave manipulator. Meanwhile, online estimation of the interaction forces is realized through the utilization of sparse technique. Two case studies of the estimation of interaction forces are performed to validate the effectiveness and feasibility of the proposed estimator. Experimental results demonstrate that the proposed estimator outperforms several existing alternatives in terms of estimation accuracy.

Interaction Force Estimation Using Extended State Observers: An Application to Impedance-Based Assistive and Rehabilitation Robotics

This letter presents a force observer that estimates the external interaction forces from the measured joint position and joint actuation for a class of robotic manipulators. This is done without an explicit (physical) force or torque sensor, through an extended state observer (ESO) assuming a known model of the dynamics of the manipulator. It is designed to be capable of estimating relatively fast time-varying external forces. The main result shows that the proposed ESO is theoretically able to estimate unknown time-varying external forces to an arbitrary accuracy for any initial condition starting in a given compact set. Simulation results demonstrate the effectiveness of the proposed ESO. Moreover, first experimental evaluations with a healthy subject and a poststroke patient on a rehabilitation robotic platform show the feasibility of the approach in this context and highlight the importance of the quality of the dynamic model.

Interaction Force Estimation Using Camera and Electrical Current Without Force/Torque Sensor

In this paper, we propose a method for the estimation of the interaction forces between the motorized system and object through the visual and electric information. In particular, we propose a new interaction force sensing method based on sequential images and the electrical current from the motor during the interaction between the system and environment to estimate the interaction force using deep learning. In the previous method, to measure the interaction force using only visual information, the prediction is inaccurate when the system interacts with an undeformable target, even though the aspect of the change appears small in the image. We use a neural network structure for estimating the interaction force from the time-series data of visual and electric information using deep learning, which combines the convolution neural network and long short-term memory models. From the evaluation to show the feasibility of the interaction force estimation, the proposed learning models successfully estimate the forces for four targets (rigid box, rigid box on sponge, sponge, and stapler), which are both deformable and undeformable objects. The proposed method demonstrates the best results in the interaction force estimation between the motorized system and object.

The effect of non-polar oil on fine hematite flocculation and flotation using sodium oleate or hydroxamic acids as a collector

The effects of non-polar oil on hydrophobic flocculation and micro-flotation of fine hematite (−20 µm) were studied when sodium oleate, octyl hydroxamic acid, or oleoyl hydroxamic acid was used as a collector. The micro-flotation test was carried out under neutral pH, and kerosene was used as the non-polar oil. Focused Beam Reflectance Measurement (FBRM) particle size analysis coupled with optical microscopy was performed to monitor the real-time evolution of hematite particles during flocculation. In addition, contact angles and zeta potentials of hematite were determined and used as parameters in the extended-DLVO theory interaction force estimation. The micro-flotation tests showed a beneficial effect of kerosene at high collector dosages regardless of the type of collectors used. The FBRM and optical image analysis indicated that the sizes of hematite aggregates induced by sodium oleate or oleoyl hydroxamic acid was much larger than by octyl hydroxamic acid; and the addition of kerosene further enhanced the aggregate size differences. The extended-DLVO theory calculation was carried out to explain the differences in the aggregation behaviors. The results indicated that the hydrophobic interaction energies between hydrophobic hematite particles induced by sodium oleate or oleoyl hydroxamic acid were stronger than by octyl hydroxamic acid. When kerosene was added, the aggregates grew to a much larger size due to the strong hydrophobic interaction between hematite particles and kerosene droplets.

Inferring Interaction Force from Visual Information without Using Physical Force Sensors.

In this paper, we present an interaction force estimation method that uses visual information rather than that of a force sensor. Specifically, we propose a novel deep learning-based method utilizing only sequential images for estimating the interaction force against a target object, where the shape of the object is changed by an external force. The force applied to the target can be estimated by means of the visual shape changes. However, the shape differences in the images are not very clear. To address this problem, we formulate a recurrent neural network-based deep model with fully-connected layers, which models complex temporal dynamics from the visual representations. Extensive evaluations show that the proposed learning models successfully estimate the interaction forces using only the corresponding sequential images, in particular in the case of three objects made of different materials, a sponge, a PET bottle, a human arm, and a tube. The forces predicted by the proposed method are very similar to those measured by force sensors.